# Importing parquet file as a 2d dense array

**URL:** <https://forum.tiledb.com/t/importing-parquet-file-as-a-2d-dense-array/758>\
**Category:** Uncategorized\
**Created:** [January 3, 2025, 3:26pm UTC](https://forum.tiledb.com/t/importing-parquet-file-as-a-2d-dense-array/758 "2025-01-03T15:26:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yoshi](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/yoshi/32/360_2.png) [@yoshi](https://forum.tiledb.com/u/yoshi)\
**Post date:** [January 3, 2025, 3:26pm UTC](https://forum.tiledb.com/t/importing-parquet-file-as-a-2d-dense-array/758/1 "2025-01-03T15:26:49Z")

</div>

Hello,  
I have a parquet file that has a matrix of Samples x Features (all features are same data type), and would like to create a TileDB object, where the row number and the feature# are the dimensions (as in I access sample 1 feature 1:2 as object[1,1:2]. I found a tutorial on importing the files as 1d arrays, and the columns become the attributes, but what I want is a single attribute per cel. The `from_pandas` has a parameter to set a column as a dimension, but this would mean changing the data from wide format to long format, which sounds very inefficient .  
A possible solution would be to fit the array into memory, and then just set the TileDB object and do a write, but the file unfortunately is too big to fit in memory.  
So far my best idea is to read the parquet file and start writing to it by a chunks, and then consolidate it, but wondering if there is an easier approach.  
thnx!

---

<div class="post-metadata">

**Author:** ![yoshi](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/yoshi/32/360_2.png) [@yoshi](https://forum.tiledb.com/u/yoshi)\
**Post date:** [January 15, 2025, 8:26pm UTC](https://forum.tiledb.com/t/importing-parquet-file-as-a-2d-dense-array/758/2 "2025-01-15T20:26:41Z")

</div>

just bumping this @nguyenv maybe you would have some insight on this? thnx!

---

<div class="post-metadata">

**Author:** ![spencerseale](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/spencerseale/32/340_2.png) [@spencerseale](https://forum.tiledb.com/u/spencerseale)\
**Post date:** [January 15, 2025, 8:58pm UTC](https://forum.tiledb.com/t/importing-parquet-file-as-a-2d-dense-array/758/3 "2025-01-15T20:58:29Z")

</div>

HI @yoshi,

Thanks for the question and sorry for the delay!

For writing large files that won’t fit into memory, I recommend you first create the array. During array creation, you’ll define the schema exactly as you’d like such as defining a single attribute.

Once you’ve written your array to disk, you can batch read from the original file and write to your array. Because TileDB arrays can handle parallel i/o, you can implement this in parallel for a faster ingestion process. Consolidation after all of these writes is a great idea as well as this will improve downstream read performance!

Once your data is written to a TileDB array, you’ll be able to perform out-of-core queries from your dataset without running into the memory constraints typical of a large tabular file.

Check out our documentation site, [TileDB Academy](https://cloud.tiledb.com/academy/home/) if you need help with any of the above!

Best,  
Spencer
