# Improved performance **URL:** \ **Category:** Uncategorized\ **Created:** [December 26, 2020, 8:46pm UTC](https://forum.tiledb.com/t/improved-performance/291 "2020-12-26T20:46:05Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Mtrl\_Scientist](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/mtrl_scientist/32/134_2.png) [@Mtrl\_Scientist](https://forum.tiledb.com/u/Mtrl_Scientist)\ **Post date:** [December 26, 2020, 8:46pm UTC](https://forum.tiledb.com/t/improved-performance/291/1 "2020-12-26T20:46:05Z")
I recently saw @stavros pointing to a [tutorial](https://nbviewer.jupyter.org/github/TileDB-Inc/TileDB-Cloud-Example-Notebooks/blob/release/tutorials/notebooks/python/dataframes/df_basics.ipynb) for the new dataframe interactions. I really like the new .df way of directly returning a dataframe from a TileDB array, and especially how TileDB now takes advantage of arrow! ``` uri = 'BTC_USDT' t1,t2 = (np.datetime64('2020-12-20'),np.datetime64('2021-12-25')) with tiledb.open(uri,'r') as A: df = A.query(attrs=('price','volume','is_buyer_maker', 'first_trade_ID','last_trade_ID' ), dims=('date',),index_col='date',use_arrow=True ).df[:,t1:t2] ``` This will read 9612287 rows × 5 columns in 6.31 s. > **Array-Schema** > > ArraySchema( > domain=Domain(\*[ > Dim(name=‘agg\_ID’, domain=(0, 9000000000000000000), tile=1000000, dtype=‘int64’), > Dim(name=‘date’, domain=(numpy.datetime64(‘1980-01-01T00:00:00.000000000’), numpy.datetime64(‘2100-01-01T00:00:00.000000000’)), tile=86400000000000 nanoseconds, dtype=‘datetime64[ns]’), > ]), > attrs=[ > Attr(name=‘price’, dtype=‘float64’, var=False, filters=FilterList([ZstdFilter(level=9),])), > Attr(name=‘volume’, dtype=‘float64’, var=False, filters=FilterList([ZstdFilter(level=9),])), > Attr(name=‘first\_trade\_ID’, dtype=‘int64’, var=False, filters=FilterList([ZstdFilter(level=9),])), > Attr(name=‘last\_trade\_ID’, dtype=‘int64’, var=False, filters=FilterList([ZstdFilter(level=9),])), > Attr(name=‘is\_buyer\_maker’, dtype=‘int8’, var=False, filters=FilterList([BitShuffleFilter(), ZstdFilter(level=9),])), > Attr(name=‘is\_best\_price\_match’, dtype=‘int8’, var=False, filters=FilterList([BitShuffleFilter(), ZstdFilter(level=9),])), > ], > cell\_order=‘row-major’, > tile\_order=‘row-major’, > capacity=10000000, > sparse=True, > allows\_duplicates=False, > coords\_filters=FilterList([ZstdFilter(level=-1),]) > ) > **Stats-dump1** > > TileDB Embedded Version: (2, 1, 3) > TileDB-Py Version: 0.7.4 > ==== READ ==== > > - Number of read queries: 3 > > - Number of attempts until results are found: 9 > > - Number of attributes read: 15 > > - Number of dimensions read: 3 > > - Number of logical tiles overlapping the query: 167 > > - Number of physical tiles read: 3006 > > - Number of cells read: 19516358 > > - Number of result cells: 17023669 > > - Percentage of useful cells read: 87.2277% > > - Number of bytes read: 163402447 bytes (0.15218 GB) > > - Number of read operations: 10186 > > - Number of bytes unfiltered: 693603523 bytes (0.645969 GB) > > - Unfiltering inflation factor: 4.24476x > > - Time to compute estimated result size: 0.043015 secs > > - Time to open array: 0.038626 secs > > - Total metadata read: 229077 bytes (0.000213345 GB) > > - Time to load array metadata: 7.5849e-05 secs > > - Time to initialize the read state: 0.000125849 secs > > - Read time: 5.36799 secs > > - Total read query time (array open + init state + read): 5.36812 secs > ==== Python Stats ==== > > - TileDB-Py Indexing Time: 6.25372 But when increasing the buffer size, I get almost 100% read performance improvement! ``` cfg = tiledb.Ctx().config() cfg.update( { 'py.init_buffer_bytes': 1024**2 * 400 } ) tiledb.default_ctx(cfg) ``` The default recommended parameter of `1024 **2 * 100` had no improvements, but setting it to `1024** 2 * 400` did the trick for me, reducing read-times to 3.27 s. > **Stats-Dump2** > > TileDB Embedded Version: (2, 1, 3) > TileDB-Py Version: 0.7.4 > ==== READ ==== > > - Number of read queries: 1 > > - Number of attempts until results are found: 1 > > - Number of attributes read: 5 > > - Number of dimensions read: 1 > > - Number of logical tiles overlapping the query: 90 > > - Number of physical tiles read: 540 > > - Number of cells read: 9634194 > > - Number of result cells: 9624052 > > - Percentage of useful cells read: 99.8947% > > - Number of bytes read: 117662004 bytes (0.109581 GB) > > - Number of read operations: 10099 > > - Number of bytes unfiltered: 472129351 bytes (0.439705 GB) > > - Unfiltering inflation factor: 4.01259x > > - Time to compute estimated result size: 0.00702653 secs > > - Time to open array: 0.0323722 secs > > - Total metadata read: 230241 bytes (0.000214429 GB) > > - Time to load array metadata: 7.5414e-05 secs > > - Time to initialize the read state: 0.000172829 secs > > - Read time: 3.3564 secs > > - Total read query time (array open + init state + read): 3.35657 secs > ==== Python Stats ==== > > - TileDB-Py Indexing Time: 3.46947 Anyway, great work and looking forward to future improvements! ---
**Author:** ![qiuwei](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/qiuwei/32/80_2.png) [@qiuwei](https://forum.tiledb.com/u/qiuwei)\ **Post date:** [December 28, 2020, 6:46am UTC](https://forum.tiledb.com/t/improved-performance/291/2 "2020-12-28T06:46:01Z")
@Mtrl_Scientist Excellent! By using the parameters you provided, I could reduce the reading time in my benchmark([Benchmarking tiledb read performance](https://forum.tiledb.com/t/benchmarking-tiledb-read-performance/278/3)) from ~16s to ~8s, only slower than parquet by a very small margin. Thanks for the tip! ---
**Author:** ![stavros](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@stavros](https://forum.tiledb.com/u/stavros)\ **Post date:** [December 28, 2020, 11:25am UTC](https://forum.tiledb.com/t/improved-performance/291/3 "2020-12-28T11:25:23Z")
Great to know folks, thanks for the update! More optimizations and tutorials are coming up. Stavros