How to speedup reading form a sparse TileDB array

1

Hello,
we use TileDB to store GPR-Data (Ground Penetrating Radar) which is in our case voxel data.
A typical Dataset has dimension of around 10000x10000x100, which we store as a sparse TileDB array.

Writing to TileDB is not really an issue since we have to do is just once for each dataset.
Our Renderer works with 323 fixed size blocks, we use TileDB to directly retrieve blocks of this size from the array.
Since we retrieve each block separately, there are many queries submitted.
In order to speed up we set the TileDB arrays ordering to match our renderer and set the space tile extends to 32 for each dimension.

I am wondering if there are further possibilities to speed up reading from the dataset.
We retrieved TileDB Statistic but do not know what to do with them.

Any help to speed up reading is greatly appreciated.

Thank you very much
Johannes

Statistics Output:

[
  {
    "timers": {
      "Context.StorageManager.Query.Subarray.read_load_relevant_rtrees.sum": 0.895914,
      "Context.StorageManager.Query.Subarray.read_load_relevant_rtrees.avg": 1.79179e-05,
      "Context.StorageManager.Query.Subarray.read_compute_tile_overlap.sum": 14.8978,
      "Context.StorageManager.Query.Subarray.read_compute_tile_overlap.avg": 0.000148975,
      "Context.StorageManager.Query.Subarray.read_compute_relevant_tile_overlap.sum": 3.84849,
      "Context.StorageManager.Query.Subarray.read_compute_relevant_tile_overlap.avg": 7.69683e-05,
      "Context.StorageManager.Query.Subarray.read_compute_relevant_frags.sum": 8.96206,
      "Context.StorageManager.Query.Subarray.read_compute_relevant_frags.avg": 0.000179238,
      "Context.StorageManager.Query.Subarray.read_compute_est_result_size.sum": 18.7467,
      "Context.StorageManager.Query.Subarray.read_compute_est_result_size.avg": 0.000374926,
      "Context.StorageManager.Query.Reader.unfilter_coord_tiles.sum": 212.037,
      "Context.StorageManager.Query.Reader.unfilter_coord_tiles.avg": 0.00106016,
      "Context.StorageManager.Query.Reader.unfilter_attr_tiles.sum": 59.204,
      "Context.StorageManager.Query.Reader.unfilter_attr_tiles.avg": 0.00118406,
      "Context.StorageManager.Query.Reader.read.sum": 581.246,
      "Context.StorageManager.Query.Reader.read.avg": 0.0116247,
      "Context.StorageManager.Query.Reader.load_tile_offsets.sum": 2.01493,
      "Context.StorageManager.Query.Reader.load_tile_offsets.avg": 1.34326e-05,
      "Context.StorageManager.Query.Reader.init_state.sum": 3.19643,
      "Context.StorageManager.Query.Reader.init_state.avg": 6.39273e-05,
      "Context.StorageManager.Query.Reader.copy_fixed_coords.sum": 4.58645,
      "Context.StorageManager.Query.Reader.copy_fixed_coords.avg": 3.05757e-05,
      "Context.StorageManager.Query.Reader.copy_fixed_attr_values.sum": 1.68753,
      "Context.StorageManager.Query.Reader.copy_fixed_attr_values.avg": 3.37499e-05,
      "Context.StorageManager.Query.Reader.copy_coordinates.sum": 25.8148,
      "Context.StorageManager.Query.Reader.copy_coordinates.avg": 0.000516287,
      "Context.StorageManager.Query.Reader.copy_attr_values.sum": 116.873,
      "Context.StorageManager.Query.Reader.copy_attr_values.avg": 0.0023374,
      "Context.StorageManager.Query.Reader.coord_tiles.sum": 23.4975,
      "Context.StorageManager.Query.Reader.coord_tiles.avg": 0.00023497,
      "Context.StorageManager.Query.Reader.compute_subarray_coords.sum": 53.5172,
      "Context.StorageManager.Query.Reader.compute_subarray_coords.avg": 0.00107032,
      "Context.StorageManager.Query.Reader.compute_sparse_result_tiles.sum": 1.90832,
      "Context.StorageManager.Query.Reader.compute_sparse_result_tiles.avg": 3.81657e-05,
      "Context.StorageManager.Query.Reader.compute_sparse_result_cell_slabs_sparse.sum": 1.6388,
      "Context.StorageManager.Query.Reader.compute_sparse_result_cell_slabs_sparse.avg": 3.27753e-05,
      "Context.StorageManager.Query.Reader.compute_result_coords.sum": 427.655,
      "Context.StorageManager.Query.Reader.compute_result_coords.avg": 0.00855292,
      "Context.StorageManager.Query.Reader.compute_range_result_coords.sum": 130.049,
      "Context.StorageManager.Query.Reader.compute_range_result_coords.avg": 0.00260092,
      "Context.StorageManager.Query.Reader.attr_tiles.sum": 48.3776,
      "Context.StorageManager.Query.Reader.attr_tiles.avg": 0.000967533,
      "Context.StorageManager.Query.Reader.SubarrayPartitioner.read_next_partition.sum": 4.18022,
      "Context.StorageManager.Query.Reader.SubarrayPartitioner.read_next_partition.avg": 8.36027e-05
    },
    "counters": {
      "Context.StorageManager.read_unfiltered_byte_num": 7221064,
      "Context.StorageManager.read_tile_offsets_size": 3938272,
      "Context.StorageManager.read_rtree_size": 3282792,
      "Context.StorageManager.VFS.read_ops_num": 194319,
      "Context.StorageManager.VFS.read_byte_num": 18992982601,
      "Context.StorageManager.Query.Subarray.precompute_tile_overlap.tile_overlap_cache_hit": 50001,
      "Context.StorageManager.Query.Subarray.precompute_tile_overlap.tile_overlap_byte_size": 43001456,
      "Context.StorageManager.Query.Subarray.precompute_tile_overlap.relevant_fragment_num": 50001,
      "Context.StorageManager.Query.Subarray.precompute_tile_overlap.ranges_requested": 50001,
      "Context.StorageManager.Query.Subarray.precompute_tile_overlap.ranges_computed": 50001,
      "Context.StorageManager.Query.Subarray.precompute_tile_overlap.fragment_num": 800016,
      "Context.StorageManager.Query.Reader.result_num": 1006485930,
      "Context.StorageManager.Query.Reader.read_unfiltered_byte_num": 46004596800,
      "Context.StorageManager.Query.Reader.overlap_tile_num": 287543,
      "Context.StorageManager.Query.Reader.loop_num": 50001,
      "Context.StorageManager.Query.Reader.dim_num": 150003,
      "Context.StorageManager.Query.Reader.dim_fixed_num": 150003,
      "Context.StorageManager.Query.Reader.cell_num": 2875287300,
      "Context.StorageManager.Query.Reader.attr_num": 50001,
      "Context.StorageManager.Query.Reader.attr_fixed_num": 50001,
      "Context.StorageManager.Query.Reader.SubarrayPartitioner.compute_current_start_end.ranges": 50001,
      "Context.StorageManager.Query.Reader.SubarrayPartitioner.compute_current_start_end.found": 50001,
      "Context.StorageManager.Query.Reader.SubarrayPartitioner.compute_current_start_end.adjusted_ranges": 50001
    }
  }
]
2

Hi @Johannes,
Thanks for posting. A few background questions:

  • What language API and version of TileDB are you using?
  • Is it possible to share your array schema, perhaps with dimension/attribute names changed?
  • What percent of the dimension cube is filled with these blocks? I would usually think of a voxel dataset as dense, unless you are storing something like a sparse voxel octree. (based on the 32^3 block size and number of queries, it looks like about 16% - is that correct?)
  • How are you storing which blocks are actually filled, in order to query those? We might be able to suggest a query strategy which can retrieve multiple blocks per query, which should help.

Some initial suggestions:

  • Make sure that the array capacity is equal to an integer multiple of your block size.
  • Given that you know ahead of time the size and (I assume) the coordinate bounds of each individual block, you may save time by avoiding the return of individual coordinates. With TileDB-Py you can do:
a = tiledb.open(uri)
a.query(dims=False)[<ranges>]

In C/C++/C#/Go APIs, you can avoid returning the coordinates by not setting any buffers for the dimensions.

Best,
Isaiah

3

Thank you for your reply,

we use the C++ API of TileDB 2.5.2 .

Our Schema is rather simply, we store a single float value for each voxel.
One can think of our data as a narrow band below the ground surface with a thickness of around 1m which equals 20 voxels in the following schema.

ArraySchema(
  domain=Domain(*[
    Dim(name='d0', domain=(0, 11213), tile='32', dtype='int32'),
    Dim(name='d1', domain=(0, 7471), tile='32', dtype='int32'),
    Dim(name='d2', domain=(0, 43), tile='32', dtype='int32'),
  ]),
  attrs=[
    Attr(name='a', dtype='float32', var=False, nullable=False),
  ],
  cell_order='col-major',
  tile_order='col-major',
  capacity=10000,
  sparse=True,
  allows_duplicates=False,
  coords_filters=FilterList([ZstdFilter(level=-1)]),
)
non_empty_domain ((0, 11213), (0, 7471), (0, 43))
spacing (40.0, 40.0, 50.0) //mm

We use 46% (76185) of all possible blocks of this dataset.
The fill statistices for our 323 active blocks in this example:
Mean 62.9% Std 29.5%, Median 37.5% Range [0.003% 100%]
In any case I will create an dense version of the array for comparison.

I will start with setting the capacity, which seems to be completely off.

We store the used blocks information in a separate active-blocks sparse TileDB array, which is 32-times smaller than the original array. By querying the complete array, the returned coordinates contain the indices of active/used blocks.

Thanks again,
Johannes

4

Hi @Johannes,

Have you tried performance with a newer version? We do provide builds via conda-forge and directly. For Linux, those should both be compatible with glibc-based Linux distros back to CentOS 7.

Best,
Isaiah