[TileDB::Task] Error: Caught std::exception: vector too long (on Azure array)

1

I am hitting a vector too long error when calling tiledb.consolidate() of TileDB-Py on an array hosted in Azure Blob Storage. There were no errors when I do the same on an identical array located on local hard disk. I have tried different config values of sm.consolidate.*, sm.memory.* and vfs.azure.*, but no luck.

    tiledb.consolidate(array_uri, ctx=ctx)
  File "tiledb\libtiledb.pyx", line 3396, in tiledb.libtiledb.consolidate
  File "tiledb\libtiledb.pyx", line 3483, in tiledb.libtiledb._consolidate_timestamp
  File "tiledb\libtiledb.pyx", line 350, in tiledb.libtiledb._raise_ctx_err
  File "tiledb\libtiledb.pyx", line 335, in tiledb.libtiledb._raise_tiledb_error
tiledb.cc.TileDBError: [TileDB::Task] Error: Caught std::exception: vector too long

[Update] I tried to rebuild the array from scratch and try to run consolidate with steps=1 repeatedly, until I hit errors again. This time I am getting

[TileDB::Task] Error: Caught std::exception: bad allocation

Could anyone give me some hints?

2

Hi @slbteam08,

Thanks for getting in touch. Would it be possible to share either:

  1. a reduced test case (example code demonstrating the issue)
    or
  2. Details of your Array schema (sparse/dense, dimensions, tiling, number of attributes) and the write pattern you encounter this with - number of writes, size of each write (cells, or number of bytes for var-size data)