More meta GetBlob requests than expected

emil.f · October 14, 2025, 1:57pm

I am trying out running TileDB with Azure and is doing some performance investigations. One thing that I have noted is that TileDB creates a lot of GetBlob requests in Azure to get the meta data.

In my test script I am writing 25 fragments and then try to get one column reaching over all fragments:

import sys
import time
import random
import uuid

import numpy as np
import tiledb

N1 = 12_060
N2 = 2500
TEST_NUM = int(sys.argv[1])
ARRAY_URI_BASE = f"azure://experimentation/array_perf_testing_{TEST_NUM}"


def _get_ctx(config: dict[str, str]):
    common_config = {
        "sm.compute_concurrency_level": 4,
        "sm.io_concurrency_level": 4,
    }
    return tiledb.Ctx(config=tiledb.Config({**common_config, **config}))


def create_array(ctx: tiledb.Ctx):
    _NUMBER_DIM_DTYPE = np.dtype(np.int32)
    _NUMBER_DIM_DTYPE = np.dtype(np.int32)
    _NUMBER_DTYPE = np.dtype(np.float64)
    time_dim = tiledb.Dim(
        name="time",
        domain=(0, np.iinfo(_NUMBER_DIM_DTYPE).max),
        tile=100,
        dtype=_NUMBER_DIM_DTYPE,
    )
    attribute_dim = tiledb.Dim(
        name="x",
        domain=(0, N1 - 1),
        tile=100,
        dtype=_NUMBER_DIM_DTYPE,
    )
    attr = tiledb.Attr(dtype=_NUMBER_DTYPE)
    domain = tiledb.Domain(time_dim, attribute_dim)
    schema = tiledb.ArraySchema(
        domain=domain,
        sparse=False,
        attrs=[attr],
        tile_order="col-major",
        cell_order="col-major",
    )
    array_uri = ARRAY_URI_BASE + "/" + uuid.uuid4().hex[0:7]
    tiledb.Array.create(array_uri, schema, ctx=ctx)
    return array_uri



def main():
    ctx = _get_ctx({})
    array_uri = create_array(ctx)

    all_data = np.array(
        [[random.random() for _ in range(N1)] for _ in range(N2)]
    )
    position = 0
    batch_size = 100
    with tiledb.open(array_uri, "w", ctx=ctx) as array:
        while True:
            next_batch = all_data[position : position + batch_size]
            if len(next_batch) == 0:
                break
            print(f"writing for position {position=}")
            array[position : position + batch_size, :] = next_batch
            position += batch_size

    with tiledb.open(array_uri, "r", ctx=ctx) as array:
        data = array[0:N2, 1]
        time.sleep(1)
        data = array[0:N2, 1000]

if __name__ == "__main__":
    main()

When I check the Azure log for all GetBlob requests produced by this there are not 25 for getting each meta blob data but instead 75! Here is the log:

"TimeGenerated [UTC]",OperationName,StatusCode,StatusText,DurationMs,ServerLatencyMs,Uri
"10/14/2025, 1:24:49.935 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__schema/__1760448286558_1760448286558_4125e9b9db15b64ee167d89e7cb26369"
"10/14/2025, 1:24:53.731 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291919_1760448291919_453a0f9df6c5b48038d3fa8edbc71a39_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.733 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292788_1760448292788_1edde05091dc35a525ae822d8663d7a0_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.734 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448289935_1760448289935_05dbeedee0e29a954b6bb44c1479949d_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.735 PM",GetBlob,206,Success,5,5,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291034_1760448291034_1eeb2d8bf1c76adbb283aa5897b28f15_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.736 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448289935_1760448289935_05dbeedee0e29a954b6bb44c1479949d_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.738 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292788_1760448292788_1edde05091dc35a525ae822d8663d7a0_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.739 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291919_1760448291919_453a0f9df6c5b48038d3fa8edbc71a39_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.740 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291034_1760448291034_1eeb2d8bf1c76adbb283aa5897b28f15_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.745 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290161_1760448290161_0130db09c947fb805b23e8f669adf904_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.746 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292928_1760448292928_66261d10fedfd60467fa41503550843f_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.747 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292062_1760448292062_485ee5e21bb214df3c18c9b4a2916b72_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.749 PM",GetBlob,206,Success,2,2,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290161_1760448290161_0130db09c947fb805b23e8f669adf904_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.751 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292928_1760448292928_66261d10fedfd60467fa41503550843f_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.752 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291174_1760448291174_78fed9df8c36dff23f72a71a82fa52f9_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.753 PM",GetBlob,206,Success,5,5,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292062_1760448292062_485ee5e21bb214df3c18c9b4a2916b72_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.756 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291174_1760448291174_78fed9df8c36dff23f72a71a82fa52f9_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.758 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290307_1760448290307_3a7e8ba9e6e59afbb5c9a05b5b230c1d_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.760 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293078_1760448293078_0bf6933a217c2c5707a0831c1c2ceee1_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.762 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292202_1760448292202_07873423d9f24f3d3ffe82fb4566b948_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.763 PM",GetBlob,206,Success,2,2,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290307_1760448290307_3a7e8ba9e6e59afbb5c9a05b5b230c1d_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.764 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293078_1760448293078_0bf6933a217c2c5707a0831c1c2ceee1_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.765 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291315_1760448291315_36841099d2ee6e89ce78cd6db0b106c6_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.768 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292202_1760448292202_07873423d9f24f3d3ffe82fb4566b948_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.769 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291315_1760448291315_36841099d2ee6e89ce78cd6db0b106c6_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.771 PM",GetBlob,206,Success,3,2,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290452_1760448290452_14195bdb6b9b869252df508f581d56ce_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.773 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293225_1760448293225_4670f7440809c8912562d3caacbce57a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.775 PM",GetBlob,206,Success,2,2,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290452_1760448290452_14195bdb6b9b869252df508f581d56ce_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.777 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293225_1760448293225_4670f7440809c8912562d3caacbce57a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.777 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292349_1760448292349_6145f23869d35d15ee0c882a9a22bc6c_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.778 PM",GetBlob,206,Success,5,5,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291464_1760448291464_779741313ec90a6ab8eab43581349b16_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.784 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291464_1760448291464_779741313ec90a6ab8eab43581349b16_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.785 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290601_1760448290601_00b9c8818e69a5776f2270c5b9d5b36a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.786 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292349_1760448292349_6145f23869d35d15ee0c882a9a22bc6c_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.786 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293374_1760448293374_16e6cf96b3c4bd1dab4807817877d08a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.789 PM",GetBlob,206,Success,2,2,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290601_1760448290601_00b9c8818e69a5776f2270c5b9d5b36a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.790 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293374_1760448293374_16e6cf96b3c4bd1dab4807817877d08a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.793 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291618_1760448291618_27e16def19ca3cba2c52bc501b29099b_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.795 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292497_1760448292497_3e9a693959d0c88bff62053330f59576_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.798 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290747_1760448290747_672e3c4b5b5b640463574582131d5c27_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.798 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291618_1760448291618_27e16def19ca3cba2c52bc501b29099b_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.798 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293521_1760448293521_593ebd755811c57bf468097555eb2dad_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.799 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292497_1760448292497_3e9a693959d0c88bff62053330f59576_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.802 PM",GetBlob,206,Success,2,2,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290747_1760448290747_672e3c4b5b5b640463574582131d5c27_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.803 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293521_1760448293521_593ebd755811c57bf468097555eb2dad_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.808 PM",GetBlob,206,Success,5,5,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291771_1760448291771_4ea9a772aec0e964d7b3d40ef98f02a6_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.811 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292641_1760448292641_792d3802d4528b69821d9a1ea34c1bc0_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.812 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290892_1760448290892_28b0fcefd2673f3ceb198df2f74a2a78_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.813 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291771_1760448291771_4ea9a772aec0e964d7b3d40ef98f02a6_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.815 PM",GetBlob,206,Success,3,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292641_1760448292641_792d3802d4528b69821d9a1ea34c1bc0_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.816 PM",GetBlob,206,Success,3,2,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290892_1760448290892_28b0fcefd2673f3ceb198df2f74a2a78_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.823 PM",GetBlob,206,Success,4,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291034_1760448291034_1eeb2d8bf1c76adbb283aa5897b28f15_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.825 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292788_1760448292788_1edde05091dc35a525ae822d8663d7a0_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.826 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448289935_1760448289935_05dbeedee0e29a954b6bb44c1479949d_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.826 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291919_1760448291919_453a0f9df6c5b48038d3fa8edbc71a39_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.829 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292928_1760448292928_66261d10fedfd60467fa41503550843f_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.830 PM",GetBlob,206,Success,4,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291174_1760448291174_78fed9df8c36dff23f72a71a82fa52f9_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.831 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292062_1760448292062_485ee5e21bb214df3c18c9b4a2916b72_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.832 PM",GetBlob,206,Success,5,5,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290161_1760448290161_0130db09c947fb805b23e8f669adf904_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.834 PM",GetBlob,206,Success,4,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291315_1760448291315_36841099d2ee6e89ce78cd6db0b106c6_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.835 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292202_1760448292202_07873423d9f24f3d3ffe82fb4566b948_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.835 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293078_1760448293078_0bf6933a217c2c5707a0831c1c2ceee1_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.837 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290307_1760448290307_3a7e8ba9e6e59afbb5c9a05b5b230c1d_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.839 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291464_1760448291464_779741313ec90a6ab8eab43581349b16_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.840 PM",GetBlob,206,Success,4,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293225_1760448293225_4670f7440809c8912562d3caacbce57a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.840 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292349_1760448292349_6145f23869d35d15ee0c882a9a22bc6c_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.842 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290452_1760448290452_14195bdb6b9b869252df508f581d56ce_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.843 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291618_1760448291618_27e16def19ca3cba2c52bc501b29099b_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.844 PM",GetBlob,206,Success,4,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293374_1760448293374_16e6cf96b3c4bd1dab4807817877d08a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.845 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292497_1760448292497_3e9a693959d0c88bff62053330f59576_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.848 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291771_1760448291771_4ea9a772aec0e964d7b3d40ef98f02a6_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.848 PM",GetBlob,206,Success,5,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290601_1760448290601_00b9c8818e69a5776f2270c5b9d5b36a_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.849 PM",GetBlob,206,Success,4,3,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448293521_1760448293521_593ebd755811c57bf468097555eb2dad_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.850 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292641_1760448292641_792d3802d4528b69821d9a1ea34c1bc0_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.853 PM",GetBlob,206,Success,4,4,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290747_1760448290747_672e3c4b5b5b640463574582131d5c27_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.859 PM",GetBlob,206,Success,5,5,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290892_1760448290892_28b0fcefd2673f3ceb198df2f74a2a78_22/__fragment_metadata.tdb"
"10/14/2025, 1:24:53.866 PM",GetBlob,206,Success,8,5,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448291174_1760448291174_78fed9df8c36dff23f72a71a82fa52f9_22/a0.tdb"
"10/14/2025, 1:24:53.868 PM",GetBlob,206,Success,8,7,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448290452_1760448290452_14195bdb6b9b869252df508f581d56ce_22/a0.tdb"
"10/14/2025, 1:24:53.868 PM",GetBlob,206,Success,8,7,"https://<REDACTED>.blob.core.windows.net:443/experimentation/array_perf_testing_1/b56417f/__fragments/__1760448292202_1760448292202_07873423d9f24f3d3ffe82fb4566b948_22/a0.tdb"
<removed getting the actual data as post was too long>

From the documentation it recommends to consolidate the meta data in these cases. So I added

    tiledb.consolidate(
        array_uri,
        ctx=ctx,
        config=tiledb.Config({"sm.consolidation.mode": "fragment_meta"}),
    )

before reading from the Array. This helps some, but the total number of GetBlob requests are the same, 2 for each meta blob during consolidation and 1 for each meta blob during reading the array. So consolidation would be better doing many reads, but would still have liked it to only read from the consolidated meta file when reading the array. (can share the Azure logs for this also if it is of interest)

Is this something that can be improved in tiledb? Is it a bug? From my point of view doing so many blob read leads to extra time and cost. Please say if you rather want this as an issue on Github. I would also be open to help solve this if that is reasonable (but have no real insight into the code base).

(using version 0.35.0 of the Python package)

teo-tsirpanis · October 27, 2025, 11:54pm

Hi @emil.f,

I ran your script with the Azurite storage emulator and managed to reproduce the behavior you observed. I noticed that you are writing to the array in batches of columns, but afterwards you read the entirety of a single row. This read query is not efficient, because the row is split into many files, and TileDB has to perform multiple GET requests to retrieve it.

After applying fragment metadata consolidation as you did, and changing array[0:N2, 1] to array[1, 0:N1], I observed that only three files are read; during the open, the consolidated fragment metadata file (should look like __fragment_meta/***.meta), and during the read, the __fragment_metadata.tdb and a0.tdb files of the fragment with the column that we asked for.

Let me know if you have further questions.

emil.f · October 28, 2025, 2:04pm

Thanks for the response @teo-tsirpanis!

Unfortunately my use case requires that data is written as columns (or rows) as this is how the data is produced/ingested. But a row (or column) is typically what the post analysis needs to be performed on so this is what is fetched.

So it fetching from many fragments is expected and acceptable (consolidation for all fragments will most likely not be worth the time it takes in majority of my cases). However, I would have liked it not fetch the meta data for each fragment multiple times. Even for your case, does it need to fetch the meta data for the fragment, should this information not be in the consolidated meta data?

So I guess my questions are:

For the case with no consolidation reading over many fragments, can we optimize so each fragment metadata is only read once? I would not mind trying to help fix this if reasonable and someone point me in the right direction.
Is there some data in the fragment metadata that are not in the consolidated metadata requiring Tiledb to read in both consolidated and fragment metadata?

teo-tsirpanis · October 29, 2025, 5:10pm

We don’t read the entire fragment metadata file at once, beacuse it can consume a lot of memory, especially on arrays with many fragments and lots of data written. We could do it when the file is “small”, but that would require design and performance trade-off considerations, and I don’t expect this to be a priority any time soon. Fragment metadata consolidation is the recommended workaround for these kinds of performance issues. You should also make sure you are reusing Array objects, in order to avoid unnecessary I/O.
Currently, the term “fragment metadata consolidation” is a bit of a misnomer; we do not consolidate the entire fragment metadata files, but only their footers. This however allows us to skip loading the metadata files of fragments we are not interested in; as I mentioned earlier, when I queried array[1, 0:N1] after fragment metadata consolidation, TileDB read from only one fragment metadata file, instead of all of them.

emil.f · October 30, 2025, 10:16am

@teo-tsirpanis Thanks a bunch for taking time explaining the technical details! This answers all my questions.

Topic		Replies	Views
How to speed up the reading from tiledb	5	2031	October 8, 2020
Write confirmation (Question)	16	2010	March 9, 2021
Nonempty-Domain for Subarrays	10	1755	April 14, 2021
TileDBError: [TileDB::S3] Error: Direct write failed!	15	6664	March 20, 2021
Tiledb performance with sparse point cloud data	8	962	March 23, 2023

More meta GetBlob requests than expected

Related topics