Write confirmation (Question)

14

Hi Stavros,

Since multiple people are actively using it, I couldn’t change to do consolidation during processes from a single machine. What I’m doing is, regularly I’m running consolidation script. However it is taking incredibly long (over 8 hours it only could do 120 consolidation loops as below). What I’m doing is

import tiledb
for i in range(1000):
    print(i)
    config = tiledb.Config({'sm.consolidation.step_min_frags':2,'sm.consolidation.step_max_frags':20,'sm.consolidation.steps':1})
    ctx = tiledb.Ctx(config)
    table_name='example.tldb'
    tiledb.consolidate(table_name, ctx=ctx)
    tiledb.vacuum(table_name)

Is it in accordance with your suggestion? Is there any other way to make it faster? The reason why I can’t just let it run without a loop when db is idle is my kernel died with 64 GB RAM, that’s why I’m doing in such a loop. I would appreciate if you can show any pointer

1 Like