Hi,
When doing large writes (>10GB arrays) on S3, it would be useful to have a tqdm-like progress bar.
Is this something that already exists? If not, what would be the easiest way for me to implement it myself?
1 Like
Hi @mukund,
Writes/assignments of the form A[ ] = ... dispatch to Array.__setitem__, which calls a C function tiledb_query_submit, and releases the Python GIL. So there is no Python interpreter involvement for the duration of the write (all of the data is provided as pointers passed in to the query object, before query_submit). A callback target to provide progress updates could be implemented on the Python side, which would re-acquire the GIL and update TQDM (or any similar target), however such a callback does not currently exist in the query class. We’ll keep this in mind as the writer code is refactored and see if it can be implemented efficiently in the future.
Best,
Isaiah
1 Like