How does locking work in tile-db?

#1

I was just wondering the inner workings of how tile-db locks the data for reading/writting. The reason that I bring it up is because I initially thought that Tile-db applies locks on data at Tile level(at least for the writes). And I also took the luxury to assume that it is an atomic lock. Meaning if one process is trying to write to a particular tile then other processes would wait until this process is done doing its job on the tile.

To test this theory, I create an array with only one tile of the dimensions (10K X 10K). Wirtting to it typically takes 12-16 seconds(I am writting on S3 nto sure if that changes anythingbut I just wanted to throw the information out there).

So far so good. So now we do have a tile of (10K X 10K) and it takes 12-16 seconds to write to it. now, I just initiated 3 processes that do write to each cell in this tile/array. What I expected to see was that array/tile would be locked for ~15 seconds when the first one is trying to write to it and then it will realse the lock for the seconds process to write ending up taking atleast(~40-45) seconds. but all three processes finished almost back to back. Which is a pleasant surprise. But makes me wonder a little bit about at what level does tile-db apply locks on?

#2

Hi there,

I would certainly start with the following docs:
https://docs.tiledb.com/developer/basic-concepts/concurrency
https://docs.tiledb.com/developer/basic-concepts/consistency

TL;DR

There is no locking when writing and reading whatsoever. You can perform multiple writes and multiple reads at the same time and they will all work without locking. The downside of that is that you need to be careful not to write to the same subarray with two processes, as any of the two writes may end up overwriting on top of the other.

The only place where TileDB locks is in a very short interval during consolidation, when it is about to delete the old consolidated fragments.

Finally, note that TileDB uses filelocking for locking, which works only on specific filesystems (e.g., it does not work on S3).

#3

Thank you. That was helpful.