Hi all!
Let me start with a quick acknowledgment: I am not familiar how TileDB handles (de)compression under the hood.
I am interested in reading raw (compressed) space tile data for a TileDB-backed API I am working on. We store raster weather data in compressed, dense 2d arrays. The array’s space tiles are aligned with Web Mercator tile dimensions for a given level of detail, and the API serves these Web Mercator tiles. Ideally, we wouldn’t need to decompress and recompress the data before shipping off to the client.
Any insight would be awesome! Thank you!
Regards,
Eric
Hi @weathernutt
During array creation, you can choose a variety of compression filters we make available: https://cloud.tiledb.com/academy/structure/arrays/foundation/key-concepts/storage/compression/
During each read, you do not have to worry about manually decompressing the data. We have many methods that will transform the query object into data objects specific to the TileDB language api being used.
I recommend you browse the filter types and play around with a couple different architectures to determine how the queries you’d like to make perform and are materialized.
Spencer
Hi @spencerseale,
Thank you for the reply! I have already profiled most of the compression filters/levels and selected the best balance of size/speed for our use case. My primary interest is bypassing decompression altogether when reading the data. Our API essentially passes the compressed data on to the user. Without knowing how to query the data without having TileDB automatically decompress it, I have to compress it again anyways to send it along to the user. The question here is: can I skip the automatic TileDB decompression, and subsequent manual compression, to save time?
I understand there are likely constraints - e.g. maybe the chunks that are automatically (de)compressed by TileDB are based on data tiles, not space tiles, or some other structure that doesn’t align with how I need it.
Overall, I’d like TileDB to do the least amount of work possible to get me something I can pass along. Any specific advice?
Thank you!
Regards,
Eric