Is there AWS session token analogues for Azure Blob storage and Google Cloud Storage TileDB backends?

1

We’ve used AWS Session Tokens to limit the availability time and access permissions for S3 compatible storage and it works well. Yet there is a question on how the same limitations could be applied on other cloud storage services (AzureBlob//GoogleCloud). There is a simmilar mechanisms in for Google cloud service accounts (https://cloud.google.com/iam/docs/creating-short-lived-service-account-credentials) and Azure SAS (https://docs.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature).

Google’s one seems to be pretty straitforward. It seems it generates .json, which could be passed to tiledb.

Is there any additional information such applications? Could it be there are some plans on adding Azure SAS integrations to tiledb?

Thank you in advance!
Dmitriy

2

Hi @DmitryLisitsin,

For Google Storage, our current recommendation is to use the GOOGLE_APPLICATION_CREDENTIALS environment variable along with the project ID passed in through the TileDB config. I believe this should also work with the temporary GCS credential system as in the link you shared (the .json file).

For Azure, we will need to add a config parameter within TileDB to pass the SAS token into the Azure storage library. We currently have a number of our own patches to the library, so we need to investigate this further to see what is required.

Best,
Isaiah

3

Hi, @ihnorton

Thank you for your reply! I didn’t find any roadmaps on tiledb development in the documentation and thus decided to post this question. Thanks for your clarifications.

Currently we are using AWS S3 backend to access storage, so the GCS/Azure support is not required for our team right now. Yet it would be convenient to be able to use tiledb with any storage provider with enhanced security by means of access tokens. It good to know that you are planning to add such support to Azure storage as well.

Waiting for new releases!

Best regards,
Dmitriy

1 Like
4

Hi @DmitryLisitsin,

I’ve put in a PR adding support for Azure SAS tokens here: [Azure] Add Azure shared access signature (SAS) support by ihnorton · Pull Request #2420 · TileDB-Inc/TileDB · GitHub

It should be available in the next TileDB release.

Best,
Isaiah

1 Like
5

Thanks for this, it was the missing piece for us to be able to implement.

Please not that there is no mention of this option in the documentation. It would be helpfull to add it to Azure Blob Storage - TileDB Embedded Docs and Configuration - TileDB Embedded Docs @ihnorton

Thanks for you good work!

1 Like
6

Hi @hernrup-modelon,

I just added a doc entry for the SAS token here – thanks for pointing this out!

Best,
Isaiah