Create tiledbvcf dataset on azure on MacOS

1

Hi, I have installed tiledbvcf-py and tiledb using conda.
The versions are:
TileDB-VCF version 0.34.1
TileDB version 2.25.0
htslib version 1.20

I am trying to create a dataset on an azure storage account authenticating using the storage account key.
However, I get an error when running ds.create_dataset():
File “/opt/miniconda3/envs/tiledbvcf_env/lib/python3.12/site-packages/tiledbvcf/dataset.py”, line 685, in create_dataset
self.writer.create_dataset()
RuntimeError: TileDB-VCF exception: TileDB internal: 404 The specified blob does not exist.

I am running this on MacOS

Thanks

2

Hi JP,

Did you configure your tiledbvcf.Dataset object to work with your Azure backend? See configuration docs here: https://docs.tiledb.com/main/how-to/configuration.

For all tiledb APIs, the URIs they manipulate are meant to be filesystem agnostic. In order to do this, each API will in someway take a configuration object to configure that backend.

Thanks,
Spencer

3

Thank you Spencer for your help!
I have vfs.azure.storage_account_name and vfs.azure.storage_account_key configured and the URI has the format f"azure://{container_name}/{dataset_name}". I am able to run the same code on Linux succesfully. Later I tested reading from this already created tiledbvcf.Dataset on MacOS and I was also able to do it. The one thing that is not working is trying to create the dataset or writing to it using MacOS.

4

Hi @jpissinis,
Just to double-check, do have identical code that is working on linux but failing on mac? Have you confirmed that the Azure configuration on mac works (for example, list the bucket prefix with Azure CLI). We’re not aware of any issues like this at the moment. It would be helpful if you’re able to share a minimal reproduction with TileDB-VCF or the TileDB Python API ( $ conda install -c conda-forge tiledb-py) (which uses the same underlying libtiledb library as TileDB-VCF, when installed w/ conda).

5

Hi @ihnorton,
Thank you for your help. I am trying to reproduce this on a fresh environment to be able to share a minimal reproduction, however, I run now into another issue on MacOS when installing the latest version of tiledbvcf. I get the following when trying to import it:
import tiledbvcf
File “/opt/miniconda3/envs/tiledbvcf_write/lib/python3.12/site-packages/tiledbvcf/init.py”, line 9, in
import tiledbvcf.libtiledbvcf
ImportError: dlopen(/opt/miniconda3/envs/tiledbvcf_write/lib/python3.12/site-packages/tiledbvcf/libtiledbvcf.cpython-312-darwin.so, 0x0002): Symbol not found: __ZN6google5cloud5v2_2823MakeInsecureCredentialsENS1_7OptionsE
Referenced from: <1A63D6D0-49DB-3A68-829B-D4DF679EC466> /opt/miniconda3/envs/tiledbvcf_write/lib/python3.12/site-packages/tiledbvcf/libtiledb.dylib
Expected in: <708779A3-90F4-34E6-944D-5EB418DB2D24> /opt/miniconda3/envs/tiledbvcf_write/lib/libgoogle_cloud_cpp_common.2.30.0.dylib

I tried the fix in Tiledbvcf installation error on MacOS but downgrading libgoogle did not solve it in my case.

The tiledb versions are now:
TileDB-VCF version 0.34.2
TileDB version 2.25.0
htslib version 1.20

And I installed them using:
conda install -c conda-forge tiledb-py
conda install -c tiledb -c conda-forge libtiledbvcf tiledbvcf-py

Thank you