# Dask Array to TileDB with Google Cloud Storage Issue

**URL:** <https://forum.tiledb.com/t/dask-array-to-tiledb-with-google-cloud-storage-issue/475>\
**Category:** Uncategorized\
**Created:** [June 28, 2022, 3:04pm UTC](https://forum.tiledb.com/t/dask-array-to-tiledb-with-google-cloud-storage-issue/475 "2022-06-28T15:04:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jgreen](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/jgreen/32/217_2.png) [@jgreen](https://forum.tiledb.com/u/jgreen)\
**Post date:** [June 28, 2022, 3:04pm UTC](https://forum.tiledb.com/t/dask-array-to-tiledb-with-google-cloud-storage-issue/475/1 "2022-06-28T15:04:46Z")

</div>

I am trying to convert a Dask Array and write directly to Google Cloud Storage in Python, but I am getting the error:  
`TileDBError: [TileDB::Array] Error: Cannot open array; Array does not exist` whenever I attempt to use `da.to_tiledb("gcs://mybucket/array-name.tldb")`.

I have configured the `GOOGLE_APPLICATION_CREDENTIALS` environment variable with my bucket key, set `config['vfs.gcs.project_id'] = 'my-project-id'`, as well as attempting to pass the credentials file to the `storage_options` keyword argument, with all yielding the same error. The [Dask documentation](https://docs.dask.org/en/stable/generated/dask.array.to_tiledb.html) mentions that this method should work with “any TileDB-supported URI, including local disk, S3, or HDFS.”

Writing to local disk works as expected, leading me to believe that perhaps Google Cloud functionality is not supported with this method. I could create an in-memory numpy array of my data, but I would much rather do this out-of-core to eventually convert larger datasets.

Here is my code, as well as some information about the `numpy.ndarray` I am attempting to write to TileDB Embedded:

 ![image](https://canada1.discourse-cdn.com/flex035/uploads/tiledb/original/1X/8bbeda5701000c9d58d0e9ea201182962ede9781.png)

Note: I know that a Python adapter to convert NetCDF to TileDB exists, but I’ve been having a lot of trouble getting it to work in my environment. Because of this I switched to the Dask Array to TileDB method.

---

<div class="post-metadata">

**Author:** ![julia](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/julia/32/116_2.png) [@julia](https://forum.tiledb.com/u/julia)\
**Post date:** [June 28, 2022, 5:45pm UTC](https://forum.tiledb.com/t/dask-array-to-tiledb-with-google-cloud-storage-issue/475/2 "2022-06-28T17:45:44Z")

</div>

Hi @jgreen, thanks for reaching out! Someone else will follow-up with you on the GCS portions of this question, but I would like to check in on the issues you were having with converting NetCDF files with the Python library. First, which library did you attempt to use? The one we are supporting internally is [tiledb-cf](https://pypi.org/project/tiledb-cf/), but it sometimes gets buried under the tiledb\_netcdf library developed by the Met Office Informatics Lab. If you were using tiledb-cf, would you be willing to provide more information about the issues you were hitting?

---

<div class="post-metadata">

**Author:** ![jgreen](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/jgreen/32/217_2.png) [@jgreen](https://forum.tiledb.com/u/jgreen)\
**Post date:** [June 28, 2022, 7:04pm UTC](https://forum.tiledb.com/t/dask-array-to-tiledb-with-google-cloud-storage-issue/475/3 "2022-06-28T19:04:07Z")

</div>

Thanks for the reply! I was unaware that there were two libraries, but I was indeed using the `tiledb_netcdf` library instead of `tiledb-cf`. Switching to latter and using the command line interface worked without issue. Looking forward to going through the API documentation, thanks again!

---

<div class="post-metadata">

**Author:** ![jgreen](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/jgreen/32/217_2.png) [@jgreen](https://forum.tiledb.com/u/jgreen)\
**Post date:** [June 29, 2022, 2:44pm UTC](https://forum.tiledb.com/t/dask-array-to-tiledb-with-google-cloud-storage-issue/475/4 "2022-06-29T14:44:25Z")

</div>

UPDATE: I figured out why I was receiving the `TileDBError: [TileDB::Array] Error: Cannot open array; Array does not exist`. When I exported the environment variable, it was on another command line instance. I am working in a proprietary middleware system, which is still in development so sometimes exhibits odd behavior when more than one terminal is open in a different browser tab.

Even when using the `os` module, odd behavior is observed (once again, thanks to the software I am using to run everything on). To set the correct environment variable with my credentials, I found the most reliable method to be `%%bash` cell magic, although I will continue to experiment for the most general way to do this.

---

<div class="post-metadata">

**Author:** ![ihnorton](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/ihnorton/32/23_2.png) [@ihnorton](https://forum.tiledb.com/u/ihnorton)\
**Post date:** [July 7, 2022, 2:27pm UTC](https://forum.tiledb.com/t/dask-array-to-tiledb-with-google-cloud-storage-issue/475/5 "2022-07-07T14:27:30Z")

</div>

Hi @jgreen,

I’ve added a story on our backlog to look at setting GCS credentials without use of environment variables.

Best,  
Isaiah
