# Network related errors when writing on gcs **URL:** \ **Category:** Uncategorized\ **Created:** [March 22, 2021, 3:30pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318 "2021-03-22T15:30:15Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![vincentschut](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/vincentschut/32/130_2.png) [@vincentschut](https://forum.tiledb.com/u/vincentschut)\ **Post date:** [March 22, 2021, 3:30pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/1 "2021-03-22T15:30:15Z")
Hi, we’ve recently replaced our network storage format by tiledb. We use gke (kubernetes on google cloud) for our processing, gcs for storage, and python as our language. Often, many simultaneous jobs try to write tiledb stuff to gcs, and sometimes we get network related errors, like: `{ "code": 504, "message": "", "errors": [{ "message": "", "domain": "global", "reason": "gatewayTimeout" }] } } [INTERNAL])` or: `Upload part failed on: uninteresting_file_name/.tdb__tiledb_14 (Error in non-idempotent operation InsertObjectMedia: Service Unavailable [UNAVAILABLE])` Is there anything we could do (tune?) to mitigate these errors? They seem like temporal network errors, so maybe there is a setting to make tiledb retry more? Unfortunately, I saw lots of S3 related settings, but only a few gcs related ones in the docs… ---
**Author:** ![joe\_maley](https://avatars.discourse-cdn.com/v4/letter/j/bc79bd/32.png) [@joe\_maley](https://forum.tiledb.com/u/joe_maley)\ **Post date:** [March 22, 2021, 6:46pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/2 "2021-03-22T18:46:06Z")
Hi Vincent – Thanks for reaching out! As you suggest, we might be able to mitigate this with retry/timeout tuning. I will introduce GCS-specific configuration parameters that mirror the configuration options that we currently have for S3. I expect to complete and release these changes this week. Joe ---
**Author:** ![vincentschut](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/vincentschut/32/130_2.png) [@vincentschut](https://forum.tiledb.com/u/vincentschut)\ **Post date:** [March 24, 2021, 2:36pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/3 "2021-03-24T14:36:41Z")
Hi Joe, thanks for that. I saw the PR “Improve GCS retries” in github passing by (i’m following the repo, just to keep myself updated). Once a new 2.2 version is released (hopefully soon!), we’ll be happy to test it! Vincent. ---
**Author:** ![joe\_maley](https://avatars.discourse-cdn.com/v4/letter/j/bc79bd/32.png) [@joe\_maley](https://forum.tiledb.com/u/joe_maley)\ **Post date:** [March 24, 2021, 8:08pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/4 "2021-03-24T20:08:18Z")
I’ve just released v2.2.7 of the core that contains the patch you referenced. It has not yet been released in a Python package, but we will do that soon. While adding the new configuration option, `vfs.gcs.request_timeout_ms`, I realized that we also had a bug in the retry path. Upgrading to the new core version may be enough to substantially improve your situation. If transient network issues persist, I would recommend increasing that configuration parameter from its default of `3000` to something higher. We’re always interested in learning about new use-cases. If you need any further assistance, please email me directly at [joe@tiledb.com](mailto:joe@tiledb.com). Thanks! ---
**Author:** ![vincentschut](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/vincentschut/32/130_2.png) [@vincentschut](https://forum.tiledb.com/u/vincentschut)\ **Post date:** [March 26, 2021, 11:19am UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/5 "2021-03-26T11:19:02Z")
Thanks for the 2.2.7 of the core lib. If I build that, would the existing python package automatically pick up the newer core library version? Or is the only way to use it to wait for a newer python package? Btw, is there a way to check which core lib version is being used by the python package, something like `tiledb.core. __version__ `)? ---
**Author:** ![joe\_maley](https://avatars.discourse-cdn.com/v4/letter/j/bc79bd/32.png) [@joe\_maley](https://forum.tiledb.com/u/joe_maley)\ **Post date:** [March 26, 2021, 12:16pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/6 "2021-03-26T12:16:18Z")
Unfortunately, you would need to build the Python package from source to link against the new version of the core library. However, you do not need to build the core from source. To get this immediately, I would recommend downloading the binary distribution of the from the release page ([Releases · TileDB-Inc/TileDB · GitHub](https://github.com/TileDB-Inc/TileDB/releases)). To build+install: ```auto git clone https://github.com/TileDB-Inc/TileDB-Py.git cd TileDB-Py pip install -r requirements_dev.txt python setup.py install --tiledb=/path/to/the/downloaded/tiledb/ ``` ---
**Author:** ![joe\_maley](https://avatars.discourse-cdn.com/v4/letter/j/bc79bd/32.png) [@joe\_maley](https://forum.tiledb.com/u/joe_maley)\ **Post date:** [March 26, 2021, 12:42pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/7 "2021-03-26T12:42:48Z")
Additionally, you can dynamically link to a new version of the core if you are using the conda package: [Python - TileDB Docs](https://docs.tiledb.com/main/solutions/tiledb-embedded/installation/building-from-source/python) Otherwise, we will have an official release soon. Thanks! ---
**Author:** ![ihnorton](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/ihnorton/32/23_2.png) [@ihnorton](https://forum.tiledb.com/u/ihnorton)\ **Post date:** [March 26, 2021, 8:38pm UTC](https://forum.tiledb.com/t/network-related-errors-when-writing-on-gcs/318/8 "2021-03-26T20:38:34Z")
Release binaries for TileDB-Py 0.8.6 with TileDB Embedded 2.2.7 are now available [on PyPI](https://pypi.org/project/tiledb/0.8.6/). Best, Isaiah