# TileDB in AWS behind a corporate proxy server

**URL:** <https://forum.tiledb.com/t/tiledb-in-aws-behind-a-corporate-proxy-server/236>\
**Category:** Uncategorized\
**Created:** [August 4, 2020, 11:40am UTC](https://forum.tiledb.com/t/tiledb-in-aws-behind-a-corporate-proxy-server/236 "2020-08-04T11:40:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nickholway](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/nickholway/32/68_2.png) [@nickholway](https://forum.tiledb.com/u/nickholway)\
**Post date:** [August 4, 2020, 11:40am UTC](https://forum.tiledb.com/t/tiledb-in-aws-behind-a-corporate-proxy-server/236/1 "2020-08-04T11:40:17Z")

</div>

I’m looking at using the Python Tiledb bindings to create & read arrays in S3. We have direct access to S3 on one site and I can create Tiledb arrays, on the site that I’m based we still have to use a corporate proxy server for this and tiledb.SparseArray.create fails with

> ```
> > TileDBError: [TileDB::S3] Error: Failed to create multipart request for object '/cdf_data.tiledb/__array_schema.tdb
> > Exception:  
> > Error message: Unable to connect to endpoint
> 
> ```

Has anyone managed to use Tiledb in AWS behind a proxy server? I thought I’d check before submitting a bug report.

I’m using 0.6.6 of the TileDB bindings on Python 3.6 on Centos7. If it makes a difference we use a non-standard port on the proxy.

---

<div class="post-metadata">

**Author:** ![ihnorton](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/ihnorton/32/23_2.png) [@ihnorton](https://forum.tiledb.com/u/ihnorton)\
**Post date:** [August 5, 2020, 2:08am UTC](https://forum.tiledb.com/t/tiledb-in-aws-behind-a-corporate-proxy-server/236/2 "2020-08-05T02:08:43Z")

</div>

Hi @nickholway,

Please try to override the default value for the `vfs.s3.proxy_scheme` [configuration option](https://docs.tiledb.com/main/configuration-parameters) to _`http`_. Our current internal default is `https`, which is not usually applicable (we will update).

I was able to reproduce part of your error locally, and resolved it with the `proxy_schema` override – here’s an example that I tested locally with the squid proxy in [this docker image](https://hub.docker.com/r/datadog/squid/):

```auto
import tiledb

cfg = {
'vfs.s3.proxy_host': 'localhost',
'vfs.s3.proxy_port': '3128',
'vfs.s3.aws_access_key_id': '...',
'vfs.s3.aws_secret_access_key': '...',
'vfs.s3.proxy_scheme': 'http',
#’vfs.s3.logging_level': 'TRACE'
}
tiledb.default_ctx(config=cfg)

a = tiledb.open("s3://bucket-name/array_path")
print(a.schema)

```

If overriding `vfs.s3.proxy_scheme` does not fix the problem, the proxy may require authentication; try setting `vfs.s3.proxy_username` and `vfs.s3.proxy_password`.

Otherwise, two less likely possibilities are:

- proxy setup actually does use https, and requires a certificate override
- that the proxy requires SOCKS5 support

The last two may take some modifications on our side or in the AWS library, so please ping us at `hello <at> tiledb.com` or `isaiah <at>` if the first two suggestions don’t help. We want to make sure TileDB is usable for proxied environments, and will need to figure out which update to prioritize.

Best,  
Isaiah

---

<div class="post-metadata">

**Author:** ![nickholway](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/nickholway/32/68_2.png) [@nickholway](https://forum.tiledb.com/u/nickholway)\
**Post date:** [August 5, 2020, 8:54am UTC](https://forum.tiledb.com/t/tiledb-in-aws-behind-a-corporate-proxy-server/236/3 "2020-08-05T08:54:49Z")

</div>

Hi Isasiah,  
Thanks for the response.  
Your suggestion of setting vfs.s3.proxy\_host, vfs.s3.proxy\_port & vfs.s3.proxy\_scheme worked for me.  
A couple of suggestion:

- For the S3 docs, it’d be useful it you could include the context setting on the page as otherwise a developer has to hunt for it.
- On our systems (Jupyter hub in this case), the proxies are normally set with environment variables, it’d be good if TileDB could pick these up and use them - when troubleshooting the above I used boto3 and it “just worked”

Thanks  
Nick

---

<div class="post-metadata">

**Author:** ![ihnorton](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/ihnorton/32/23_2.png) [@ihnorton](https://forum.tiledb.com/u/ihnorton)\
**Post date:** [August 5, 2020, 1:23pm UTC](https://forum.tiledb.com/t/tiledb-in-aws-behind-a-corporate-proxy-server/236/4 "2020-08-05T13:23:20Z")

</div>

Hi Nick,

I’m glad that it worked, and thanks for the suggestion: I’ve added the proxy settings to the [AWS config docs](https://docs.tiledb.com/main/backends/s3).

Good point about the environment variables; I’ve opened a ticket to take a look at this, because it would definitely simplify such use-cases. Are you using the cURL-style `HTTP_PROXY`/`HTTPS_PROXY` variables?

Best,  
Isaiah

---

<div class="post-metadata">

**Author:** ![nickholway](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/nickholway/32/68_2.png) [@nickholway](https://forum.tiledb.com/u/nickholway)\
**Post date:** [August 5, 2020, 4:35pm UTC](https://forum.tiledb.com/t/tiledb-in-aws-behind-a-corporate-proxy-server/236/5 "2020-08-05T16:35:23Z")

</div>

> I’m glad that it worked, and thanks for the suggestion: I’ve added the proxy settings to the [AWS config docs](https://docs.tiledb.com/main/backends/s3).

Great, that’s much clearer

> Are you using the cURL-style `HTTP_PROXY` / `HTTPS_PROXY` variables?

Correct. It’d also be good to be able to override them through a) $NO\_PROXY and b) via vfs.s3.something in case you have a direct connection or are using an internal S3-like object store.

Nick