# AWS S3, basic read/write, "Unable to connect to endpoint with address..." **URL:** \ **Category:** Uncategorized\ **Created:** [October 17, 2019, 12:06pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112 "2019-10-17T12:06:50Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![luotzi](https://avatars.discourse-cdn.com/v4/letter/l/848f3c/32.png) [@luotzi](https://forum.tiledb.com/u/luotzi)\ **Post date:** [October 17, 2019, 12:06pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/1 "2019-10-17T12:06:50Z")
Hi, - I’m trying to write a sparse array in AWS S3 from python 3.6, Ubuntu 18.04 with the pip-installed stock version 0.4.3. conda install -c conda-forge tiledb would install a v1.\*, with which import of tiledb fails. Using conda pip, though. - The array works fine when written and read locally, but rewriting wrom scratch, adding the s3 bucket and prefix doesn’t seem to work. - First tried with no explicitly set configs, e.g. just the env vars set - also tried aws-syncing it to the bucket first, then reading it - tried to set vfs configs to the credentials and region - nevertheless I seem to get an =\> “Error message: Unable to connect to endpoint with address x.x.x.x” My AWS user has r/w privileges to the bucket. I’m using the usual AWS env vars AWS\_SECRET\_ACCESS\_KEY, AWS\_DEFAULT\_REGION, AWS\_ACCESS\_KEY\_ID , which operate fine with the AWS CLI. First tried with ‘s3://my-bucket/my-prefix/my-array-name’, then ‘s3://my-bucket/my-array-name’ , no change. Should be a fairly vanilla operation, isn’t it… any ideas, thanks in advance ? ---
**Author:** ![seth](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@seth](https://forum.tiledb.com/u/seth)\ **Post date:** [October 17, 2019, 12:22pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/2 "2019-10-17T12:22:58Z")
Hello @luotzi, The problem you’ve encountered is likely due to curl CA certificate mismatches. If you are using conda the simple fix is to uninstall tiledb from pip and use our python conda package `tiledb-py`. `conda install -c conda-forge tiledb-py` It is also possible to install from pip without using the binary wheels and compile from source, you would need to install cmake, g++ and build-essentials. `pip install tiledb --no-binary tiledb` The specifics about the issue are when curl is compiled it will detect the location of the CA certificates and it hard codes that path into the binary. Pip binary package is build for manywheel1 which uses an older centos and has the CA certificates in a separate location. We have a fix for this issue coming in the next TileDB release 1.7 by doing [runtime detection of CA Cert locations on Linux](https://github.com/TileDB-Inc/TileDB/pull/1393). Let us know if you have problems with the `tiledb-py` conda package. ---
**Author:** ![luotzi](https://avatars.discourse-cdn.com/v4/letter/l/848f3c/32.png) [@luotzi](https://forum.tiledb.com/u/luotzi)\ **Post date:** [October 17, 2019, 12:47pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/3 "2019-10-17T12:47:58Z")
(Many thanks for a swift response!) Apparently, filters have been relocated, error below… ? tiledb 1.2.2 0 conda-forge tiledb-py 0.1.1 py36h7eb728f\_1 conda-forge AttributeError: module ‘tiledb’ has no attribute ‘LZ4Filter’ ---
**Author:** ![ihnorton](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/ihnorton/32/23_2.png) [@ihnorton](https://forum.tiledb.com/u/ihnorton)\ **Post date:** [October 17, 2019, 1:18pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/4 "2019-10-17T13:18:08Z")
Hi @luotzi, That is an old version of tiledb and tiledb-py, and it doesn’t have S3 support (plus apparently some other linking issue). The latest version of tiledb-py is 0.4.3. Without seeing the rest of the environment, it is not clear why conda installed the old version, but my guess is that you have something else installed in your existing environment which prevents conda from installing the latest version due to pinning/compatibility checks. One option for testing purposes would be to create a new conda environment and ensure that the latest tiledb-py is installed there, and works for your needs: ```auto conda create -p /tmp/testenv conda activate /tmp/testenv conda install -c conda-forge tiledb-py python conda list ``` (if it doesn’t work, please send the full output of `conda list`) Hope that helps, Isaiah ---
**Author:** ![luotzi](https://avatars.discourse-cdn.com/v4/letter/l/848f3c/32.png) [@luotzi](https://forum.tiledb.com/u/luotzi)\ **Post date:** [October 23, 2019, 12:52pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/5 "2019-10-23T12:52:34Z")
Thanks, some upgrades helped. Now, I’m trying to optimize a write of a large array into S3, or actually a small part of it first. With default configs, the write succeeds but is quite slow. I have a total of 8 threads on 4 cores. I use Bzip filtering for the data, that part seems to work in parallel. Looks like S3 writes are not being parallelized at all, though. Below, a config attempt to enable all but one thread for s3 writes. Any further configuration tips ? ``` config['vfs.max_parallel_ops'] = 7 config['sm.num_writer_threads'] = 7 ``` It seems to have no effect - from the stats dump, I have . . . vfs\_s3\_num\_parts\_written, 81 vfs\_s3\_write\_num\_parallelized, 0 ## Summary: Hardware concurrency: 8 Reads: Read query submits: 0 Tile cache hit ratio: 0 / 0 Fixed-length tile data copy-to-read ratio: 0 / 0 bytes Var-length tile data copy-to-read ratio: 0 / 0 bytes Total tile data copy-to-read ratio: 0 / 0 bytes Read compression ratio: 518 / 450 bytes (1.2x) Writes: Write query submits: 20 Tiles written: 23646 Write compression ratio: 710357323 / 67085935 bytes (10.6x) ---
**Author:** ![stavros](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@stavros](https://forum.tiledb.com/u/stavros)\ **Post date:** [October 23, 2019, 1:00pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/6 "2019-10-23T13:00:34Z")
Could you please share your array schema and the write query layout? From the number of bytes and tiles written, I suspect your tile size is very small, which may be affecting the compression time. I also suspect that bzip2 compression dominates the total time (this is the slowest compressor). Finally, note that the parameter that controls parallelism on S3 is `"vfs.s3.max_parallel_ops"` (all S3 parameters have a `vfs.s3.*` prefix). `"vfs.s3.multipart_part_size"` is also relevant here. ---
**Author:** ![luotzi](https://avatars.discourse-cdn.com/v4/letter/l/848f3c/32.png) [@luotzi](https://forum.tiledb.com/u/luotzi)\ **Post date:** [October 23, 2019, 1:43pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/7 "2019-10-23T13:43:05Z")
For the complete array of which I’m now only testing small slices, 27 min out of 1h 10 min seemed to have been used for bzip2, deciding from compressor\_bzip\_compress of the dump. Tried with gzip, did not reduce total time very much, so perhaps the bottleneck really is s3 parallelization? Setting `vfs.s3.max_parallel_ops` had no effect, in the stats dump still ```auto vfs_s3_write_num_parallelized, 0 ``` although in config, ```auto config['vfs.max_parallel_ops'] = 7 config['vfs.s3.max_parallel_ops'] = 7 config['sm.num_writer_threads'] = 7 ``` Schema: ```auto CHUNK_SIZE_DIM1 = CHUNK_SIZE_DIM2 = 1000 DIMENSIONS_DATA_TYPE = np.int64 ATTR_DATA_TYPE = np.int16 CHUNK_SIZE_DIM3 = int(1e6) CHUNK_SIZE_DIM4 = int(3.5e6) COORDS_COMPRESSION = [tiledb.DoubleDeltaFilter(), tiledb.Bzip2Filter()] DATA_COMPRESSION = [tiledb.Bzip2Filter()] CHUNK_CAPACITY = 2500 def create_array(array_path, ctx=None): dom = tiledb.Domain( tiledb.Dim(name='dim1, domain=(0, DIM1_MAX), tile=CHUNK_SIZE_DIM1, dtype=DIMENSIONS_DATA_TYPE), tiledb.Dim(name='dim2', domain=(0, DIM2_MAX), tile=CHUNK_SIZE_DIM2, dtype=DIMENSIONS_DATA_TYPE), tiledb.Dim(name='dim3', domain=(DIM3_MIN, DIM3_MAX), tile=CHUNK_SIZE_DIM3, dtype=DIMENSIONS_DATA_TYPE), tiledb.Dim(name='dim4', domain=(DIM4_MIN, DIM4_MAX), tile=CHUNK_SIZE_DIM4, dtype=DIMENSIONS_DATA_TYPE)) schema = tiledb.ArraySchema(ctx=ctx, domain=dom, sparse=True, capacity = CHUNK_CAPACITY, coords_filters=tiledb.FilterList(COORDS_COMPRESSION), attrs = \ [tiledb.Attr(name='attr1', dtype=ATTR_DATA_TYPE, filters=tiledb.FilterList(DATA_COMPRESSION)), tiledb.Attr(name='attr2', dtype=ATTR_DATA_TYPE, filters=tiledb.FilterList(DATA_COMPRESSION))] ) tiledb.SparseArray.create(array_path, schema) ``` Write profile consists of hypercubes limited by dim1 and dim2, written in patches of 1000x1000. dim3 and dim4 are piecewise linearly behaving values, hence DoubleDelta (which seems to work great). All million cells in each write op have attribute values. versions: ```auto tiledb-py 0.4.3 py37hb78b526_1 conda-forge Python 3.7.4 ``` ---
**Author:** ![luotzi](https://avatars.discourse-cdn.com/v4/letter/l/848f3c/32.png) [@luotzi](https://forum.tiledb.com/u/luotzi)\ **Post date:** [October 23, 2019, 2:30pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/8 "2019-10-23T14:30:02Z")
The above stats were just for a small slice, since the whole array takes quite a long time. Here are the stats for one complete run I made with tile size of (dim1, dim2) = (10,10). Changing to (1000,1000) didn’t seem to make a difference… . . . writer\_num\_attr\_tiles\_written, 910971 writer\_num\_bytes\_before\_filtering, 27327986064 writer\_num\_bytes\_written, 2675107932 sm\_contexts\_created, 1 sm\_query\_submit\_layout\_col\_major, 0 sm\_query\_submit\_layout\_row\_major, 0 sm\_query\_submit\_layout\_global\_order, 0 sm\_query\_submit\_layout\_unordered, 792 sm\_query\_submit\_read, 0 sm\_query\_submit\_write, 792 tileio\_read\_num\_bytes\_read, 450 tileio\_read\_num\_resulting\_bytes, 518 tileio\_write\_num\_bytes\_written, 9560597 tileio\_write\_num\_input\_bytes, 39039583 vfs\_read\_total\_bytes, 450 vfs\_write\_total\_bytes, 2684819009 vfs\_read\_num\_parallelized, 0 vfs\_read\_all\_total\_regions, 0 vfs\_posix\_write\_num\_parallelized, 0 vfs\_win32\_write\_num\_parallelized, 0 vfs\_s3\_num\_parts\_written, 3169 vfs\_s3\_write\_num\_parallelized, 0 ## Summary: Hardware concurrency: 8 Reads: Read query submits: 0 Tile cache hit ratio: 0 / 0 Fixed-length tile data copy-to-read ratio: 0 / 0 bytes Var-length tile data copy-to-read ratio: 0 / 0 bytes Total tile data copy-to-read ratio: 0 / 0 bytes Read compression ratio: 518 / 450 bytes (1.2x) Writes: Write query submits: 792 Tiles written: 910971 Write compression ratio: 27367025647 / 2684668529 bytes (10.2x) ---
**Author:** ![stavros](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@stavros](https://forum.tiledb.com/u/stavros)\ **Post date:** [October 23, 2019, 2:33pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/9 "2019-10-23T14:33:29Z")
OK, a couple of interesting comments to make. ## On write parallelism - TileDB parallelizes writes across attributes (i.e., tiles across attributes are written in parallel) - TileDB _potentially_ parallelizes the write of _sequential_ tiles on each attribute. That is, it takes the tiles that will be placed contiguously in the attribute file, and if their collective size is big enough, it issues a multipart upload to S3. This is based on `vfs.s3.max_parallel_ops` and `vfs.s3.multipart_part_size`, so if the tiles you are writing in a single write session are smaller than `vfs.s3.multipart_part_size`, the tiles are written serially. Since you are writing 1,000,000 cells at a time, this means that for the coordinate tiles you end up writing `1,000,000 * 4 (dimensions) * 8 (bytes) = 32,000,000 bytes`. But since you experience ~10x compression ratio, TileDB really ends up writing ~3MB. The default `vfs.s3.multipart_part_size` is 5MB, and this is why you don’t see any parallelism here. Note though that you do get parallelism across attributes, i.e., TileDB writes the coordinate tiles, and the attribute tiles on `attr1` and `attr2` in parallel, but essentially you only utilize only 3 of your threads. Also note that the sizes of `attr1` and `attr2` are considerably smaller (~200KB compressed each). In other words, each of your writes is too small, thus not being able to fully take advantage of TileDB’s parallelism and while the S3 latency must be killing you. ## On compression parallelism - TileDB parallelizes across attributes (and coordinates) - TileDB parallelizes within each tile by chunking. Default chunking is 64KB. You tile capacity (`CHUNK_CAPACITY`) is 2500, so each coordinate tile is `2500 * 4 * 8 = 80,000` bytes, and your attribute tile is `2500 * 2 = 5000 bytes`. This is too small to take advantage of (i) good compression ratio from the compressors, and (ii) parallelism across tile chunks and L1 cache locality. ## Solution 1. Increase your chunk capacity to be in the order of 100,000 or 1,000,000, so that the tiles are big enough to be decomposed into chunks that will be compressed in parallel. 2. Perform much larger writes. Make sure that each time you go to S3, you go for real, i.e., for many MBs at a time for each attribute. So perhaps buffer many more cells and dispatch big writes in `TILEDB_UNORDERED` mode (so that you don’t worry about sorting - TileDB will do it for you). ## On array design Are you sure that the array should be sparse? The first 2 of your dimensions seem dense. How about you move `DIM3` and `DIM4` to attributes and create a 2D dense array instead? If the first two dimensions are selective enough, you can filter `DIM3` and `DIM4` later outside of TileDB without too much overhead. I expect this to be _much_ more efficient. ---
**Author:** ![luotzi](https://avatars.discourse-cdn.com/v4/letter/l/848f3c/32.png) [@luotzi](https://forum.tiledb.com/u/luotzi)\ **Post date:** [October 23, 2019, 2:40pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/10 "2019-10-23T14:40:14Z")
(Thanks again for fast and to-the-point replies. ) Will study those tips in detail… In the meantime, a quick answer to the sparse vs. dense question: the main motivation of the application really is to be able to slice the read queries by dim3 and dim4, which are indeed sparse… ---
**Author:** ![stavros](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@stavros](https://forum.tiledb.com/u/stavros)\ **Post date:** [October 23, 2019, 2:43pm UTC](https://forum.tiledb.com/t/aws-s3-basic-read-write-unable-to-connect-to-endpoint-with-address/112/11 "2019-10-23T14:43:00Z")
Regarding “bigger writes”, you can also consider writing in `TILEDB_GLOBAL_ORDER` ([https://docs.tiledb.io/en/stable/tutorials/writing-sparse.html#writing-in-global-layout](https://docs.tiledb.io/en/stable/tutorials/writing-sparse.html#writing-in-global-layout)), as TileDB will be doing the proper buffering before writing to S3. The downside is that you need to provide the cells sorted to TileDB in the global order, which is cumbersome. If slicing needs to be done by `DIM3` and `DIM4`, I would suggest making those the two dimensions of a sparse array, and move `DIM1` and `DIM2` to attributes. In the very least, I would suggest moving those two as the _first_ two dimensions, if you think that those are indeed the selective dimensions.