MyTile issue, following the README instructions


I’m trying to experiment with TileDB MariaDB MyTile engine, following instructions there: GitHub - TileDB-Inc/TileDB-MariaDB: MyTile is a MariaDB storage engine for accessing TileDB arrays.

docker run --rm -e MYSQL_ALLOW_EMPTY_PASSWORD=1 -e AWS_ACCESS_KEY_ID=somekey -e AWS_SECRET_ACCESS_KEY=somesecret -it tiledb/tiledb-mariadb

I have a tiledb array on s3, let’s say s3://my-bucket/assay-1.tiledb. My understanding is there’s no cataloging needed, and I should be able to query that array with:

> select * from `s3://my-bucket/assay-1.tiledb`;
ERROR 1146 (42S02): Table 'test.s3://my-bucket/assay-1.tiledb' doesn't exist

The credentials I passed are correct, as I can use them to open that array directly from python console, pointing to that same s3 url. Not sure how to skip the test database, and it’s even needed.

Creating an array gives an error too:

> CREATE TABLE `s3://my-bucket/regions`(
  regionkey bigint WITH (dimension=true),
  name varchar,
  comment varchar
  ) uri = 's3://my-bucket/region' array_type='SPARSE';
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'WITH (dimension=true),   name varchar,   comment varchar   ) uri = 's3://my-bucket...' at line 1

I checked the MyTile engine is indeed available with SHOW ENGINES, it seems fine.

I’m not sure what I’m doing wrong there, I tried previous docker image versions, 0.10.0 and 0.9.5, same results. Any hints?


@slelong two immediate things come to mind. The first is if you need to to set a region? You can tell TileDB the region of the bucket with SET mytile_tiledb_config="vfs.s3.region=us-east-1".

The other issue that might be happening is there is a max of 64 characters for the “table name” in MariaDB. If the s3 path is more than 64 characters you can query it by first calling create table and just passing the uri field, see the example at: Usage - TileDB MariaDB .

When creating an array you can similarly set a short “table name” and the uri option to the full s3 path if it is longer than 64 characters.

@seth , thanks that was helpful. I tried to set the region as you mentioned, still didn’t work, but passing the whole config with SET mytile_tiledb_config, including vfs.s3.aws_access_key_id, vfs.s3.aws_secret_access_key fixed the issue. It’s pretty magical :slight_smile:

I also had a look at the max 64 chars for a table name, this works fine, except I have a @ char in my s3 keys, which gives an error ERROR 1030 (HY000): Got error 155 "The table does not exist in the storage engine" from storage engine MyTile. Using the engine Aria, I could create a table with an @ by quoting the name, eg:

CREATE TABLE `products@tbl` (product_id INT NOT NULL AUTO_INCREMENT, PRIMARY KEY (product_id));

But I couldn’t find my way doing the same with MyTile:

# quotes
create table my_array4 engine=mytile uri='s3://my-bucket/project_id@version/assay-1.tiledb';
# backticks
create table my_array4 engine=mytile uri=`s3://my-bucket/project_id@version/assay-1.tiledb`;

Is there way to do that? Thanks!

Hello @slelong, we have released a new version of MyTile 0.10.2 which fixes the issue with @ in the middle of a URI. Can you please pull the 0.10.2 docker image (tiledb/tiledb-mariadb:0.10.2) and try again?

Hi @seth, I confirm, it works with 0.10.2, that’s awesome, thanks!

1 Like