Unable to query an array created by using tiledb functionality

I have created a tiledb array using tiledb function and I want to query on those array using tiledb-presto then it throws error like

Query 20200825_073220_00002_g9pwj failed: line 1:15: Table tiledb.tiledb.s3://testbucket/processed/csv_dowjones does not exist
select * from tiledb.tiledb.“s3://testbucket/processed/csv_dowjones”

I am able to create, insert rows and query on table which is created by using tiledb-presto sql command.
I don’t know what I am missing in this.
@seth @ihnorton

@Shashikant255 Can you enable s3 logging and post the log? I’ve successfully tested both our docker image and a presto cluster accessing existing s3 arrays. I suspect your problem might be related to minio. The trace log should provide more details for us to help debug what settings need tweaking for your environment.

set session tiledb.tiledb_config='vfs.s3.logging_level=TRACE';

Please note you likely will also need the minio config settings besides just the s3 log level for tiledb_config session parameter.

Thanks @seth for your reply.

I don’t think this is related to minio because even if I give local system path in query then also it gives same issue. If I create any array using CREATE TABLE, then I am able to access it whether I kept it on local or on minio. But I if have some array which is created by tiledb function : tiledb.DenseArray.create , then I am not able to access it either it is in minio or in local system.

I have set tiledb_config session parameter. Below is the log:

presto_1  | 2020-08-28T05:17:37.787Z    DEBUG   dispatcher-query-52     io.prestosql.execution.QueryStateMachine        Query 20200828_051737_00010_eqt5v is QUEUED
presto_1  | 2020-08-28T05:17:37.788Z    DEBUG   dispatcher-query-54     io.prestosql.execution.QueryStateMachine        Query 20200828_051737_00010_eqt5v is WAITING_FOR_RESOURCES
presto_1  | 2020-08-28T05:17:37.794Z    DEBUG   dispatcher-query-51     io.prestosql.execution.QueryStateMachine        Query 20200828_051737_00010_eqt5v is FAILED
presto_1  | 2020-08-28T05:17:37.794Z    DEBUG   dispatcher-query-53     io.prestosql.execution.QueryStateMachine        Query 20200828_051737_00010_eqt5v failed
presto_1  | io.prestosql.sql.analyzer.SemanticException: line 1:15: Table tiledb.tiledb.s3://testbucket/processed/csv_dowjones does not exist
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:931)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:272)
presto_1  |     at io.prestosql.sql.tree.Table.accept(Table.java:53)
presto_1  |     at io.prestosql.sql.tree.AstVisitor.process(AstVisitor.java:27)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:286)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.analyzeFrom(StatementAnalyzer.java:1879)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:1040)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:272)
presto_1  |     at io.prestosql.sql.tree.QuerySpecification.accept(QuerySpecification.java:144)
presto_1  |     at io.prestosql.sql.tree.AstVisitor.process(AstVisitor.java:27)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:286)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:296)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:742)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:272)
presto_1  |     at io.prestosql.sql.tree.Query.accept(Query.java:107)
presto_1  |     at io.prestosql.sql.tree.AstVisitor.process(AstVisitor.java:27)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:286)
presto_1  |     at io.prestosql.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:264)
presto_1  |     at io.prestosql.sql.analyzer.Analyzer.analyze(Analyzer.java:76)
presto_1  |     at io.prestosql.sql.analyzer.Analyzer.analyze(Analyzer.java:68)
presto_1  |     at io.prestosql.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:184)
presto_1  |     at io.prestosql.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:92)
presto_1  |     at io.prestosql.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:706)
presto_1  |     at io.prestosql.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:118)
presto_1  |     at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
presto_1  |     at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
presto_1  |     at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
presto_1  |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
presto_1  |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
presto_1  |     at java.lang.Thread.run(Thread.java:748)
presto_1  |
presto_1  |

@Shashikant255 thanks for this log. There should also be a separate log file written by the aws sdk, something like tiledb_s3_2020-08-27-13.log, where the format it year-month-day-hour. This will log the S3 requests and any failures or errors. Can you post this log?

Can you also post the array schema? I’ll create a similar array and test access.

@seth It seems that issue is due to datatype of attribute.
I have created an array with following schema:-

schema = tiledb.ArraySchema(
tiledb.Dim(name=‘rows’, domain=(1, 252), tile=11, dtype=‘uint64’),
tiledb.Attr(name=‘Date’, dtype=“datetime64[Y]”),
tiledb.Attr(name=‘Open’, dtype=‘float64’),
tiledb.Attr(name=‘High’, dtype=‘float64’),
tiledb.Attr(name=‘Low’, dtype=‘float64’),
tiledb.Attr(name=‘Close’, dtype=‘float64’),
tiledb.Attr(name=‘AdjClose’, dtype=‘float64’),
tiledb.Attr(name=‘Volume’, dtype=‘int64’),
tile_order=‘row-major’, sparse=False, ctx=ctx)

In this case it say that array does not exits. If I remove that Date attribute from above schema then I am able to query the array. To overcome this issue I have changed the datatype to np.dtype('S1'). With this I am able to query it from tiledb-presto.

Is datetime64 datatype not supported in tiledb-presto?

Thank for your quick response and support.

@Shashikant255 Thanks for the schema and info. It does appear the problem is the Date field which is a TILEDB_DATETIME_YEAR datatype. Give us a few days and we’ll expand presto’s support to all DATETIME precision levels. Prestodb is slated for several upgrades over the next few weeks. We’ll get the datetime support added in 1-2 days, and then we’ll add support for heterogeneous dimensions, before moving on to other more general improvements.

Hi @Shashikant255. Thank you for reporting the issue and providing us all the info. We are working towards supporting all TileDB datetime datatypes in TileDB-Presto, and we will have released a new version with all the updates by the end of the day.


Thanks @victor_tiledb for your response.

Hi @Shashikant255. We just released a new TileDB-Presto version 1.6.1 which includes the Datetime data types support (https://github.com/TileDB-Inc/TileDB-Presto/releases/tag/1.6.1).

Let us know if you run any further issues.