I have created a tiledb array using tiledb function and I want to query on those array using tiledb-presto then it throws error like
Query 20200825_073220_00002_g9pwj failed: line 1:15: Table tiledb.tiledb.s3://testbucket/processed/csv_dowjones does not exist
select * from tiledb.tiledb.“s3://testbucket/processed/csv_dowjones”
I am able to create, insert rows and query on table which is created by using tiledb-presto sql command.
I don’t know what I am missing in this.
@seth @ihnorton
@Shashikant255 Can you enable s3 logging and post the log? I’ve successfully tested both our docker image and a presto cluster accessing existing s3 arrays. I suspect your problem might be related to minio. The trace log should provide more details for us to help debug what settings need tweaking for your environment.
set session tiledb.tiledb_config='vfs.s3.logging_level=TRACE';
Please note you likely will also need the minio config settings besides just the s3 log level for tiledb_config session parameter.
Thanks @seth for your reply.
I don’t think this is related to minio because even if I give local system path in query then also it gives same issue. If I create any array using CREATE TABLE, then I am able to access it whether I kept it on local or on minio. But I if have some array which is created by tiledb function : tiledb.DenseArray.create , then I am not able to access it either it is in minio or in local system.
I have set tiledb_config session parameter. Below is the log:
presto_1 | 2020-08-28T05:17:37.787Z DEBUG dispatcher-query-52 io.prestosql.execution.QueryStateMachine Query 20200828_051737_00010_eqt5v is QUEUED presto_1 | 2020-08-28T05:17:37.788Z DEBUG dispatcher-query-54 io.prestosql.execution.QueryStateMachine Query 20200828_051737_00010_eqt5v is WAITING_FOR_RESOURCES presto_1 | 2020-08-28T05:17:37.794Z DEBUG dispatcher-query-51 io.prestosql.execution.QueryStateMachine Query 20200828_051737_00010_eqt5v is FAILED presto_1 | 2020-08-28T05:17:37.794Z DEBUG dispatcher-query-53 io.prestosql.execution.QueryStateMachine Query 20200828_051737_00010_eqt5v failed presto_1 | io.prestosql.sql.analyzer.SemanticException: line 1:15: Table tiledb.tiledb.s3://testbucket/processed/csv_dowjones does not exist presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:931) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitTable(StatementAnalyzer.java:272) presto_1 | at io.prestosql.sql.tree.Table.accept(Table.java:53) presto_1 | at io.prestosql.sql.tree.AstVisitor.process(AstVisitor.java:27) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:286) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.analyzeFrom(StatementAnalyzer.java:1879) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:1040) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:272) presto_1 | at io.prestosql.sql.tree.QuerySpecification.accept(QuerySpecification.java:144) presto_1 | at io.prestosql.sql.tree.AstVisitor.process(AstVisitor.java:27) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:286) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:296) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:742) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:272) presto_1 | at io.prestosql.sql.tree.Query.accept(Query.java:107) presto_1 | at io.prestosql.sql.tree.AstVisitor.process(AstVisitor.java:27) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:286) presto_1 | at io.prestosql.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:264) presto_1 | at io.prestosql.sql.analyzer.Analyzer.analyze(Analyzer.java:76) presto_1 | at io.prestosql.sql.analyzer.Analyzer.analyze(Analyzer.java:68) presto_1 | at io.prestosql.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:184) presto_1 | at io.prestosql.execution.SqlQueryExecution.<init>(SqlQueryExecution.java:92) presto_1 | at io.prestosql.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:706) presto_1 | at io.prestosql.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:118) presto_1 | at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) presto_1 | at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) presto_1 | at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) presto_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) presto_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) presto_1 | at java.lang.Thread.run(Thread.java:748) presto_1 | presto_1 |
@Shashikant255 thanks for this log. There should also be a separate log file written by the aws sdk, something like tiledb_s3_2020-08-27-13.log, where the format it year-month-day-hour. This will log the S3 requests and any failures or errors. Can you post this log?
Can you also post the array schema? I’ll create a similar array and test access.
@seth It seems that issue is due to datatype of attribute.
I have created an array with following schema:-
schema = tiledb.ArraySchema(
domain=tiledb.Domain(*[
tiledb.Dim(name=‘rows’, domain=(1, 252), tile=11, dtype=‘uint64’),
]),
attrs=[
tiledb.Attr(name=‘Date’, dtype=“datetime64[Y]”),
tiledb.Attr(name=‘Open’, dtype=‘float64’),
tiledb.Attr(name=‘High’, dtype=‘float64’),
tiledb.Attr(name=‘Low’, dtype=‘float64’),
tiledb.Attr(name=‘Close’, dtype=‘float64’),
tiledb.Attr(name=‘AdjClose’, dtype=‘float64’),
tiledb.Attr(name=‘Volume’, dtype=‘int64’),
],
cell_order=‘row-major’,
tile_order=‘row-major’, sparse=False, ctx=ctx)
In this case it say that array does not exits. If I remove that Date attribute from above schema then I am able to query the array. To overcome this issue I have changed the datatype to np.dtype('S1'). With this I am able to query it from tiledb-presto.
Is datetime64 datatype not supported in tiledb-presto?
Thank for your quick response and support.
@Shashikant255 Thanks for the schema and info. It does appear the problem is the Date field which is a TILEDB_DATETIME_YEAR datatype. Give us a few days and we’ll expand presto’s support to all DATETIME precision levels. Prestodb is slated for several upgrades over the next few weeks. We’ll get the datetime support added in 1-2 days, and then we’ll add support for heterogeneous dimensions, before moving on to other more general improvements.
Hi @Shashikant255. Thank you for reporting the issue and providing us all the info. We are working towards supporting all TileDB datetime datatypes in TileDB-Presto, and we will have released a new version with all the updates by the end of the day.
Victor
Thanks @victor_tiledb for your response.
Hi @Shashikant255. We just released a new TileDB-Presto version 1.6.1 which includes the Datetime data types support (https://github.com/TileDB-Inc/TileDB-Presto/releases/tag/1.6.1).
Let us know if you run any further issues.
Best,
Victor