Adding attributes to existing arrays

Mtrl_Scientist · March 10, 2020, 5:46pm

As far as I can tell, attributes have to be added before creating the schema and the array.

Would it be possible to allow the addition of other attributes to already existing arrays in the future or is this fundamentally not possible with the tileDB tech?

stavros · March 10, 2020, 5:54pm

Yeap, it is in our backlog. Nothing fundamentally difficult. In fact, the opposite; TileDB stores the values of each attribute in separate files, so removing/adding attributes should be easy. There are a couple of nuances with respect to updating the array schema file (e.g., to work well with object stores and provide time traveling). We’ll try to add the functionality very soon. Thanks.

Hoeze · October 29, 2020, 1:15am

Is this feature implemented for now?
It would be cool to append columns to existing dataframes

stavros · October 29, 2020, 5:56am

Not yet but it is in our roadmap. Hopefully we will start working on it before the end of the year.

Mtrl_Scientist · October 29, 2020, 11:20am

Looking forward!

evanb · May 28, 2021, 9:40pm

This feature would be really useful. Have there been any updates on this front? @stavros

Looking forward to using this! If there’s help that needs to be done in terms of a PR, please let me know.

stavros · May 29, 2021, 9:07am

This PR sets us up for implementing full-fledged schema evolution. We are about to release 2.3 over the next couple of days, and the first schema evolution features (which will include adding/removing attributes) will be added in 2.4 in a few weeks.

We of course always welcome contributions, please feel free to contact us if you’d like to learn more about the design and participate in the implementation.

Bin_Deng · October 4, 2021, 3:59pm

We already implemented adding or dropping attributes to existing arrays. Please checkout the latest release(2.4).

Bin_Deng · October 4, 2021, 4:01pm

Starting from release 2.4, we can add or drop attributes to the existing array. More implementation on array schema features will come soon

ihnorton · October 4, 2021, 4:06pm

The doc updates are still in progress for Python, but please see usage example here (compatible with TileDB-Py 0.10):

github.com

TileDB-Inc/TileDB-Py/blob/dev/tiledb/tests/test_schema_evolution.py#L37-L48

    
      
          newattr = tiledb.Attr("a3", dtype=np.int8)
          se.add_attribute(newattr)
          se.array_evolve(uri)
          
          
data2 = {
              "a1": np.arange(5, 9),
              "a2": np.random.randint(0, 1e7, size=4).astype(np.int32),
              "a3": np.random.randint(0, 255, size=4).astype(np.int8),
          }
          
          
with tiledb.open(uri, "w") as A:
              A[:] = data2

Topic		Replies	Views
Reading array at previous timestamp after schema evolution	2	534	July 11, 2022
Python Documentation	1	874	February 25, 2021
Variable-length Attributes in Python	2	881	August 1, 2019
N+1 problem and append	3	1082	June 17, 2020
[Feature Request] Automatic Schema Recommendations	2	701	October 3, 2021

Adding attributes to existing arrays

Related topics