# Am I wrongly filling a sparse array that has a variable length string attribute? Or is this a bug?

**URL:** https://forum.tiledb.com/t/am-i-wrongly-filling-a-sparse-array-that-has-a-variable-length-string-attribute-or-is-this-a-bug/293
**Category:** Uncategorized
**Created:** [January 4, 2021, 11:20am UTC](https://forum.tiledb.com/t/am-i-wrongly-filling-a-sparse-array-that-has-a-variable-length-string-attribute-or-is-this-a-bug/293 "2021-01-04T11:20:52Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![imiller](https://avatars.discourse-cdn.com/v4/letter/i/8e7dd6/32.png) [@imiller](https://forum.tiledb.com/u/imiller)
#### Post date: [January 4, 2021, 11:20am UTC](https://forum.tiledb.com/t/am-i-wrongly-filling-a-sparse-array-that-has-a-variable-length-string-attribute-or-is-this-a-bug/293/1 "2021-01-04T11:20:52Z")

</div>

Hello,

I would like to create a sparse array that has a variable length string attribute, but I’m currently reading back invalid values when filling the array based on the tiledb examples. So I’m wondering if I’m wrongly filling the array, or if this is perhaps a bug (I’m using tiledb python 0.7.4 with numpy 1.19.4, under python 3.8).

Here is a minimum reproducible example, including also an integer-valued variable length attribute that works as expected:

```auto
#
# Generic initialization and array creation
#
import tiledb
import numpy as np

array_name = "array_test"

def create_array(array_name):
    ctx = tiledb.Ctx()

    dom = tiledb.Domain(
        tiledb.Dim(name="rows", domain=(None, None), dtype=np.bytes_),
        tiledb.Dim(name="cols", domain=(None, None), dtype=np.bytes_),
        ctx=ctx
    )

    attr_1 = tiledb.Attr(name="lab", var=True, dtype=np.bytes_, ctx=ctx)
    attr_2 = tiledb.Attr(name="val", var=True, dtype=np.int64, ctx=ctx)

    schema = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[attr_1, attr_2], ctx=ctx)

    tiledb.SparseArray.create(array_name, schema)

create_array(array_name)

```

Now, if the string attribute is set only with a single value per cell:

```auto
#
# Generate data for 2 cells: [(a, c), (b, d)]
#

keys_rows = ['a', 'b']
keys_cols = ['c', 'd']

# Single label per cell: this works as expected
labs = np.array([
    np.array(['xx'], dtype=np.bytes_), 
    np.array(['zzz'], dtype=np.bytes_)
], dtype='O')

vals = np.array([
    np.array([1, 2], dtype=np.int64), 
    np.array([3], dtype=np.int64)
], dtype='O')

```

then I get the expected result:

```auto
with tiledb.open(array_name, "w") as array:
    array[keys_rows, keys_cols] = dict(lab=labs, val=vals)

with tiledb.SparseArray(array_name, mode='r') as arr:
    out = arr['a', 'c']

# out
OrderedDict([('lab', array([b'xx'], dtype=object)),
             ('val', array([array([1, 2])], dtype=object)),
             ('rows', array([b'a'], dtype=object)),
             ('cols', array([b'c'], dtype=object))])

```

On the other hand, with multiple values for the string attribute:

```auto
#
# Generate data for 2 cells: [(a, c), (b, d)]
#
keys_rows = ['a', 'b']
keys_cols = ['c', 'd']

# Multiple labels per cell: invalid data returned
labs = np.array([
    np.array(['xx', 'yyy'], dtype=np.bytes_), 
    np.array(['zzz', 'llll', 'ppppp'], dtype=np.bytes_)
], dtype='O')

vals = np.array([
    np.array([1, 2], dtype=np.int64), 
    np.array([3], dtype=np.int64)
], dtype='O')

```

then I get invalid values when reading back:

```auto
# out
OrderedDict([('lab', array([b'\xf0\xde'], dtype=object)),
             ('val', array([array([1, 2])], dtype=object)),
             ('rows', array([b'a'], dtype=object)),
             ('cols', array([b'c'], dtype=object))])

```

Best,  
Ian

---

<div class="post-metadata">

### Author: ![stavros](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@stavros](https://forum.tiledb.com/u/stavros)
#### Post date: [January 4, 2021, 12:47pm UTC](https://forum.tiledb.com/t/am-i-wrongly-filling-a-sparse-array-that-has-a-variable-length-string-attribute-or-is-this-a-bug/293/2 "2021-01-04T12:47:06Z")

</div>

Hi Ian,

It appears that you wish to store a list of strings in a cell. Unfortunately, this functionality is not supported yet. You can store a variable number of values (e.g., characters, numbers, etc), but not a **list** of variable-length values (e.g., a list of strings). We expect to add this functionality in Q1 2021. Please stay tuned.

Stavros

---

<div class="post-metadata">

### Author: ![mukund](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/mukund/32/285_2.png) [@mukund](https://forum.tiledb.com/u/mukund)
#### Post date: [August 5, 2023, 6:22pm UTC](https://forum.tiledb.com/t/am-i-wrongly-filling-a-sparse-array-that-has-a-variable-length-string-attribute-or-is-this-a-bug/293/3 "2023-08-05T18:22:13Z")

</div>

Hi Stavros, is it possible to do this now?

---

<div class="post-metadata">

### Author: ![seth](https://avatars.discourse-cdn.com/v4/letter/s/f14d63/32.png) [@seth](https://forum.tiledb.com/u/seth)
#### Post date: [August 7, 2023, 7:14pm UTC](https://forum.tiledb.com/t/am-i-wrongly-filling-a-sparse-array-that-has-a-variable-length-string-attribute-or-is-this-a-bug/293/4 "2023-08-07T19:14:23Z")

</div>

Hi @mukund, thanks for reaching out. Lists of var-length strings is still on our roadmap. This was delayed previously in favor of other datatypes like, nullables, enumerations (categoricals/factors), interoperability with apache arrow and others. Our current plan is to address list of var length strings and lists of other datatypes before the end of the year.

We’ve love to learn more about your use case. Are you able to share details? If you’d prefer to discuss in private, please feel free to email me at `seth@tiledb.com`. Also happy to setup a time to discuss over the phone.

---

<div class="post-metadata">

### Author: ![mukund](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/mukund/32/285_2.png) [@mukund](https://forum.tiledb.com/u/mukund)
#### Post date: [August 15, 2023, 6:13am UTC](https://forum.tiledb.com/t/am-i-wrongly-filling-a-sparse-array-that-has-a-variable-length-string-attribute-or-is-this-a-bug/293/5 "2023-08-15T06:13:48Z")

</div>

I’m using tiledb to store Single Cell RNAseq and other omics data.

I want to create a dataframe of shape `samples` x `annotations`

Sometimes an annotation column can be a tuple of strings. E.g. If a sample is treated with 2 drugs the annotations will be `["drugA", "drugB"]`. So far I haven’t been able to figure out how I can store this data as a sparse array without resorting to hacks like JSON-encoding the list or something.
