# N+1 problem and append

**URL:** <https://forum.tiledb.com/t/n-1-problem-and-append/205>\
**Category:** Uncategorized\
**Created:** [June 17, 2020, 1:16pm UTC](https://forum.tiledb.com/t/n-1-problem-and-append/205 "2020-06-17T13:16:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![michael-imbeault](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@michael-imbeault](https://forum.tiledb.com/u/michael-imbeault)\
**Post date:** [June 17, 2020, 1:16pm UTC](https://forum.tiledb.com/t/n-1-problem-and-append/205/1 "2020-06-17T13:16:49Z")

</div>

Interested in using TileDB for genomics (not variants, more like genome-wide signals) and been looking at the implementation for variants (GenomicsDB). They claim that using TileDB gets around the N+1 problem (can add a new sample easily) - but I’m looking at the docs and I don’t see a way to append to an already created array.

The only way I can think of is at the time of building the space array, specificy a very high dimension for samples, as I think these are not materialized if they do not contain any data.

Does TileDB support appending a row or a column to an existing array, or is the above a way to do it instead? What is the maximum size of dimensions if it is the case?

Thanks!

---

<div class="post-metadata">

**Author:** ![stavros](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@stavros](https://forum.tiledb.com/u/stavros)\
**Post date:** [June 17, 2020, 1:35pm UTC](https://forum.tiledb.com/t/n-1-problem-and-append/205/2 "2020-06-17T13:35:18Z")

</div>

I think you are looking at the wrong project :). Our genomic variant work is [TileDB-VCF](https://github.com/TileDB-Inc/TileDB-VCF) and here are the related [docs](https://docs.tiledb.com/genomics/).

In general, you can “append” to a TileDB array by just writing to any slice “at the end” of a huge domain, like `[1,MAX_UINT64]`. The only problem with this currently is that you need to keep track of the last domain value you wrote. You can do this through the [non-empty domain](https://docs.tiledb.com/main/api-usage/reading-arrays/basic-reading#getting-the-non-empty-domain).

We are happy to add new APIs to provide a better user experience.

---

<div class="post-metadata">

**Author:** ![michael-imbeault](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@michael-imbeault](https://forum.tiledb.com/u/michael-imbeault)\
**Post date:** [June 17, 2020, 1:40pm UTC](https://forum.tiledb.com/t/n-1-problem-and-append/205/3 "2020-06-17T13:40:03Z")

</div>

Indeed, got confused with the GenomicsDB reference, I was reading the documentation on TileDB-VCF and is what I was refering to.

OK so the smart way at the moment is to create a very large array, keep track of the last ‘sample’ you added and put it at the end. Would have expected some append function like zarr has but this can do as well. Maybe some mentions of this in the docs could be good 🙂

Thanks for speedy replies, I’m looking to adopt TileDB for a project so far so good 🙂

---

<div class="post-metadata">

**Author:** ![stavros](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@stavros](https://forum.tiledb.com/u/stavros)\
**Post date:** [June 17, 2020, 1:45pm UTC](https://forum.tiledb.com/t/n-1-problem-and-append/205/4 "2020-06-17T13:45:17Z")

</div>

Yeah indeed, “append” would be useful as an API. The main reason we have avoided “append” is that it isn’t process-safe when you have multiple writers. You need some blocking “registration” step, which the TileDB-VCF project specifically takes care of with the `register` step. We need to implement something similar for the general array case. In our roadmap as well 🙂 .