Best way to write large number of strings to Array?

1

I have a large Json with around 175,000 properties each with two strings giving around 4 million characters. At the moment I have been writing them as characters to two different arrays which i index with comma’s that i use to separate them when i read from the array back intro separate strings. This process takes 5-10 seconds to do and is very slow compared to reading the double properties from the Json which takes under a second. I understand this is a very bad way to do it but i am just trying to learn TileDB and just did this to get some results.

So my question is what is the best way i can represent this information in a TileDB array? It seems that it must use a sparse array and can only be written unordered. As when i read the information i have to reassemble the characters as they are not in order which slows down reading a lot as well. Is it possible to keep the characters indexed somehow so that i can look for say string 55 in the array which will read all the characters associated with that string only?

2

Hi @DGoerke,

Thanks for posting the question here. We have a few follow-up questions to try to clarify here so we can help with some examples.

  • Which API are you using?
  • It sounds like you have key/value pairs mapping a string to another string – is that right? And the second string might have sub-cells that need to be stored?
    • Could you give an example of a few items of this data (simulated if necessary) – eg a snippet of JSON.

It sounds like string dimensions and/or attribute filtering would be a solution here. Once we have a better understanding of the problem, we’ll link examples or create some new ones, so that we can start from the same place.

Best,
Isaiah

3

I am using the C# API, and currently have a Name string, and a parent string that maps to another string. A snippet of the JSON looks like this:

{

"Name": "SB08_901",

"Parent": "TMST_EA_CT_PR",

"Bounds": {

  "Centre": {

    "X": 556.3621,

    "Y": 95.15,

    "Z": 112.021713

  },

  "Size": {

    "X": 0.377031237,

    "Y": 9.28,

    "Z": 0.4711328

  }

}

},

Should be noted some Parent strings have “” as their value, and all have a reference to another property in the JSON.