# Ran out of memory during vacuuming

**URL:** <https://forum.tiledb.com/t/ran-out-of-memory-during-vacuuming/275>\
**Category:** Uncategorized\
**Created:** [November 23, 2020, 4:56pm UTC](https://forum.tiledb.com/t/ran-out-of-memory-during-vacuuming/275 "2020-11-23T16:56:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mtrl\_Scientist](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/mtrl_scientist/32/134_2.png) [@Mtrl\_Scientist](https://forum.tiledb.com/u/Mtrl_Scientist)\
**Post date:** [November 23, 2020, 4:56pm UTC](https://forum.tiledb.com/t/ran-out-of-memory-during-vacuuming/275/1 "2020-11-23T16:56:29Z")

</div>

Ran out of memory during consolidation & vacuuming:

```
[ERROR/2020-11-23 14:22:13,534] root: Error: Error: Internal TileDB uncaught exception; std::bad_alloc, ../Material Indicators/mnt/volume-nbg1-1/trade_data3/Binance/ANT_USDT
Traceback (most recent call last):
  File "<ipython-input-3-4d265a553e82>", line 61, in f
    tiledb.consolidate(file,ctx=ctx)
  File "tiledb/libtiledb.pyx", line 4902, in tiledb.libtiledb.consolidate
  File "tiledb/libtiledb.pyx", line 469, in tiledb.libtiledb._raise_ctx_err
  File "tiledb/libtiledb.pyx", line 454, in tiledb.libtiledb._raise_tiledb_error
tiledb.libtiledb.TileDBError: Error: Internal TileDB uncaught exception; std::bad_alloc

```

Now getting tons of errors about missing .ok files.

```
[ERROR/2020-11-23 17:24:18,598] root: Error: [TileDB::IO] Error: Cannot delete file '/root/mnt3/Material Indicators/mnt/volume-nbg1-1/indicator_data/Binance/CVD/raw/VITE_USDT/__1605762118400_1605762118400_c94337e75f474eac99c27c885a47605b_5.ok'; No such file or directory, ../Material Indicators/mnt/volume-nbg1-1/indicator_data/Binance/CVD/raw/VITE_USDT
Traceback (most recent call last):
  File "<ipython-input-3-4d265a553e82>", line 62, in f
    tiledb.vacuum(file,config=config)
  File "tiledb/libtiledb.pyx", line 6155, in tiledb.libtiledb.vacuum
  File "tiledb/libtiledb.pyx", line 484, in tiledb.libtiledb.check_error
  File "tiledb/libtiledb.pyx", line 480, in tiledb.libtiledb._raise_ctx_err
  File "tiledb/libtiledb.pyx", line 465, in tiledb.libtiledb._raise_tiledb_error
tiledb.libtiledb.TileDBError: [TileDB::IO] Error: Cannot delete file '/root/mnt3/Material Indicators/mnt/volume-nbg1-1/indicator_data/Binance/CVD/raw/VITE_USDT/__1605762118400_1605762118400_c94337e75f474eac99c27c885a47605b_5.ok'; No such file or directory

```

Code used to consolidate & vacuum:

```
config = tiledb.Config()

config = tiledb.Config({
    "sm.tile_cache_size":str(120_000_000),
    "sm.consolidation.step_min_frags":"2",
    "sm.consolidation.step_max_frags":"60",
    "sm.consolidation.steps":"100",
    "sm.consolidation.buffer_size":str(120_000_000),
    "sm.consolidation.step_size_ratio": "0.5",
})

ctx = tiledb.Ctx(config)

tiledb.consolidate(file,ctx=ctx)
tiledb.vacuum(file,config=config)

```

It seems I can still read the data and write to the array but the error appears every time I try to consolidate & vacuum. What should I do now?

Link to faulty example array:  
[https://1drv.ms/u/s!ArP7\_EkyioIBwuJ8lydSoSFqzN7knw?e=IlfGBv](https://1drv.ms/u/s!ArP7_EkyioIBwuJ8lydSoSFqzN7knw?e=IlfGBv)

Note:  
It’s not just one array. It’s hundreds that have been affected by this…

---

<div class="post-metadata">

**Author:** ![joe\_maley](https://avatars.discourse-cdn.com/v4/letter/j/bc79bd/32.png) [@joe\_maley](https://forum.tiledb.com/u/joe_maley)\
**Post date:** [November 23, 2020, 7:22pm UTC](https://forum.tiledb.com/t/ran-out-of-memory-during-vacuuming/275/2 "2020-11-23T19:22:42Z")

</div>

@Mtrl_Scientist

We’ve encountered a similar scenario in the past and fixed the issue with the following python script. This script attempts to get the array back into a good state for retrying the vacuum. Please back up your array before using the script.

I’d suggest saving the following snippet as `sanitize_array.py` and running it with `python sanitize_array.py <array name>`.

```
import os
import shutil

if len(sys.argv) < 2:
  print("usage: python sanitize_array.py <array name>")
  quit()

array_dir = sys.argv[1]

# Gather sets for fragment file names and ok-fragment file names.
fragments_set = set()
fragments_ok_set = set()
for file_name in os.listdir(array_dir):
  if file_name.endswith(".tdb") or \
      file_name.endswith(".vac") or \
      file_name.endswith(".meta"):
    continue

  if file_name.startswith("__"):
    if file_name.endswith(".ok"):
      fragments_ok_set.add(file_name)
    else:
      fragments_set.add(file_name)

print("found " + str(len(fragments_set)) + " fragments")
print("found " + str(len(fragments_ok_set)) + " ok-fragments")

# Remove all fragments without a "__fragment_metadata.tdb" file.
removed_fragments = []
for fragment in fragments_set:
  fragment_path = os.path.join(array_dir, fragment)
  metadata_path = os.path.join(fragment_path, "__fragment_metadata.tdb")
  if not os.path.isfile(metadata_path):
    shutil.rmtree(fragment_path)
    removed_fragments.append(fragment)

# Remove the removed fragment file names from `fragments_set`.
print("removed " + str(len(removed_fragments)) + " fragments")
for removed_fragment in removed_fragments:
  fragments_set.remove(removed_fragment)

# Create missing ok-fragment files
added_fragment_ok_files = 0
for fragment in fragments_set:
  fragment_ok = fragment + ".ok"
  fragment_ok_path = os.path.join(array_dir, fragment_ok)
  if not os.path.isfile(fragment_ok_path):
    open(fragment_ok_path, "a").close()
    added_fragment_ok_files += 1

print("added " + str(added_fragment_ok_files) + " ok-fragments")
```

---

<div class="post-metadata">

**Author:** ![Mtrl\_Scientist](https://yyz1.discourse-cdn.com/flex035/user_avatar/forum.tiledb.com/mtrl_scientist/32/134_2.png) [@Mtrl\_Scientist](https://forum.tiledb.com/u/Mtrl_Scientist)\
**Post date:** [November 25, 2020, 9:05am UTC](https://forum.tiledb.com/t/ran-out-of-memory-during-vacuuming/275/3 "2020-11-25T09:05:11Z")

</div>

It seems to work, thanks!