SparseNDArray write bounding box not multi-writer safe

[bkmartinjr]

The sparse array bounding box can easily become incorrect due to point-in-time consistency race between the array and metadata updates.

Test case:

import tiledbsoma as soma
import pyarrow as pa

path = "foo_array"
soma.SparseNDArray.create(
    path,
    type=pa.float32(),
    shape=(100, 10),
).close()

A = soma.open(path, mode="w")  # at t+0
B = soma.open(path, mode="w")  # at t+1

# at t+2
A.write(
    pa.Table.from_pydict({"soma_dim_0": [99], "soma_dim_1": [9], "soma_data": [1.0]})
)
A.close()

# at t+3
B.write(
    pa.Table.from_pydict({"soma_dim_0": [0], "soma_dim_1": [0], "soma_data": [1.0]})
)
B.close()


# print the bounding box as stored in various forms
with soma.open(path) as _A:
    m = _A.metadata
    bb = (
        (m["soma_dim_0_domain_lower"], m["soma_dim_1_domain_lower"]),
        (m["soma_dim_0_domain_upper"], m["soma_dim_1_domain_upper"]),
    )
    print(f"SOMA metadata bounding box is: {bb}")
    print(f"Non-empty domain is: {_A._handle._handle.nonempty_domain()}")

    tbl = _A.read().tables().concat()
    minmax = (
        pa.compute.min_max(tbl["soma_dim_0"]).as_py(),
        pa.compute.min_max(tbl["soma_dim_1"]).as_py(),
    )
    values_bb = (
        (minmax[0]["min"], minmax[1]["min"]),
        (minmax[0]["max"], minmax[1]["max"]),
    )
    print(f"Coordinate value bounding box is {values_bb}")

Expected output would have all bonding boxes returning the same region. however, as of 1.6, it returns:

Bounding box is: ((0, 0), (0, 0))
Non-empty domain is: ((0, 99), (0, 9))
Coordinate value bounding box is ((0, 0), (99, 9))

package version info:

tiledbsoma.__version__        1.6.0
TileDB-Py tiledb.version()    (0, 24, 0)
TileDB core version           2.18.2
libtiledbsoma version()       libtiledb=2.18.2
python version                3.10.12.final.0
OS version                    Linux 6.2.0-1017-aws

[bkmartinjr]

CC: @mojaveazure @johnkerl

[johnkerl]

Discussion here #1971 (comment) whether we try to bug-fix the bounding box, or simply jettison it.

[bkmartinjr]

Update: I have confirmed that this bug affects the Census "builder", which uses multiple writers to concurrently create large sparse arrays. The issue had not been previously noticed, likely because the bounding box is only used in a handful of places on the read path.

If there is a work-around, short of disabling all concurrent writes, it would be useful to know about it. The only solution I have come up with is to have the writer (i.e., the TileDB-SOMA user, aka Census builder) go back and monkey-patch the metadata after all writes are complete.

In the Census build I checked, we had (on dim 0):

• nonempty-domain (0, 5800086)
• shape: (0, 5800086)
• bounding box: (0, 3999999)

Side note: AFAIK, the only user of this incorrect metadata is in the soma "io" packages, which are primarily used to do full dataset import/export. They are unlikely to be used by a Census user, but in principle they are supported. We just need to make sure we don't use this metadata anywhere else until this bug is resolved.

CC: @aaronwolen

[johnkerl]

Pending #2407

[johnkerl]

Current progress is on #2407

[johnkerl]

This is now a duplicate of #2407