Curate datasets of any format¶

Our previous guide explained how to validate, standardize & annotate DataFrame and AnnData. In this guide, we’ll walk through the basic API that lets you work with any format of data.

# !pip install 'lamindb[bionty,zarr]'
!lamin init --storage ./test-curate-any --schema bionty

import lamindb as ln
import bionty as bt
import zarr
import numpy as np

data = zarr.create((10,), dtype=[('value', 'f8'), ("gene", "U15"), ('disease', 'U16')], store='data.zarr')
data["gene"] = ["ENSG00000139618", "ENSG00000141510", "ENSG00000133703", "ENSG00000157764", "ENSG00000171862", "ENSG00000091831", "ENSG00000141736", "ENSG00000133056", "ENSG00000146648", "ENSG00000118523"]
data["disease"] = np.random.choice(['MONDO:0004975', 'MONDO:0004980'], 10)

→ connected lamindb: testuser1/test-curate-any

Define validation criteria¶

Entities that don’t have a dedicated registry (“are not typed”) can be validated & registered using ULabel:

criteria = {
    "disease": bt.Disease.ontology_id,
    "project": ln.ULabel.name,
    "gene": bt.Gene.ensembl_gene_id,
}

Validate and standardize metadata¶

validate() validates passed values against reference values in a registry. It returns a boolean vector indicating whether a value has an exact match in the reference values.

bt.Disease.validate(data["disease"], field=bt.Disease.ontology_id)

! Your Disease registry is empty, consider populating it first!
   → use `.import_from_source()` to import records from a source, e.g. a public ontology

array([False, False, False, False, False, False, False, False, False,
       False])

When validation fails, you can call inspect() to figure out what to do.

inspect() applies the same definition of validation as validate(), but returns a rich return value InspectResult. Most importantly, it logs recommended curation steps that would render the data validated.

Note: you can use standardize() to standardize synonyms.

bt.Disease.inspect(data["disease"], field=bt.Disease.ontology_id);

! received 2 unique terms, 8 empty/duplicated terms are ignored

! 2 unique terms (100.00%) are not validated for ontology_id: MONDO:0004980, MONDO:0004975

   detected 2 Disease terms in Bionty for ontology_id: 'MONDO:0004975', 'MONDO:0004980'

→  add records from Bionty to your Disease registry via .from_values()

Following the suggestions to register new labels:

Bulk creating records using from_values() only returns validated records:

Note: Terms validated with public reference are also created with .from_values, see Manage biological registries for details.

diseases = bt.Disease.from_values(data["disease"], field=bt.Disease.ontology_id)
ln.save(diseases)

Repeat the process for more labels:

projects = ln.ULabel.from_values(
    ["Project A", "Project B"], 
    field=ln.ULabel.name, 
    create=True, # create non-existing labels rather than attempting to load them from the database
)
ln.save(projects)

genes = bt.Gene.from_values(data["gene"], field=bt.Gene.ensembl_gene_id)
ln.save(genes)

Annotate and save dataset with validated metadata¶

Register the dataset as an artifact:

artifact = ln.Artifact("data.zarr", description="a zarr object").save()

Link the artifact to validated labels. You could directly do this, e.g., via artifact.ulabels.add(projects) or artifact.diseases.add(diseases).

However, often, you want to track the features that measured labels. Hence, let’s try to associate our labels with features:

from lamindb.core.exceptions import ValidationError

try:
    artifact.features.add_values({"project": projects, "disease": diseases})
except ValidationError as e:
    print(e)

This errored because we hadn’t yet registered features. After copy and paste from the error message, things work out:

ln.Feature(name='project', dtype='cat[ULabel]').save()
ln.Feature(name='disease', dtype='cat[bionty.Disease]').save()
artifact.features.add_values({"project": projects, "disease": diseases})
artifact.features

Since genes are the measurements, we register them as features:

feature_set = ln.FeatureSet(genes)
feature_set.save()
artifact.features.add_feature_set(feature_set, slot="genes")
artifact.describe()

Show code cell content Hide code cell content

# clean up test instance
!lamin delete --force test-curate-any
!rm -r data.zarr

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.10/x64/bin/lamin", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/lamin_cli/__main__.py", line 203, in delete
    return delete(instance, force=force)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/lamindb_setup/_delete.py", line 102, in delete
    n_objects = check_storage_is_empty(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/lamindb_setup/core/upath.py", line 776, in check_storage_is_empty
    raise InstanceNotEmpty(message)
lamindb_setup.core.upath.InstanceNotEmpty: Storage /home/runner/work/lamindb/lamindb/docs/test-curate-any/.lamindb contains 2 objects ('_is_initialized' ignored) - delete them prior to deleting the instance