How to manage a database¶

Input4MIPs validation includes the ability to create a database of file entries. This database captures key metadata about the files. In this notebook, we go through how this database management works.

If you're reading this notebook, we assume that you're already familiar with input4MIPs validation's key concepts, e.g. its CV handling. If you're not, have a look at "How to write a single valid file".

In [1]:

Copied!





import tempfile
from pathlib import Path

import netCDF4
import pint
import pint_xarray  # noqa: F401
from loguru import logger

from input4mips_validation.cvs import load_cvs
from input4mips_validation.database import (
    load_database_file_entries,
)
from input4mips_validation.testing import create_files_in_tree
import tempfile
from pathlib import Path

import netCDF4
import pint
import pint_xarray  # noqa: F401
from loguru import logger

from input4mips_validation.cvs import load_cvs
from input4mips_validation.database import (
    load_database_file_entries,
)
from input4mips_validation.testing import create_files_in_tree

In [2]:

Copied!

logger.disable("input4mips_validation")
logger.disable("input4mips_validation")

In [3]:

Copied!

UR = pint.get_application_registry()
UR.define("ppb = ppm * 1000")
UR.define("ppt = ppb * 1000")
UR = pint.get_application_registry()
UR.define("ppb = ppm * 1000")
UR.define("ppt = ppb * 1000")

Create some files¶

We start by adding some files, so that we have a tree of files to work with.

In [4]:

Copied!

FILE_TREE_ROOT = Path(tempfile.mkdtemp())
FILE_TREE_ROOT
FILE_TREE_ROOT = Path(tempfile.mkdtemp())
FILE_TREE_ROOT

Out[4]:

PosixPath('/tmp/tmpff1r9f2z')

In [5]:

Copied!

cv_version = "v6.6.0"
cvs = load_cvs(f"gh:{cv_version}")
cvs.source_id_entries.source_ids
cv_version = "v6.6.0"
cvs = load_cvs(f"gh:{cv_version}")
cvs.source_id_entries.source_ids

Downloading file 'input4MIPs_activity_id.json' from 'https://raw.githubusercontent.com/PCMDI/input4MIPs_CVs/v6.6.0/CVs/input4MIPs_activity_id.json' to '/home/docs/checkouts/readthedocs.org/user_builds/input4mips-validation/conda/latest/lib/python3.9/site-packages/input4mips_validation/cvs/input4MIPs_CVs_v6.6.0'.

Downloading file 'input4MIPs_DRS.json' from 'https://raw.githubusercontent.com/PCMDI/input4MIPs_CVs/v6.6.0/CVs/input4MIPs_DRS.json' to '/home/docs/checkouts/readthedocs.org/user_builds/input4mips-validation/conda/latest/lib/python3.9/site-packages/input4mips_validation/cvs/input4MIPs_CVs_v6.6.0'.

Downloading file 'input4MIPs_institution_id.json' from 'https://raw.githubusercontent.com/PCMDI/input4MIPs_CVs/v6.6.0/CVs/input4MIPs_institution_id.json' to '/home/docs/checkouts/readthedocs.org/user_builds/input4mips-validation/conda/latest/lib/python3.9/site-packages/input4mips_validation/cvs/input4MIPs_CVs_v6.6.0'.

Downloading file 'input4MIPs_license.json' from 'https://raw.githubusercontent.com/PCMDI/input4MIPs_CVs/v6.6.0/CVs/input4MIPs_license.json' to '/home/docs/checkouts/readthedocs.org/user_builds/input4mips-validation/conda/latest/lib/python3.9/site-packages/input4mips_validation/cvs/input4MIPs_CVs_v6.6.0'.

Downloading file 'input4MIPs_source_id.json' from 'https://raw.githubusercontent.com/PCMDI/input4MIPs_CVs/v6.6.0/CVs/input4MIPs_source_id.json' to '/home/docs/checkouts/readthedocs.org/user_builds/input4mips-validation/conda/latest/lib/python3.9/site-packages/input4mips_validation/cvs/input4MIPs_CVs_v6.6.0'.

Out[5]:

('CEDS-CMIP-2024-07-08',
 'CEDS-CMIP-2024-07-08-supplemental',
 'CEDS-CMIP-2024-10-21',
 'CEDS-CMIP-2024-10-21-supplemental',
 'CR-CMIP-0-2-0',
 'CR-CMIP-0-3-0',
 'DRES-CMIP-BB4CMIP7-1-0',
 'MRI-JRA55-do-1-6-0',
 'PCMDI-AMIP-1-1-9',
 'PCMDI-AMIP-ERSST5-1-0',
 'PCMDI-AMIP-Had1p1-1-0',
 'PCMDI-AMIP-OI2p1-1-0',
 'SOLARIS-HEPPA-CMIP-4-1',
 'SOLARIS-HEPPA-CMIP-4-2',
 'SOLARIS-HEPPA-CMIP-4-3',
 'SOLARIS-HEPPA-CMIP-4-4',
 'UOEXETER-CMIP-0-1-0',
 'UOEXETER-CMIP-1-1-2',
 'UOEXETER-CMIP-1-1-3',
 'UofMD-landState-3-0')

In [6]:

Copied!





starting_files = create_files_in_tree(
    (
        "mole_fraction_of_carbon_dioxide_in_air",
        "mole_fraction_of_methane_in_air",
        "areacella",
    ),
    ("ppm", "ppb", "%"),
    fixed_fields=(False, False, True),
    tree_root=FILE_TREE_ROOT,
    cvs=cvs,
    dataset_category="GHGConcentrations",
)
print("To start with, these are the files in our tree: ")
for file in starting_files:
    print(f"- {file}")
starting_files = create_files_in_tree(
    (
        "mole_fraction_of_carbon_dioxide_in_air",
        "mole_fraction_of_methane_in_air",
        "areacella",
    ),
    ("ppm", "ppb", "%"),
    fixed_fields=(False, False, True),
    tree_root=FILE_TREE_ROOT,
    cvs=cvs,
    dataset_category="GHGConcentrations",
)
print("To start with, these are the files in our tree: ")
for file in starting_files:
    print(f"- {file}")

To start with, these are the files in our tree: 
- /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
- /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
- /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc

We're going to deliberately break one of the files here, so you can see how validation of broken files works below too.

In [7]:

Copied!





broken_file = starting_files[0]
ncds = netCDF4.Dataset(broken_file, "a")
# Add units to bounds variable, which isn't allowed
ncds["lat_bnds"].setncattr("units", "degrees_north")
ncds.close()
broken_file = starting_files[0]
ncds = netCDF4.Dataset(broken_file, "a")
# Add units to bounds variable, which isn't allowed
ncds["lat_bnds"].setncattr("units", "degrees_north")
ncds.close()

Create the database¶

The first step is to create our database. Below, we use our command-line interface. There is also a Python API, in case you want to do this directly from Python (note, the logging is setup slightly differently in the Python API so the default shown messages are different, but the behaviour is the same and you can always adjust the logging to suit your own preferences).

The first step is to define where you want to create the database. Note, this must be a directory because the database is serialised as a collection of files in this directory (with the file's sha acting as the filename to avoid clashes).

In [8]:

Copied!

DB_DIR_ROOT = Path(tempfile.mkdtemp()) / "example-db"
DB_DIR_ROOT
DB_DIR_ROOT = Path(tempfile.mkdtemp()) / "example-db"
DB_DIR_ROOT

Out[8]:

PosixPath('/tmp/tmpzeunmtzd/example-db')

In [9]:

Copied!





# The full docs of this command can be accessed with
# !input4mips-validation db create --help
!input4mips-validation db create \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT} \
    {FILE_TREE_ROOT}
# The full docs of this command can be accessed with
# !input4mips-validation db create --help
!input4mips-validation db create \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT} \
    {FILE_TREE_ROOT}

files:   0%|                                              | 0/3 [00:00<?, ?it/s]4525 - 131209903998784 - 2025-10-01T14:41:36.279227+0000 - INFO_DB_ENTRY - input4mips_validation.database.database:database.py:80 - Creating file database entry for /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4525 - 131209903998784 - 2025-10-01T14:41:36.373816+0000 - INFO_DB_ENTRY - input4mips_validation.database.database:database.py:80 - Creating file database entry for /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

files:  67%|█████████████████████████▎            | 2/3 [00:00<00:00, 15.49it/s]4525 - 131209903998784 - 2025-10-01T14:41:36.408536+0000 - INFO_DB_ENTRY - input4mips_validation.database.database:database.py:80 - Creating file database entry for /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc

files: 100%|██████████████████████████████████████| 3/3 [00:00<00:00, 19.99it/s]
4525 - 131209903998784 - 2025-10-01T14:41:36.429430+0000 - INFO - input4mips_validation.cli.db:db.py:120 - Dumping the 3 created entries to the new database in /tmp/tmpzeunmtzd/example-db

Entries to write:   0%|                                   | 0/3 [00:00<?, ?it/s]
Entries to write: 100%|█████████████████████████| 3/3 [00:00<00:00, 2152.77it/s]
4525 - 131209903998784 - 2025-10-01T14:41:36.431233+0000 - SUCCESS - input4mips_validation.cli.db:db.py:126 - Created new database in /tmp/tmpzeunmtzd/example-db

As we said, the database is serialised as a number of files, one for each file, with the filename being defined by the file's hash to avoid filename clashes.

In [10]:

Copied!

list(DB_DIR_ROOT.rglob("*"))
list(DB_DIR_ROOT.rglob("*"))

Out[10]:

[PosixPath('/tmp/tmpzeunmtzd/example-db/ceb53db3c1b81f8456bf85b5e6a94ccc588622ba057c19ed154309a7c042ca32.json'),
 PosixPath('/tmp/tmpzeunmtzd/example-db/f08847d3e03454a6b49d4eebab135147082615f29dcfc064ccaf29cdcda3b766.json'),
 PosixPath('/tmp/tmpzeunmtzd/example-db/63e3267f9d800749b1cf4d52da5ede37bb2f1a2a5626e4175b1491dd43eadc5a.json')]

The format of the database on disk is really a detail though. A database can be loaded trivially like below.

In [11]:

Copied!

db = load_database_file_entries(DB_DIR_ROOT)
db = load_database_file_entries(DB_DIR_ROOT)

Validating the entries in the database¶

When the database is created, all the entries are assigned a validation status of None. This signals that we have not yet tried to validate the entries.

In [12]:

Copied!

[(f"Validation status: {v.validated_input4mips}", Path(v.filepath).name) for v in db]
[(f"Validation status: {v.validated_input4mips}", Path(v.filepath).name) for v in db]

Out[12]:

[('Validation status: None',
  'mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: None',
  'mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: None',
  'areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc')]

We can go through and update the validation status using the db validate command.

In [13]:

Copied!





# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT}
# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT}

4530 - 132761356183360 - 2025-10-01T14:41:37.881760+0000 - INFO - input4mips_validation.cli.db:db.py:442 - Determining entries to validate

database entries:   0%|                                   | 0/3 [00:00<?, ?it/s]4530 - 132761356183360 - 2025-10-01T14:41:37.891789+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4530 - 132761356183360 - 2025-10-01T14:41:37.892476+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4530 - 132761356183360 - 2025-10-01T14:41:37.892596+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4530 - 132761356183360 - 2025-10-01T14:41:38.548929+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4530 - 132761356183360 - 2025-10-01T14:41:38.585187+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4530 - 132761356183360 - 2025-10-01T14:41:38.585361+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries:  33%|█████████                  | 1/3 [00:00<00:01,  1.44it/s]4530 - 132761356183360 - 2025-10-01T14:41:38.585760+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4530 - 132761356183360 - 2025-10-01T14:41:38.586480+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4530 - 132761356183360 - 2025-10-01T14:41:38.586608+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4530 - 132761356183360 - 2025-10-01T14:41:38.914857+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4530 - 132761356183360 - 2025-10-01T14:41:38.950997+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4530 - 132761356183360 - 2025-10-01T14:41:38.951222+0000 - INFO_DB_ENTRY_ERROR - input4mips_validation.validation.database:database.py:268 - Validation failed with res_validation.exception_type='ValidationResultsStoreError' for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries:  67%|██████████████████         | 2/3 [00:01<00:00,  2.00it/s]4530 - 132761356183360 - 2025-10-01T14:41:38.951660+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4530 - 132761356183360 - 2025-10-01T14:41:38.952029+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4530 - 132761356183360 - 2025-10-01T14:41:38.952128+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc

4530 - 132761356183360 - 2025-10-01T14:41:39.258907+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4530 - 132761356183360 - 2025-10-01T14:41:39.279608+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4530 - 132761356183360 - 2025-10-01T14:41:39.279782+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc

database entries: 100%|███████████████████████████| 3/3 [00:01<00:00,  2.37it/s]
database entries: 100%|███████████████████████████| 3/3 [00:01<00:00,  2.16it/s]
4530 - 132761356183360 - 2025-10-01T14:41:39.280357+0000 - INFO - input4mips_validation.cli.db:db.py:462 - Updating 3 validated entries in the database in /tmp/tmpzeunmtzd/example-db

Entries to write:   0%|                                   | 0/3 [00:00<?, ?it/s]
Entries to write: 100%|█████████████████████████| 3/3 [00:00<00:00, 1824.67it/s]
4530 - 132761356183360 - 2025-10-01T14:41:39.282375+0000 - SUCCESS - input4mips_validation.cli.db:db.py:477 - Validated the entries which hadn't been validated in the database in /tmp/tmpzeunmtzd/example-db

This updates the validation status of the files in our database. We can see that the file that is broken failed validation, while the other file passed.

In [14]:

Copied!





[
    (f"Validation status: {v.validated_input4mips}", Path(v.filepath).name)
    for v in load_database_file_entries(DB_DIR_ROOT)
]
[
    (f"Validation status: {v.validated_input4mips}", Path(v.filepath).name)
    for v in load_database_file_entries(DB_DIR_ROOT)
]

Out[14]:

[('Validation status: True',
  'mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: False',
  'mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: True',
  'areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc')]

Imagine that more files were added to our tree¶

For example, new files were published on ESGF. We simulate this below by adding some more files to our file tree.

In [15]:

Copied!





added_files = create_files_in_tree(
    (
        "mole_fraction_of_halon1211_in_air",
        "mole_fraction_of_pfc6116_in_air",
    ),
    ("ppt", "ppt"),
    fixed_fields=(False, False),
    tree_root=FILE_TREE_ROOT,
    cvs=cvs,
)
print("Added these files in our tree: ")
for file in added_files:
    print(f"- {file}")
added_files = create_files_in_tree(
    (
        "mole_fraction_of_halon1211_in_air",
        "mole_fraction_of_pfc6116_in_air",
    ),
    ("ppt", "ppt"),
    fixed_fields=(False, False),
    tree_root=FILE_TREE_ROOT,
    cvs=cvs,
)
print("Added these files in our tree: ")
for file in added_files:
    print(f"- {file}")

Added these files in our tree: 
- /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
- /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

Adding files to the database¶

We can add these files to our database using db add-tree. This adds any files in a tree, which are not already in the database, to the database.

In [16]:

Copied!





# The full docs of this command can be accessed with
# !input4mips-validation db add-tree --help
!input4mips-validation --logging-level INFO_FILE \
    db add-tree \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT} \
    {FILE_TREE_ROOT}
# The full docs of this command can be accessed with
# !input4mips-validation db add-tree --help
!input4mips-validation --logging-level INFO_FILE \
    db add-tree \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT} \
    {FILE_TREE_ROOT}

4544 - 130825811777344 - 2025-10-01T14:41:41.135501+0000 - INFO - input4mips_validation.cli.db:db.py:284 - Found 2 new files to add to the database

files:   0%|                                              | 0/2 [00:00<?, ?it/s]4544 - 130825811777344 - 2025-10-01T14:41:41.147519+0000 - INFO_DB_ENTRY - input4mips_validation.database.database:database.py:80 - Creating file database entry for /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4544 - 130825811777344 - 2025-10-01T14:41:41.240544+0000 - INFO_DB_ENTRY - input4mips_validation.database.database:database.py:80 - Creating file database entry for /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

files: 100%|██████████████████████████████████████| 2/2 [00:00<00:00, 15.53it/s]
files: 100%|██████████████████████████████████████| 2/2 [00:00<00:00, 15.51it/s]
4544 - 130825811777344 - 2025-10-01T14:41:41.276536+0000 - INFO - input4mips_validation.cli.db:db.py:299 - Dumping 2 new entries to the database in /tmp/tmpzeunmtzd/example-db

Entries to write:   0%|                                   | 0/2 [00:00<?, ?it/s]
Entries to write: 100%|█████████████████████████| 2/2 [00:00<00:00, 1755.67it/s]
4544 - 130825811777344 - 2025-10-01T14:41:41.278100+0000 - SUCCESS - input4mips_validation.cli.db:db.py:303 - Added missing entries from /tmp/tmpff1r9f2z to the database in /tmp/tmpzeunmtzd/example-db

Initially, the new entries have a validation status of None, while the entries for the other files are left unchanged.

In [17]:

Copied!





[
    (f"Validation status: {v.validated_input4mips}", Path(v.filepath).name)
    for v in load_database_file_entries(DB_DIR_ROOT)
]
[
    (f"Validation status: {v.validated_input4mips}", Path(v.filepath).name)
    for v in load_database_file_entries(DB_DIR_ROOT)
]

Out[17]:

[('Validation status: None',
  'mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: True',
  'mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: False',
  'mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: True',
  'areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc'),
 ('Validation status: None',
  'mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc')]

Re-validate the database¶

Now that we have added more entries to the database, we can also validate them.

In [18]:

Copied!





# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT}
# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT}

4549 - 137726895130432 - 2025-10-01T14:41:42.883131+0000 - INFO - input4mips_validation.cli.db:db.py:442 - Determining entries to validate

database entries:   0%|                                   | 0/2 [00:00<?, ?it/s]4549 - 137726895130432 - 2025-10-01T14:41:42.893201+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4549 - 137726895130432 - 2025-10-01T14:41:42.893921+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4549 - 137726895130432 - 2025-10-01T14:41:42.894029+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4549 - 137726895130432 - 2025-10-01T14:41:43.288671+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4549 - 137726895130432 - 2025-10-01T14:41:43.322054+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4549 - 137726895130432 - 2025-10-01T14:41:43.322167+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries:  50%|█████████████▌             | 1/2 [00:00<00:00,  2.33it/s]4549 - 137726895130432 - 2025-10-01T14:41:43.322427+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4549 - 137726895130432 - 2025-10-01T14:41:43.323045+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4549 - 137726895130432 - 2025-10-01T14:41:43.323117+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4549 - 137726895130432 - 2025-10-01T14:41:43.649589+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4549 - 137726895130432 - 2025-10-01T14:41:43.682125+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4549 - 137726895130432 - 2025-10-01T14:41:43.682234+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries: 100%|███████████████████████████| 2/2 [00:00<00:00,  2.57it/s]
database entries: 100%|███████████████████████████| 2/2 [00:00<00:00,  2.53it/s]
4549 - 137726895130432 - 2025-10-01T14:41:43.682655+0000 - INFO - input4mips_validation.cli.db:db.py:462 - Updating 2 validated entries in the database in /tmp/tmpzeunmtzd/example-db

Entries to write:   0%|                                   | 0/2 [00:00<?, ?it/s]
Entries to write: 100%|█████████████████████████| 2/2 [00:00<00:00, 1539.76it/s]
4549 - 137726895130432 - 2025-10-01T14:41:43.684222+0000 - SUCCESS - input4mips_validation.cli.db:db.py:477 - Validated the entries which hadn't been validated in the database in /tmp/tmpzeunmtzd/example-db

You will see that only the entries which were not previously validated were validated. The validation status of the previously unvalidated files in the database has been updated too.

In [19]:

Copied!





[
    (f"Validation status: {v.validated_input4mips}", Path(v.filepath).name)
    for v in load_database_file_entries(DB_DIR_ROOT)
]
[
    (f"Validation status: {v.validated_input4mips}", Path(v.filepath).name)
    for v in load_database_file_entries(DB_DIR_ROOT)
]

Out[19]:

[('Validation status: True',
  'mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: True',
  'mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: False',
  'mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 ('Validation status: True',
  'areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc'),
 ('Validation status: True',
  'mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc')]

If we try and validate again, nothing happens because the function realises that all the files in the database have already been validated.

In [20]:

Copied!





# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT}
# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT}

4560 - 127549380822848 - 2025-10-01T14:41:45.153900+0000 - INFO - input4mips_validation.cli.db:db.py:442 - Determining entries to validate

4560 - 127549380822848 - 2025-10-01T14:41:45.154100+0000 - INFO - input4mips_validation.cli.db:db.py:447 - All files in /tmp/tmpzeunmtzd/example-db have already been validated

If we want to re-validate all the files in our database, irrespective of their current validation status, we can use the --force flag.

In [21]:

Copied!





# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT} \
    --force
# The full docs of this command can be accessed with
# !input4mips-validation db validate --help
!input4mips-validation --logging-level INFO_FILE \
    db validate \
    --cv-source "gh:{cv_version}" \
    --db-dir {DB_DIR_ROOT} \
    --force

4563 - 132075224196928 - 2025-10-01T14:41:46.578923+0000 - INFO - input4mips_validation.cli.db:db.py:438 - `--force` used, hence all entries will be re-validated

database entries:   0%|                                   | 0/5 [00:00<?, ?it/s]4563 - 132075224196928 - 2025-10-01T14:41:46.588920+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:46.589523+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:46.589604+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4563 - 132075224196928 - 2025-10-01T14:41:46.972843+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4563 - 132075224196928 - 2025-10-01T14:41:47.008326+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.008478+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_halon1211_in_air/gn/v20251001/mole-fraction-of-halon1211-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries:  20%|█████▍                     | 1/5 [00:00<00:01,  2.38it/s]4563 - 132075224196928 - 2025-10-01T14:41:47.008851+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.009541+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.009664+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4563 - 132075224196928 - 2025-10-01T14:41:47.335726+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4563 - 132075224196928 - 2025-10-01T14:41:47.370401+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.370575+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_methane_in_air/gn/v20251001/mole-fraction-of-methane-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries:  40%|██████████▊                | 2/5 [00:00<00:01,  2.59it/s]4563 - 132075224196928 - 2025-10-01T14:41:47.370997+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.371813+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.371930+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4563 - 132075224196928 - 2025-10-01T14:41:47.710217+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.743303+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.743483+0000 - INFO_DB_ENTRY_ERROR - input4mips_validation.validation.database:database.py:268 - Validation failed with res_validation.exception_type='ValidationResultsStoreError' for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_carbon_dioxide_in_air/gn/v20251001/mole-fraction-of-carbon-dioxide-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries:  60%|████████████████▏          | 3/5 [00:01<00:00,  2.63it/s]4563 - 132075224196928 - 2025-10-01T14:41:47.743892+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.744186+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4563 - 132075224196928 - 2025-10-01T14:41:47.744323+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc

4563 - 132075224196928 - 2025-10-01T14:41:48.062512+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4563 - 132075224196928 - 2025-10-01T14:41:48.082991+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc
4563 - 132075224196928 - 2025-10-01T14:41:48.083143+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/fx/areacella/gn/v20251001/areacella_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn.nc

database entries:  80%|█████████████████████▌     | 4/5 [00:01<00:00,  2.75it/s]4563 - 132075224196928 - 2025-10-01T14:41:48.083503+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:134 - Checking the SHA for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:48.084203+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:148 - Creating validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:48.084309+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:105 - Creating validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

4563 - 132075224196928 - 2025-10-01T14:41:48.425966+0000 - INFO_FILE - input4mips_validation.validation.file:file.py:190 - Created validation results for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:48.458665+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:192 - Created validation results for the entry for file: /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4563 - 132075224196928 - 2025-10-01T14:41:48.458823+0000 - INFO_DB_ENTRY - input4mips_validation.validation.database:database.py:261 - Validation passed for the entry pointing to /tmp/tmpff1r9f2z/input4MIPs/CMIP6Plus/CMIP/CR/CR-CMIP-0-2-0/atmos/mon/mole_fraction_of_pfc6116_in_air/gn/v20251001/mole-fraction-of-pfc6116-in-air_input4MIPs_GHGConcentrations_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc

database entries: 100%|███████████████████████████| 5/5 [00:01<00:00,  2.72it/s]
database entries: 100%|███████████████████████████| 5/5 [00:01<00:00,  2.67it/s]
4563 - 132075224196928 - 2025-10-01T14:41:48.459383+0000 - INFO - input4mips_validation.cli.db:db.py:462 - Updating 5 validated entries in the database in /tmp/tmpzeunmtzd/example-db

Entries to write:   0%|                                   | 0/5 [00:00<?, ?it/s]
Entries to write: 100%|█████████████████████████| 5/5 [00:00<00:00, 2829.40it/s]
4563 - 132075224196928 - 2025-10-01T14:41:48.461496+0000 - SUCCESS - input4mips_validation.cli.db:db.py:477 - Re-validated all the entries in the database in /tmp/tmpzeunmtzd/example-db

Next steps¶

Now that you know how to manage a database, you can manage a database of your own. As always, if you hit any issues, please raise an issue.