How to upload to an FTP server¶

Here we document how to upload file(s) to an FTP server. For input4MIPs, the most common target server is LLNL's FTP server, but this tool should, in theory, work for any FTP server. This is the last step in submitting files to the input4MIPs collection and, ultimately, publication in the ESGF index. This document assumes that you already have file(s) that you have validated and want to upload. If you don't already have files, check out "How to write a single valid file". If you haven't validated your files, check out "How to validate a single file".

In [1]:

from pathlib import Path

from pathlib import Path

Starting point¶

We assume that you are starting from existing file(s). For this demo, we use the files which are used for other demos.

Note: This is just a demo, so we don't actually upload files. Instead we do a dry run, so you can see what would happen, without actually having to do the upload.

In [2]:

tree_root = Path(".")

tree_root = Path(".")

In [4]:

# Show the files that we would upload
list(tree_root.rglob("*.nc"))

# Show the files that we would upload list(tree_root.rglob("*.nc"))

Out[4]:

[PosixPath('docs/how-to-guides/fixed_CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'),
 PosixPath('docs/how-to-guides/CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc')]

The CVs¶

A note before we continue. In the below, you will notice that there is an option, --cv-source. This points to the source of the controlled vocabularies (CVs). You can pick different sources for the CVs. For example, you can load the CVs from local files, or from the input4MIPs CVs GitHub (or any other web source).

In this example, we're going to use a specific commit from the input4MIPs CVs GitHub to avoid anything breaking, even if we make further changes to the CVs. For your own work, you will probably want to use either:

1. local files, e.g. --cv-source path/to/local/files
2. the branch where you have added your information to the CVs, --cv-source https://raw.githubusercontent.com/PCMDI/input4MIPs_CVs/branch_name/CVs/
3. a tagged version of the input4MIPs CVs GitHub --cv-source gh:tag-id
4. the main branch of the input4MIPs CVs GitHub --cv-source gh:main
5. a specific commit in the input4MIPs CVs GitHub, like we do below, --cv-source gh:commit-hash

This is definitely not the best documented feature of the library, so if anything is unclear, please raise an issue.

Upload the files¶

Uploading the files is quite simple. Below, we demonstrate how to do this. Note that we do a dry run here, i.e. we don't actually upload any files. When you are using this command, remove the --dry-run flag to actually upload the files.

If you find a bug while using this tool, please raise an issue.

Below, we use our command-line interface. There is also a Python API, in case you want to do this directly from Python (note, the logging is setup slightly differently in the Python API so the default shown messages are different, but the behaviour is the same and you can always adjust the logging to suit your own preferences).

In [6]:

# Below we do a dry run, i.e. we don't actually upload the files.
# Make sure you remove the --dry-run flag for your own uploads.
!input4mips-validation \
    --logging-level DEBUG \
    upload-ftp . \
    --password "your-email-goes-here@invalid.com" \
    --cv-source "gh:v6.6.0" \
    --ftp-dir-rel-to-root "dir-to-upload-in-goes-here-eg-your-institute-name" \
    --n-threads 10 \
    --dry-run

# Below we do a dry run, i.e. we don't actually upload the files. # Make sure you remove the --dry-run flag for your own uploads. !input4mips-validation \ --logging-level DEBUG \ upload-ftp . \ --password "your-email-goes-here@invalid.com" \ --cv-source "gh:v6.6.0" \ --ftp-dir-rel-to-root "dir-to-upload-in-goes-here-eg-your-institute-name" \ --n-threads 10 \ --dry-run

4603 - 132520188020544 - 2025-10-01T14:41:51.495470+0000 - DEBUG - input4mips_validation.cvs.loading:loading.py:46 - raw_cvs_loader=RawCVLoaderKnownRemoteRegistry(registry=<pooch.core.Pooch object at 0x7886a5256af0>, force_download=False)
4603 - 132520188020544 - 2025-10-01T14:41:51.499766+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:71 - Dry run. Would log in to ftp.llnl.gov using username='anonymous'
4603 - 132520188020544 - 2025-10-01T14:41:51.499836+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:355 - Dry run. Would ensure that /incoming/dir-to-upload-in-goes-here-eg-your-institute-name existed on the server
4603 - 132520188020544 - 2025-10-01T14:41:51.499870+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:83 - Dry run. Would close connection to ftp.llnl.gov
4603 - 132520188020544 - 2025-10-01T14:41:51.499896+0000 - INFO - input4mips_validation.upload_ftp:upload_ftp.py:369 - Uploading in parallel using up to 10 threads

4603 - 132520188020544 - 2025-10-01T14:41:51.501510+0000 - WARNING - input4mips_validation.upload_ftp:upload_ftp.py:390 - Filepath could not be resolved with the DRS, we will upload the following file without any directory structure. file=PosixPath('docs/how-to-guides/fixed_CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'). cvs.DRS.directory_path_template='<activity_id>/<mip_era>/<target_mip>/<institution_id>/<source_id>/<realm>/<frequency>/<variable_id>/<grid_label>/v<version>'
4603 - 132519686878784 - 2025-10-01T14:41:51.501764+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:71 - Dry run. Would log in to ftp.llnl.gov using username='anonymous'
4603 - 132519686878784 - 2025-10-01T14:41:51.501826+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:178 - Uploading docs/how-to-guides/fixed_CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4603 - 132519686878784 - 2025-10-01T14:41:51.501859+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:180 - Dry run. Would cd on the FTP server to /incoming/dir-to-upload-in-goes-here-eg-your-institute-name
4603 - 132519686878784 - 2025-10-01T14:41:51.501896+0000 - INFO_FILE - input4mips_validation.upload_ftp:upload_ftp.py:186 - Relative to /incoming/dir-to-upload-in-goes-here-eg-your-institute-name on the FTP server, will upload docs/how-to-guides/fixed_CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc to fixed_CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4603 - 132519686878784 - 2025-10-01T14:41:51.501939+0000 - INFO_FILE - input4mips_validation.upload_ftp:upload_ftp.py:210 - Dry run. Would upload docs/how-to-guides/fixed_CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4603 - 132519686878784 - 2025-10-01T14:41:51.501967+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:83 - Dry run. Would close connection to ftp.llnl.gov
4603 - 132520188020544 - 2025-10-01T14:41:51.502027+0000 - WARNING - input4mips_validation.upload_ftp:upload_ftp.py:390 - Filepath could not be resolved with the DRS, we will upload the following file without any directory structure. file=PosixPath('docs/how-to-guides/CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc'). cvs.DRS.directory_path_template='<activity_id>/<mip_era>/<target_mip>/<institution_id>/<source_id>/<realm>/<frequency>/<variable_id>/<grid_label>/v<version>'
4603 - 132519686878784 - 2025-10-01T14:41:51.503548+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:71 - Dry run. Would log in to ftp.llnl.gov using username='anonymous'
4603 - 132519686878784 - 2025-10-01T14:41:51.503629+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:178 - Uploading docs/how-to-guides/CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4603 - 132519686878784 - 2025-10-01T14:41:51.503664+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:180 - Dry run. Would cd on the FTP server to /incoming/dir-to-upload-in-goes-here-eg-your-institute-name
4603 - 132519686878784 - 2025-10-01T14:41:51.503705+0000 - INFO_FILE - input4mips_validation.upload_ftp:upload_ftp.py:186 - Relative to /incoming/dir-to-upload-in-goes-here-eg-your-institute-name on the FTP server, will upload docs/how-to-guides/CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc to CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4603 - 132519686878784 - 2025-10-01T14:41:51.503748+0000 - INFO_FILE - input4mips_validation.upload_ftp:upload_ftp.py:210 - Dry run. Would upload docs/how-to-guides/CH4-em-biomassburning_input4MIPs_emissions_CMIP_CR-CMIP-0-2-0_gn_200001-201012.nc
4603 - 132519686878784 - 2025-10-01T14:41:51.503777+0000 - DEBUG - input4mips_validation.upload_ftp:upload_ftp.py:83 - Dry run. Would close connection to ftp.llnl.gov
4603 - 132520188020544 - 2025-10-01T14:41:51.504033+0000 - SUCCESS - input4mips_validation.upload_ftp:upload_ftp.py:429 - Uploaded all files

Next steps¶

Once you have your files, now you are ready to upload them. Before/during/just after you have upload the files, please make an issue at PCMDI/input4MIPs_CVs so we can ensure that your files are published as quickly as possible.