Validate and deliver

With your CSV, MCF, and config.json ready, the last steps are to check the formatting, record what you changed, describe and check the delivery, and package it all for submission.

Validate the files

Data Commons provides a Java tool that checks the formatting of the files before import. It validates that the CSV, MCF, and config.json follow the structure the pipeline expects, flagging errors in variable definitions, entities, or column mappings.

The only prerequisite is Java 11 or later. Download it from Java Downloads | Oracle. Confirm the installation with:

java -version

Then download the .jar from the Assets section of the latest release and run:

java -jar <path-to-jar> lint <list of mcf/tmcf/csv files>

The lint command can evaluate MCF files on their own. To evaluate CSV files, you must also provide a matching TMCF file. If it finds warnings or errors, the tool produces a JSON report with examples of the problems.

For more on the tool and its options, see the import tool usage docs.

Document the transformations

After adapting the data to the format Data Commons requires, document the transformations you applied during preparation. This is captured in a “Formatted data” template — its purpose is to transparently record the changes made to the original data (structural changes, column-name standardization, derived variables, and any other transformations needed for compatibility with the Data Commons schema).

General information

Field	Description
change description	Short description of the transformations relative to the original version
original file	Name of the source file before any modification
final file	Name of the file produced after processing, to be used for ingestion

Transformations applied

Field	Description
transformation type	Main type applied — column renaming, format change, aggregation, record filtering, or creation of derived variables
modified columns	Columns changed during processing
created columns	New columns or variables created during processing
conversions applied	Conversions such as unit changes, data-type transformations, or date standardization
missing-data handling	How missing or invalid values were handled

Final state of the dataset

Field	Description
number of rows	Total records in the final CSV
number of columns	Total columns in the final CSV
final column list	Columns present in the final file
period covered	Time span covered by the dataset after processing
entity type represented	Type of entity in the dataset (municipality, state, or country)
percentage of rows removed	Percentage of records removed during processing

Notes

Field	Description
additional notes	Free field for relevant information that doesn’t fit the fields above

Describe the delivery: `bdc.yaml`

Every delivery carries a small manifest, bdc.yaml, at the top level of the archive. It records what we can’t reconstruct from the files later: the id the data is published under, where it came from, when you downloaded it, and whom to ask about it. Print the template from a clone of the repository (it needs uv) and fill it in:

./datatools/datatools.sh drop-template > bdc.yaml

Field	Description
`schemaVersion`	Keep `1`
`source`	Required. Short id the data is published under: lowercase, no spaces, specific (`funai_terras_indigenas`, not `result` or `dados`). It becomes the source’s directory
`title`	Required. Human-readable title, in Portuguese
`contributor.name`, `contributor.email`	Who built the delivery, so questions reach a person
`upstream.name`, `upstream.url`	The publisher and page the data actually came from
`upstream.retrieved`	When you downloaded it (`YYYY-MM-DD`). Sources change silently; this is the only record of which version you processed
`upstream.license`	Licence or terms of use, if stated
`notes`	Anything a reviewer should know: known gaps, decisions you made, series you left out

Leaving source as the template’s my_source_id stops the check below. Every other blank field is reported but doesn’t block, so fill them in anyway.

Check the delivery before sending

check-drop runs exactly the check your delivery will face when we bring it in, on your machine, and writes nothing:

./datatools/datatools.sh check-drop fonte_nome_do_responsavel.zip   # or the unzipped folder

It finds each config.json that declares inputFiles (your result/ directory), confirms every declared file is present, reads every .mcf under that directory as one schema, and applies the rules of this guide. It exits 0 when the delivery passes, 1 with a list of what to fix, and 2 when the delivery can’t be read at all (no config.json, or no usable source id). Send only a delivery that passes.

Inside result/, only the files config.json lists and the .mcf files are imported. Anything else there (a .tmcf, notes) is kept for reference, and the rest of the archive (original data and PDFs) is for the reviewers.

Deliver the data

The final step is to submit the data. Upload a compressed file named <source>_<responsible>.zip containing:

the filled-in bdc.yaml, at the top level;
a directory named after the source, holding the original data as extracted, with no changes or treatment;
a PDF describing the original data (from Document your source);
a directory named result, holding the JSON, CSV, and MCF files produced after processing;
a PDF describing the transformations applied to the data.

The archive should have a structure similar to this:

fonte_nome_do_responsavel.zip
├── bdc.yaml
├── source/
│   └── nome_da_fonte/
│       └── dados_originais.ext
├── result/
│   ├── dataset_final.csv
│   ├── variables.mcf
│   ├── config.json
│   └── outros_arquivos_necessarios.ext
├── descricao_dados_originais.pdf
└── descricao_transformacoes.pdf