Write the config

Finally, define a config.json file. There must be exactly one config file per delivery, located in the root directory. It tells the pipeline which files to read and what entity type they define, names the data’s sources and provenance, and specifies the column mapping. The mapping is only needed when the column names were customized.

Here’s an example configuration:

{
  "includeInputSubdirs": true,
  "inputFiles": {
    "atividade_fisica_adultos.csv": {
      "provenance": "MINISTERIO_SAUDE_BR",
      "format": "variablePerRow",
      "entityType": "AdministrativeArea2",
      "columnMappings": {
        "variable": "sv",
        "entity": "municipio",
        "date": "ano",
        "value": "valor"
      }
    }
  },
  "sources": {
    "dados.saude.gov.br": {
      "url": "https://dados.saude.gov.br",
      "provenances": {
        "MINISTERIO_SAUDE_BR": "https://dados.saude.gov.br/dataset/atividade-fisica-adultos"
      }
    }
  }
}

The config has three main fields: includeInputSubdirs, inputFiles, and sources.

includeInputSubdirs (optional)

Whether the pipeline should consider files in subdirectories of the input folder. When true, files can be referenced using relative paths within those subdirectories. If omitted, the default is false.

inputFiles (required)

Defines the input files the pipeline will process. Each CSV is one entry, keyed by its filename — so if files live in subdirectories of the input folder, specify their paths relative to it. List every CSV of the delivery by its own path: a CSV with no entry, or an entry that names a file the delivery doesn’t contain, fails validation.

The parameters for each CSV:

Parameter	When	Meaning
format	required	Always variablePerRow.
provenance	optional	Identifies the specific dataset the data came from. Each provenance is a dataset published by a given source and must match a provenance defined under sources.
importType	required for entities	The type of data in the file. When the data defines new entities, use entities; otherwise the default is observations.
entityType	required for observations	The type of the entities the observations belong to — for example AdministrativeArea2 for municipalities, AdministrativeArea1 for states, Country for countries. Every observation file needs it, and all entities in one file must be of that type. For files that define custom entities, use rowEntityType.
rowEntityType	required for entities	The DCID of the entity type — new or existing — associated with the custom entities defined in the file.
entityColumns	optional, entities only	Columns whose values are references to other entities. List only properties whose values are entities (not text, dates, or numbers), and fill those cells with DCIDs — several per cell may be separated by commas. A text column such as name must stay out of this list.
columnMappings	required for custom columns	Fill this in when the CSV column headers don’t follow the standard names.

Using the CSV from the formatting step, the mapping looks like:

"columnMappings": {
  "variable": "sv",
  "entity": "municipio",
  "date": "ano",
  "value": "valor"
}

groupStatVarsByProperty

Leave this field out (or set it to false). Variables are grouped only by the groups declared in your MCF files, so a delivery that sets it to true is refused.

sources (required)

Defines the sources and provenances associated with the input data. Each named source contains a mapping between provenances and their URLs.

Parameter	Meaning
url	The URL of the source.
provenances	A set of NAME: URL pairs identifying the specific datasets obtained from that source. The provenance name must match the value used in the provenance field under inputFiles.

Final file layout

After the steps above, you should end up with a structure similar to this:

meus_dados/
├── config.json
├── vars1.mcf
├── dados1.csv
├── dados2.csv
└── outros_dados/
    ├── vars2.mcf
    ├── dados3.csv
    └── dados4.csv

The number, names, and organization of the CSV and MCF files inside the folder are up to you. The config.json, however, must live in the root directory.

With the files in place, validate and deliver the dataset.