Extract, derive & infer metadata from a CSV (local path or URL) - using the statistical profile of a dataset, mapped and driven by a configurable metadata scheming YAML spec (DCAT-US v3, DCAT-AP v3 and Croissant 1.1 bundled; Geoconnex when built with the
geoconnexfeature), with optional CKAN/DCAT metadata discovery for URL inputs. This enables FAIRification at scale.
Table of Contents | Source: src/cmd/profile.rs | 📇🧠🤖🌐📚🌎⛩️ 
Description | Examples | Usage | Arguments | Profile Options | Common Options
Description ↩
Profile a CSV (local path or URL) and emit a .metadata.json file carrying
five top-level blocks:
dpp — inferred dataset signals: lat/lon/date columns, file size,
row count, encoding, etc. (the legacy datapusher-plus
inference block).
stats — per-column summary statistics from qsv stats.
frequency — per-column value counts from qsv frequency.
ckan — a CKAN-shaped block (package + resources) that
datapusher-plus consumes to prepopulate CKAN packages.
projection — the dataset re-expressed in the active profile's metadata
vocabulary. Default is DCAT-US v3, emitted in its canonical
form: plain JSON with unprefixed keys and no @context,
validating against the schemas data.gov's own validator at
https://harvest.data.gov/validate/ runs. Bundled alternates are
dcat-ap-v3 (EU portals), croissant (ML/AI registries) and
geoconnex (water-data federations). Consumable directly by
data.gov harvesters, EU DCAT-AP catalogs, mlcommons /
Hugging Face / Kaggle, and Internet of Water tooling.
Behind the scenes qsv runs the same statistical + frequency analysis
datapusher-plus (DP+) runs in CKAN, builds a Jinja2 evaluation context from
the results, and — when an optional CKAN scheming YAML spec is supplied —
evaluates the spec's formula / suggestion_formula templates against that
context. Jinja2 helpers and filters are a native Rust port of DP+'s
jinja2_helpers.py, built on minijinja.
For DCAT-US v3 the projection also COMPUTES properties that are tedious to
author by hand, from the same statistics: temporal coverage and resolution
(an exact xsd:duration: a regular series' step, else recording precision),
spatial bounding box (excluding out-of-range and (0, 0) placeholder
coordinates) and resolution - from latitude/longitude columns or, without
them, a WKT POINT (longitude latitude) column - an update schedule (only for
a regular calendar series), the content language, DQV quality measurements
(completeness, plus duplicate rows and CSV/UTF-8 validity with --validate),
and a CSVW table schema with per-column constraints (required, ranges,
lengths, enumerations, primary key). Values
that are generated, detected or estimated rather than measured (a
description, the language, the schedule, the temporal and spatial resolution)
form an "inferred" tier that the publisher's own metadata replaces;
projection_provenance lists them (and LLM column annotations, which only an
override replaces). A generated
description carries a disclosure and a review warning, so it never passes
for a human-authored one.
When the input is a URL whose response carries DCAT markup (HTTP
Link: rel=describedBy), qsv discovers the publisher's stated metadata and
merges it as a base layer beneath the inferred projection.
For an example CKAN scheming YAML spec, see:
https://github.com/dathere/datapusher-plus/blob/main/ckanext/datapusher_plus/dataset-druf.yaml
For more extensive examples, see https://github.com/dathere/qsv/blob/master/tests/test_profile.rs. See also https://github.com/dathere/qsv/wiki/Metadata-Profiling
Examples ↩
Quick: dpp/stats/frequency + default DCAT-US v3 projection.
qsv profile data.csvPipe stdin; output defaults to
stdin.metadata.json.
cat data.csv | qsv profileURL input: discover the publisher's DCAT markup and merge it as a base layer.
qsv profile https://data.example.gov/datasets/sample.csvSeed publisher/contact info from a JSON file; write to a chosen output path.
qsv profile data.csv --initial-context publisher.json -o data.metadata.jsondata.gov-style harvest: validate against DCAT-US v3 JSON Schema, abort on mandatory-field violations, wrap in a Catalog envelope.
qsv profile data.csv --validate --strict --catalog -o data.metadata.jsonDCAT-AP v3 for EU data portals;
pyshaclvalidates the bundled SHACL shapes.
qsv profile open-data.csv --profile dcat-ap-v3 --validate --strictCroissant JSON-LD for an ML dataset;
mlcroissantvalidates the output.
qsv profile train.csv --profile croissant --validate -o train.croissant.jsonGeoconnex JSON-LD for hydrologic data (qsv built with the
geoconnexfeature).
qsv profile gages.csv --profile geoconnex --validate --strictEvaluate a CKAN scheming spec: Jinja2 formulas compute spatial/temporal extents, accrual periodicity, and other derived fields.
qsv profile data.csv --spec dataset-druf.yaml -o data.metadata.jsonCKAN-only output: drop the
projectionblock, keep dpp/stats/frequency/ckan.
qsv profile data.csv --no-projection --spec dataset-druf.yamlCustom YAML profile from disk (embedded names always win over same-named files, so use a non-clashing name for custom profiles).
qsv profile data.csv --profile ./my-org-dcat.yaml --validateUsage ↩
qsv profile [options] [<input>]
qsv profile --helpArguments ↩
| Argument | Description |
|---|---|
<input> |
Path or URL to the CSV to profile. When - or omitted, reads from stdin. When the URL has DCAT markup, qsv will attempt to discover and ingest it as a base layer of metadata (unless --no-dcat-discovery is set). See --no-dcat-discovery and --dcat-discovery-timeout for details and opt-out. |
Profile Options ↩
| Option | Type | Description | Default |
|---|---|---|---|
‑‑spec |
string | CKAN scheming YAML spec file. If omitted, only the inferred dpp block (lat/lon/date columns, dataset stats) is emitted; no formulas are evaluated. |
|
‑‑initial‑context |
string | JSON file providing seed values for the package / resource dicts plus optional JSON-Pointer overrides for the final projection block. Replaces the older --package-meta / --resource-meta flags. Top-level keys: package, resource, dataset_info. Each leaf value may be wrapped as {"value": ..., "force": true} to mark it as overriding any value discovered from URL DCAT markup AND any value qsv inferred. Force is honored across all three subtrees: dataset_info entries override their target path verbatim; package / resource entries route through the active profile's field_mappings: table (e.g. package.title force=true lands at /projection/title, beating inference and discovery). Forced values for slots the profile does not surface are silently dropped (no-op). dataset_info keys are JSON Pointers into the output (e.g. /projection/title): the value replaces what is there, and missing objects are created. Inside an array, use an existing index, or a final - to append (e.g. /projection/hasQualityMeasurement/- adds a measurement after the computed ones). An override that cannot be applied (not a pointer, an index out of range, - with no array to append to) is reported in projection_warnings (or, with --no-projection, on stderr) and fails --strict. See tests/resources/profile/dcat-init-context.README.md for a fully-populated example. |
|
‑‑no‑projection |
flag | Skip the metadata projection block (dcat/croissant/ geoconnex, depending on the active profile). | |
‑‑no‑ckan |
flag | Skip the CKAN-shape block. | |
‑‑croissant‑frequency |
flag | Embed per-column value-frequency distributions in the metadata projection. The croissant profile renders them as inline cr:RecordSets (one <col>-frequency RecordSet of {value, count, percentage} rows per column), per the spec's "distribution of values is a statistic on the field" guidance. Off by default (keeps the projection compact); the raw counts always remain in the top-level frequency block regardless. Other bundled profiles ignore this flag. |
|
‑‑locate |
string | Name the places the data covers, as DCAT-US v3 spatial Locations, by looking up the densest coordinate cells (up to 40, row-weighted) with qsv geocode. is one of: geonames - label the bounding box with the place, admin1 or country that 80% of the sampled rows share (offline Geonames index). census - add each US state and county holding at least 5% of the sampled rows (the heaviest state always; up to 10 counties), keyed by Census GEOID (US Census Geocoder: online, keyless, cached). State and county columns, when they resolve, supply these instead, with or without coordinates, as they name every row exactly: states as USPS, FIPS or names; counties as names or GEOIDs, matched exactly within each row's state (TIGERweb county names). A name several counties share is noted, not guessed. A state column lists states only when at least 90% of its values are US states. both - geonames and census. Also adds the coordinates' centroid. A lookup that cannot run is noted in projection_warnings; the bounding box is kept. Default: off. |
|
‑‑describegpt |
string | Use a qsv describegpt --format json result as LLM-inferred metadata (see also --infer). Its Description becomes the description, its Tags the keywords and its Dictionary labels and descriptions annotate the CSVW columns - all as inferred values, which supplied or discovered metadata replaces. describegpt's attribution footer moves to the llm_attribution block and the description gets a one-line disclosure. |
|
‑‑infer |
flag | Run qsv describegpt --all on the input and use its result as with --describegpt. The LLM is configured through describegpt's QSV_LLM_* environment variables. If it cannot run, a warning is added and the statistics-based description is used. |
|
‑‑tag‑vocab |
string | With --describegpt or --infer: a local tag,description CSV (the format describegpt uses for its own tag vocabulary). Only tags in it are kept - describegpt only asks the LLM to keep to the vocabulary - and they also become theme Concepts carrying the vocabulary's description. |
|
‑‑dcat‑legacy‑license |
flag | Also emit license on the Dataset, alongside the Distribution-level copy. Both are valid DCAT-US v3 (the Dataset-level property was added upstream), but the migration guide directs publishers to put license on each Distribution, so that is what qsv emits by default. Default: off. |
|
‑‑no‑dcat‑discovery |
flag | Skip DCAT-markup discovery on URL inputs. Discovery sniffs HTTP Link: rel=describedBy (and, in future, sibling .metadata.json / JSON-LD <script> blocks) to use the publisher's stated metadata as a base layer. | |
‑‑dcat‑discovery‑timeout |
integer | Per-request timeout for DCAT-markup discovery probes. Default: 5. | |
‑‑validate |
flag | Validate the emitted projection block against the active profile's declared validators. For dcat-us-v3 that's the vendored GSA JSON Schema bundle (see resources/dcat-us-v3/); for dcat-ap-v3 / geoconnex it's pyshacl over the bundled SHACL shapes; for croissant it's mlcroissant. Catches missing mandatory fields, cardinality issues, and shape violations. Violations append to projection_warnings by default. Also runs qsv validate over the input (CSV structure + UTF-8), so a passing file is declared UTF-8 encoded and its syntactic validity is recorded as a quality measurement. A file that fails it (or a pass that cannot run) also gets a Required syntacticValidity warning. Ignored, with a warning on stderr, when combined with --no-projection. |
|
‑‑strict |
flag | Fail the command, instead of just warning, when an --initial-context override cannot be applied, and, with --validate, when the input fails qsv validate, on Required-severity schema violations or on non-Info external-validator findings. Severity comes from the DCAT-US v3 schema's own requirementLevel annotations, so a missing mandatory property aborts while a recommended-field advisory does not. |
|
‑‑allow‑external‑validator |
flag | Opt in to spawning the validator binary declared by validation.external when the profile was loaded from an arbitrary YAML file. Bundled profiles (dcat-us-v3, dcat-ap-v3, croissant, geoconnex) always run their declared external validators because the profile content is vetted at qsv release time. Without this flag, file-loaded profiles emit a Recommended-severity warning instead of running the binary, so an untrusted YAML can't silently execute arbitrary commands. Default: off. |
|
‑‑catalog |
flag | Wrap the emitted DCAT-US v3 Dataset inside a Catalog envelope (Catalog{dataset:[...]}). Useful for federation harvesters (data.gov, CKAN ingest) that expect Catalog-shaped top-level metadata. Default: off (Dataset-only, backwards-compatible). | |
‑‑profile |
string | Metadata projection profile to use. Embedded names: dcat-us-v3 (default), dcat-ap-v3, croissant; geoconnex (when built with the geoconnex feature, which no prebuilt binary includes: build qsv with -F feature_capable,geoconnex, or qsvdp with -F datapusher_plus,geoconnex). A path to a custom YAML profile is also accepted; embedded names always win over same-named files. See resources/profiles/README.md for the schema and authoring guide. |
|
‑‑force |
flag | Force recomputing cardinality and unique values even if a stats cache file exists. | |
‑j,‑‑jobs |
integer | The number of jobs to run in parallel for the underlying stats/frequency passes. When not set, the number of jobs is set to the number of CPUs detected. | |
‑o,‑‑output |
string | Output JSON path. Default: .metadata.json. |
Common Options ↩
| Option | Type | Description | Default |
|---|---|---|---|
‑h,‑‑help |
flag | Display this message | |
‑n,‑‑no‑headers |
flag | When set, the first row will not be interpreted as headers. Namely, it will be processed with the rest of the rows. Otherwise, the first row will always appear as the header row in the output. | |
‑d,‑‑delimiter |
string | The field delimiter for reading CSV data. Must be a single character. | |
‑‑memcheck |
flag | Check if there is enough memory to load the entire CSV into memory using CONSERVATIVE heuristics. |
Source: src/cmd/profile.rs
| Table of Contents | README