undatum

A command-line tool for data processing and analysis

undatum (pronounced un-da-tum) is a CLI for converting, analyzing, validating, and transforming datasets across many formats. Conversion, statistics, selection, validation and database loads stream records, so file size is not limited by memory; each command page states how much memory the command needs.

Features

• 140+ formats via iterabledata: CSV, JSON, JSON Lines, BSON, XML, XLS/XLSX, Parquet, AVRO, ORC, plus geospatial, lakehouse (Delta/Iceberg/Lance/DuckLake/Paimon), scientific, RDF, log, config, graph, and feed formats. Run undatum formats list to see every supported format and its read/write capabilities.
• Compression support: GZ, XZ, BZ2, ZIP, ZSTD, LZ4, 7Z, Brotli, Snappy, LZO
• Verified repacking: Recompress local streams and native Parquet, ORC and Avro files, or compact offline DuckDB and SQLite databases (undatum repack). Outputs are staged, independently verified and published atomically; optimization keeps the original unless the candidate saves space. Measure savings with --dry-run --json and set optional time, temporary-space and memory budgets
• Multi-cloud I/O: Read and write s3://, gs:///gcs://, and az:///abfs:///abfss:// URIs natively via iterabledata (pip install "undatum[cloud]")
• Databases: Query and dump PostgreSQL, MySQL/MariaDB, SQLite, SQL Server, ClickHouse, MongoDB, and Elasticsearch/OpenSearch (undatum db query, undatum db dump); load files into DuckDB, SQLite, PostgreSQL, MySQL, ClickHouse, SQL Server, MongoDB, and Elasticsearch/OpenSearch (undatum db load, with create-table, replace and upsert where the database supports them)
• Large files: record commands stream with bounded memory; sort, dedup and reverse spill to disk; DuckDB runs SQL and many transforms out of core. Each command page states its memory use (for example uniq and frequency grow with the number of distinct values)
• Unix pipelines: - reads standard input and results go to standard output in the input's format: cat data.csv | undatum sort - --by amount | undatum head - -n 10
• SQL expressions: --where "amount > 100 AND city = 'Berlin'" filters and --add "total = price * qty" adds computed fields on select, convert and other commands, evaluated by DuckDB on typed values
• Partitioned output: undatum convert data.csv out/ --partition-by year,month -O parquet writes Hive-style field=value/ directories that DuckDB, Spark, Athena and BigQuery read
• Automatic detection: Encoding, delimiters (comma, semicolon, tab, pipe), headers, compression and file types
• Data validation: rule files with a built-in library — emails, URLs, dates, E.164 phones, ISO country/currency/language codes, IBAN, UUID, patterns, ranges, unique and references to another file (undatum validate --rules; --list-rules prints the catalogue)
• Quality reports: undatum quality profiles a dataset (empty values, distinct and top values, type conformance), checks it against an expected schema and rules, and fails CI when a threshold is not met (Markdown, HTML or JSON)
• Schema drift: undatum diff --schema and undatum schema-drift report added, removed, retyped and renamed fields across deliveries; --fail-on turns drift into a non-zero exit code
• Machine-readable output: --json on informational commands (count, headers, sniff, stats, schema, diff, validate, ...) prints versioned JSON documents; errors become JSON too, with documented exit codes
• Frictionless Data Packaging: Create, extend, and validate datapackage.json descriptors (undatum package)
• Ad-hoc SQL on files: Run DuckDB SQL over CSV, JSONL, Parquet, and other formats (undatum sql)
• Python SDK: a lazy Dataset API — chained steps build a plan that runs as one DuckDB query when every step has a SQL form
• AI-powered tooling: undatum ai and --autodoc on analyze, schema and doc share iterabledata's providers (OpenAI, Anthropic, Gemini, Azure, OpenRouter, Ollama, LM Studio, Perplexity, OpenAI-compatible endpoints); values in personal-data columns are masked in samples sent to remote providers
• Agent tools & MCP server: one tool per operation for MCP and LangChain; undatum mcp serve also exposes dataset listings, schemas and samples as resources and ships ready-made prompts (mcp 1.x and 2.x)
• Optional TUI and web UI: Explore a bounded sample in the terminal (undatum tui) or a local browser (undatum web)
• Optional Data API: Serve file-backed datasets over HTTP (FastAPI + DuckDB)

Documentation

The full documentation site (Docusaurus) lives in docs/ and is published at datenoio.github.io/undatum.

Section	What it covers
Getting started	Install, quick start, positioning
Cookbook	Task index by role
CLI reference	Every command
Formats	Honest capability matrix
Python SDK	Fluent Dataset API
MCP / agents	Agent tools and MCP server
JSON output	Versioned JSON layouts for scripts and agents
Troubleshooting	Exit codes and common errors
Migrating to 2.0	Renamed commands and options, changed defaults

Source pages: docs/docs/. Changelog: CHANGELOG.md. Contributor workflow: WORKFLOW_GUIDE.md.

Installation

Using uv or pipx (recommended for CLI use)

uv tool install undatum
# or
pipx install undatum

Using pip

pip install --upgrade pip setuptools
pip install undatum

Dependencies are declared in pyproject.toml and will be installed automatically by modern versions of pip (23+), including pyarrow for Parquet.

macOS

brew install pipx && pipx install undatum
# or
uv tool install undatum

Since 1.8.0, release tags also publish PyInstaller single-file binaries (Linux, macOS, Windows) on GitHub Releases; 1.7.0 and earlier are on PyPI only. pipx/uv remain the supported install paths for most users.

A man page ships with the package (man undatum after install, or make man to regenerate man/undatum.1).

Container

Release images are published to GitHub Container Registry:

docker run --rm -v "$PWD:/data" ghcr.io/datenoio/undatum convert data.csv data.parquet

:<version> and :latest hold the core install, :<version>-full and :latest-full add the common extras. A Homebrew tap and a conda-forge package are planned; see the installation guide.

Optional extras

Extra	Enables
api	Data API server (undatum api, FastAPI + uvicorn + httpx)
extract	Document extraction (undatum extract, PDF/DOC/DOCX tables and text)
plot	Plotting (undatum plot, matplotlib)
phone	National phone numbers in validation rules (format: phone with region, phonenumbers)
tui	Interactive terminal UI (undatum tui, Textual)
web	Local web UI (undatum web, FastAPI + Jinja2)
mcp	MCP server for AI agents (undatum mcp serve)
langchain	LangChain agent tools
polars, dask	DataFrame interop from the Dataset SDK
s3	S3 cloud storage support (boto3)
gcs	Google Cloud Storage (gs:// / gcs://, gcsfs)
azure	Azure Blob / ADLS (az:// / abfs://, adlfs)
cloud	Multi-cloud storage via fsspec (S3 + GCS + Azure)
postgres, mysql, mssql, clickhouse	Database connectors (MongoDB is in the base install)
elastic	Elasticsearch / OpenSearch
frictionless	Full Frictionless Data Package validation
lakehouse	Delta / Iceberg / Lance / DuckLake / Hudi via iterabledata
gis	Geospatial and LiDAR formats
scientific	MATLAB, geophysical, and HDF5 formats
access	Microsoft Access (.mdb / .accdb)
compression	Extra codecs (snappy, brotli, lzo)
repack	Process memory monitoring for repack --memory-limit (psutil)
all	Every extra

pip install "undatum[extract,api]"

After installation both undatum and the shorter data command are available:

undatum --version
undatum headers data.csv
data headers data.csv   # same thing

Shell completion

undatum --install-completion   # for the shell you run it from (bash, zsh, fish, PowerShell)
undatum --show-completion      # print the script instead of installing it

Requirements

• Python 3.10 or greater (CI tests 3.10–3.13)

Install from source

python -m pip install --upgrade pip setuptools wheel
python -m pip install .

Quick start

# Inspect supported formats
undatum formats list --capabilities

# Convert
undatum convert people.csv people.parquet
undatum convert --tagname item data.xml data.jsonl

# Recompress local files with independent read-back verification
undatum repack data.csv.gz
undatum repack data.parquet --dry-run --json

# Inspect
undatum headers data.jsonl
undatum analyze data.jsonl
undatum stats data.csv
undatum table data.csv --limit 20

# Query and transform
undatum sql "SELECT city, COUNT(*) AS n FROM data GROUP BY city" cities.csv
undatum select sales.csv --where "amount > 100" --add "total = price * qty" --output big.csv
undatum convert sales.csv sales/ --partition-by year -O parquet

# Pipelines: '-' reads stdin; CSV in, CSV out (-O jsonl to change it)
cat data.csv | undatum sort - --by amount --numeric amount | undatum head - -n 5

# Validate, check quality and drift, package
undatum validate data.csv --rules rules.yml
undatum quality data.csv --rules rules.yml --thresholds quality.yml --output report.html
undatum schema-drift deliveries/ --baseline schema.json --fail-on removed,type
undatum package create data.csv --output datapackage.json

# JSON for scripts and agents
undatum count data.csv --json

# Document
undatum ai doc data.csv
undatum doc data.jsonl --format-out markdown --output dataset.md

More first-success paths: quick start and the cookbook.

Recompress and compact files

# Change the outer codec without changing the decoded bytes
undatum repack data.csv.gz data.csv.zst

# Compact offline databases into new files
undatum repack warehouse.duckdb compact.duckdb
undatum repack app.sqlite compact.sqlite

# After closing all SQLite users, compact in place
undatum repack app.sqlite --sqlite-offline

repack accepts local files and retains their data format. It preserves the source when encoding or verification fails and skips a same-representation candidate that does not save space; use --allow-grow to accept it. Replacing an existing SQLite destination requires --sqlite-offline; active database sidecars cause refusal. DuckDB's initial supported profile is limited to 256 MiB input and 1 million rows. See the repack reference for format limits, profiles, resource budgets and the undatum.repack/1 JSON report.

Commands

All commands are available as undatum <command> or via the shorter data alias.

Top-level data commands: convert, repack, extract, analyze, doc, stats, validate, quality, schema, schema-bulk, schema-drift, diff, sql, select, search, mask, plot, tui, web, migrate-script, and the other transform/inspection commands in the CLI reference.

Upgrading: the older names document, profile, ingest and scheme and the old option spellings (--filetype, --outtype, --n, ...) still work with a deprecation warning until 2.0 (use doc, stats, db load, schema --format cerberus). undatum migrate-script scripts/ --write rewrites them in shell scripts, Makefiles, Markdown and pipeline YAML; the migration guide and the changelog list the changed defaults.

Command groups:

Group	Subcommands
ai	doc, filter, plan, suggest
api	discover, serve, run, openapi
db	query, load, dump
package	create, add-resource, validate
pipeline	run, validate, doc, templates list, templates init
formats	list, describe, export, tables
mcp	serve, tools
examples	list, show, run
plugins	list, info, validate
config	show

undatum convert --help
undatum sql --help

Contributing

Contributions are welcome. See CONTRIBUTING.md and the development docs.

License

MIT License — see LICENSE. Third-party attribution and license terms are in THIRD_PARTY_NOTICES.

Links

• Documentation
• GitHub
• PyPI
• Changelog
• Issue tracker
• iterabledata (streaming I/O engine)