Home

Welcome to the Extract wiki! This is where we document workflows, guides and details about the inner workings of a piece of software that is essential in helping ICIJ fulfil its mission.

What is Extract?

Extract is a cross-platform tool for parallel, distributed content extraction. Built on top of Apache Tika, it pulls text and metadata from a wide range of different formats — including OCR of images and scanned PDFs via Tesseract.

Extract streams the output from Tika instead of buffering it all into memory before writing. This lets it operate on very large files without running out of memory.

It ships in two parts:

• extract-lib — the extraction engine: scanning, queueing, parallel Tika parsing, OCR, embedded-document handling, reporting and the pluggable "spewer" output layer. This is the library that powers Datashare (see Extract and Datashare).
• extract-cli — a command line front end around the library, with queue, spew, commit and related commands. It supports distributed queueing (Redis, AMQP or a SQL database) and can write to plain text files, standard output or a search index.

Use in Investigations

Extract was first built for the Panama Papers investigation, and was also an essential part of the engineering behind Swiss Leaks and Luxembourg Leaks.

For the Panama Papers, ICIJ used Extract to pull text and metadata from over 11.5 million files leaked from Mossack Fonseca. For more about this, see Wrangling 2.6TB of data: The people and the technology behind the Panama Papers.

In short, we ran an array of 35 g2.2xlarge EC2 machines, each spewing text and metadata at an Apache Solr core over HTTP, all using the open source distribution you see here. We then ran a customised Blacklight frontend in front of Solr, giving over 400 journalists simultaneous access to the corpus.

How Extract is used today

Since 2019, the primary way Extract is used — at ICIJ and by the wider community of journalists, researchers and NGOs — is embedded inside Datashare, ICIJ's open source document-analysis platform. In Datashare, extract-lib is the engine that turns raw files into searchable documents, and the combined platform has been used to index more than half a billion documents across investigations worldwide.

Compared with the original Panama Papers stack, two things have changed:

• Solr → Elasticsearch. Datashare indexes into Elasticsearch rather than Solr. The standalone CLI's Solr spewer is retained for historical and bespoke use, but new work goes through Datashare and Elasticsearch.
• Manual cluster → managed pipeline. Datashare wraps the same scan/extract/index steps in an orchestrated pipeline (SCAN → INDEX → NLP) and adds named-entity recognition, a web UI, batch search and access control on top.

If you just want to search a set of documents, you almost certainly want Datashare, not the raw CLI. Reach for extract-cli directly when you need a scriptable, headless extraction step in a custom pipeline. See Extract and Datashare for how the two relate.

Where to go next

• Getting Started — install and run Extract.
• Building — compile from source.
• Workflows — single-instance and distributed extraction recipes.
• Supported Formats — what Tika (and Extract's extras) can parse.
• Metadata — the fields Extract emits.
• Extract and Datashare — how the library powers Datashare.