Metadata

Unless you disable the --outputMetadata option when using the spew command, Extract adds all the metadata fields extracted by Tika from each file parsed.

Document identity

The document ID is a hash digest of the file's bytes by default. Two options control how it is computed:

• --digestAlgorithm — the hash digest method, for example SHA256.
• --digestProjectName — include a project name in the hash so that the same file ingested into two different projects gets two distinct IDs.

Because the ID is derived from content, re-running extraction on the same file yields the same ID, which keeps the index free of duplicates. This same digest-identifier mechanism is what Datashare uses to address embedded documents (attachments inside a .zip, .pst or .docx) by content hash rather than by their position in the container.

Fields added for an index

When outputting to a search index, a few extra fields are added that make working with the index easier. These are as follows.

• extract_id: a unique ID for the document, a hash digest of the file by default.
• extract_base_type: the Content-Type without any parameters. Useful for file type based faceting.
• extract_paths: the original file path. This field is multivalued when using a file hash digest as the method for calculating IDs.
• extract_parent_paths: the file's parent path. Useful for drill-down faceting. While faceting on the path field is technically possible, it's not desirable because you'll always get a facet for the file itself, with a document count of one.

All metadata field names are converted to lowercase, prefixed with tika_, and non-alphanumeric characters are converted to underscores.

How Datashare stores metadata

Datashare indexes into Elasticsearch rather than Solr, so the Solr schema below does not apply to it. Datashare's ElasticsearchSpewer writes the document content and Tika metadata into Elasticsearch, while a DatabaseSpewer persists document metadata (content type, charset, content length, language, parent/root IDs and the embedded extractionLevel) into a relational database. Embedded children record their parentId, rootId and extraction level so the document tree can be reconstructed. If you are running Datashare, you don't need to define a schema yourself — it ships with its own index mappings.

Example Solr Schema (legacy)

The following schema dates from the Apache Solr setup originally used for the Panama Papers (see Home). It is kept here as a reference for the standalone CLI's Solr spewer; new deployments should use Datashare and Elasticsearch instead. It shows how to define the metadata fields in your schema, using path tokenisation on the parent_path and path fields. You will then be able to use drill-down faceting on the parent_path field in a client UI to navigate a hierarchy and view results per-directory.

<?xml version="1.0" encoding="UTF-8" ?>
<schema name="test" version="1.5">
	<types>
		<fieldtype name="string"  class="solr.StrField" sortMissingLast="true" omitNorms="true" />
		<fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0" />

		<fieldType name="text" class="solr.TextField">
			<analyzer>
				<tokenizer class="solr.StandardTokenizerFactory" />
				<filter class="solr.StandardFilterFactory" />
				<filter class="solr.LowerCaseFilterFactory" />
				<filter class="solr.StopFilterFactory" />
				<filter class="solr.PorterStemFilterFactory" />
			</analyzer>
		</fieldType>

		<fieldType name="descendant_path" class="solr.TextField">
			<analyzer type="index">
				<tokenizer class="solr.PathHierarchyTokenizerFactory" delimiter="/" />
			</analyzer>
			<analyzer type="query">
				<tokenizer class="solr.KeywordTokenizerFactory" />
			</analyzer>
		</fieldType>
	</types>

	<fields>
		<field name="_version_" type="long" indexed="true" stored="true" />
		<field name="extract_id" type="string" indexed="true" stored="true" multiValued="false" required="true" />
		<field name="extract_base_type" type="string" indexed="true" stored="true" />
		<field name="extract_paths" type="descendant_path" indexed="true" stored="true" multiValued="true" />
		<field name="extract_parent_paths" type="descendant_path" indexed="true" stored="true" multiValued="true" />
		<field name="text" type="text" indexed="true" stored="false" />

		<!-- Main body of document. NOTE: This field is not indexed by default, since it is also copied to "text" using copyField below. This is to save space. Use this field for returning and highlighting document content. Use the "text" field to search the content. -->
		<field name="tika_content" type="text" indexed="false" stored="true" />

		<!-- Dynamic fields for arbitrary metadata. -->
		<dynamicField name="metadata_*" type="string" indexed="true"  stored="true" />
	</fields>

	<defaultSearchField>text</defaultSearchField>

	<!-- Field to use to determine and enforce document uniqueness. -->
	<uniqueKey>extract_id</uniqueKey>

	<!-- Text field to search by default. -->
	<copyField source="tika_content" dest="text" />
</schema>