DCM2BQ

DCM2BQ (DICOM to BigQuery) is a tool for extracting metadata and generating vector embeddings from DICOM files, loading both into Google BigQuery. It can be run as a standalone CLI or as a containerized service, making it easy to integrate into data pipelines.

By generating vector embeddings for DICOM images, Structured Reports, and PDFs, DCM2BQ enables powerful semantic search and similarity-based retrieval across your medical imaging data. This allows you to find related studies, cases, or reports even when traditional metadata fields do not match exactly.

This open-source package can be used as an alternative to the DICOM metadata streaming feature in the Google Cloud Healthcare API, enabling similar functionality for DICOM data stored in Google Cloud Storage. It can also be used to complement a Healthcare API DICOM store by generating embeddings for existing or new data.

Why DCM2BQ?

Traditional imaging systems like PACS and VNAs offer limited query capabilities over DICOM metadata. By ingesting the complete metadata and vector embeddings into BigQuery, you unlock powerful, large-scale analytics and insights from your imaging data.

Benefits of Embedding-Based Search:

Go beyond exact field matching: Find similar images, reports, or studies based on visual or textual content, not just metadata.
Enable content-based retrieval: Search for "cases like this one" or "find similar findings" using embeddings.
Support multi-modal queries: Use embeddings from images, SRs, and PDFs for unified search across modalities.
Improve research, cohort discovery, and clinical decision support by surfacing relevant cases that would be missed by keyword or tag-based search alone.

Features

Parse DICOM Part 10 files.
Convert DICOM metadata to a flexible JSON representation.
Load DICOM metadata and vector embeddings into a BigQuery table.
Enable semantic and similarity search over your imaging archive using embeddings.
Run as a containerized service, ideal for event-driven pipelines.
Run as a command-line interface (CLI) for manual or scripted processing.
Handle Google Cloud Storage object lifecycle events (creation, deletion) to keep BigQuery synchronized.
Process zip and tar.gz/tgz archives containing multiple DICOM files with a single event.
Generate vector embeddings from DICOM images, Structured Reports, and encapsulated PDFs using Google's multi-modal embedding model.
Highly configurable to adapt to your needs.

BigQuery schema

The project uses two BigQuery tables: an instances table for DICOM metadata and a separate embeddings table for per-frame vector embeddings.

Instances table (`instances`)

id: STRING (REQUIRED) - Deterministic SHA256 hash of DICOM UIDs
timestamp: TIMESTAMP (REQUIRED) - When the record was written
path: STRING (REQUIRED) - Full path to the DICOM file
version: STRING (NULLABLE) - Object version identifier
info: RECORD (REQUIRED) - Processing metadata with structured fields:
- event: STRING - Event type (e.g., OBJECT_FINALIZE)
- input: RECORD - DICOM file metadata (size, type, storageClass)
metadata: JSON (NULLABLE) - Complete DICOM JSON metadata

Embeddings table (`embeddings`)

Stores one row per embedding (one per frame for multi-frame DICOM images, one for SR/PDF):

id: STRING (REQUIRED) - Composite key <instanceId>_<frameNumber> (frame 0 for single-frame/text content), deterministic per frame
instanceId: STRING (REQUIRED) - Foreign key to the instances table
timestamp: TIMESTAMP (REQUIRED) - When the embedding was generated
frameNumber: INT64 (NULLABLE) - 0-based frame index (null for non-image content)
info: RECORD (NULLABLE) - Embedding metadata (model, input path/size/mimeType)
embeddingVector: FLOAT ARRAY - The vector embedding

Instances view (`instancesView`)

A BigQuery view that resolves the latest row per instance (both tables are append-only, so reprocessing a file adds rows rather than replacing them) and joins in a count of embeddings per instance, exposing an embedding_count column.

Embeddings view (`embeddingsView`)

A BigQuery view over the embeddings table that keeps only the latest row per embedding id, guaranteeing a single embedding per frame even after a file has been processed multiple times. Consumers should read per-frame embeddings through this view rather than the raw table.

The Cloud Run service is configured with table IDs via gcpConfig.bigQuery.instancesTableId and gcpConfig.bigQuery.embeddingsTableId settings (see config.defaults.js). Use the embeddingVector column on the embeddings table when running vector searches or creating vector indexes.

Note: the project includes sample DDL and queries — see src/bq-samples.sql.

Example queries

You can find example queries and DDL for creating the embedding model and vector index in src/bq-samples.sql. The file includes:

example SELECTs against the instances and embeddings tables,
sample aggregation queries for vector search,
and DDL samples to create an embedding model and a vector index on the embeddings.embeddingVector column.

Before running vector searches, ensure you have created the embedding model and vector index (the samples show how to do this with bq query).

Installation

Dependencies

dcm2bq uses dcmnorm, a very fast, Rust-based DICOM parser and renderer.

For DICOM parsing and image rendering, dcm2bq relies on the external dcmnorm CLI binary in the execution environment.

If you use MPEG4 rendering via dcmnorm, ffmpeg must also be installed and available on PATH.

It is included in the provided Docker image. If you are building from source or running the CLI locally, you will need to install it manually.

On Debian/Ubuntu/macOS/Linux (recommended):

./helpers/install-dcmnorm.sh

You can pin a version by passing it as the only argument, for example ./helpers/install-dcmnorm.sh 0.1.3.

Docker

The service is distributed as a container image. You can find the latest releases on Docker Hub.

docker pull jasonklotzer/dcm2bq:latest

From Source (for CLI)

To use the CLI, you can install it from the source code.

Ensure you have node and npm installed. We recommend using nvm.
Ensure you have installed the required Dependencies including dcmnorm.

Clone the repository:

git clone https://github.com/googlecloudplatform/dcm2bq.git

Navigate to the directory and install dependencies and the CLI:
```
cd dcm2bq
npm install
npm install -g .
```
Verify the installation:
```
dcm2bq --help
```

Usage

As a Service (Cloud Run)

The recommended deployment uses Google Cloud Storage, Pub/Sub, and Cloud Run.

The workflow is as follows:

An object operation (e.g., creation, deletion) occurs in a GCS bucket.
A notification is sent to a Pub/Sub topic.
A Pub/Sub subscription pushes the message to a Cloud Run service running the dcm2bq container.
The dcm2bq container processes the message:
- It validates the message schema and checks for a DICOM-like file extension (e.g., .dcm) or supported archive (.zip, .tar.gz, .tgz).
- For new objects, it reads the file from GCS and parses the DICOM metadata.
- For archive files, it extracts all .dcm files and processes each one individually.
- If embeddings are enabled, it generates vector embeddings from the DICOM data (for supported types like images, SRs, and PDFs) by calling the Vertex AI Embeddings API. For multi-frame images (e.g., WSI), it generates one embedding per frame.
- It inserts DICOM metadata into the instances table and embeddings into the separate embeddings table in BigQuery.
- For deleted objects, it records the deletion event in BigQuery.
If an error occurs, the message is NACK'd for retry. After maximum retries, it's sent to a dead-letter topic for analysis.

Note: When deploying to Cloud Run, ensure the container has enough memory allocated to handle your largest DICOM files.

The service also supports processing archives (.zip, .tar.gz, .tgz) containing multiple DICOM files. See docs/ARCHIVE_SUPPORT.md for details.

Local Mode (test the full pipeline without Pub/Sub)

You can exercise the entire workflow on locally ingested files, without GCS/HCAPI notifications or Pub/Sub. Input is read from a local folder; metadata still goes to BigQuery, and generated assets go wherever the config points (GCS by default, or a local folder). A prior deployment is the prerequisite that creates the BigQuery dataset/tables and buckets.

Start the service:
```
DCM2BQ_CONFIG_FILE=test/testconfig.json dcm2bq service
```
DCM2BQ_LOCAL_ROOT (or localConfig.rootPath in the config file) is optional. When set, the service restricts local file access to that directory — useful when the endpoint is reachable by others. When unset, any accessible path is accepted.

Index a file or folder. This synthesizes push events (the same envelope shape Pub/Sub push delivers) and POSTs them to the running service:

dcm2bq index /path/to/dicom               # one-shot: index all supported files recursively
dcm2bq index /path/to/dicom --watch       # keep watching for new or changed files
dcm2bq index /path/to/dicom --force       # reprocess unchanged files as new rows
dcm2bq index file.dcm --service-url http://localhost:8080

Rows created this way have file:// paths and info.input.type = "LOCAL" in BigQuery. The event generation defaults to the file's mtime (in microseconds), mirroring GCS generation semantics: re-indexing an unchanged file is deduplicated, while a modified file lands as a new row.

To keep extracted images/text local as well, set gcpConfig.embedding.input.gcsBucketPath to a file:///path URI instead of gs://bucket/path.

The admin console also supports local rows: it reads file:// assets from disk (optionally scoped to DCM2BQ_LOCAL_ROOT) and reprocesses them by POSTing directly to the service (set DCM2BQ_SERVICE_URL) instead of publishing to Pub/Sub. VS Code users can start both processes with the "Local Mode: Service + Admin Console" compound launch configuration.

HTTP Service UI (local/admin usage)

When running in HTTP service mode (dcm2bq service [port]), a static admin website is available at /ui.

Admin Console (standalone deployment)

The admin console can also be deployed as a standalone service with its own Node.js backend and static frontend. It connects directly to BigQuery and GCS to query instances, show metadata, and download extracted assets. For full deployment and usage instructions, see admin-console/README.md.

As a CLI

The CLI is useful for testing, development, and batch processing.

Example: Dump DICOM metadata as JSON

dcm2bq dump test/files/dcm/ct.dcm | jq

This command will output the full DICOM metadata in JSON format, which can be piped to tools like jq for filtering and inspection.

Example: Generate a vector embedding

dcm2bq embed test/files/dcm/ct.dcm

This command will process the DICOM file, generate a vector embedding using the configured model, and output the embedding as a JSON array.

Example: Extract rendered image or text from a DICOM file

dcm2bq extract test/files/dcm/ct.dcm

This command will extract and save a rendered image (JPG) or extracted text (TXT) from the DICOM file, depending on its type (image, SR, or PDF). The output file extension is chosen automatically unless you specify --output.

Example: Extract with summarization (SR/PDF only)

dcm2bq extract test/files/dcm/sr.dcm --summary

By default, summarization is disabled for extracted text. If you pass --summary, the extracted text from Structured Reports (SR) or PDFs will be summarized using Gemini before saving. This is useful for generating concise, embedding-friendly text.

Example: Extract without summarization (explicitly)

dcm2bq extract test/files/dcm/sr.dcm

If you do not pass --summary, the full extracted text will be saved (subject to length limits for embedding).

Example: Index a local file or folder through a locally running service

dcm2bq index test/files/dcm --watch

This command posts synthetic events for each supported file (.dcm, .dicom, .zip, .tar.gz, .tgz) to a locally running dcm2bq service, which processes them through the full pipeline (parse, extract, embed, persist to BigQuery). With --watch, it keeps watching the folder and indexes new or changed files as they arrive. See Local Mode for setup.

Example: Process a DICOM file and retrieve results from BigQuery

dcm2bq process test/files/dcm/ct.dcm

This command uploads a DICOM file to GCS, triggers CloudRun processing via Pub/Sub, polls BigQuery for results, and displays a formatted overview. It uses test/testconfig.json if available, or you can specify a config file with --config deployment-config.json. See docs/PROCESS_COMMAND.md for detailed usage and archive file support.

Example: List items in the dead letter queue

dcm2bq dlq list

This command queries the BigQuery dead letter table and displays a summary showing the total count of failed messages and a list of distinct failed files with their failure counts. Useful for monitoring and troubleshooting processing failures.

Example: Requeue failed items from the dead letter queue

dcm2bq dlq requeue

This command reads the dead letter queue, identifies failed GCS and HCAPI objects, and republishes schema-compliant Pub/Sub messages so they are processed again via Cloud Run. You can limit the number of items with --limit 50. By default it publishes to topic dcm2bq-gcs-events; override with PUBSUB_REQUEUE_TOPIC if needed.

Configuration

Configuration options can be found in the default config file.

You can override these defaults in two ways.

Important: When providing an override via environment variable or a file, you must supply the entire configuration object. The default configuration is not merged with your overrides; your provided configuration will be used as-is.

Environment Variable: Set DCM2BQ_CONFIG to a JSON string containing the full configuration.

export DCM2BQ_CONFIG='{"bigquery":{"datasetId":"my_dataset","instancesTableId":"my_table"},"gcpConfig":{"projectId":"my-gcp-project","embeddings":{"enabled":true,"model":"multimodalembedding@001"}},"jsonOutput":{...}}'

Config File: Set DCM2BQ_CONFIG_FILE to the path of a JSON file containing your full configuration.

# config.json
# {
#   "bigquery": {
#     "datasetId": "my_dataset",
#     "instancesTableId": "my_table"
#   },
#   "gcpConfig": {
#     "projectId": "my-gcp-project",
#     "embeddings": {
#       "enabled": true,
#       "model": "multimodalembedding@001"
#     }
#   },
#   "jsonOutput": {
#      ...
#   }
# }
export DCM2BQ_CONFIG_FILE=./config.json

Embedding and Summarization Configuration

To enable vector embedding generation and input extraction, configure the embedding.input section within gcpConfig. The configuration uses a hierarchical structure where the presence of settings indicates they are enabled.

Example config.json override:

{
  "gcpConfig": {
    "embedding": {
      "input": {
        "gcsBucketPath": "gs://my-bucket/processed-data",
        "summarizeText": {
          "model": "gemini-2.5-flash-lite",
          "maxLength": 1024
        },
        "vector": {
          "model": "multimodalembedding@001"
        }
      }
    }
  }
}

Note: The JSON snippet above is a partial example showing only the embeddings-related settings. When providing an override (via DCM2BQ_CONFIG or DCM2BQ_CONFIG_FILE), you must supply the entire configuration object — partial merges are not supported.

Embedding Input Configuration

embedding.input.gcsBucketPath: GCS bucket path where processed images (.jpg) and text (.txt) files will be saved. Format: gs://bucket-name/optional-path. Files are organized as {gcsBucketPath}/{StudyInstanceUID}/{SeriesInstanceUID}/{SOPInstanceUID}.{jpg|txt}. If this is omitted or empty, no files will be saved. Important: This bucket should be separate from the DICOM source bucket to avoid triggering unwanted events when processed files are created.
embedding.input.vector.model: If present, vector embeddings will be generated using the specified Vertex AI model (e.g., multimodalembedding@001). Omit this section to only extract and save inputs without generating embeddings.

Text Summarization Configuration

embedding.input.summarizeText.model: If present, long text extracted from SR/PDF will be summarized using the specified Gemini model before processing. Omit this section to skip summarization. This can be overridden at runtime by the CLI --summary flag.
embedding.input.summarizeText.maxLength: Maximum character length for summarized text (default: 1024). The summarization prompt instructs the model to keep output under this limit. This also controls when summarization is triggered: text longer than maxLength will be summarized when embedding compatibility is required.

Documentation

Additional documentation on new features and development guides can be found in the docs directory:

CLI Process Command

docs/PROCESS_COMMAND.md - Overview and usage of the dcm2bq process command for uploading DICOM files and retrieving results
docs/PROCESS_COMMAND_IMPLEMENTATION.md - Implementation details of the process command

Archive Support

docs/ARCHIVE_SUPPORT.md - Comprehensive guide to archive file processing (.zip, .tar.gz, .tgz), including usage, timeouts, and troubleshooting
docs/QUICK_REFERENCE_ARCHIVE.md - Quick examples and reference table for common archive scenarios

Testing

docs/TEST_COVERAGE_PROCESS_COMMAND.md - Comprehensive test coverage documentation for the process command, including unit and integration tests

Development

To get started with development, follow the installation steps for the CLI.

The test directory contains numerous examples, unit tests, and integration tests that are helpful for understanding the codebase and validating changes.

Running Tests

The unit tests are fully mocked and can be run without any GCP dependencies or configuration files. All external service calls (BigQuery, Cloud Storage, Vertex AI, Gemini) are stubbed to ensure fast, reliable test execution.

To run the unit test suite:

npm test
# or using the helper script
./helpers/run-unit-tests.sh

The tests use a mock configuration defined in test/test-config.js and don't require any real GCP resources or the test/testconfig.json file.

Integration Tests

For testing against real GCP services, integration test files are available that require:

A properly configured test/testconfig.json file (generated by running ./helpers/deploy.sh my-project-name)
GCP authentication (gcloud auth application-default login)
Deployed GCP resources (BigQuery dataset/table, GCS buckets)
Docker CLI/runtime (for container smoke checks run as part of integration tests)

When to run integration tests

Run unit tests (npm test) locally before sending a PR or release tag; they are fully mocked and fast.
Run integration tests only after deploying the test stack (e.g., ./helpers/deploy.sh upload <project>) or promoting a build to a staging environment, because they need live GCP resources.
Recommended checkpoints: after dependency or schema changes, before a release cut once the candidate container is deployed to the test/staging project, and periodically in CI on a schedule against that deployed environment.

Available integration test suites:

semantic_compare.integration.js - Tests semantic similarity between text and image embeddings
pipeline.integration.js - End-to-end pipeline tests (GCS upload → processing → BigQuery insertion)
storage-embeddings.integration.js - Storage and embedding feature tests
config-validation.integration.js - Configuration, schema, and permissions validation tests
dead-letter.integration.js - Dead letter queue functionality (failed message handling, BigQuery writes, IAM permissions)
error-handling.integration.js - Error handling and HTTP status code validation
docker-admin-ui.integration.js - Docker image smoke checks (container boot, /, /ui, static assets, and WebSocket endpoint)

To run all integration tests:

npm run test:integration
# or using the helper script directly
./helpers/run-integration-tests.sh

Or manually with mocha:

DCM2BQ_CONFIG_FILE=test/testconfig.json mocha test/*.integration.js

To run a specific integration test suite:

DCM2BQ_CONFIG_FILE=test/testconfig.json mocha test/pipeline.integration.js

Note: Integration tests make real API calls to Google Cloud services and may incur costs. They also upload test files to GCS and insert rows into BigQuery (cleanup is performed automatically).

Integration runs now also execute Docker smoke checks in the same pass, so expect a longer overall runtime.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details on how to contribute to this project.

License

This project is licensed under the Apache 2.0 License.

Deployment with Terraform

The recommended way to deploy the service and all required Google Cloud resources is using Terraform. This will provision:

Google Cloud Storage bucket(s)
Pub/Sub topics and subscriptions
BigQuery dataset and tables
Cloud Run service
All necessary IAM permissions

A helper script is provided to automate the process:

./helpers/deploy.sh [OPTIONS] [destroy|upload] <gcp_project_id>

upload: Upload test DICOM files from test/files/dcm/*.dcm to the GCS bucket created by Terraform (standalone; does not deploy).
destroy: Destroy all previously created resources (cleanup).
--debug: Enable debug mode with verbose logging in the Cloud Run service.
--no-embeddings: Disable vector embedding generation.
--no-embedding-input: Disable extraction/storage of embedding input files (also disables embeddings).
--no-admin-console: Skip standalone admin-console deployment.
--help or -h: Show usage instructions.

The Terraform default for dcm2bq_concurrency is 80, so new Cloud Run deployments allow up to 80 simultaneous requests per instance by default. If you need a different value, set it through Terraform before running the helper script, for example:

TF_VAR_dcm2bq_concurrency=120 ./helpers/deploy.sh my-gcp-project-id

When deploy_admin_console is enabled, admin-console IAP is configured using Cloud Run native IAP (iap_enabled = true). The legacy HTTPS load balancer + OAuth client flow is no longer required.

Examples

Deploy infrastructure:
```
./helpers/deploy.sh my-gcp-project-id
```

Deploy with debug mode enabled:

./helpers/deploy.sh --debug my-gcp-project-id

Upload test data only (no deploy):

./helpers/deploy.sh upload my-gcp-project-id

Deploy and then upload test data (two steps):

./helpers/deploy.sh my-gcp-project-id
./helpers/deploy.sh upload my-gcp-project-id

Destroy all resources:

./helpers/deploy.sh destroy my-gcp-project-id

The script will:

Ensure all dependencies (Terraform, gcloud, gsutil) are installed.
Create a GCS bucket for Terraform state (if needed).
Generate a backend config for Terraform.
Deploy all infrastructure using Terraform.
Optionally upload test DICOM files if the flag is supplied.

Note: All resource names (buckets, datasets, tables, etc.) are made unique per deployment to avoid collisions.

Name		Name	Last commit message	Last commit date
Latest commit History 358 Commits
.vscode		.vscode
admin-console		admin-console
assets		assets
docs		docs
helpers		helpers
src		src
test		test
tf		tf
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
deployment-config.example.json		deployment-config.example.json
package-lock.json		package-lock.json
package.json		package.json
renovate.json		renovate.json

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

DCM2BQ

Why DCM2BQ?

Features

BigQuery schema

Instances table (instances)

Embeddings table (embeddings)

Instances view (instancesView)

Embeddings view (embeddingsView)

Example queries

Installation

Dependencies

Docker

From Source (for CLI)

Usage

As a Service (Cloud Run)

Local Mode (test the full pipeline without Pub/Sub)

HTTP Service UI (local/admin usage)

Admin Console (standalone deployment)

As a CLI

Configuration

Embedding and Summarization Configuration

Embedding Input Configuration

Text Summarization Configuration

Documentation

CLI Process Command

Archive Support

Testing

Development

Running Tests

Integration Tests

Contributing

License

Deployment with Terraform

About

Topics

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 5

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Instances table (`instances`)

Embeddings table (`embeddings`)

Instances view (`instancesView`)

Embeddings view (`embeddingsView`)

Packages