DWR Publication Inventory

This repository builds a searchable inventory of peer-reviewed publications funded and/or authored by the California Department of Water Resources (DWR). The inventory is intended to provide a maintained record of DWR-related scientific publications for discovery, reporting, quality control, and dashboard use. It combines Scopus publication metadata with project-specific review decisions, DWR funding and authorship indicators, division assignments, institutional affiliation cleanup, and scientific topic classifications.

This README is primarily for maintainers who need to refresh the inventory, review candidate records, resolve author and affiliation metadata, and publish updated data products.

The repository uses a targets pipeline to retrieve records from Scopus, support efficient manual review of funder and author matches, classify publications into a custom scientific taxonomy with the accompanying pubclassify R package, and write publication datasets for downstream use.

TL;DR

For a runnable, step-by-step version of this workflow, use scripts/inventory_update_tutorial.R from the repository root.

To refresh the inventory:

# One-time/local setup:
# - Restore dependencies with renv::restore()
# - Set required API credentials
# - Add data/lookups/author_division_lookup.csv if doing author review

# In config/pipeline.yml, set scopus.allow_api_calls: true for an intentional harvest.
# Optionally set refresh.id; blank defaults to today's date.

# Build both review queues.
targets::tar_make(names = c(funder_review_queue_file, author_review_queue_file))

# After the harvest, set scopus.allow_api_calls: false to avoid re-querying Scopus.

# Review funder candidates.
shiny::runApp("shiny/funder_review_app.R")

# Review author candidates and resolve unresolved author divisions.
shiny::runApp("shiny/author_review_app.R")
shiny::runApp("shiny/author_division_resolution_app.R")

# Build/refresh the affiliation lookup, then review unresolved rows.
targets::tar_make(affiliation_lookup_file)
shiny::runApp("shiny/affiliation_review_app.R")

# Publish accepted records and update the funding division lookup.
targets::tar_make(names = c(accepted_publications_updated, funding_division_lookup_updated))

# In data/lookups/funding_division_lookup.csv, fill division for rows where new == TRUE.

# Rebuild dashboard exports and complete the refresh log.
targets::tar_make(names = c(dashboard_csv, dashboard_parquet, refresh_log_completed))

Setup

Open the project from the repository root so .Rprofile can activate renv. Before running the pipeline or Shiny apps for the first time, restore the locked R package environment:

renv::restore()

The required R version is recorded under the R key in renv.lock. If renv::restore() fails — for example after upgrading R — see the setup troubleshooting guidance in CONTRIBUTING.md.

The pubclassify package is not on CRAN; it is recorded in renv.lock as a GitHub dependency, so renv::restore() should install it automatically. If you need to install or update it manually, use pak:

pak::pak("DWR-Open-Science/pubclassify")

The refresh workflow also requires API credentials in environment variables. Set these before running the pipeline:

• SCOPUS_API_KEY
• SCOPUS_INSTTOKEN
• PUBCLASSIFY_EMAIL
• PUBCLASSIFY_LLM_KEY

These values are secrets or user-specific settings and should not be committed to the repository. They can be set in a local .Renviron file, through the shell, or in the active R session before running targets::tar_make().

Applications

There are four applications in this repo that can be run locally for quality control:

• shiny/funder_review_app.R: manual review of Scopus funder search candidates.
• shiny/author_review_app.R: manual review of Scopus affiliation search candidates and per-author DWR affiliation status.
• shiny/author_division_resolution_app.R: division assignment for confirmed DWR authors whose division could not be resolved automatically.
• shiny/affiliation_review_app.R: manual review of unresolved canonical institution assignments with DOI/title context and canonical-name browsing.

For the shared evidence standard, decision definitions, score explanations, and review precedents, use the Review Quality Control Guide.

Two dashboard applications provide views of the publication inventory:

• shiny/dashboard_app_internal.R: DWR staff dashboard. In addition to the shared dashboard features, it displays DWR division information and provides an LLM-powered chat assistant ("Ask the data") that accepts natural-language questions about the inventory. The Institution Map tab shows a choropleth world map of co-author affiliated institutions by geography with hover/click popups. The Publishing Network tab shows an interactive force-directed co-authorship network (institution or people mode) with click-through paper lists. The Science Fields tab lists the taxonomy categories, fields, and definitions used for publication classification. See shiny/README.md, docs/CHAT_TOOLS.md, and docs/DASHBOARD.md for details.
• shiny/dashboard_app_external.R: public dashboard. It provides the same publication browsing, charts, map, network, and science-fields functionality without DWR division information or the LLM chat assistant.

To run an application, pass its file path as the first argument to shiny::runApp():

shiny::runApp("shiny/<app-file>.R")

For the DWR staff dashboard, use shiny::runApp("shiny/dashboard_app_internal.R"). For the public dashboard, use shiny::runApp("shiny/dashboard_app_external.R").

The application will then run locally—Shiny will serve it as a local URL, such as http://127.0.0.1:xxxx/, that can be opened in a browser.

Refresh Workflow

Refresh behavior is controlled through config/pipeline.yml. Before starting a refresh, review this file to confirm whether Scopus API calls are enabled, which refresh ID will be used, which large language model API endpoint and model are used, and whether the pipeline should process only new records or reclassify all reviewed records.

The pipeline uses an append-oriented refresh model. Previously accepted records are not overwritten in the default mode. Each refresh cycle is organized around review queues, durable decision files, and a durable accepted publications table.

1. Build Review Queues

Scopus API calls are disabled by default. To intentionally refresh harvested records, temporarily set this in config/pipeline.yml:

scopus:
  allow_api_calls: true

refresh:
  id: 2026-06-15
  default_mode: new_records_only

Then run:

targets::tar_make(names = c(funder_review_queue_file, author_review_queue_file))

This harvests Scopus candidates, assigns stable record_key values, saves a harvest snapshot under data/harvests/, and writes the current review queues:

• data/queues/funder_review_queue.parquet
• data/queues/author_review_queue.parquet

Set scopus.allow_api_calls back to false after the harvest step. This keeps local review, affiliation lookup, and publish work from accidentally hitting Scopus when working with later steps in the pipeline. Leave refresh.id blank to default to today's date.

2. Review Candidates

Funded

Run the funder review app to quality control articles that Scopus reports were funded by DWR:

shiny::runApp("shiny/funder_review_app.R")

Funder decisions are written to data/decisions/funding_review_decisions.csv with keep, drop, or unsure decisions keyed by record_key.

Kept funder records also need a DWR funding division assignment. This is not done in the Shiny app. During the publish step, the pipeline syncs kept funder records into data/lookups/funding_division_lookup.csv. After that sync, manually fill the division column for rows where new == TRUE by retrieving contract numbers and associated divisions from SAP, then rerun the dashboard export targets so funding_division is populated in the final outputs.

Authored

Author division assignment uses a private HR-derived lookup file, followed by manual review when the lookup cannot resolve a single division. The private file is intentionally not included in this repository. Before running author review, add it locally at:

data/lookups/author_division_lookup.csv

The file must be a CSV with these columns:

Column	Description
division	DWR division or office name for the employee in that year
year	Calendar year for the employee/division assignment
name	Employee name in FIRST MIDDLE LAST format; case is normalized by the pipeline

Example structure:

division,year,name
SWP - Engineering,2024,JANE Q SMITH
SWP - Engineering,2025,JANE Q SMITH
Flood Operations,2025,Alex R Johnson

During author review, confirmed DWR authors are matched against this lookup by name and publication year. The app first tries to resolve the author's division from the publication year, then the prior year, then the following year. If this produces one division, the division is saved automatically. If the lookup finds multiple possible divisions, or no match, the author is sent to the division resolution app for manual review.

Division names from the HR lookup are normalized against data/lookups/dwr_org_lookup.csv. This lookup maps historical names, abbreviations, and renamed offices to the canonical division names used in the inventory, so division assignments remain comparable even when DWR organization names change over time.

Run the author review app to quality control articles that Scopus reports were authored by DWR-affiliated authors:

shiny::runApp("shiny/author_review_app.R")

Author review writes two files:

• data/decisions/author_review_decisions.csv: publication-level keep/drop/unsure decisions for DWR affiliation candidates.
• data/decisions/author_division_decisions.csv: author-level DWR/not-DWR decisions and resolved division metadata where available.

Resolve any confirmed DWR authors that still need manual division assignment by running:

shiny::runApp("shiny/author_division_resolution_app.R")

3. Refresh The Affiliation Lookup

Next, run the lookup-maintenance target:

targets::tar_make(affiliation_lookup_file)

This harvests and prepends any previously unseen raw affiliation strings to data/lookups/affiliation_lookup.csv. The durable lookup has one row per publication/raw-affiliation occurrence, with DOI and title context.

This step also uses the configured LLM to propose canonical institution names. The model and OpenAI-compatible API endpoint are set in config/pipeline.yml, and the API key is read from PUBCLASSIFY_LLM_KEY. To keep the API call smaller, the pipeline sends only raw affiliation strings that still need labels, reuses trusted canonical values from the existing lookup when possible, and batches the remaining strings for LLM canonicalization. The LLM is prompted with prompts/affiliation_system_prompt.txt, prompts/affiliation_user_template.txt, and the known institution reference list in data/lookups/institution_reference.txt.

The LLM output is treated as a draft lookup value, not as final reviewed data. Unresolved or uncertain results are retained as canonical = "Unknown" with new = TRUE.

Review unresolved rows in the affiliation review app:

shiny::runApp("shiny/affiliation_review_app.R")

The app defaults to Unknown and new rows, provides autocomplete from established canonical institution names, and includes a searchable canonical institution browser. Set each reviewed row's canonical value and save it; saved rows are marked new = FALSE.

After the affiliation lookup is reviewed, the full publish step automatically runs institution_geo_lookup_file to geolocate any newly seen canonical institutions. This queries the LLM for country and US state using prompts/geo_system_prompt.txt and prompts/geo_user_template.txt and writes results to data/lookups/institution_geo_lookup.csv. Institutions where the country cannot be determined confidently are recorded as NA; those rows are not re-queried in future refreshes. No manual review step is needed for the geo lookup.

4. Classify Publications

Disciplinary classification happens during the full publish step; there is no separate command to run. The pipeline uses the custom taxonomy in taxonomy/dwr_disciplines_taxonomy.csv, which defines DWR-relevant scientific fields and their broader categories. Each taxonomy row contains a field, a plain-language definition, and a top-level category used later in exports and the dashboard.

The classifier sends publication metadata to the configured LLM through the pubclassify package. The prompt combines:

• prompts/classify_system_prompt.txt: the classifier role and general rules.
• prompts/classify_user_instructions.txt: priority rules for hard boundary cases.
• The taxonomy field definitions from taxonomy/dwr_disciplines_taxonomy.csv.
• Each publication's title, abstract, and related metadata.

The LLM is asked to assign each publication to exactly one taxonomy field based on the paper's central scientific objective, not only on method, data type, or study system. Under the hood, pubclassify expects structured model output, parses the JSON response returned by the model, and adds classification fields such as pc_field. The pipeline then joins the corresponding top-level taxonomy category as pc_category.

If the LLM returns a malformed response, a rate limit response, or an unrecognized category, the pipeline retries the affected batch after a delayed backoff. If it still cannot classify the batch, the target stops before new records are appended; it does not silently accept missing taxonomy labels.

LLM rate limits

The active Azure Foundry endpoint has an effective per-user/model limit of 2,000 output tokens per minute. config/pipeline.yml records that constraint under llm.rate_limit. The pipeline uses a shared on-disk rolling-window ledger (data/generated/llm_rate_limit.csv) across affiliation labeling, geocoding, and classification, so one LLM phase cannot immediately exhaust capacity needed by the next.

When changing to another endpoint or deployment, update these settings based on the provider's documented output token limit: output_tokens_per_minute, safety_fraction, the per-task batch sizes, and maximum output token budgets. Keep the safety margin unless the limit is exclusive to this pipeline. The pipeline stops after max_attempts delayed retries rather than writing partial classifications. At the default 2,000-output-token/minute limit, large refreshes will intentionally pause between LLM batches; that is expected behavior.

scripts/patch_classify_na.R remains available to repair records accepted by older pipeline versions that already have missing classification fields:

Rscript scripts/patch_classify_na.R

This classifies any accepted_publications.parquet rows with missing or literal "NA" pc_field values and patches the file in place using the same rate-limit settings. Afterwards, run targets::tar_make(names = c(dashboard_csv, dashboard_parquet)) to refresh the dashboard exports.

5. Publish The Updated Inventory

targets::tar_make(names = c(accepted_publications_updated, funding_division_lookup_updated))

This applies review decisions, canonicalizes affiliations, classifies records that are not already in the accepted publications table, appends newly accepted records to data/generated/accepted_publications.parquet, and updates the keep-only funding division lookup.

Open data/lookups/funding_division_lookup.csv and fill the division column for any current-refresh rows where new == TRUE. These rows are kept funder records that need a manual DWR division assignment before the dashboard exports are final. Leave new unchanged; the pipeline recalculates it on the next run.

Then rebuild the dashboard exports and complete the refresh log:

targets::tar_make(names = c(dashboard_csv, dashboard_parquet, refresh_log_completed))

This joins funding and author division fields and writes dashboard exports.

The targeted publish step uses the saved harvest snapshot for the current refresh.id, so it can run after scopus.allow_api_calls has been set back to false. It updates data/lookups/affiliation_lookup.csv before canonicalization and stops if any lookup rows are still marked new = TRUE.

Refresh Modes

Set refresh.default_mode in config/pipeline.yml before running the publish command:

Mode	Behavior
new_records_only (default)	Classify only records not already accepted
reclassify_all	Reclassify every record in the current reviewed harvest

Project Structure

_targets.R                              # Pipeline definition
README.md                               # Project overview and operator quickstart
CONTRIBUTING.md                         # Contribution guidelines and setup troubleshooting
CODE_OF_CONDUCT.md                      # Contributor covenant
LICENSE
.betterleaks.toml                       # Secrets scanner config (extends betterleaks defaults)
betterleaks-baseline.json               # Baseline of known-clean findings for CI comparison
.pre-commit-config.yaml                 # Optional local pre-commit hook (betterleaks)

.github/
  CODEOWNERS                            # Default reviewers for pull requests
  workflows/
    r-tests.yml                         # CI: restore renv and run unit tests
    secrets-scan.yml                    # CI: scan for accidentally committed secrets

docs/
  ARCHITECTURE.md                       # Pipeline and dashboard architecture reference
  DASHBOARD.md                          # Dashboard design and behavior reference
  CHAT_TOOLS.md                         # Chat assistant tool reference
  DATA_REFERENCE.md                     # Data schemas and file formats
  LIMITATIONS.md                        # Known dataset and data-model limitations

config/
  pipeline.yml                          # Non-secret pipeline settings and paths

R/
  README.md                             # R/ directory overview
  load_pipeline_config.R                # Read config/pipeline.yml
  add_record_keys.R                     # Assign stable record_key values
  create_refresh_id.R                   # Refresh identity and refresh log management
  save_harvest_candidates.R             # Persist harvest snapshots per refresh
  build_funder_review_queue.R           # Build funder candidates needing review
  build_author_review_queue.R           # Build author/affiliation candidates needing review
  apply_review_decisions.R              # Apply keep/drop decisions from review files
  append_accepted_publications.R        # Append new records to accepted publications
  update_funding_division_lookup.R      # Maintain keep-only funder division lookup
  join_funding_division.R               # Join funding divisions into exports
  join_author_division.R                # Join resolved author divisions into exports
  flag_dwr_contributions.R              # Add contribution-type boolean flags
  classify_publications.R               # Classify publications using pubclassify and the LLM
  dashboard_chat_tools.R                # Tool definitions for the dashboard chat assistant
  author_name_utils.R                   # Author lookup/name matching utilities
  score_dwr_relevance.R                 # Funder-review suspicion scoring
  score_author_affiliation.R            # Author-review suspicion scoring
  apply_affiliation_lookup.R            # Canonicalize affiliation strings
  build_affiliation_lookup.R            # Build affiliation lookup table
  build_institution_reference.R         # Build institution reference list
  build_institution_geo_lookup.R        # Geolocate canonical institutions (country, state)
  join_institution_countries.R          # Join institution countries onto publications
  require_scopus_api_allowed.R          # Guard Scopus API calls behind config flag
  resolve_harvest_candidates_file.R     # Locate saved harvest candidates for a refresh

scripts/
  patch_classify_na.R                   # Back-fill pc_field for publications left NA after LLM failures

taxonomy/
  dwr_disciplines_taxonomy.csv          # DWR field taxonomy

prompts/
  README.md                             # Prompt authoring guidance
  classify_system_prompt.txt            # LLM system prompt for disciplinary classification
  classify_user_instructions.txt        # LLM user instructions for disciplinary classification
  affiliation_system_prompt.txt         # LLM system prompt for affiliation canonicalization
  affiliation_user_template.txt         # LLM user template for affiliation canonicalization
  geo_system_prompt.txt                 # LLM system prompt for institution geolocation
  geo_user_template.txt                 # LLM user template for institution geolocation
  chat_system_prompt.txt                # LLM system prompt for the dashboard chat assistant

data/
  README.md                             # Data directory overview and file ownership
  refresh_log.csv                       # One row per refresh cycle with counts
  decisions/
    funding_review_decisions.csv        # Funder keep/drop/unsure decisions
    author_review_decisions.csv         # Author-publication keep/drop/unsure decisions
    author_division_decisions.csv       # Per-author DWR status and division decisions
  lookups/
    funding_division_lookup.csv         # Kept funder records with division assignments
    dwr_org_lookup.csv                  # Raw org label to canonical division mapping
    institution_reference.txt           # Institution reference list for lookup generation
    affiliation_lookup.csv              # Occurrence-level canonical institution review table
    institution_geo_lookup.csv          # Country and US state per canonical institution
    author_division_lookup.csv          # Required local HR-derived lookup; ignored by Git
  generated/
    accepted_publications.parquet       # Durable source of truth; generated by pipeline
    dwr_publications.csv                # dashboard_csv target; list columns collapsed
    dwr_publications.parquet            # dashboard_parquet target; native list columns
  queues/
    funder_review_queue.parquet         # Generated funder review queue; ignored by Git
    author_review_queue.parquet         # Generated author review queue; ignored by Git
  harvests/                             # Per-refresh candidate snapshots; ignored by Git

shiny/                                  # Review and visualization apps
  README.md                             # Shiny apps overview
  funder_review_app.R
  author_review_app.R
  author_division_resolution_app.R
  affiliation_review_app.R
  dashboard_app_internal.R
  dashboard_app_external.R
  content/
    about.md                            # Dashboard about panel content
    classification.md                   # Dashboard classification panel content
  www/
    dwr-logo-new.png                    # Dashboard logo asset

tests/
  README.md                             # Testing overview
  TESTING.md                            # Test writing guidance
  testthat/
    helper-fixtures.R                   # Shared test fixtures
    test-*.R                            # Unit tests (one file per R/ function)

renv/
  activate.R
  settings.json
renv.lock                               # R dependency lockfile

Outputs

• data/generated/accepted_publications.parquet: durable source of truth; records are appended across refreshes in the default mode.
• data/lookups/funding_division_lookup.csv: manual lookup for funder-query records with explicit keep decisions in data/decisions/funding_review_decisions.csv. Newly accepted current-refresh rows are prepended; new == TRUE means the row still needs a manual funding division assignment from the current refresh.
• data/generated/dwr_publications.csv: dashboard export with list columns collapsed to semicolon-delimited strings. Includes funding_division, author_division, and affiliation_countries when available.
• data/generated/dwr_publications.parquet: full-fidelity dashboard export used by shiny/dashboard_app_internal.R. Includes native list columns (including affiliation_countries) plus joined division fields.

Data Reference

For stable record keys, refresh log fields, decision file schemas, and lookup table schemas, see docs/DATA_REFERENCE.md. For interpretation caveats, see docs/LIMITATIONS.md.