DRAP Data Extraction Toolkit

Two scripts for pulling structured drug data out of the Drug Regulatory Authority of Pakistan's (DRAP) public web properties, since DRAP doesn't publish a bulk export of either dataset.

Data sources

1. Registered Product Registry — drap_registry_detail_scraper.py (primary)

Endpoint: https://eapp.dra.gov.pk/productView.php (POST, webRegNo=<6-digit>)

This is the AJAX endpoint behind DRAP's registry search page (eapp.dra.gov.pk/WebProductIndex.php). It takes a plain registration number and returns the full registration record: product name, registration status, dosage form, composition, route of administration, company name and address, pack sizes, and more.

Registration numbers are sequential-ish integers, so this script gets comprehensive coverage by walking the number range directly rather than searching for terms. It auto-detects the real upper bound of the range on first run (see Auto range detection), checkpoints progress, and can be safely stopped and resumed.

This is the authoritative source — use it when you need registration status, composition, or company details.

2. Pharmaceutical Product Price Index — drap_price_index_scraper.py (secondary)

Endpoint: https://e.dra.gov.pk/public/price?page=N

A plain, paginated, server-rendered table of currently priced pharmaceutical products (~1,069 pages as of Aug 2026, ~20 rows each). Gives brand name, composition, registration number, manufacturer, category, pack size, current MRP (maximum retail price), and the date that price took effect.

Use this when you need current market pricing — it's a snapshot of what's actively priced/sold, not the full registration history.

3. Update script — drap_update.py (routine use, after the first full pull)

Not a separate data source — it orchestrates the two scripts above to keep the datasets current without repeating the expensive full registry scan. See Keeping this up to date for the strategy and Usage for commands.

Approaches we tried and abandoned

For context, in case future changes to the site bring these back into play:

• Scraping the registry search UI directly (WebProductIndex.php) — it's a search form with no bulk listing, so comprehensive coverage would've required enumerating search terms. Superseded once we found productView.php is a direct-lookup endpoint that doesn't need that.
• Parsing individual MRP-fixation PDFs from dra.gov.pk/tag/costing_pricing/ — DRAP publishes these as one-off notifications (e.g. "MRP fixation of twenty drugs"), not a single master list, so covering the full catalog this way means discovering and parsing many separate PDFs. Superseded by the price index above, which is a single live, paginated source.

Setup

pip install -r requirements.txt

or individually:

pip install requests beautifulsoup4 lxml

Usage

Registry detail scraper

# First run: auto-detects the real max registration number, then scans 1..max
python drap_registry_detail_scraper.py

# Skip detection, scan a known range directly
python drap_registry_detail_scraper.py --no-auto-detect --end 130000

# Scan a specific slice, faster (less polite) rate
python drap_registry_detail_scraper.py --start 50000 --end 60000 --delay 0.5

# Resume an interrupted run — just re-run the same command
python drap_registry_detail_scraper.py

Run python drap_registry_detail_scraper.py --help for the full flag list (output path, checkpoint file, probe window size, etc.).

Before running at full scale: try a small range first to confirm parsing still matches the live site — DRAP can change markup without notice:

python drap_registry_detail_scraper.py --no-auto-detect --start 1 --end 2000

Price index scraper

python drap_price_index_scraper.py

No flags — it fetches page 1 to read the total page count off the page itself, then walks every page sequentially.

Registry scraper — targeted reg_number list

Instead of scanning a sequential range, fetch details only for reg_numbers already known from another CSV (e.g. every reg_number in the price index):

python drap_registry_detail_scraper.py \
    --reg-list-csv drap_price_index.csv \
    --reg-list-column reg_number

This is what drap_update.py uses internally to fetch only newly-appearing reg_numbers instead of re-running the full range scan.

Update script (routine refresh)

Run this after the initial full pull, instead of re-running the registry scraper's full range scan:

# Normal routine run: refresh price index, diff, fetch only new reg_numbers
python drap_update.py

# Periodically (e.g. monthly): also re-check registry status for
# already-known, currently-active reg_numbers -- catches status changes
# (e.g. active -> cancelled) that a delta-only run would miss
python drap_update.py --refresh-existing-registry

# Re-run the diff/registry-fetch step without re-fetching the price index
# (e.g. to retry a failed registry fetch without re-pulling all 1069 pages)
python drap_update.py --skip-price-refresh

See Keeping this up to date for what each run actually does.

Output schema

drap_registry_details.csv

Column	Notes
reg_no_queried	The number the script requested (zero-padded, e.g. 012333)
Product Name
Registration No	As returned by DRAP — should match reg_no_queried
Registration Date
Company Name
Company Address
Registration Status	e.g. "Provisionally Active"
Route of Admin	Often blank in older/legacy records
Used For	Human / Veterinary
Dosage Form
Product Specification	Often blank
Manufacturing Type	Often blank
Label Claim	Often blank
Container Closure	Often blank
Composition	Often blank — cross-reference with the price index's composition field when needed
Pack Size(s)	Often blank

Several fields are frequently empty in the live data, including on real records — that's DRAP's underlying data, not a parsing gap.

drap_price_index.csv

Column	Notes
brand_name
composition
reg_number	Join key with drap_registry_details.csv
manufacturer
license_no	DML (manufacturing license) or DSL (sale license) number
category	e.g. "Essential Drugs"
pack_size
price_pkr	Cleaned to a plain number, e.g. 3257.96
effective_from	Date this price took effect
status	active or deleted — only present when this file has been through drap_update.py; a plain drap_price_index_scraper.py run doesn't add it. See below.
run_date	Date this row was last added, updated, or soft-deleted — only present alongside status

drap_price_index.csv is a persistent master file once drap_update.py has run at least once, not just a snapshot of the latest pull:

• Rows are never deleted outright. A product missing from the latest pull gets status=deleted and run_date set to the date that was detected (soft delete) — its last known price/details are preserved.
• A row unchanged between runs keeps its original run_date; only an addition, an update to a tracked field (price, effective_from, category, manufacturer, brand name, composition), or a soft delete bumps it to today.
• A previously deleted row reappearing in a later pull is reactivated (status=active, run_date = the reactivation date).
• Rows are matched between pulls by (reg_number, pack_size, effective_from, price_pkr, composition), not just (reg_number, pack_size) — DRAP's price index lists multiple price-history entries for the same pack (e.g. last year's price alongside this year's), so a narrower key silently collapsed distinct rows together. A practical side effect: a plain price update for an already-active product shows up as its old price row going deleted and a new row appearing active with today's run_date, rather than as an in-place field change — which matches how DRAP itself keeps the old entry around instead of overwriting it.
• Running drap_price_index_scraper.py directly (not via drap_update.py) overwrites this file with a plain raw pull with no status/run_date columns — that's expected; drap_update.py is what maintains the master file. See Keeping this up to date.

drap_price_index_changes.csv (append-only, written by drap_update.py)

Column	Notes
date	Date the change was detected
change_type	new_reg_number, delisted_reg_number, or field_changed
reg_number
pack_size	Only set for field_changed rows (diffed at reg_number+pack_size granularity)
brand_name
field / old_value / new_value	Only set for field_changed rows

drap_delisted.csv (append-only, written by drap_update.py)

Column	Notes
reg_number
brand_name
last_seen_price / last_effective_from	Values from the last run where this reg_number was still in the price index
delisted_detected_on	Date it first disappeared from the price index

A row here means the product dropped out of the price index (no longer actively priced/sold) — its record in drap_registry_details.csv is left as-is, not deleted, since it's still part of the registration history.

Resuming interrupted runs

Both scripts checkpoint progress:

• drap_registry_detail_scraper.py writes drap_registry_state.json (last completed registration number + the auto-detected max) roughly every 50 requests, and on Ctrl+C. Re-running the same command picks up where it left off.
• drap_price_index_scraper.py appends to the CSV as it goes; if interrupted, check the last row written and re-run with the page number it stopped at (this one doesn't checkpoint automatically — worth adding if you expect frequent interruptions).

Logging

All three scripts write a timestamped log file in addition to console output, and record permanently-failed requests (after retries) to a separate CSV for review:

Script	Log file	Failures CSV
drap_registry_detail_scraper.py	drap_registry_scraper.log	drap_registry_failures.csv
drap_price_index_scraper.py	drap_price_index_scraper.log	drap_price_index_failures.csv
drap_update.py	drap_update.log	(delegates to the two above)

A transient network error (timeout, connection reset) is retried automatically and only logged as a warning; an entry only lands in the failures CSV if all retries were exhausted. To retry just the failed items without re-running the whole scrape, feed the failures CSV back in as a targeted reg_number list:

python drap_registry_detail_scraper.py \
    --reg-list-csv drap_registry_failures.csv --reg-list-column reg_no \
    --state-file /tmp/retry_state.json --failures-file /tmp/retry_failures.csv

(Use a separate --state-file/--failures-file for a retry run so it doesn't collide with the main run's checkpoint.)

Auto range detection

The registry scraper doesn't know DRAP's true max registration number in advance, so by default it finds it automatically:

1. Exponential search — starting from --probe-start (default 50,000), it doubles the candidate number until it finds one where a whole window of consecutive numbers comes back empty.
2. Binary search — it then narrows between the last "has data" point and the first "empty" point to pinpoint the boundary.

A window (default 25 consecutive numbers, --probe-window) is checked at each probe point rather than a single number, because registration numbers have gaps — cancelled or withdrawn products still hold their number, so a single miss doesn't mean you've reached the end of the range.

This adds a few hundred extra requests up front but saves you from either guessing wrong and truncating real data, or grinding through a much larger range than necessary.

Rate limiting and politeness

Both scripts default to 1 request/second. This is a government server, not commercial infrastructure — before increasing request rate or adding concurrency:

• Check robots.txt on both hosts (eapp.dra.gov.pk, e.dra.gov.pk).
• The registry scraper's range is much larger (~150,000 requests vs. ~1,069 for the price index) — at the default rate that's roughly 40+ hours. It's designed to be safely interrupted and resumed across multiple sessions rather than rushed through in one sitting.

Legal and terms of use

This toolkit accesses two public, unauthenticated DRAP web endpoints. Before using it, especially at scale or for anything beyond personal research:

• Check robots.txt on both hosts (eapp.dra.gov.pk, e.dra.gov.pk) and any terms of use published on dra.gov.pk — this repo does not constitute legal advice on whether scraping is permitted.
• DRAP is a Pakistani government regulatory body; the data reflects their public disclosures, not this project's own research. This code is offered as a scraping tool only — it doesn't grant any rights to DRAP's underlying data, and output should be attributed to DRAP if redistributed.
• Respect the rate limits described below. This is a government server, not commercial infrastructure built for bulk access.
• Use responsibly — this project is intended for research, tooling, and personal/organizational reference use, not for republishing DRAP's data as an authoritative or commercial data source.

Data caveats

• DRAP's registry search page carries a disclaimer that its data is a provisional list, not warranted for legal, research, or statistical use, and intended for stakeholders reviewing their own products. Treat scraped output the same way — useful for building tooling, not as a legally authoritative source.
• Registration numbers have gaps (cancelled, withdrawn, or otherwise inactive products retain their number but return empty).
• Site markup can change without notice on both endpoints; re-run a small test range periodically if you're maintaining this long-term.

Keeping this up to date

The registry (drap_registry_details.csv) contains every registration DRAP has ever issued, including long-discontinued products — there's no need to re-enumerate the whole thing on a schedule. The price index (drap_price_index.csv), by contrast, only lists currently active/priced products, which makes it the right source for detecting what's changed.

drap_update.py implements that division of labor:

1. Snapshots the current drap_price_index.csv (the master file) to drap_price_index.prev.csv.
2. Refreshes the price index in full as a raw pull (cheap — ~1,069 pages, ~18 minutes).
3. Merges the raw pull into the master file, keyed by (reg_number, pack_size), rather than overwriting it — see the status/run_date columns described in Output schema. A row present before and still present now with no field changes keeps its old run_date; an addition, a tracked-field update, or a disappearance (soft delete) stamps today's date. Nothing is ever deleted outright.
   • Newly active reg_numbers (new products, or a soft-deleted one reappearing) → the only ones that trigger a registry detail fetch, and only for those specific reg_numbers (seconds to minutes, not hours) — skipped if a registry record for that reg_number already exists.
   • Fully delisted reg_numbers (every row for that reg_number is now status=deleted) → logged to drap_delisted.csv; their existing registry record is left alone.
   • Field changes on rows that stayed active → logged to drap_price_index_changes.csv. No network calls needed for this step.
4. Registry details for reg_numbers you already have are not re-checked by default — that's what makes routine runs cheap. Registration status can still change on an existing drug (e.g. active → cancelled) between runs, so periodically (e.g. monthly, not every run) pass --refresh-existing-registry to re-check the currently-active set.

# Routine (e.g. weekly): cheap delta update
python drap_update.py

# Occasional (e.g. monthly): also re-check known reg_numbers' registry status
python drap_update.py --refresh-existing-registry

Never re-run drap_registry_detail_scraper.py's full sequential range scan as part of routine updates — that's a one-time bootstrap operation (or a last resort if you suspect a large block of new registration numbers was never captured), not something to schedule.

Combining the two datasets

Both CSVs share a registration number column (reg_no_queried / Registration No in the registry file, reg_number in the price file). Joining on that gives you, per product: registration status and composition from the registry, plus current market pricing from the price index — useful as a single reference table for something like NexCare's drug lookup.

License

MIT — see LICENSE. The license covers this code only, not DRAP's underlying data (see Legal and terms of use).