chore: purge committed sample/corpus data#45
Open
Airwhale wants to merge 3 commits into
Open
Conversation
Removes 191 tracked data files that were bloating the repo (~1.4M lines):
scraped Reddit user histories (Scrapers/users/, reddit_sample_data/users/),
the sample corpus + extraction-output caches (reddit_sample_data/), the
output/ slices, and binary zips.
All of this is preserved externally and recoverable:
- Raw source (Arctic Shift NDJSON, covidlonghaulers corpus slices,
subreddit_posts.json) lives in s3://patientpunk/raw_data/ and
s3://patientpunk/6_11_hackathon/.
- The 185 per-user scraped histories were synced to
s3://patientpunk/raw_data/scraped_user_histories/ and also live in the
local PatientPunk_data/repo_export/ evacuation copy.
- The outputs/ caches are regenerable intermediates.
.gitignore is updated to keep data, databases, zips, and pipeline temp files
out of the repo going forward.
This is the base of the stack that splits PR #43 into reviewable pieces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the .gitignore file to exclude various runtime data, scraped corpus data, dev scratch directories, pipeline intermediate files, and database files, while removing several local JSON files containing scraped Reddit posts and comments. The review feedback correctly points out that ignoring the data/ directory directly prevents Git from tracking !data/.gitkeep, and suggests ignoring data/* instead to ensure the placeholder file is tracked.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Airwhale
changed the title
Airwhale
changed the title
This was referenced Jun 15, 2026
Airwhale
pushed a commit
that referenced
this pull request
Jun 15, 2026
Independent-of-the-extraction-stack repo-level changes from #43: - Agent config: .cursorrules, AGENTS.md, CLAUDE.md - Docs: root README, and the kept ADRs (001-pydantic-v2, 003-two-pipeline, 004-variable-picker). ADRs 002-marimo-for-ui and 005-analysis-warning-policy are omitted — they document the UI/stats-engine being dropped (superseded by the research-assistant skill). - Scrapers: CONTRIBUTING/README/SCRAPER_HELP docs + scrape_corpus.py. - .env.example: LLM provider/endpoint config. - src/utilities/__init__.py: shared LLM config + get_git_commit provenance helper (used by the posts pipeline; load_db has a local fallback). Working-notes docs (MVP_PLAN, ldn_notes, DATA_NOTES) intentionally left out — they read as drafts, not settled docs. Based on main, not the extraction stack — none of these files are touched by #45-#49, so it can land in any order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Airwhale
pushed a commit
that referenced
this pull request
Jun 15, 2026
Independent-of-the-extraction-stack repo-level changes from #43: - Agent config: .cursorrules (AGENTS.md / CLAUDE.md are gitignored as local-only agent files — see #45.) - Docs: root README, and the kept ADRs (001-pydantic-v2, 003-two-pipeline, 004-variable-picker). ADRs 002-marimo-for-ui and 005-analysis-warning-policy are omitted — they document the UI/stats-engine being dropped (superseded by the research-assistant skill). - Scrapers: CONTRIBUTING/README/SCRAPER_HELP docs + scrape_corpus.py. - .env.example: LLM provider/endpoint config. - src/utilities/__init__.py: shared LLM config + get_git_commit provenance helper (used by the posts pipeline; load_db has a local fallback). Working-notes docs (MVP_PLAN, ldn_notes, DATA_NOTES) intentionally left out. Based on main, not the extraction stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These are personal agent-workflow files (they reference the git-ignored CONTINUITY.md session ritual), not shared docs. Keep them local; the shared project context lives in README.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Airwhale
force-pushed
the
chore/purge-committed-data
branch
from
2bd19de to
0fbc78e
Compare
Airwhale
pushed a commit
that referenced
this pull request
Jun 15, 2026
Independent-of-the-extraction-stack repo-level changes from #43: - Docs: root README, and the kept ADRs (001-pydantic-v2, 003-two-pipeline, 004-variable-picker). ADRs 002-marimo-for-ui and 005-analysis-warning-policy are omitted — they document the UI/stats-engine being dropped (superseded by the research-assistant skill). - Scrapers: CONTRIBUTING/README/SCRAPER_HELP docs + scrape_corpus.py. - .env.example: LLM provider/endpoint config. - src/utilities/__init__.py: shared LLM config + get_git_commit provenance helper (used by the posts pipeline; load_db has a local fallback). Agent config (.cursorrules, AGENTS.md, CLAUDE.md) is gitignored as local-only files — see #45. Working-notes docs (MVP_PLAN, ldn_notes, DATA_NOTES) omitted. Based on main, not the extraction stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of an ongoing effort to split the oversized #43 into smaller, reviewable PRs. Pure repo hygiene — no product code.
What
Scrapers/users/,reddit_sample_data/users/), the sample corpus + extraction-output caches (reddit_sample_data/),output/slices, and binary zips..gitignoreupdated to keep data,*.db, zips, pipeline temp files, and local-only agent config (.cursorrules,AGENTS.md,CLAUDE.md,CONTINUITY.md) out of the repo going forward.Nothing is lost — all recoverable
subreddit_posts.json):s3://patientpunk/raw_data/+s3://patientpunk/6_11_hackathon/.s3://patientpunk/raw_data/scraped_user_histories/and in localPatientPunk_data/repo_export/.outputs/caches: regenerable intermediates.Review
191 deletions + the
.gitignoreedits. Confirm the deleted paths are all data (no code).