Skip to content

ci: drop pytest in favor of Sprocket test#263

Open
a-frantz wants to merge 63 commits intomainfrom
demonstration/sprocket-test
Open

ci: drop pytest in favor of Sprocket test#263
a-frantz wants to merge 63 commits intomainfrom
demonstration/sprocket-test

Conversation

@a-frantz
Copy link
Copy Markdown
Member

@a-frantz a-frantz commented Sep 3, 2025

This PR is for demonstration purposes of the RFC document here

this PR has morphed from a demonstration into a real PR for swapping our test infra from pytest to Sprocket. Still a WIP as not all the tests are written yet and it has not been hooked into CI yet

@a-frantz a-frantz force-pushed the demonstration/sprocket-test branch from 535b997 to 7bc1e3a Compare October 8, 2025 13:13
@github-advanced-security
Copy link
Copy Markdown

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

@a-frantz a-frantz changed the title DEMO: sprocket test example ci: drop pytest in favor of Sprocket test Dec 29, 2025
a-frantz and others added 10 commits March 27, 2026 09:47
_Describe the problem or feature in addition to a link to the issues._

the non summary mode of validate_bam can output multiple lines _per
read_ in the input BAM, using more disk than the input file. This
passing that output through gzip.

This also removes the x2 factor on disk size allocation for this task.
We may instead want to keep that x2 factor but only if we're in full
report mode?

Before submitting this PR, please make sure:

- [ ] You have added a few sentences describing the PR here.
- [ ] The code passes all CI tests without any errors or warnings.
- [ ] You have added tests (when appropriate).
- [ ] You have added an entry in any relevant CHANGELOGs (when
appropriate).
- [ ] If you have made any changes to the `scripts/` or `docker/`
directories, please ensure any image versions have been incremented
accordingly!
- [ ] You have updated the README or other documentation to account for
these changes (when appropriate).

---------

Co-authored-by: Andrew Thrasher <adthrasher@gmail.com>
Add filtering of sex chromosomes to the UMAP generation. Also generate a
list of probes that have SNPs.

Before submitting this PR, please make sure:

- [x] You have added a few sentences describing the PR here.
- [x] The code passes all CI tests without any errors or warnings.
- [ ] You have added tests (when appropriate).
- [x] You have added an entry in any relevant CHANGELOGs (when
appropriate).
- [x] If you have made any changes to the `scripts/` or `docker/`
directories, please ensure any image versions have been incremented
accordingly!
- [ ] You have updated the README or other documentation to account for
these changes (when appropriate).

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Ari Frantz <ari.frantz@stjude.org>
Co-authored-by: Jobin Sunny <38107318+jsunny23@users.noreply.github.com>
Currently, we compute file sizes in `GiB`. We then use those to request
disk space, but in `GB`. This is silently undercounting the amount of
disk space we need for these tasks. `GB` was chosen to be the common
unit because our variables already end in `_gb` so this is the least
disruptive change.

Before submitting this PR, please make sure:

- [x] You have added a few sentences describing the PR here.
- [ ] The code passes all CI tests without any errors or warnings.
- [ ] You have added tests (when appropriate).
- [x] You have added an entry in any relevant CHANGELOGs (when
appropriate).
- [ ] If you have made any changes to the `scripts/` or `docker/`
directories, please ensure any image versions have been incremented
accordingly!
- [ ] You have updated the README or other documentation to account for
these changes (when appropriate).
_Describe the problem or feature in addition to a link to the issues._

Before submitting this PR, please make sure:

- [ ] You have added a few sentences describing the PR here.
- [ ] The code passes all CI tests without any errors or warnings.
- [ ] You have added tests (when appropriate).
- [ ] You have added an entry in any relevant CHANGELOGs (when
appropriate).
- [ ] If you have made any changes to the `scripts/` or `docker/`
directories, please ensure any image versions have been incremented
accordingly!
- [ ] You have updated the README or other documentation to account for
these changes (when appropriate).
@a-frantz a-frantz force-pushed the demonstration/sprocket-test branch from 824d01b to a7b661d Compare March 27, 2026 13:49
Copy link
Copy Markdown

@github-advanced-security github-advanced-security AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Snyk Container found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

@a-frantz a-frantz requested a review from adthrasher April 15, 2026 16:08
@a-frantz a-frantz marked this pull request as ready for review April 15, 2026 16:08
@a-frantz
Copy link
Copy Markdown
Member Author

all tests passing 🎉
@adthrasher for local testing, I picard::mark_duplicates uses more memory than available on my MPB, so I've annotated that test with high_mem. Running sprocket dev test -f high_mem at the root of this branch should be all green. I'm seeing all green on the cluster as well. For executing tests there, try a parallelism = 300 to get a good balance of speed and resource utilization. See stjude-rust-labs/sprocket#821 for why just lifting the parallelism throttle completely doesn't work

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't gotten further, but my assumption was that we'd call sprocket dev test as the second part of this to test the newly built images.

with:
lint: true
except: KnownRules
sprocket_lint:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're doing lint and format, I'd probably just shorten this to sprocket.

Suggested change
sprocket_lint:
sprocket:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe lengthen to sprocket_lint_format

Comment thread tools/picard.wdl
Comment on lines -203 to -205
String stringency_arg = if index_validation_stringency_less_exhaustive
then "--INDEX_VALIDATION_STRINGENCY LESS_EXHAUSTIVE"
else ""
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't recall what this parameter does. Why are we removing the argument?

adthrasher added a commit that referenced this pull request Apr 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants