Patch delays#140
Closed
YaphetKG wants to merge 11 commits into
Closed
Conversation
* Sundry updates and cleanup, pre-refactor * checkpointing the big move of dags/roger to src/roger. No packaging info added, this is almost certainly broken * First round of code updates for new dug data model * Setuptools and python library adjustments * Tweaking a couple of pipelines * First cut at a DDM pipeline class * Refactoring CLI to use pipelines instead of dug_util * Starting DM2 ingestion pipeline. * Finished refactor so that cli runs same code as dags (pipelines module) * Removing dead code * just a tiny PEP8 tweak * Breaking test dag into separate file for clarity's sake. * Bringing old tests up to speed * Updated Makefile to use new cli entrypoint * Fix to make subsubclasses of DugPipeline work as pipelines * Tweak to make simpler override of pipelines, just focusing on filename pattern * Numerous fixes to CLI to make CLI-based testing easier, more pipeline-focused * Update to allow disabling of redis http cache in config (pass to dug) * touchup heal studies pipeline * Added ability to override config file location, test config (no redis cache) * heal-cdes pipeline added * Update dockerfile image, temp bump to github workflows to build * Backing up to python 3.12 because of issues with airflow and python 3.13 * Version subrevision bumped to trigger build for vulnerabilities * update healthcheck * update pipeline indexing * add index config, fix typo * Fix errors in failing subtasks, update dockerfile * remove expand_to_dug_element and update pipeline to filter DDM2 json files * update pipeline validation * update validation * add roger heal_studies pipeline back * update validation pagination * fix search validation for 2 char concept names * fix redis dependency version * change concept indexing to use index_element * return to index_concept * add debug logs for concept validation * fix set serialization in dbug * Add KG-expansion for variable optional terms * add crawl_task as dependencies of validation tasks * Ensure concept validation does not strip hyphens from search terms * Fix regex * replace hyphens with spaces * Fix concept validation to only pick concept search terms from variables * Fix concept skipping for curies without associated variable elements * update concept validation search_term cleaning regex * update regex search term cleaning for validation * add gt lt as non cleaned chars for validation search terms * add colon to nonstripped validation chars * bump page size in search_elements * Updated upstream versions for security reasons. * pyflake cleanup. * bump trivy version --------- Co-authored-by: frostyfan109 <groupe@renci.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.