Skip to content

fix: update Job status to FAILED when job_pipeline raises an exception - #3972

Open
Aditya30ag wants to merge 1 commit into
intelowlproject:developfrom
Aditya30ag:fix/job-pipeline-stuck-running-3653
Open

fix: update Job status to FAILED when job_pipeline raises an exception#3972
Aditya30ag wants to merge 1 commit into
intelowlproject:developfrom
Aditya30ag:fix/job-pipeline-stuck-running-3653

Conversation

@Aditya30ag

@Aditya30ag Aditya30ag commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown

Description

When job.execute() raises inside job_pipeline, the except block only marked individual plugin reports as failed it never updated the Job row itself.
Since execute() sets job.status = RUNNING as its first action, and the Celery chain (including job_set_final_status) never dispatches on exception, the job gets permanently stuck in RUNNING.

Consequences:

  • Frontend shows an infinite spinner no WebSocket notification is ever sent
  • remove_old_jobs never cleans the job finished_analysis_time stays NULL
  • Parent Investigation status is never updated
  • Exception message is silently discarded

Fix: In the except block, immediately set job.status = FAILED, job.finished_analysis_time = now(), append the error, persist with job.save(update_fields=[...]), push a WebSocket notification, and propagate failure to the parent Investigation if one exists.
This matches the pattern already used in run_plugin (line 295 of tasks.py).

Type of change

  • Bug fix (non-breaking change which fixes an issue).

Issue #3653

Checklist

  • I have read and understood the rules about how to Contribute to this project
  • The pull request is for the branch develop
  • A new plugin (analyzer, connector, visualizer, playbook, pivot or ingestor) was added or changed — not applicable
  • I have inserted the copyright banner at the start of the file
  • No new libraries were added as requirements
  • Linters (Ruff) gave 0 errors
  • I have added tests for the bug I solved (see tests/ folder). All tests gave 0 errors
  • I have reviewed and verified any LLM-generated code included in this PR. I have explicitly stated that I used an LLM (Claude) to assist in identifying the bug, writing the fix, and the verification script. All output was manually reviewed and validated against the real source.

Copilot AI lite review requested due to automatic review settings August 28, 2026 11:36
@Aditya30ag
Aditya30ag changed the base branch from master to develop August 28, 2026 11:38
Copilot AI reviewed Aug 28, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a Celery job lifecycle bug where exceptions raised during Job.execute() inside job_pipeline could leave the Job row stuck in RUNNING, preventing UI updates, cleanup, and investigation status propagation.

Changes:

  • Mark the Job as FAILED and set finished_analysis_time when job.execute() raises.
  • Persist the exception message into job.errors.
  • Send a WebSocket update and propagate status recalculation to the root Investigation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread intel_owl/tasks.py
Comment thread intel_owl/tasks.py
Comment on lines 254 to +270
@@ -262,6 +267,12 @@ def job_pipeline(
):
report.status = report.STATUSES.FAILED.value
report.save()
JobConsumer.serialize_and_send_job(job)
Copilot AI review requested due to automatic review settings August 28, 2026 11:51
Copilot AI reviewed Aug 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Aditya30ag

Aditya30ag commented Aug 28, 2026

Copy link
Copy Markdown
Author

Hey @berardifra,
Could u please review the changes?

@Aditya30ag
Aditya30ag force-pushed the fix/job-pipeline-stuck-running-3653 branch from b6bcfd4 to fbb27a0 Compare August 28, 2026 12:03
Copilot AI review requested due to automatic review settings August 28, 2026 12:03
Copilot AI reviewed Aug 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 28, 2026 12:05
@Aditya30ag
Aditya30ag force-pushed the fix/job-pipeline-stuck-running-3653 branch from fbb27a0 to c6fed10 Compare August 28, 2026 12:05
Copilot AI reviewed Aug 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings August 28, 2026 12:07
@Aditya30ag
Aditya30ag force-pushed the fix/job-pipeline-stuck-running-3653 branch from c6fed10 to baf4d46 Compare August 28, 2026 12:07
Copilot AI reviewed Aug 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@mlodic @Aditya30ag
fix: update Job status to FAILED when job_pipeline raises an exception.
93bd6b8 
Closes intelowlproject#3653

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@Aditya30ag
Aditya30ag force-pushed the fix/job-pipeline-stuck-running-3653 branch from baf4d46 to 93bd6b8 Compare August 28, 2026 12:09
Copilot AI review requested due to automatic review settings August 28, 2026 12:09
Copilot AI reviewed Aug 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@berardifra berardifra mentioned this pull request Aug 30, 2026
Merged
13 tasks
@berardifra

berardifra commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Hi @Aditya30ag, thanks for tagging me.

I opened #3976 with a small branch against develop for the same issue, keeping the fix focused and adding a regression test for the job.execute() exception path.

I also referenced your PR in the description so the context is preserved.

@Aditya30ag

Aditya30ag commented Aug 30, 2026

Copy link
Copy Markdown
Author

@berardifra
Thank you for taking the time to look into this and for opening a PR with a regression test. I really appreciate it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Aditya30ag @berardifra @mlodic