dagProcessor liveness probe always fails #69048

Activity

1. IanSweeneyKodiak

   added

   kind:bugThis is a clearly a bugThis is a clearly a bug

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Jun 26, 2026
2. boring-cyborg commented on Jun 26, 2026

   

   boring-cyborgBot

   on Jun 26, 2026 – with boring-cyborg

   More actions

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
3. dosubot

   added

   area:helm-chartAirflow Helm ChartAirflow Helm Chart

   on Jun 26, 2026
4. IanSweeneyKodiak commented on Jun 26, 2026

   

   IanSweeneyKodiak

   on Jun 26, 2026

   Last edited by IanSweeneyKodiak

   Author

   More actions

   Turns out there is a running dagProcessor job in the db but the cli used for the health check returns no alive jobs
   select * from job where job_type='DagProcessorJob' order by start_date desc;
206 | | running | DagProcessorJob | 2026-06-26 17:33:29.530223+00 | | 2026-06-26 17:37:04.932384+00 | | airflow-dag-processor-5df45489f7-92r7z | airflow

   airflow jobs check --job-type DagProcessorJob
No alive jobs found

   It seems that maybe the dagProcessor is not sending a heartbeat?

   Guessing this is because the heartbeat is only_if_necessary
   

   airflow/airflow-core/src/airflow/jobs/dag_processor_job_runner.py

   Line 55 in ba8cfe4

   only_if_necessary=True,

   Maybe due to the config option
   scheduler.job_heartbeat_sec: 30
5. Miretpl commented on Jun 28, 2026

   

   Miretpl

   on Jun 28, 2026

   Contributor

   More actions

   If you don't see missing dags or in the dag-processor logs, you see that the dag-processor is working, I would assume that there is high CPU usage + maybe some bottlenecks on the database, and the healthcheck command just passes the execution timeout. If that is the case, you can modify resources, maybe optimise Dags code regarding parsing and execution layers (https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code, which in general should decrease workload on the Dag Processor) and tweak a little Dag Processor configuration like parsing processor, etc.

   From more fixes like workarounds, there are some options regarding the probe itself within the chart: https://github.com/apache/airflow/blob/helm-chart/1.22.0/chart/values.yaml#L3052.
6. tirkarthi commented on Jun 30, 2026

   

   tirkarthi

   on Jun 30, 2026

   Contributor

   More actions

   What is the value of hostname command output inside the pod? Does the value match the entries found in the database with heartbeat timestamp updated?
7. arkadiuszbach commented on Jul 7, 2026

   

   arkadiuszbach

   on Jul 7, 2026

   Contributor

   More actions

   Turns out there is a running dagProcessor job in the db but the cli used for the health check returns no alive jobs select * from job where job_type='DagProcessorJob' order by start_date desc; 206 | | running | DagProcessorJob | 2026-06-26 17:33:29.530223+00 | | 2026-06-26 17:37:04.932384+00 | | airflow-dag-processor-5df45489f7-92r7z | airflow

   airflow jobs check --job-type DagProcessorJob No alive jobs found

   It seems that maybe the dagProcessor is not sending a heartbeat?

   Guessing this is because the heartbeat is only_if_necessary

   airflow/airflow-core/src/airflow/jobs/dag_processor_job_runner.py

   Line 55 in ba8cfe4

   only_if_necessary=True,
   Maybe due to the config option scheduler.job_heartbeat_sec: 30

   Yes its because of scheduler.job_heartbeat_sec: 30 and dag_processor.health_check_threshold: 30, see: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#health-check-threshold

   just increase health_check_threshold for the dag_processor
8. eajimmy commented on Jul 8, 2026

   

   eajimmy

   on Jul 8, 2026

   More actions

   Same problem here on Airflow 3.2.1 (KubernetesExecutor + CeleryExecutor).
   I confirmed it with this query, running it repeatedly:

   SELECT id, hostname, state,
          latest_heartbeat,
          EXTRACT(EPOCH FROM (now() - latest_heartbeat)) AS age_seconds
   FROM job
   WHERE job_type = 'DagProcessorJob'
   ORDER BY latest_heartbeat DESC
   LIMIT 30;

   age_seconds oscillates continuously between ~30 and ~60s and never drops close to 0.

   Setting AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120 fixes it, as mentioned above.

   I also tried increasing parsing_processes and min_file_process_interval, but these had no effect on age_seconds.
   It seems that the staleness is intrinsic to the heartbeat mechanism, not to parsing load.
   I even confirmed this with the processor fully idle (min_file_process_interval=300, no files to parse) and the age still stayed in the 30–60s range.
9. IanSweeneyKodiak commented on Jul 8, 2026

   

   IanSweeneyKodiak

   on Jul 8, 2026

   Author

   More actions

   Would it be possible/correct to update this default in the helm chart then?
   AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120
10. Miretpl commented on Jul 9, 2026

    

    Miretpl

    on Jul 9, 2026

    Contributor

    More actions

    I also tried increasing parsing_processes and min_file_process_interval, but these had no effect on age_seconds.
    It seems that the staleness is intrinsic to the heartbeat mechanism, not to parsing load.
    I even confirmed this with the processor fully idle (min_file_process_interval=300, no files to parse) and the age still stayed in the 30–60s range.

    This indicates insufficient resources on the database, such as CPU or IOPS.

    Would it be possible/correct to update this default in the Helm chart then?
    AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120

    To be honest, I don't see the reason to do that. At least for now, there is a lack of information regarding particular components' configurations and resources, as well as the dags' characteristics themselves, which are impacting the load on both the dag processor and the database. From my experience with quite big Airflow environments (comparing, e.g. to the data from the Airflow Survey), when the database is correctly configured with quality gates for dags code and some tweaks to dag processor configuration (depending on the dags characteristics), the default settings are sufficient. Also, adjusting this configuration is easy, and I would say not problematic. If adjusting it wasn't possible, it would be a totally different story.

    I will convert this issue to a discussion since we don't have enough data to determine whether it is environment-specific or an Apache Airflow issue.
11. apache

    locked and limited conversation to collaborators on Jul 9, 2026
12. Miretpl

    converted this issue into a discussion #69667 on Jul 9, 2026