Scheduler goes to unhealthy state and auto heals after a while

[anuragmq]

Hi,
I am running the Airflow in Kubernetes environment using EKS .

when looking at the Gantt chart for one of the dags (call it Dag1) .I can notice around 30-45 mins idle time between tasks when no task is executing and the scheduler status turns to unhealthy.

Looking at the dag_processor_manager logs, I can see the following errors coinciding with the idle timing

{dag_processing.py:1334} ERROR - Processor for /airflow-dags/abc.py with PID 26155 started at 2020-12-09T17:31:53.953789+00:00 has timed out, killing it.

The abc.py dag is not even scheduled to run at this time so not sure why the scheduler/ processor is trying to run this task .
After the above process is killed, the tasks starts running in the Dag1.

Any idea why would this error be coming up and how to fix this error .

My environment details :

Airflow. - 1.10.10
Kubernetes - 1.16.5

Thanks,
Anurag