Cannot view logs for running task in Airflow standalone #42984

rcheatham-q · 2024-10-10T21:32:02Z

rcheatham-q
Oct 10, 2024

Apache Airflow version

2.10.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When viewing the logs of a running task in the UI of an airflow standalone instance, the logs do not show. The output of the standalone command shows a lot of SIGSEGV signals being sent to the worker. Once the task completed, the SIGSEGV signals stopped and everything functioned normally, including viewing logs. Additionally, the following dialog was displayed.

What you think should happen instead?

When viewing the logs for a running task in the UI of an Airflow instance run with airflow standalone, the logs are shown.

How to reproduce

Initialize a new Airflow environment using the following commands:

python3.10 -m venv venv
source venv/bin/activate
pip install apache-airflow==2.10.2
AIRFLOW_HOME=$(pwd)/.airflow venv/bin/airflow standalone

Modify the file ./venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py to include some code that takes a while to complete. I have tried both sleeping using the sleep function in the time package, and logging thousands of lines in a loop and both caused the issue.

Trigger the DAG and view the logs for the modified task.

Operating System

OSX

Versions of Apache Airflow Providers

None

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

standalone.log

❯ AIRFLOW_HOME=$(pwd)/.airflow venv/bin/airflow standalone
standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
standalone | Database ready
/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/flask_limiter/extension.py:333 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
WARNI [airflow.example_dags.plugins.workday] Could not import pandas. Holidays will not be considered.
triggerer  | ____________       _____________
triggerer  | ____    |__( )_________  __/__  /________      __
triggerer  | ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
triggerer  | ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
triggerer  | _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
triggerer  | [2024-10-10T15:23:44.871-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
triggerer  | [2024-10-10T15:23:44.873-0600] {triggerer_job_runner.py:181} INFO - Setting up TriggererHandlerWrapper with handler 
triggerer  | [2024-10-10T15:23:44.873-0600] {triggerer_job_runner.py:237} INFO - Setting up logging queue listener with handlers [ (NOTSET)>, ]
triggerer  | [2024-10-10T15:23:44.876-0600] {triggerer_job_runner.py:338} INFO - Starting the triggerer
scheduler  | ____________       _____________
scheduler  | ____    |__( )_________  __/__  /________      __
scheduler  | ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
scheduler  | ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
scheduler  | _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
scheduler  | [2024-10-10T15:23:44.972-0600] {_client.py:1038} INFO - HTTP Request: GET https://apacheairflow.gateway.scarf.sh/scheduler?version=2.10.2&python_version=3.10&platform=Darwin&arch=arm64&database=sqlite&db_version=3.46&executor=SequentialExecutor "HTTP/1.1 200 OK"
scheduler  | [2024-10-10T15:23:45.010-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
scheduler  | [2024-10-10T15:23:45.011-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
scheduler  | [2024-10-10T15:23:45.061-0600] {scheduler_job_runner.py:935} INFO - Starting the scheduler
scheduler  | [2024-10-10T15:23:45.062-0600] {scheduler_job_runner.py:942} INFO - Processing each file at most -1 times
scheduler  | [2024-10-10T15:23:45.065-0600] {manager.py:174} INFO - Launched DagFileProcessorManager with pid: 90224
scheduler  | [2024-10-10T15:23:45.066-0600] {scheduler_job_runner.py:1847} INFO - Adopting or resetting orphaned tasks for active dag runs
triggerer  | [2024-10-10 15:23:45 -0600] [90221] [INFO] Starting gunicorn 23.0.0
triggerer  | [2024-10-10 15:23:45 -0600] [90221] [INFO] Listening at: http://[::]:8794 (90221)
triggerer  | [2024-10-10 15:23:45 -0600] [90221] [INFO] Using worker: sync
triggerer  | [2024-10-10 15:23:45 -0600] [90225] [INFO] Booting worker with pid: 90225
triggerer  | [2024-10-10 15:23:45 -0600] [90226] [INFO] Booting worker with pid: 90226
webserver  | /Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/flask_limiter/extension.py:333 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
scheduler  | [2024-10-10 15:23:45 -0600] [90223] [INFO] Starting gunicorn 23.0.0
scheduler  | [2024-10-10 15:23:45 -0600] [90223] [INFO] Listening at: http://[::]:8793 (90223)
scheduler  | [2024-10-10 15:23:45 -0600] [90223] [INFO] Using worker: sync
scheduler  | [2024-10-10 15:23:45 -0600] [90227] [INFO] Booting worker with pid: 90227
scheduler  | [2024-10-10T15:23:45.798-0600] {settings.py:63} INFO - Configured default timezone UTC
scheduler  | [2024-10-10 15:23:45 -0600] [90228] [INFO] Booting worker with pid: 90228
scheduler  | [2024-10-10T15:23:45.806-0600] {manager.py:406} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2) when using sqlite. So we set parallelism to 1.
webserver  | [2024-10-10T15:23:46.003-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
webserver  | /Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py:547 FutureWarning: section/key [webserver/update_fab_perms] has been deprecated, you should use[fab/update_fab_perms] instead. Please update your `conf.get*` call to use the new name
webserver  | [2024-10-10 15:23:46 -0600] [90219] [INFO] Starting gunicorn 23.0.0
webserver  | [2024-10-10 15:23:46 -0600] [90219] [INFO] Listening at: http://0.0.0.0:8080 (90219)
webserver  | [2024-10-10 15:23:46 -0600] [90219] [INFO] Using worker: sync
webserver  | [2024-10-10 15:23:46 -0600] [90230] [INFO] Booting worker with pid: 90230
webserver  | [2024-10-10 15:23:46 -0600] [90231] [INFO] Booting worker with pid: 90231
webserver  | [2024-10-10 15:23:46 -0600] [90232] [INFO] Booting worker with pid: 90232
webserver  | [2024-10-10 15:23:46 -0600] [90233] [INFO] Booting worker with pid: 90233
standalone | Airflow is ready
standalone | Login with username: admin  password: [REDACTED]
standalone | Airflow Standalone is for development purposes only. Do not use this in production!
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:51 -0600] "GET /dags/tutorial_taskflow_api/grid HTTP/1.1" 200 9755 "http://localhost:8080/home?status=active" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:51 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 3222 "http://localhost:8080/dags/tutorial_taskflow_api/grid" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:51 -0600] "GET /object/next_run_datasets/tutorial_taskflow_api HTTP/1.1" 200 39 "http://localhost:8080/dags/tutorial_taskflow_api/grid" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:51 -0600] "GET /api/v1/dags/tutorial_taskflow_api HTTP/1.1" 200 1189 "http://localhost:8080/dags/tutorial_taskflow_api/grid" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:51 -0600] "GET /api/v1/dags/tutorial_taskflow_api/details HTTP/1.1" 200 1962 "http://localhost:8080/dags/tutorial_taskflow_api/grid" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:55 -0600] "POST /dags/tutorial_taskflow_api/trigger?origin=%2Fdags%2Ftutorial_taskflow_api%2Fgrid HTTP/1.1" 302 413 "http://localhost:8080/dags/tutorial_taskflow_api/grid" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:55 -0600] "GET /dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00 HTTP/1.1" 200 9885 "http://localhost:8080/dags/tutorial_taskflow_api/grid" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:55 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4159 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:55 -0600] "GET /object/next_run_datasets/tutorial_taskflow_api HTTP/1.1" 200 39 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:57 -0600] "GET /api/v1/dags/tutorial_taskflow_api/tasks/transform HTTP/1.1" 200 1116 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:57 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 751 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:57 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform/dependencies HTTP/1.1" 200 608 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
scheduler  | Dag run  in running state
scheduler  | Dag information Queued at: 2024-10-10 21:23:55.333863+00:00 hash info: 38e56a72b18ab9d01e9f2f0cd6695560
scheduler  | [2024-10-10T15:23:57.796-0600] {scheduler_job_runner.py:423} INFO - 1 tasks up for execution:
scheduler  | 
scheduler  | [2024-10-10T15:23:57.797-0600] {scheduler_job_runner.py:495} INFO - DAG tutorial_taskflow_api has 0/16 running and queued tasks
scheduler  | [2024-10-10T15:23:57.797-0600] {scheduler_job_runner.py:634} INFO - Setting the following tasks to queued state:
scheduler  | 
scheduler  | [2024-10-10T15:23:57.798-0600] {scheduler_job_runner.py:736} INFO - Trying to enqueue tasks: [] for executor: SequentialExecutor(parallelism=32)
scheduler  | [2024-10-10T15:23:57.798-0600] {scheduler_job_runner.py:680} INFO - Sending TaskInstanceKey(dag_id='tutorial_taskflow_api', task_id='extract', run_id='manual__2024-10-10T21:23:55.317397+00:00', try_number=1, map_index=-1) to SequentialExecutor with priority 3 and queue default
scheduler  | [2024-10-10T15:23:57.798-0600] {base_executor.py:168} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'tutorial_taskflow_api', 'extract', 'manual__2024-10-10T21:23:55.317397+00:00', '--local', '--subdir', '/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py']
scheduler  | [2024-10-10T15:23:57.799-0600] {sequential_executor.py:84} INFO - Executing command: ['airflow', 'tasks', 'run', 'tutorial_taskflow_api', 'extract', 'manual__2024-10-10T21:23:55.317397+00:00', '--local', '--subdir', '/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py']
scheduler  | [2024-10-10T15:23:58.459-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
scheduler  | [2024-10-10T15:23:58.463-0600] {dagbag.py:588} INFO - Filling up the DagBag from /Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py
scheduler  | [2024-10-10T15:23:58.491-0600] {example_local_kubernetes_executor.py:40} WARNING - Could not import DAGs in example_local_kubernetes_executor.py
scheduler  | Traceback (most recent call last):
scheduler  | File "/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 38, in 
scheduler  | from kubernetes.client import models as k8s
scheduler  | ModuleNotFoundError: No module named 'kubernetes'
scheduler  | [2024-10-10T15:23:58.492-0600] {example_local_kubernetes_executor.py:41} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:58 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4254 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
scheduler  | [2024-10-10T15:23:58.636-0600] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes]
scheduler  | [2024-10-10T15:23:58.654-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
scheduler  | [2024-10-10T15:23:58.669-0600] {task_command.py:467} INFO - Running  on host Q-QKQT6Y31P2-rcheatham
webserver  | 127.0.0.1 - - [10/Oct/2024:15:23:58 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform/logs/1?full_content=false HTTP/1.1" 200 2 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
scheduler  | [2024-10-10T15:23:58.975-0600] {scheduler_job_runner.py:764} INFO - Received executor event with state success for task instance TaskInstanceKey(dag_id='tutorial_taskflow_api', task_id='extract', run_id='manual__2024-10-10T21:23:55.317397+00:00', try_number=1, map_index=-1)
scheduler  | [2024-10-10T15:23:58.980-0600] {scheduler_job_runner.py:801} INFO - TaskInstance Finished: dag_id=tutorial_taskflow_api, task_id=extract, run_id=manual__2024-10-10T21:23:55.317397+00:00, map_index=-1, run_start_date=2024-10-10 21:23:58.727434+00:00, run_end_date=2024-10-10 21:23:58.804088+00:00, run_duration=0.076654, state=success, executor=SequentialExecutor(parallelism=32), executor_state=success, try_number=1, max_tries=0, job_id=17, pool=default_pool, queue=default, priority_weight=3, operator=_PythonDecoratedOperator, queued_dttm=2024-10-10 21:23:57.797448+00:00, queued_by_job_id=16, pid=90241
scheduler  | [2024-10-10T15:23:59.765-0600] {scheduler_job_runner.py:423} INFO - 1 tasks up for execution:
scheduler  | 
scheduler  | [2024-10-10T15:23:59.765-0600] {scheduler_job_runner.py:495} INFO - DAG tutorial_taskflow_api has 0/16 running and queued tasks
scheduler  | [2024-10-10T15:23:59.765-0600] {scheduler_job_runner.py:634} INFO - Setting the following tasks to queued state:
scheduler  | 
scheduler  | [2024-10-10T15:23:59.766-0600] {scheduler_job_runner.py:736} INFO - Trying to enqueue tasks: [] for executor: SequentialExecutor(parallelism=32)
scheduler  | [2024-10-10T15:23:59.766-0600] {scheduler_job_runner.py:680} INFO - Sending TaskInstanceKey(dag_id='tutorial_taskflow_api', task_id='transform', run_id='manual__2024-10-10T21:23:55.317397+00:00', try_number=1, map_index=-1) to SequentialExecutor with priority 2 and queue default
scheduler  | [2024-10-10T15:23:59.766-0600] {base_executor.py:168} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'tutorial_taskflow_api', 'transform', 'manual__2024-10-10T21:23:55.317397+00:00', '--local', '--subdir', '/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py']
scheduler  | [2024-10-10T15:23:59.767-0600] {sequential_executor.py:84} INFO - Executing command: ['airflow', 'tasks', 'run', 'tutorial_taskflow_api', 'transform', 'manual__2024-10-10T21:23:55.317397+00:00', '--local', '--subdir', '/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py']
scheduler  | [2024-10-10T15:24:00.435-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
scheduler  | [2024-10-10T15:24:00.440-0600] {dagbag.py:588} INFO - Filling up the DagBag from /Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py
scheduler  | [2024-10-10T15:24:00.468-0600] {example_local_kubernetes_executor.py:40} WARNING - Could not import DAGs in example_local_kubernetes_executor.py
scheduler  | Traceback (most recent call last):
scheduler  | File "/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 38, in 
scheduler  | from kubernetes.client import models as k8s
scheduler  | ModuleNotFoundError: No module named 'kubernetes'
scheduler  | [2024-10-10T15:24:00.468-0600] {example_local_kubernetes_executor.py:41} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]
scheduler  | [2024-10-10T15:24:00.600-0600] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes]
scheduler  | [2024-10-10T15:24:00.618-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
scheduler  | [2024-10-10T15:24:00.634-0600] {task_command.py:467} INFO - Running  on host Q-QKQT6Y31P2-rcheatham
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:01 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:01 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:04 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:04.649-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:04 -0600] [90219] [ERROR] Worker (pid:90232) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:04 -0600] [90245] [INFO] Booting worker with pid: 90245
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:04 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:05.284-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:05 -0600] [90219] [ERROR] Worker (pid:90233) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:05 -0600] [90246] [INFO] Booting worker with pid: 90246
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:07 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:07 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:08.331-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:08 -0600] [90219] [ERROR] Worker (pid:90231) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:08 -0600] [90247] [INFO] Booting worker with pid: 90247
webserver  | [2024-10-10T15:24:08.919-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:08 -0600] [90219] [ERROR] Worker (pid:90246) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:08 -0600] [90250] [INFO] Booting worker with pid: 90250
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:10 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:10 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:11.993-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:11 -0600] [90219] [ERROR] Worker (pid:90245) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:12 -0600] [90252] [INFO] Booting worker with pid: 90252
webserver  | [2024-10-10T15:24:12.554-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:12 -0600] [90219] [ERROR] Worker (pid:90252) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:12 -0600] [90255] [INFO] Booting worker with pid: 90255
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:13 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:13 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:15.623-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:15 -0600] [90219] [ERROR] Worker (pid:90250) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:15 -0600] [90260] [INFO] Booting worker with pid: 90260
webserver  | [2024-10-10T15:24:16.194-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:16 -0600] [90219] [ERROR] Worker (pid:90230) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:16 -0600] [90261] [INFO] Booting worker with pid: 90261
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:16 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:16 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:19.532-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:19 -0600] [90219] [ERROR] Worker (pid:90255) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:19 -0600] [90273] [INFO] Booting worker with pid: 90273
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:19 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:19 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:20.073-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:20 -0600] [90219] [ERROR] Worker (pid:90260) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:20 -0600] [90274] [INFO] Booting worker with pid: 90274
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:22 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:22 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:23.110-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:23 -0600] [90219] [ERROR] Worker (pid:90261) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:23 -0600] [90275] [INFO] Booting worker with pid: 90275
webserver  | [2024-10-10T15:24:23.672-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:23 -0600] [90219] [ERROR] Worker (pid:90273) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:23 -0600] [90276] [INFO] Booting worker with pid: 90276
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:26 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:26 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:26.733-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:26 -0600] [90219] [ERROR] Worker (pid:90274) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:26 -0600] [90277] [INFO] Booting worker with pid: 90277
webserver  | [2024-10-10T15:24:27.301-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:27 -0600] [90219] [ERROR] Worker (pid:90247) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:27 -0600] [90278] [INFO] Booting worker with pid: 90278
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:29 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1001 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:29 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4380 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | [2024-10-10T15:24:30.373-0600] {executor_loader.py:254} INFO - Loaded executor: SequentialExecutor
webserver  | [2024-10-10 15:24:30 -0600] [90219] [ERROR] Worker (pid:90277) was sent SIGSEGV!
webserver  | [2024-10-10 15:24:30 -0600] [90279] [INFO] Booting worker with pid: 90279
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:30 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform/logs/1?full_content=false HTTP/1.1" 200 3950 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
scheduler  | [2024-10-10T15:24:30.957-0600] {scheduler_job_runner.py:764} INFO - Received executor event with state success for task instance TaskInstanceKey(dag_id='tutorial_taskflow_api', task_id='transform', run_id='manual__2024-10-10T21:23:55.317397+00:00', try_number=1, map_index=-1)
scheduler  | [2024-10-10T15:24:30.960-0600] {scheduler_job_runner.py:801} INFO - TaskInstance Finished: dag_id=tutorial_taskflow_api, task_id=transform, run_id=manual__2024-10-10T21:23:55.317397+00:00, map_index=-1, run_start_date=2024-10-10 21:24:00.685070+00:00, run_end_date=2024-10-10 21:24:30.765587+00:00, run_duration=30.080517, state=success, executor=SequentialExecutor(parallelism=32), executor_state=success, try_number=1, max_tries=0, job_id=18, pool=default_pool, queue=default, priority_weight=2, operator=_PythonDecoratedOperator, queued_dttm=2024-10-10 21:23:59.765812+00:00, queued_by_job_id=16, pid=90244
scheduler  | [2024-10-10T15:24:30.974-0600] {job.py:229} INFO - Heartbeat recovered after 35.07 seconds
scheduler  | [2024-10-10T15:24:31.917-0600] {scheduler_job_runner.py:423} INFO - 1 tasks up for execution:
scheduler  | 
scheduler  | [2024-10-10T15:24:31.917-0600] {scheduler_job_runner.py:495} INFO - DAG tutorial_taskflow_api has 0/16 running and queued tasks
scheduler  | [2024-10-10T15:24:31.917-0600] {scheduler_job_runner.py:634} INFO - Setting the following tasks to queued state:
scheduler  | 
scheduler  | [2024-10-10T15:24:31.918-0600] {scheduler_job_runner.py:736} INFO - Trying to enqueue tasks: [] for executor: SequentialExecutor(parallelism=32)
scheduler  | [2024-10-10T15:24:31.918-0600] {scheduler_job_runner.py:680} INFO - Sending TaskInstanceKey(dag_id='tutorial_taskflow_api', task_id='load', run_id='manual__2024-10-10T21:23:55.317397+00:00', try_number=1, map_index=-1) to SequentialExecutor with priority 1 and queue default
scheduler  | [2024-10-10T15:24:31.918-0600] {base_executor.py:168} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'tutorial_taskflow_api', 'load', 'manual__2024-10-10T21:23:55.317397+00:00', '--local', '--subdir', '/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py']
scheduler  | [2024-10-10T15:24:31.919-0600] {sequential_executor.py:84} INFO - Executing command: ['airflow', 'tasks', 'run', 'tutorial_taskflow_api', 'load', 'manual__2024-10-10T21:23:55.317397+00:00', '--local', '--subdir', '/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py']
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:32 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1036 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:32 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4444 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
scheduler  | [2024-10-10T15:24:32.603-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
scheduler  | [2024-10-10T15:24:32.607-0600] {dagbag.py:588} INFO - Filling up the DagBag from /Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/tutorial_taskflow_api.py
scheduler  | [2024-10-10T15:24:32.639-0600] {example_local_kubernetes_executor.py:40} WARNING - Could not import DAGs in example_local_kubernetes_executor.py
scheduler  | Traceback (most recent call last):
scheduler  | File "/Users/rcheatham/workspace/airflow/bug-test/venv/lib/python3.10/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 38, in 
scheduler  | from kubernetes.client import models as k8s
scheduler  | ModuleNotFoundError: No module named 'kubernetes'
scheduler  | [2024-10-10T15:24:32.640-0600] {example_local_kubernetes_executor.py:41} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes]
scheduler  | [2024-10-10T15:24:32.783-0600] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes]
scheduler  | [2024-10-10T15:24:32.800-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
scheduler  | [2024-10-10T15:24:32.815-0600] {task_command.py:467} INFO - Running  on host Q-QKQT6Y31P2-rcheatham
scheduler  | [2024-10-10T15:24:33.099-0600] {scheduler_job_runner.py:764} INFO - Received executor event with state success for task instance TaskInstanceKey(dag_id='tutorial_taskflow_api', task_id='load', run_id='manual__2024-10-10T21:23:55.317397+00:00', try_number=1, map_index=-1)
scheduler  | [2024-10-10T15:24:33.102-0600] {scheduler_job_runner.py:801} INFO - TaskInstance Finished: dag_id=tutorial_taskflow_api, task_id=load, run_id=manual__2024-10-10T21:23:55.317397+00:00, map_index=-1, run_start_date=2024-10-10 21:24:32.866040+00:00, run_end_date=2024-10-10 21:24:32.936800+00:00, run_duration=0.07076, state=success, executor=SequentialExecutor(parallelism=32), executor_state=success, try_number=1, max_tries=0, job_id=19, pool=default_pool, queue=default, priority_weight=1, operator=_PythonDecoratedOperator, queued_dttm=2024-10-10 21:24:31.917929+00:00, queued_by_job_id=16, pid=90282
scheduler  | [2024-10-10T15:24:33.877-0600] {dagrun.py:854} INFO - Marking run  successful
scheduler  | Dag run in success state
scheduler  | Dag run start:2024-10-10 21:23:57.783848+00:00 end:2024-10-10 21:24:33.878082+00:00
scheduler  | [2024-10-10T15:24:33.878-0600] {dagrun.py:905} INFO - DagRun Finished: dag_id=tutorial_taskflow_api, execution_date=2024-10-10 21:23:55.317397+00:00, run_id=manual__2024-10-10T21:23:55.317397+00:00, run_start_date=2024-10-10 21:23:57.783848+00:00, run_end_date=2024-10-10 21:24:33.878082+00:00, run_duration=36.094234, state=success, external_trigger=True, run_type=manual, data_interval_start=2024-10-10 21:23:55.317397+00:00, data_interval_end=2024-10-10 21:23:55.317397+00:00, dag_hash=38e56a72b18ab9d01e9f2f0cd6695560
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:33 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform/logs/1?full_content=false HTTP/1.1" 200 3950 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:35 -0600] "GET /api/v1/dags/tutorial_taskflow_api/dagRuns/manual__2024-10-10T21:23:55.317397+00:00/taskInstances/transform HTTP/1.1" 200 1036 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
webserver  | 127.0.0.1 - - [10/Oct/2024:15:24:35 -0600] "GET /object/grid_data?dag_id=tutorial_taskflow_api&num_runs=25 HTTP/1.1" 200 4535 "http://localhost:8080/dags/tutorial_taskflow_api/grid?dag_run_id=manual__2024-10-10T21%3A23%3A55.317397%2B00%3A00&task_id=transform&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36"
scheduler  | [2024-10-10T15:24:37.819-0600] {workday.py:41} WARNING - Could not import pandas. Holidays will not be considered.
triggerer  | [2024-10-10T15:24:44.938-0600] {triggerer_job_runner.py:510} INFO - 0 triggers currently running
triggerer  | [2024-10-10T15:25:44.999-0600] {triggerer_job_runner.py:510} INFO - 0 triggers currently running
triggerer  | [2024-10-10T15:26:45.066-0600] {triggerer_job_runner.py:510} INFO - 0 triggers currently running

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Answered by rcheatham-q

Oct 15, 2024

By looking into the issue in the gunicorn repository I found this issue with a workaround. I created a file in the plugins directory with the workaround code and can confirm that it did allow for viewing of logs while the task is running, however this time with a different error. The error printed in the logs did not seem to affect any behavior in the UI, so I consider it a functional workaround.

View full answer

potiuk · 2024-10-13T21:46:38Z

potiuk
Oct 13, 2024
Collaborator

Yes you have some problems with your Mac. This is quite clear because this is a "system error" - and likely details have some more information about it. But you have not shown the details - they are available when you click it.

There are a number of reasons why it can happen - memory issues for example, or permission issues, or maybe your python is using wrong architecture (this happened before by a number of users who migrated from Intel to M1 Mac OS using backup/restor) but only you can tell it by looking at logs.

Converted into discussion - if you need help, you should dig deeper and maybe the logs will contain more information.

0 replies

potiuk · 2024-10-13T21:49:02Z

potiuk
Oct 13, 2024
Collaborator

BTW. SIGSEGV is often a problem that occures when some of libraries you use uses multithreading - and this is a Python, not Airflow issues. search for SIGSEGV in historical issues and you will find a number of workarounds that you can implement - depending which library is failing Python (again - this is Python problem not Airflow and we cannot do anything about)

Relevant issues can be found by following this link https://github.com/apache/airflow/issues?q=is%3Aissue+SIGSEGV+is%3Aclosed

0 replies

rcheatham-q · 2024-10-15T15:42:58Z

rcheatham-q
Oct 15, 2024
Author

Thank you for sharing this information. By following your advice, I was led to this PR and attempted the debugging steps listed, which revealed the crash to be in urllib/request.py in the proxy_bypass_macosx_sysconf method. Searching that error led me to this issue which establishes this error as resulting from improper forking behavior on OSX systems. As the crash occurs in the webserver and based on the logs, I believe it is the gunicorn package that is not forking properly. I concur that this is not an Airflow issue, so I'll close this discussion.

0 replies

rcheatham-q · 2024-10-15T15:59:14Z

rcheatham-q
Oct 15, 2024
Author

By looking into the issue in the gunicorn repository I found this issue with a workaround. I created a file in the plugins directory with the workaround code and can confirm that it did allow for viewing of logs while the task is running, however this time with a different error. The error printed in the logs did not seem to affect any behavior in the UI, so I consider it a functional workaround.

1 reply

potiuk Oct 15, 2024
Collaborator

Also you can searc for NO_PROXY env variable if the reason is with proxy #24463

Fyime · 2024-10-25T08:31:43Z

Fyime
Oct 25, 2024

run airflow with sys env "OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES"

1 reply

rcheatham-q Nov 6, 2024
Author

This did not work. Setting NO_PROXY="*" works, albeit it logs an error every time logs are fetched

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot view logs for running task in Airflow standalone #42984

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cannot view logs for running task in Airflow standalone #42984

rcheatham-q Oct 10, 2024

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Replies: 5 comments · 2 replies

potiuk Oct 13, 2024 Collaborator

potiuk Oct 13, 2024 Collaborator

rcheatham-q Oct 15, 2024 Author

rcheatham-q Oct 15, 2024 Author

potiuk Oct 15, 2024 Collaborator

Fyime Oct 25, 2024

rcheatham-q Nov 6, 2024 Author

rcheatham-q
Oct 10, 2024

Replies: 5 comments 2 replies

potiuk
Oct 13, 2024
Collaborator

potiuk
Oct 13, 2024
Collaborator

rcheatham-q
Oct 15, 2024
Author

rcheatham-q
Oct 15, 2024
Author

potiuk Oct 15, 2024
Collaborator

Fyime
Oct 25, 2024

rcheatham-q Nov 6, 2024
Author