Can't launch cross-namespace DAG with Kubernetes Pod Operator

[imishchuk-tsgs]

Hey, folks.

We have an Airflow 2.10.2 deployed in EKS.
EKS has two namespaces of interest data and ai and Airflow webserver, scheduler and workers by default run in data namespace.
I want to execute some specific DAGs in ai namespace. I've added following configuration to the Airflow

[kubernetes_executor]
...
multi_namespace_mode = True
multi_namespace_mode_namespace_list = data,ai
...

DAG is a placeholder for now

from datetime import datetime

from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

from include.constants import EKSNamespaces, EKSServiceAccounts
from include.utils import get_default_dag_args

DAG_NAME = "ai_test"

with DAG(
    dag_id=DAG_NAME,
    default_args=get_default_dag_args(namespace=EKSNamespaces.AI.value), # "ai"
    schedule=None,
    start_date=datetime(2024, 12, 3),
    tags=["test"],
) as dag:
    KubernetesPodOperator(
        name=DAG_NAME,
        image="busybox:latest",
        arguments=["sh", "-c", "echo hello && sleep 300"],
        task_id=DAG_NAME,
        service_account_name=EKSServiceAccounts.DATA_AIRFLOW_WORKER.value, # "data-airflow-worker"
    )

Service account data-airflow-worker has following permissions in ai namespace

  airflow_serviceaccounts = [
    "data-airflow-migrate-database-job",
    "data-airflow-reconciler",
    "data-airflow-scheduler",
    "data-airflow-webserver",
    "data-airflow-worker",
  ]

resource "kubernetes_role" "ai_airflow" {
  metadata {
    name      = "ai-airflow"
    namespace = kubernetes_namespace.ai.id
  }

  rule {
    api_groups = [""]
    resources = [
      "pods",
      "jobs",
    ]
    verbs = [
      "get",
      "list",
      "create",
      "update",
      "delete",
      "watch",
      "patch",
    ]
  }
}

resource "kubernetes_role_binding" "ai_airflow" {
  for_each = toset(local.airflow_serviceaccounts)
  metadata {
    name      = "ai-${each.key}"
    namespace = kubernetes_namespace.ai.id
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.ai_airflow.metadata[0].name
  }
  subject {
    kind      = "ServiceAccount"
    name      = each.key
    namespace = kubernetes_namespace.data.id
  }
}

When I trigger the DAG, KPO pod is started in data namespace and then fails with following error

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods \"ai-test-kxotpx8z\" is forbidden: error looking up service account ai/data-airflow-worker: serviceaccount \"data-airflow-worker\" not found",
  "reason": "Forbidden",
  "details": {
    "name": "ai-test-kxotpx8z",
    "kind": "pods"
  },
  "code": 403
}

data-airflow-worker service account exists only in data namespace but has necessary permissions to work with ai namespace.
It is unclear for me, why KPO pod is looking for data-airflow-worker in ai namespace and how to make sure that it uses data-airflow-worker in data namespace instead.

Obvious thing to do would be to create a service account in ai namespace, but that feels wrong, as cross-namespace operations should be supported.

Can you, please, suggest, what am I missing?

Thank you.