Macos python 3.11/3.12 calling (some) subprocesses from executor

[ryandvmartin]

Looking for some guidance on a strange error on Python 3.11/3.12 on MacOS with this executor. I experienced the issue using joblib but managed to reproduce here. The modified example is borrowed from the docs. Calling some programs with a subprocess inside the executor causes a strange python error:

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor

Current thread 0x00000001ec17cc00 (most recent call first):
  <no Python frame>

Going back to Python 3.10 on the same system with the same code, it runs as expected. Code also runs fine on Windows/Linux Python 3.11/3.12.

The example below calls jupyter in a subprocess, but swapping this to python --version or gcc --version works just fine... (Seems to be these python "shims" like jupyter --version or flake8 --version that are failing...)

Reproducing code:

import os
from time import sleep
from loky import get_reusable_executor
from subprocess import run


def call_program(k):
    pid = os.getpid()
    sleep(0.01)
    print(run(["jupyter", "--version"]).check_returncode())
    return pid

executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(call_program, 1)

On MacOS, python 3.11, 3.12 this raises with:

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor

Current thread 0x00000001ec17cc00 (most recent call first):
  <no Python frame>
---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/python/envs/test311/lib/python3.11/site-packages/loky/process_executor.py", line 463, in _process_worker
    r = call_item()
        ^^^^^^^^^^^
  File "/Users/python/envs/test311/lib/python3.11/site-packages/loky/process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<ipython-input-3-56017b8caf15>", line 9, in call_program
  File "/Users/python/envs/test311/lib/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['jupyter', '--version']' returned non-zero exit status 1.
"""

Tracing with python -m trace -t test.py shows the error:

fork_exec.py(26):      for i in open_fds - keep_fds:
fork_exec.py(41):          os.execve(sys.executable, cmd, child_env)
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor

Current thread 0x00000001ec17cc00 (most recent call first):
  <no Python frame>

Environments created with:
mamba create --name test310 python=3.10 loky ipython jupyter
mamba create --name test311 python=3.11 loky ipython jupyter
mamba create --name test311 python=3.12 loky ipython jupyter

Any thoughts or ideas? Am I even in the right place!?

Thanks :)

[ryandvmartin]

Hoping some more triage might help. I installed in dev mode and through some trials noticed that if I add 0 to this list of fds to keep open:

    # Make sure to keep stdout and stderr open for logging purpose
    keep_fds = {*keep_fds, 0, 1, 2}

My error goes away. Perhaps that gives some hints at where the root of the issue might be?