Attempt to fix static image export hanging by flushing after printing JSON #149
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a potentially "wishful thinking" attempt to fix (a subset of?) the "writing image stalls" problem with a dozen or so open issues. Unfortunately I haven't been able to test whether it fixes the issue, because I do not have the tooling at hand to build the library (assuming this indeed involves the 50 GB chromium compile). If there's a low-profile way to compile this patch I'd love to test it locally on my "known bad" setup.
My situation:
fig.write_image()
will hang on my machine, and it will always succeed on coworker's machine. (So it's not the "sometimes hangs on the same system" case some people see.)self._proc.stdout.readline()
line in_ensure_kaleido()
, see e.g. traceback in Python 3.6 container image: First fig.to_image call is very slow (~ 5 minutes) #36 (comment)Buffering?
The only thing I can suspect here is buffering. If for some reason the process is not line buffered (and kaleido runs the subprocess with a binary stdout stream, so I'm not even sure line buffering is a thing there), the
readline()
call might be stuck, waiting for the stdout stream to flush, which for some reason might not happen on some systems.Why it might not be buffering:
universal_newlines=True
doesn't make the problem go away. Manually callingself._proc.stdout.flush()
doesn't make the problem go away. Accessingself._proc.stdout.raw
which might be less buffered doesn't make the problem go away. Callingself._proc.stdout.read(70)
or even.read(50)
(since we expect 70 characters for the JSON) halts all the same.subprocess.Popen
setup) doesn't stall.bufsize=50
(say) which is shorter than the 70 bytes of the expected JSON doesn't make the problem go away (it still stalls).Why it might be buffering:
readline()
call freezes in the subprocess call (JSON is stuck in the buffer).std::endl
which would flush.Unfortunately I couldn't find any exact information on what kind of buffering options are available on a given OS, and what defaults are. The only thing I could figure out is that both in my native Windows and in my WSL the
io.DEFAULT_BUFFER_SIZE
is 8192, so if the output is not line buffered then it would take a lot of output to get it flushed (assuming I even understand the mechanics here correctly).