Re-interpret error in lost worker scenario
Previously when a worker would die unexpectedly
(such as with a sys.exit call)
the nanny's multiprocessing.Queue to that worker would get a strange
TypeError because the process went away.
We now identify and interpret this error message and tell the nanny to
clean up accordingly.
@@ -851,7 +851,12 @@ class WorkerProcess:
Wait for an incoming stop message and then stop the
worker cleanly.
"""
+ try:
msg = child_stop_q.get()
+ except (TypeError, OSError):
+ logger.error("Worker process died unexpectedly")
+ msg = {"op": "stop"}
+ finally:
child_stop_q.close()
assert msg.pop("op") == "stop"
loop.add_callback(do_stop, **msg)
