Python's subprocess.Popen(... stdout=PIPE, stderr=PIPE ...) hangs in certain situations

22 May 2017

After reading this post, I couldn't believe that it was gonna be true. So, I setup my little experiment.

I have yes.py:

import sys for i in xrange(65537): sys.stdout.write('a') sys.stderr.write('b') sys.stdout.flush() sys.stderr.flush()

Then, I have main.py:

import subprocess proc = subprocess.Popen('python yes.py', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) count = 0 for chunk in iter(lambda: proc.stderr.read(100), b''): count += 1 print count print 'Done'

Running python main.py hangs.

Now it does NOT hang if one of the below is true:

  • I change 65537 to 65536
  • I don't write to sys.stdout.

To be fair, the doc does warn about the deadlock.

But this is certainly unexpected from a high-level language like Python. The mentioned blog post was written on 2008 (9 years ago), and Python hasn't fixed it yet.


Here's one workaround:

import subprocess proc = subprocess.Popen( 'python yes.py', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) count = 0 for chunk in iter(lambda: proc.stdout.read(100), b''): count += len(chunk) more_stdout, stderr = proc.communicate() print 'Done'

I still can't get over the fact that they don't solve this on the Python level.