Calling Python Script With Subprocess.Popen And Flushing The Data

Ok so i've seen dozen of threads like that , but none of them gives a complete answer and everything i tried so far foes not work for me. 1) Script that constantly outputs some dat

Solution 1:

As @Warren Weckesser's comment says, your problem is unrelated to buffering issues.

.readline() in the parent process won't return until it reads a newline or reaches EOF. Your child process doesn't print any newlines at all so your parent process doesn't print anything until the child process ends.

The minimal fix is just to remove comma at the end of print i, in the child script.

This also works:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

p = Popen([sys.executable or 'python',
           '-u', # unbuffer stdout (or make it line-buffered on Python 3)
           '-c',
           """
import time

for i in range(5):
    print(i) # <-- no comma i.e., each number is on its own line
    time.sleep(1)
"""], stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, b''):
    print(int(line)**2)

Example:

 $ python parent.py
 0
 1
 4
 9
 16

The numbers are printed every seconds without waiting for the child process to end.

If you don't want to change the child script then you should use readline() that stops at whitespace instead of a newline character e.g.:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

p = Popen(['python2', 'child.py'], stdout=PIPE, bufsize=0)
for token in generate_tokens(p.stdout):
    print(int(token))

where generate_tokens() yields whitespace-separated tokens:

def generate_tokens(pipe):
    buf = []
    while True:
        b = pipe.read(1) # read one byte
        if not b: # EOF
            pipe.close()
            if buf:
                yield b''.join(buf)
            return
        elif not b.isspace(): # grow token
            buf.append(b)
        elif buf: # full token read
            yield b''.join(buf)
            buf = []

It also prints integers as soon as they are printed by the child.