invoking pipe command with Popen in Python
What is the correct way to invoke a command containing multiple pipes to Popen so that its output can be read? I tried:
Popen(shlex.split("mycmd arg1 | mysecondcmd - | thirdcmd -", stdout=PIPE)")
But I don't believe shlex.split is right here. What's the correct syntax?
You have a few options -- You can pass shell=True :
Popen('command1 | command2 | command3',shell=True)
Or, you can break it up into a bunch of Popen calls hooking their stdout to the next Popen's stdin as demonstrated in the documentation.
Using the sh module, pipes become function composition:
import sh
output = sh.thirdcmd(sh.mysecondcmd(sh.mycmd("arg1")))
If you want to do it with subprocess without shell = True , there is an example in the docs which shows how to write shell pipelines using subprocess.Popen . Note that you are supposed to close the proc.stdout s so that SIGPIPE s can be received properly:
import subprocess
proc1 = subprocess.Popen(shlex.split('mycmd arg1'), stdout = subprocess.PIPE)
proc2 = subprocess.Popen(shlex.split('mysecondcmd'), stdin = proc1.PIPE,
stdout = subprocess.PIPE)
proc3 = subprocess.Popen(shlex.split('thirdcmd'), stdin = proc2.PIPE,
stdout = subprocess.PIPE)
# Allow proc1 to receive a SIGPIPE if proc2 exits.
proc1.stdout.close()
# Allow proc2 to receive a SIGPIPE if proc3 exits.
proc2.stdout.close()
out, err = proc3.communicate()
This might look like a lot more work than using shell = True . The reason why you might want to avoid shell = True is because it can be a security risk (page down to the "Warning" box), especially if you are running a command supplied by a (potentially malicious) user.