Python:subprocess.call，stdout to file，stderr to file，在屏幕上实时显示stderr

(Note that there are some issues using for line in proc.stderr:—basically, if what you're reading turns out not to be line-buffered for any reason, you can sit around waiting for a newline even though there's actually half a line worth of data to process. You can read chunks at a time with, say, read(128), or even read(1), to get the data more smoothly if necessary. If you need to actually get every byte as soon as it arrives, and can't afford the cost of read(1), you'll need to put the pipe in non-blocking mode and read asynchronously.)

但如果你在 Unix 上，使用 tee 命令可能会更简单.

But if you're on Unix, it might be simpler to use the tee command to do it for you.

对于一个快速而肮脏的解决方案，您可以使用外壳通过它进行管道传输.像这样的:

For a quick&dirty solution, you can use the shell to pipe through it. Something like this:

subprocess.call('path_to_tool -option1 option2 2|tee log_file 1>2', shell=True,
                stdout=file_out)

但我不想调试 shell 管道；让我们在 Python 中完成，如 在文档中:

But I don't want to debug shell piping; let's do it in Python, as shown in the docs:

tool = subprocess.Popen(['path_to_tool', '-option1', 'option2'],
                        stdout=file_out, stderr=subprocess.PIPE)
tee = subprocess.Popen(['tee', 'log_file'], stdin=tool.stderr)
tool.stderr.close()
tee.communicate()

<小时>

最后，PyPI 上的子进程和/或 shell 周围有十几个或更多更高级别的包装器——sh、shell、shell_command、shellout、iterpipes、sarge、cmd_utils、commandwrapper 等.对于shell"、subprocess"、process"、command line"等，找到一个你喜欢的，让问题变得微不足道.

Finally, there are a dozen or more higher-level wrappers around subprocesses and/or the shell on PyPI—sh, shell, shell_command, shellout, iterpipes, sarge, cmd_utils, commandwrapper, etc. Search for "shell", "subprocess", "process", "command line", etc. and find one you like that makes the problem trivial.

如果您需要同时收集 stderr 和 stdout 怎么办?

What if you need to gather both stderr and stdout?

正如 Sven Marnach 在评论中所建议的那样，简单的方法就是将一个重定向到另一个.只需像这样更改 Popen 参数:

The easy way to do it is to just redirect one to the other, as Sven Marnach suggests in a comment. Just change the Popen parameters like this:

tool = subprocess.Popen(['path_to_tool', '-option1', 'option2'],
                        stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

然后在您使用 tool.stderr 的任何地方，都使用 tool.stdout 代替——例如，对于最后一个示例:

And then everywhere you used tool.stderr, use tool.stdout instead—e.g., for the last example:

tee = subprocess.Popen(['tee', 'log_file'], stdin=tool.stdout)
tool.stdout.close()
tee.communicate()

但这有一些权衡.最明显的是，将两个流混合在一起意味着您不能将 stdout 记录到 file_out 并将 stderr 记录到 log_file，或者将 stdout 复制到您的 stdout 并将 stderr 复制到您的 stderr.但这也意味着排序可能是不确定的——如果子进程总是在向 stdout 写入任何内容之前向 stderr 写入两行，那么一旦混合流，您最终可能会在这两行之间得到一堆 stdout.这意味着它们必须共享 stdout 的缓冲模式，所以如果您依赖 linux/glibc 保证 stderr 是行缓冲的事实(除非子进程显式更改它)，那可能不再正确.

But this has some tradeoffs. Most obviously, mixing the two streams together means you can't log stdout to file_out and stderr to log_file, or copy stdout to your stdout and stderr to your stderr. But it also means the ordering can be non-deterministic—if the subprocess always writes two lines to stderr before writing anything to stdout, you might end up getting a bunch of stdout between those two lines once you mix the streams. And it means they have to share stdout's buffering mode, so if you were relying on the fact that linux/glibc guarantees stderr to be line-buffered (unless the subprocess explicitly changes it), that may no longer be true.

如果您需要分别处理这两个过程，则会变得更加困难.之前，我说过，只要您只有一根管道并且可以同步维护它，就可以轻松地在运行中维护管道.如果你有两个管道，那显然不再正确.想象一下，您正在等待 tool.stdout.read()，而新数据来自 tool.stderr.如果数据过多，可能会导致管道溢出和子进程阻塞.但即使没有发生这种情况，您显然也无法读取和记录 stderr 数据，直到从 stdout 中输入内容.

If you need to handle the two processes separately, it gets more difficult. Earlier, I said that servicing the pipe on the fly is easy as long as you only have one pipe and can service it synchronously. If you have two pipes, that's obviously no longer true. Imagine you're waiting on tool.stdout.read(), and new data comes in from tool.stderr. If there's too much data, it can cause the pipe to overflow and the subprocess to block. But even if that doesn't happen, you obviously won't be able to read and log the stderr data until something comes in from stdout.

如果您使用 pipe-through-tee 解决方案，则可以避免最初的问题……但只能通过创建一个同样糟糕的新项目来解决.您有两个 tee 实例，当您在一个实例上调用 communicate 时，另一个实例一直在等待.

If you use the pipe-through-tee solution, that avoids the initial problem… but only by creating a new project that's just as bad. You have two tee instances, and while you're calling communicate on one, the other one is sitting around waiting forever.

因此，无论哪种方式，您都需要某种异步机制.您可以使用线程、select 反应器、gevent 之类的东西来做到这一点.

So, either way, you need some kind of asynchronous mechanism. You can do this is with threads, a select reactor, something like gevent, etc.

这是一个快速而肮脏的例子:

Here's a quick and dirty example:

proc = subprocess.Popen(['path_to_tool', '-option1', 'option2'],
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
def tee_pipe(pipe, f1, f2):
    for line in pipe:
        f1.write(line)
        f2.write(line)
t1 = threading.Thread(target=tee_pipe, args=(proc.stdout, file_out, sys.stdout))
t2 = threading.Thread(target=tee_pipe, args=(proc.stderr, log_file, sys.stderr))
t3 = threading.Thread(proc.wait)
t1.start(); t2.start(); t3.start()
t1.join(); t2.join(); t3.join()

但是，在某些极端情况下这不起作用.(问题是 SIGCHLD 和 SIGPIPE/EPIPE/EOF 到达的顺序.我认为这些都不会影响我们，因为我们没有发送任何输入……但不要不假思索就相信我通过和/或测试.)subprocess.communicate<3.3+ 中的/code> 函数可以正确处理所有繁琐的细节.但是您可能会发现使用您可以在 PyPI 和 ActiveState 上找到的异步子流程包装器实现之一，甚至是来自像 Twisted 这样成熟的异步框架的子流程的东西会简单得多.