如何在 Node.JS 中限制(或排队)对外部进程的调用?

How to limit (or queue) calls to external processes in Node.JS?(如何在 Node.JS 中限制(或排队)对外部进程的调用?)

问题描述

场景

我有一个 Node.JS 服务(使用 ExpressJS 编写)接受通过 DnD (示例).上传图片后，我会对其进行一些操作:

I have a Node.JS service (written using ExpressJS) that accepts image uploads via DnD (example). After an image is uploaded, I do a few things to it:

1. 从中提取 EXIF 数据
2. 调整大小

目前正在通过 node-imagemagick 模块和我的代码处理这些调用看起来像这样:

These calls are being handled via the node-imagemagick module at the moment and my code looks something like this:

app.post('/upload', function(req, res){
  ... <stuff here> ....

  im.readMetadata('./upload/image.jpg', function(err, meta) {
      // handle EXIF data.
  });

  im.resize(..., function(err, stdout, stderr) {
      // handle resize.
  });
});

问题

正如你们中的一些人已经发现的那样，问题在于，如果我同时上传了足够多的内容，那么这些上传中的每一个都会产生一个身份"调用，然后是一个调整大小操作(来自 Image Magick)，从而有效地杀死服务器加载.

As some of you already spotted, the problem is that if I get enough simultaneous uploads, every single one of those uploads will spawn an 'identity' call then a resize operation (from Image Magick), effectively killing the server under high load.

使用 ab -c 100 -n 100 进行测试会锁定我的小型 512 Linode 开发服务器，因此我必须强制重启.我知道我的测试可能会给服务器带来过多的负载，但我想要一种更强大的方法来处理这些请求，这样我的失败就会比虚拟机完全自杀更优雅.

Just testing with ab -c 100 -n 100 locks my little 512 Linode dev server up such that I have to force a reboot. I understand that my test may just be too much load for the server, but I would like a more robust approach to processing these requests so I have a more graceful failure then total VM suicide.

在 Java 中我 通过创建一个固定线程的 ExecutorService 解决了这个问题，该服务将工作排队并在最多 X 个线程上执行.

In Java I solved this issue by creating a fixed-thread ExecutorService that queues up the work and executes it on at most X number of threads.

在 Node.JS 中，我什至不知道从哪里开始解决这样的问题.我不太了解非线程性质以及如何创建一个异步 JavaScript 函数来排队工作，而另一个......(线程?)处理队列.

In Node.JS, I am not even sure where to start to solve a problem like this. I don't quite have my brain wrapped around the non-threaded nature and how I can create a async JavaScript function that queues up the work while another... (thread?) processes the queue.

任何关于如何考虑或如何解决此问题的指针将不胜感激.

Any pointers on how to think about this or how to approach this would be appreciated.

附录

这与这个关于FFMpeg的问题不同，虽然我想象一下，一旦他的 web 应用程序处于负载状态，这个人就会有同样的问题，因为它归结为同一个问题(并行触发太多同时的本地进程).

This is not the same as this question about FFMpeg, although I imagine that person will have this exact same question as soon as his webapp is under load as it boils down to the same problem (firing off too many simultaneous native processes in parallel).

推荐答案

threads 模块应该正是你所需要的:

The threads module should be just what you need:

https://github.com/robtweed/threads

这篇关于如何在 Node.JS 中限制(或排队)对外部进程的调用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！