ode.js请求随机开始挂起，直到服务器重启才会清除(ode.js requests randomly begin to hang and won't clear until server restart)

ode.js请求随机开始挂起，直到服务器重启才会清除(ode.js requests randomly begin to hang and won't clear until server restart)

我在我们的网络应用程序上遇到了一个非常奇怪且看似随机的问题，我似乎无法成功调试。 它可以在10分钟到6小时之间运行正常，然后突然间没有远程请求进出服务器，它们只是挂起（这包括常规的http和Web套接字请求）。 奇怪的是，定期访问该站点仍然有效，直到达到操作系统文件描述符限制，然后http完全崩溃所有停顿的连接。

没有错误，但是当问题开始时会抛出以下错误（我认为这是正在发生的任何事情的副作用而不是原因）。

TypeError: Cannot read property '0' of null at null.<anonymous> (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/collection.js:504:22) at args.(anonymous function) (/app/node_modules/strong-agent/lib/proxy.js:85:18) at g (events.js:175:14) at (events.js:98:17) at Base.__executeAllServerSpecificErrorCallbacks (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/base.js:15:29) at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:27:22 at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:70:11 at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:52:28 at _callback (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/db.js:670:5) at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/auth/mongodb_cr.js:47:1

我已尝试提高文件描述符限制和全局代理maxSockets，但不会影响此行为。 当这种情况发生时，没有流量涌入，并且在高峰时段和非高峰时段经常发生。 CPU使用率始终保持在5％以下，并且在崩溃前或崩溃期间没有任何可察觉的变化。 服务器也永远不会低于1GB的可用内存。

堆栈：SmartOS云服务器（Joyent），Express，Socket.io，MongoDB和Redis。

我已经调试了好几天了，完全没有想法去哪看。 希望有人在SO上遇到过类似的东西或对可以尝试或测试的东西有不同的想法。

I've been running into a really odd and seemingly random issue on our web app that I just can't seem to successfully debug. It runs fine for anywhere from 10 minutes to 6 hours, and then all of a sudden no remote requests to or from the server can be made, they just hang (this includes regular http and web socket requests). The odd thing is that going to the site regularly still works, until the OS file descriptor limit is reached and then http completely crashes with all of the stalled connecti.

There are no errors, though the following error is thrown when the issue begins (I assume this is a side-effect of whatever is going on rather than the cause).

TypeError: Cannot read property '0' of null at null.<anonymous> (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/collection.js:504:22) at args.(anonymous function) (/app/node_modules/strong-agent/lib/proxy.js:85:18) at g (events.js:175:14) at (events.js:98:17) at Base.__executeAllServerSpecificErrorCallbacks (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/base.js:15:29) at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:27:22 at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:70:11 at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:52:28 at _callback (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/db.js:670:5) at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/auth/mongodb_cr.js:47:1

I've tried raising the file descriptor limits and the global agent maxSockets with no affect on this behavior. There's no influx of traffic when this happens, and it happens equally as often during peak and off-peak times. The CPU usage cistently stays below 5% and doesn't have any perceptible changes leading up to or during the crash. The server also never drops below 1GB of free memory.

The stack: SmartOS cloud server (Joyent), Express, Socket.io, MongoDB and Redis.

I've been debugging this for several days and have completely run out of ideas where to look. Hoping someone on SO has run into something similar or has different ideas of what can be tried or tested.

最满意答案

经过无数小时的调试和更多的调试，我终于到了罪魁祸首。 几个不同的mongojs回调中出现了错误，这些回调似乎已经冒泡并阻止关闭连接。 随着时间的推移，这已达到临界点，连接开始挂起，直到达到文件描述符限制。

错误结果出现在ow.js节点模块中（已被放弃）。 如果有人在使用ow.js遇到这个问题，我会分叉并修补bug。 你可以在这里到提交： https ： ///goldfire/now/commit/b5bd54f8950602f752a710c606be6754b759cab2 。

我发现此错误的方法是将错误侦听器附加到数据库对象：

var db = require('mongojs').connect('...', ['collection']); ('error', function(err){ cole.log(err.stack); });

After countless hours of debugging and more debugging, I finally found the culprit. An error was being thrown inside of several different mongojs callbacks, which appears to have bubbled up and blocked the connecti from closing. Over time, this got to a tipping point and connecti started hanging until the file descriptor limit was reached.

The error turned out to be in the ow.js node module (which has been abandoned). If there is anyone out there that is running into this issue using ow.js, I forked it and patched the bug. You you can find the commit here: https:///goldfire/now/commit/b5bd54f8950602f752a710c606be6754b759cab2.

The way I found this bug was to attach an error listener to the DB object:

var db = require('mongojs').connect('...', ['collection']); ('error', function(err){ cole.log(err.stack); });