FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory #6527

D4V3M0NK · 2024-11-11T18:28:45Z

D4V3M0NK
Nov 11, 2024

Good morning all! I've spent the last couple of hours reading through a number of articles regarding MC2 running into memory issues since I'm now experiencing the same on my server. Nothing appears to have changed for our system over the past few weeks, the number of remote systems has remained stable and I've not upgraded the VM MC2 is running on, nor have I updated MC2 to the latest version.

The host is an Ubuntu 20 server AWS instance (2 vCPUs running with 2.0GB RAM on an AMD).
I'm running MC2 1.1.30 (just updated to 1.1.33 with no change) using node 20.15.0, about 40 remote systems, using the standard database format (ie, NeDB I believe, not MySQL or PostgresQL)
Disk space is running at 32% utilization.
There is an nginx proxy server between the server and the remotes.

config.json isn't anything to write home about (and the updates as of 24-11-11 aren't making any difference) :

{
  "$schema": "http://info.meshcentral.com/downloads/meshcentral-config-schema.json",
  "settings": {
    "cert": "mc2.domain.com",
    "WANonly": true,
    "trustedproxy": ["10.100.30.0/24","10.100.40.0/24"],
    "cookieIpCheck": false,
    "plugins": { "enabled": true },
    "log": "agent",
    "_agentPong": 20,

   "_addedForDebugging": "24-11-11",
    "AgentsInRAM": false,
    "AgentUpdateBlockSize": 2048,
    "agentUpdateSystem": 1,
    "noAgentUpdate": 1,
    "WsCompression": false,
    "AgentWsCompression": false


  },
  "domains": {
    "": {
      "title": "MC2",
      "title2": "Remote Support Services",
      "passwordRequirements": {
              "force2factor": true,
              "min": 12,
              "upper": 1,
              "lower": 1,
              "numeric": 1,
              "nonalpha": 1,
              "oldPasswordBan": 24
      },
      "autocomplete": false,
      "orphanAgentUser" : "[email protected]",
      "certUrl": "https://mc2.domain.com",
      "trustedCert": true,
      "_certUrl": "file:///home/ubuntu/MC2/meshcentral-data/myexternalcert.crt",
      "_ignoreAgentHashCheck": true
    }
  }
}

The server is now consistently bailing out on me:

Installing modules [ '[email protected]' ]
Installing modules [ '[email protected]' ]
MeshCentral HTTP redirection server running on port 80.
MeshCentral v1.1.30, WAN mode.
MeshCentral Intel(R) AMT server running on mc2.domain.com:4433.
MeshCentral HTTPS server running on mc2.domain.com:443.
Loaded web certificate from "https://mc2.domain.com", host: "mc2.domain.com"
  SHA384 cert hash: 75e3ec9678bc91f61b5a9b69dd8044e952802c7724e5be62077b893a88add11e4b2bed08e9f1087161c25290532dce54
  SHA384 key hash: d9fbc866033ffc98d3d310b499d249180c672a79b1921ca191ef07bcfc80452b9ff65f33d7877af15431e86c5adeb823
ERR:
<--- Last few GCs --->

[292432:0x5e387c0]    49681 ms: Mark-Compact (reduce) 960.9 (980.4) -> 960.4 (980.9) MB, 1015.36 / 0.01 ms  (+ 632.5 ms in 59 steps since start of marking, biggest step 21.4 ms, walltime since start of marking 1867 ms) (average mu = 0.470, current mu = 0.[292432:0x5e387c0]    51531 ms: Mark-Compact (reduce) 961.5 (980.9) -> 961.0 (981.6) MB, 1826.25 / 0.00 ms  (average mu = 0.286, current mu = 0.013) allocation failure; scavenge might not succeed
ERR:
<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xb80c78 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 2: 0xeee1b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 3: 0xeee497 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 4: 0x1100035  [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 5: 0x1117eb8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 6: 0x10edfd1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 7: 0x10ef165 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 8: 0x10cc7b6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 9: 0x15285e6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
10: 0x1961ef6  [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
Error: Command failed: /home/ubuntu/.nvm/versions/node/v20.15.0/bin/node /home/ubuntu/mq9/node_modules/meshcentral --launch 292409

<--- Last few GCs --->

[292432:0x5e387c0]    49681 ms: Mark-Compact (reduce) 960.9 (980.4) -> 960.4 (980.9) MB, 1015.36 / 0.01 ms  (+ 632.5 ms in 59 steps since start of marking, biggest step 21.4 ms, walltime since start of marking 1867 ms) (average mu = 0.470, current mu = 0.[292432:0x5e387c0]    51531 ms: Mark-Compact (reduce) 961.5 (980.9) -> 961.0 (981.6) MB, 1826.25 / 0.00 ms  (average mu = 0.286, current mu = 0.013) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xb80c78 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 2: 0xeee1b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 3: 0xeee497 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 4: 0x1100035  [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 5: 0x1117eb8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 6: 0x10edfd1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 7: 0x10ef165 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 8: 0x10cc7b6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
 9: 0x15285e6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]
10: 0x1961ef6  [/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node]

    at genericNodeError (node:internal/errors:984:15)
    at wrappedFn (node:internal/errors:538:14)
    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:531:35)
    at maybeClose (node:internal/child_process:1105:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
  code: null,
  killed: false,
  signal: 'SIGABRT',
  cmd: '/home/ubuntu/.nvm/versions/node/v20.15.0/bin/node /home/ubuntu/mq9/node_modules/meshcentral --launch 292409'
}
ERROR: MeshCentral failed with critical error, check mesherrors.txt. Restarting in 5 seconds...
Installing modules [ '[email protected]' ]
MeshCentral HTTP redirection server running on port 80.
MeshCentral v1.1.30, WAN mode.
MeshCentral Intel(R) AMT server running on mc2.domain.com:4433.
MeshCentral HTTPS server running on mc2.domain.com:443.
Loaded web certificate from "https://mc2.domain.com", host: "mc2.domain.com"
  SHA384 cert hash: 75e3ec9678bc91f61b5a9b69dd8044e952802c7724e5be62077b893a88add11e4b2bed08e9f1087161c25290532dce54
  SHA384 key hash: d9fbc866033ffc98d3d310b499d249180c672a79b1921ca191ef07bcfc80452b9ff65f33d7877af15431e86c5adeb823

I appreciate that there's an update, but I'd like to understand if there was a memory leak in the version that I am using, is that fixed in the newer version? Is there any more that I can do to work out what's going on?

D4V3M0NK · 2024-11-11T21:28:33Z

D4V3M0NK
Nov 11, 2024
Author

So ... the answer was a corrupt database. I backed up the database files, then removed them, restarted the service ... everything came up without issue.

4 replies

si458 Nov 11, 2024
Collaborator

This is another reason we are looking to move away from nedb, as the current nedb module we use doesn't work with node 23!
I would try moving to sqlite and see how u get on :)

D4V3M0NK Nov 12, 2024
Author

Hey Simon, good to hear from you. Question for you, if I may... trashing the NeDB files and then restarting the service, all the remotes checked in and grouped themselves together, albeit into random group names - but that's OK, they did the grouping. A simple rename of those groups got me back into working mode. I take it this was because the "orphanAgentUser" : "[email protected]" config is set. Makes perfect sense.

Out of interest, I decided to adjust the config to use SQLite3 as opposed to NeDB, and restarted the service again. I created the admin account (as one would expect) and after logging in, I see remotes checking in again ... but (oddly) only one group of remotes appeared to have "made it" - the others were nowhere to be seen.

I'm assuming that the config is global, not worrying about the backend database that's being used: so it leads me to ask - why do you think that only one group of those remotes were able to check in with the new SQLite3 backend, whereas the default NeDB appeared to have responded to all of them? As I'm sure you can imagine, this causes some concern when we plan a migration from NeDB to SQLite3 - if they all appeared, great! But that wasn't the case... thoughts?

D4V3M0NK Nov 12, 2024
Author

🫣 Timing ... give it enough time, and they do all check in ... I'm so impatient. Sorry!

si458 Nov 12, 2024
Collaborator

@D4V3M0NK hey no worries!

I was going to say u can use the migrate feature to move from nedb to any db

Run meshcentral with nedb and no config for db set
Stop meshcentral
Set db settings in config.json (this case set sqlite: true)
Run node node_modules/meshcentral --nedbtodb
Process should finish itself if not stop process
Start meshcentral again

Should now Run on ur set database :)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory #6527

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory #6527

D4V3M0NK Nov 11, 2024

Replies: 1 comment · 4 replies

D4V3M0NK Nov 11, 2024 Author

si458 Nov 11, 2024 Collaborator

D4V3M0NK Nov 12, 2024 Author

D4V3M0NK Nov 12, 2024 Author

si458 Nov 12, 2024 Collaborator

D4V3M0NK
Nov 11, 2024

Replies: 1 comment 4 replies

D4V3M0NK
Nov 11, 2024
Author

si458 Nov 11, 2024
Collaborator

D4V3M0NK Nov 12, 2024
Author

D4V3M0NK Nov 12, 2024
Author

si458 Nov 12, 2024
Collaborator