Not in sync?

[amdp]

trafficstars

What's going wrong?

"Current process list running is not in sync with saved list. npm differs. Type 'pm2 save' to synchronize." It does not start over at boot and sometimes gives different errors.

The "npm differs" seems a good hint but I don't know what to do as npm is the latest version globally (npm install -g npm)

How could we reproduce this issue?

I don't really know, it was working on a different server, now it just tells me the process list is not in sync. I installed npm, installed my repo and then installed npm2. Same dynamic as the old server.

Maybe installing from root wasn't good? I don't really think so..

Supporting information

--- PM2 report ----------------------------------------------------------------
Date                 : Thu Dec 19 2019 17:35:40 GMT+0100 (Central European Standard Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 4.2.0
node version         : 13.3.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 3min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 4.2.0
node version         : 13.3.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Core(TM) i5 CPU         750  @ 2.67GHz
cpus nb              : 4
freemem              : 13558943744
totalmem             : 16812118016
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 0  │ npm                │ fork     │ 0    │ online    │ 0.1%     │ 45.5mb   │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/root/.pm2/pm2.log last 20 lines:
PM2        | 2019-12-19T17:31:42: PM2 log: BUS socket file      : /root/.pm2/pub.sock
PM2        | 2019-12-19T17:31:42: PM2 log: Application log path : /root/.pm2/logs
PM2        | 2019-12-19T17:31:42: PM2 log: Worker Interval      : 30000
PM2        | 2019-12-19T17:31:42: PM2 log: Process dump file    : /root/.pm2/dump.pm2
PM2        | 2019-12-19T17:31:42: PM2 log: Concurrent actions   : 2
PM2        | 2019-12-19T17:31:42: PM2 log: SIGTERM timeout      : 1600
PM2        | 2019-12-19T17:31:42: PM2 log: ===============================================================================
PM2        | 2019-12-19T17:31:42: PM2 log: App [npm:0] starting in -fork mode-
PM2        | 2019-12-19T17:31:42: PM2 log: App [npm:0] online
PM2        | 2019-12-19T17:31:46: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
PM2        | 2019-12-19T17:31:46: PM2 log: App [npm:0] starting in -fork mode-
PM2        | 2019-12-19T17:31:46: PM2 log: App [npm:0] online
PM2        | 2019-12-19T17:31:49: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
PM2        | 2019-12-19T17:31:49: PM2 log: App [npm:0] starting in -fork mode-
PM2        | 2019-12-19T17:31:49: PM2 log: App [npm:0] online
PM2        | 2019-12-19T17:31:49: PM2 log: Stopping app:npm id:0
PM2        | 2019-12-19T17:31:49: PM2 log: App [npm:0] exited with code [0] via signal [SIGINT]
PM2        | 2019-12-19T17:31:49: PM2 log: pid=26339 msg=process killed
PM2        | 2019-12-19T17:32:27: PM2 log: App [npm:0] starting in -fork mode-
PM2        | 2019-12-19T17:32:27: PM2 log: App [npm:0] online

[owenallenaz]

I've actually had this same issue this week. If I run a ps aux | grep node I don't see any Node processes that aren't showing up in the pm2 list, but it tells me the same warning.

[jasonk]

What it's telling you is that the processes currently running are different from the processes that are listed in the dump file (generally ~/.pm2/dump.pm2). This is why it doesn't restart those applications at boot, because at boot it's going to start whatever is listed in the dump file.

If you want things to restart on boot, you should do what it says and run pm2 save to update the dump file with what is currently running.

If (like me) you are using pm2 for temporary development processes and you don't want it to restart anything on boot then you can get rid of the warning by just deleting the dump file.

[amdp]

Alas, pm2 changes nothing and I need apps to be restarted

[zgtnf]

I have same issue on window system, same warn. Pm2 version is 4.2.3.

[VirtueMe]

I hit this problem today, running pm2 update restarted all processes.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[AnxiousDarkly]

well stalebot is probably one of the stupidest things I have ever heard of, this is still an issue BTW.

[mxmauro]

Lukily @VirtueMe hint fixed my server but seems there is an issue. In my case, happened after a hard reboot but autosave was active.

[sierraa]

I'm running into this issue as well. As a work around I've added this to the rc.local on the host:

PM2_HOME=/opt/my-service /usr/bin/pm2 resurrect
chown -R root:my-team /opt/my-service/

[nsssim]

I hit this problem today, running pm2 update restarted all processes.

This seem to solve the problem , but if I reboot the system ... I have to run this command again ...

[nsssim]

I found it !
make sure you specify the user after the startup option pm2 startup -u nodeuser for me I used root and reboot server, and voila ! pm2 startup -u root

[mxmauro]

Tested @nsssim solution and no longer got the sync issue. Thanks!

[amdp]

This is how I solved.

TLDR: you need these three steps

1. npm install pm2@latest -g (Upgrade to 4.4.0)

2. pm2 set pm2:autodump true

3. Stop all processes, reboot, start them again

LONG VERSION:

the latest updates gave me hints on what to do with messages by pm2. First message:

Be sure to have the latest version by doing `npm install pm2@latest -g` before 
doing this procedure.

so I upgraded to 4.4.0 . Second message I got AFTER the update:

>>>> In-memory PM2 is out-of-date, do:
>>>> $ pm2 update

so I updated all processes, or stopped them rebooted and started them again. Third message:

[PM2][WARN] Current process list running is not in sync with saved list. 
Type 'pm2 save' to synchronize or enable autosync via 
'pm2 set pm2:autodump true'

so I set the autodump as true.

Now everything works. Great update. Thanks everyone.

[noHaiMan]

I found it ! make sure you specify the user after the startup option pm2 startup -u nodeuser for me I used root and reboot server, and voila ! pm2 startup -u root

helpful!

[egekhter]

Since I couldn't reliably get my PM2 processes to restart on reboot, here's what was successful:

crontab -e

@reboot sudo pm2 resurrect

[mxmauro]

Hi @egekhter , run pm2 startup -u username, where username is, of course, the user where PM2 runs. It will show you a command line.

Run that command line with sudo and that should work. Basically creates a service in root that starts the daemon in the target user.

[kadalcj]

Hi,

I currently having this issue on RHEL 8, I am a bit confused because previously I was testing it on CentOS and it was perfectly fine. On CentOS, saved process was running after reboot. But on Red Hat, it was not

[adamreisnz]

In my case, the issue for this error was that the process was not being started after boot due to an error thrown from /etc/rc.local

You can use journalctl --boot to look for potential issues.

[factorycreacom]

I found it ! make sure you specify the user after the startup option pm2 startup -u nodeuser for me I used root and reboot server, and voila ! pm2 startup -u root

Thanks. Helpull

[alionur07]

I hit this problem today, running pm2 update restarted all processes.

thank you it works for my problem.

[crshmk]

if you got this because you saved and deleted, pm2 cleardump will solve the disconnect.

[animatedcreativity]

Like @mxmauro suggested. sudo pm2 startup worked form on Mac OS X. Thanks.

[titoBouzout]

This software is trash. If you have 3 process running, and then you go to the terminal and type "reboot". And then none of the 3 process are running for any nonsense "out of sync" stuff, then it's useless and not acomplishing the goal.

I am looking forward to replace this with a thing that just doesnt trash my processes with bogus reasons, and in the case they do, THEY SHOULD TELL ME, YOU RUNNING OUT OF SYNC BUT IF YOU WANT TO RUN A, B, C THEN DO THIS, not the crap that is showing that its to run pm2 save, THAT WE DO, so what the fuck

[titoBouzout]

Its ridiculous lol, the fucking server is down, because they though that "out of sync" is more important than the server running, what a joke lol, and its still down, people complaining and shit, it WAS UP, WHAT THE FUCK IS IN YOUR MIND TO NOT START SOMETHING THAT WAS UP.

Developers are crazy, you lose your mind you know, you think you are more important than anyone else with your purist shit, Im getting tired of such careless "engineering".

[titoBouzout]

Bottom line, do you use this use something else, what? i dont know right now the server is down, so cant tell you. the thing is that you run pm2 start whatever1 pm2 start whatever2 pm2 save

then maybe someone else run some pm2 start whateverelse and because they dont run "pm2 save", they have the power that, whatever you typed first doesnt give a shit, so the whole thing go down, because maybe someone forgot to run a pm2 save But hold, because Im sure we ran "pm2 save", so well.....

if a software its capable to just not run your stuff because of some "out of sync because I am confused and nothing should run", then the whole thing is not fullfiling the purpose and its just noise and trash, things people shoulnt be using

[titoBouzout]

Just drop it or say it's deprecated @Unitech, it doesn't do what it should do, for whatever reason and yeah keep them for yourself. you seem to want to deal with an out of sync stuff, well guess what, I do not want. I just want to run processes.

[animatedcreativity]

@titoBouzout I am surprised what are you angry about. This is a free open source script. Fork it and feel free to modify it if you do not like it.

Is anyone forcing you to use it? I love this script, many people do. It is so easy to handle the processes if you know how to do it. You can create a JSON file with all the processes and run it like pm2 start processes.json. All processes will start in one go.

Never had a problem with pm2 startup & pm2 save. If you see a problem, please open a bug/feature request.

[titoBouzout]

@animatedcreativity you are absolutely right, but still piss me off, this problem comes up every once in a while at the wrong times and having to deal with this is annoying.

[SgtErnestBilko]

This problem still persists. I have installed on a mac mini running Monterey. I used 'pm2 startup -u nodeuser' and pm2 produced a .plit file in launchd. but I still get "Current process list is not synchronized with saved list. App node-red differs. Type 'pm2 save' to synchronize." I would be very grateful if anyone could help me troubleshoot this please.

Thank you

--- PM2 report ----------------------------------------------------------------
Date                 : Thu Jan 05 2023 17:12:15 GMT+0000 (Western European Standard Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.2.2
node version         : 18.12.1
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : nodeuser
uid                  : 501
gid                  : 20
uptime               : 0min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.2.2
node version         : 18.12.1
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/bin/pm2,report
argv0                : node
user                 : nodeuser
uid                  : 501
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i7-3615QM CPU @ 2.30GHz
cpus nb              : 8
freemem              : 14897061888
totalmem             : 17179869184
home                 : /Users/nodeuser
===============================================================================
--- PM2 list -----------------------------------------------
┌─────┬───────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name      │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
└─────┴───────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
[PM2][WARN] Current process list is not synchronized with saved list. App node-red differs. Type 'pm2 save' to synchronize.
===============================================================================
--- Daemon logs --------------------------------------------
/Users/nodeuser/.pm2/pm2.log last 20 lines:
PM2        | 2023-01-05T17:10:40: PM2 log: pid=350 msg=process killed
PM2        | 2023-01-05T17:11:46: PM2 log: pm2 has been killed by signal, dumping process list before exit...
PM2        | 2023-01-05T17:11:46: PM2 log: Deleting process 0
PM2        | 2023-01-05T17:11:46: PM2 log: Exited peacefully
PM2        | 2023-01-05T17:12:15: PM2 log: ===============================================================================
PM2        | 2023-01-05T17:12:15: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
PM2        | 2023-01-05T17:12:15: PM2 log: Time                 : Thu Jan 05 2023 17:12:15 GMT+0000 (Western European Standard Time)
PM2        | 2023-01-05T17:12:15: PM2 log: PM2 version          : 5.2.2
PM2        | 2023-01-05T17:12:15: PM2 log: Node.js version      : 18.12.1
PM2        | 2023-01-05T17:12:15: PM2 log: Current arch         : x64
PM2        | 2023-01-05T17:12:15: PM2 log: PM2 home             : /Users/nodeuser/.pm2
PM2        | 2023-01-05T17:12:15: PM2 log: PM2 PID file         : /Users/nodeuser/.pm2/pm2.pid
PM2        | 2023-01-05T17:12:15: PM2 log: RPC socket file      : /Users/nodeuser/.pm2/rpc.sock
PM2        | 2023-01-05T17:12:15: PM2 log: BUS socket file      : /Users/nodeuser/.pm2/pub.sock
PM2        | 2023-01-05T17:12:15: PM2 log: Application log path : /Users/nodeuser/.pm2/logs
PM2        | 2023-01-05T17:12:15: PM2 log: Worker Interval      : 30000
PM2        | 2023-01-05T17:12:15: PM2 log: Process dump file    : /Users/nodeuser/.pm2/dump.pm2
PM2        | 2023-01-05T17:12:15: PM2 log: Concurrent actions   : 2
PM2        | 2023-01-05T17:12:15: PM2 log: SIGTERM timeout      : 1600
PM2        | 2023-01-05T17:12:15: PM2 log: ===============================================================================

[animatedcreativity]

@SgtErnestBilko On reboot if it says: "Current process list is not synchronised with saved list."

Please try this command: pm2 resurrect It should work.

[SgtErnestBilko]

Thank you, very much, I appreciate your input.

Yes, 'pm2 resurrect' works as does 'pm2 update' but the point of installing pm2 is that I want node-red to restart when the mac reboots. I don't want to be typing 'pm2 resurrect' at the command line, in that case there is no point in having pm2.

The plist contains "/usr/local/lib/node_modules/pm2/bin/pm2 reserrect" so I'm not sure why it doesn't 'resurrect'.

[SgtErnestBilko]

I don't expect anyone to 'solve' my problem, but is there anyone who could 'point me in the right direction' / help me troubleshoot this problem ? Please.

[SgtErnestBilko]

OK.... it works now ! I'm not sure if I fixed it but......the plist file (I'm on a Mac) log section contained the lines "/tmp/com.PM2.err" and "tmp/com.PM2.out". In order to be able to load those files into console I added ".log" to the end of both strings and rebooted. I opened console and it confirmed that node-red was running as did pm2 list.

[honato11]

Many years ago pm2 always restarted stopped processes after a system restart/reboot. That's how a god daemon process manager should work, no? You start some processes, save the config state with pm2 save and then expect that this state is restored after a system reboot and not use the current state. Imagine an admin stopped a process and forgot to start or deliberately did not start it again (coz a non-headless version had to run for a while) and pm2 restores this current process state before a reboot rather than the state described in the config file created by pm2 save.

Many years ago this worked very well (pm2 adhered to what was configured in the pm2 save file) but then a change was made that the current process state is restored after a reboot.

Exploring this autodump option now and hope it works (this should be false per default IMHO. It just would make much more sense).

pm2 set pm2:autodump false

May the force be with me: https://github.com/Unitech/pm2/issues/4763

[honato11]

Many years ago pm2 always restarted stopped processes after a system restart/reboot. That's how a god daemon process manager should work, no? You start some processes, save the config state with pm2 save and then expect that this state is restored after a system reboot and not use the current state. Imagine an admin stopped a process and forgot to start or deliberately did not start it again (coz a non-headless version had to run for a while) and pm2 restores this current process state before a reboot rather than the state described in the config file created by pm2 save.

Many years ago this worked very well (pm2 adhered to what was configured in the pm2 save file) but then a change was made that the current process state is restored after a reboot.

Exploring this autodump option now and hope it works (this should be false per default IMHO. It just would make much more sense).

pm2 set pm2:autodump false

May the force be with me: #4763

Tried the following: 0. Started all node.js processes via pm2 start xyz. Then executed: pm2 save.

1. Stopped one node.js process (status=stopped). Then NOT executed: pm2 save.
2. Rebooted
3. node.js process which I stopped in step 1. was not restarted. It remains in the stopped status.

Tried the same with: pm2 set pm2:autodump false pm2 set pm2:autodump true

pm2 resurrect (it's even mentioned in the .plist)

...
    <key>KeepAlive</key>
    <true/>
          <key>ProgramArguments</key>
          <array>
                  <string>/bin/sh</string>
                  <string>-c</string>
                  <string>/usr/local/lib/node_modules/pm2/bin/pm2 resurrect</string>
          </array>
          <key>RunAtLoad</key>
...

Source: /Users/ourusername/Library/LaunchAgents/pm2.ourusername.plist

How can we have pm2 behave like this: Always honor the configuration saved with pm2 save. And do not autosave the current state of processes.