No graceful exit using NPM Scripts that trigger multiple SIGINT

[guillaumeprevost]

This issue appeared already last year (issue #2507) and was resolved in August 2020 by the following PRs :
https://github.com/firebase/firebase-tools/pull/2415
https://github.com/firebase/firebase-tools/pull/2517

However, I experience the issue now with firebase-tools latest version.

[REQUIRED] Environment info

firebase-tools: 9.2.0 (latest at the time of posting)

Platform: Windows (Using Git Bash so commands such as rm, mv ... are available)

[REQUIRED] Test case

Set up a Firebase project that uses the emulators.

[REQUIRED] Steps to reproduce

Use npm to run the project, with several processes chained by &&

Start the emulators through npm start :

"start": "npm run build && firebase emulators:start --import=./emulators-data-import --export-on-exit=./emulators-data",

Stop emulators using CTRL-C

Several CTRL+C are triggered by npm: as far as I can count in my logs, there is one SIGINT for each process chained : the more you add, the more SIGINT are sent through npm.

Note that this happens regardless of whether we specify --import and/or --export, but the issue is about the behavior of Firebase Emulators when it happens

From what I understand in my research and from the comments in #2507, is an NPM issue but Firebase should mitigate it by preventing multiple SIGINT to interrupt a graceful exit.

[REQUIRED] Expected behavior

A graceful exit, all processes being properly stopped, and the the data exported correctly to the specified folder.

i  emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown.
i  emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now.
i  Automatically exporting data using --export-on-exit "./emulators-data" please wait for the export to finish...        
i  Found running emulator hub for project devstorybee at http://localhost:4400
i  Creating export directory C:\dev\storybee\back\emulators-data
i  Exporting data to: C:\dev\storybee\back\emulators-data
i  emulators: Received export request. Exporting data to C:\dev\storybee\back\emulators-data.
+  emulators: Export complete.
+  Export complete
i  emulators: Shutting down emulators.
i  ui: Stopping Emulator UI
!  Emulator UI has exited upon receiving signal: SIGINT
i  functions: Stopping Functions Emulator
i  hosting: Stopping Hosting Emulator
i  firestore: Stopping Firestore Emulator
!  Firestore Emulator has exited upon receiving signal: SIGINT
i  auth: Stopping Authentication Emulator
i  hub: Stopping emulator hub
i  logging: Stopping Logging Emulator

Terminate batch job (Y/N)?

[REQUIRED] Actual behavior

i  emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown.
i  emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now.
i  Automatically exporting data using --export-on-exit "./emulators-data" please wait for the export to finish...
i  Found running emulator hub for project devstorybee at http://localhost:4400
i  Creating export directory C:\dev\storybee\back\emulators-data
i  Exporting data to: C:\dev\storybee\back\emulators-data
Terminate batch job (Y/N)? i  emulators: Received export request. Exporting data to C:\dev\storybee\back\emulators-data.
Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? Terminate batch job (Y/N)? +  emulators: Export complete.
+  Export complete
i  emulators: Shutting down emulators.
i  ui: Stopping Emulator UI
!  Emulator UI has exited upon receiving signal: SIGINT
i  functions: Stopping Functions Emulator
i  hosting: Stopping Hosting Emulator
i  firestore: Stopping Firestore Emulator

!  emulators: Received SIGINT (Ctrl-C) 2 times. You have forced the Emulator Suite to exit without waiting for 1 subprocess to finish. These processes may still be running on your machine:

┌────────────────────┬────────────────┬──────┐
│ Emulator           │ Host:Port      │ PID  │
├────────────────────┼────────────────┼──────┤
│ Firestore Emulator │ localhost:8080 │ 4020 │
└────────────────────┴────────────────┴──────┘

To force them to exit run:

TASKKILL /PID 4020 /T

^C^C^C^C^C

The behavior is amplified here : we can see there are 7 SIGINT : with the first, the emulators do what is expected, starting a clean exit. With the second, the exit (and export) gets interrupted and shutdown, the 5 other SIGINT are shown as ^C .

[phenry20]

Any updates on this issue?

[phenry20]

Seems as though it's being looked into here:

https://github.com/firebase/firebase-tools/issues/3578

[dereekb]

I'm running into an issue where the emulators aren't shutting down properly. Right now since the emulators don't support hot reloading I'm running the following within a docker container:

find dist/apps/demo-api | entr -rp -s 'npx firebase emulators:start --import=/root/data/emulators --export-on-exit'"

The entr command is piped folder changes from the dist/apps/demo-api directory, and waits for the emulators to shut down before relaunching.

It seems like when the --export-on-exit argument is added the emulators tell entr that they shut down but don't actually shut down. When entr runs the following command to start the emulators again, the emulators say the port is in use. Here's where it recieves SIGTERM from entr and tries to restart:

dbcomponents-demo-api-server-1  | i  emulators: Received SIGTERM for the first time. Starting a clean shutdown.
dbcomponents-demo-api-server-1  | i  emulators: Please wait for a clean shutdown or send the SIGTERM signal again to stop right now.
dbcomponents-demo-api-server-1  | i  Automatically exporting data using --export-on-exit "/root/data/emulators" please wait for the export to finish...
dbcomponents-demo-api-server-1  | i  Found running emulator hub for project dereekb-components at http://localhost:4400
dbcomponents-demo-api-server-1  | i  Exporting data to: /root/data/emulators
dbcomponents-demo-api-server-1  | i  emulators: Received export request. Exporting data to /root/data/emulators.
dbcomponents-demo-api-server-1  | 
dbcomponents-demo-api-server-1  | Error: Storage Emulator Rules runtime exited unexpectedly.
dbcomponents-demo-api-server-1  | ⚠  emulators: You are not currently authenticated so some features may not work correctly. Please run firebase login to authenticate the CLI.
dbcomponents-demo-api-server-1  | i  emulators: Starting emulators: auth, functions, firestore, hosting, pubsub, storage
dbcomponents-demo-api-server-1  | ⚠  emulators: It seems that you are running multiple instances of the emulator suite for project dereekb-components. This may result in unexpected behavior.
dbcomponents-demo-api-server-1  | ⚠  functions: The following emulators are not running, calls to these services from the Functions emulator will affect production: database
dbcomponents-demo-api-server-1  | ✔  functions: Using node@16 from host.
dbcomponents-demo-api-server-1  | ⚠  functions: You are not signed in to the Firebase CLI. If you have authorized this machine using gcloud application-default credentials those may be discovered and used to access production services.
dbcomponents-demo-api-server-1  | i  emulators: Shutting down emulators.
dbcomponents-demo-api-server-1  | i  functions: Stopping Functions Emulator
dbcomponents-demo-api-server-1  | i  hub: Stopping emulator hub
dbcomponents-demo-api-server-1  | ⚠  firestore: Port 9904 is not open on 0.0.0.0, could not start Firestore Emulator.
dbcomponents-demo-api-server-1  | ⚠  firestore: To select a different host/port, specify that host/port in a firebase.json config file:
dbcomponents-demo-api-server-1  |       {
dbcomponents-demo-api-server-1  |         // ...
dbcomponents-demo-api-server-1  |         "emulators": {
dbcomponents-demo-api-server-1  |           "firestore": {
dbcomponents-demo-api-server-1  |             "host": "HOST",
dbcomponents-demo-api-server-1  |             "port": "PORT"
dbcomponents-demo-api-server-1  |           }
dbcomponents-demo-api-server-1  |         }
dbcomponents-demo-api-server-1  |       }
dbcomponents-demo-api-server-1  | i  emulators: Shutting down emulators.

When running ps -A to list all the running tasks within the Docker container, this is what shows up:

root@a0899939a477:/code# ps -A
  PID TTY          TIME CMD
    1 pts/0    00:00:00 npm exec nx run
 ...
  162 pts/0    00:00:00 npm <defunct>
  172 pts/0    00:00:00 npm exec fireba <defunct>
  184 pts/0    00:00:05 node <defunct>
  196 ?        00:00:04 java
  224 ?        00:00:03 java
  273 ?        00:00:00 node
  368 pts/1    00:00:00 bash
  425 pts/1    00:00:00 ps

pid 196, 224, and 273 here represent the pub-sub emulator, the firestore emulator, and the emulator UI, respectively.

Can use netstat to see the port still in use.

root@a0899939a477:/code# netstat -tulpn | grep :9904
tcp        0      0 0.0.0.0:9904            0.0.0.0:*               LISTEN      335/java      

To get around this, I added a script that waits for the ports to shut off (never happens from the emulators) and then after a period of time kills the tasks associated with those ports, and then goes onward.

https://github.com/dereekb/dbx-components/blob/0e6fb186add4dc003660d4501200de40ca911b20/wait-for-ports.sh

[yuchenshi]

Hi, thanks for filing this issue! We are unable to promise any timeline for this, but if others also have this issue, adding a +1 on this issue can help us prioritize adding this to the roadmap.

(Googler-only internal tracking bug: b/240330572)

[mikila85]

+1

[joehan]

Hey all, for organizational purposes, I'm going to close this since it duplicates #3578.