KeyDB icon indicating copy to clipboard operation
KeyDB copied to clipboard
verified publisher icon Snapchat

[BUG] Data lost after loading

Open tronglamitmo142 opened this issue 2 years ago • 0 comments

Hi KeyDB Team,

I would like to take about the KeyDB's issue that I face when running in production environment in GCP.

I ran the KeyDB with Flash feature (I put the keydb.conf below). The number of keys is 5 million. After few minutes, almost the key is gone and I checked the amount data of volume in flash storage, it was also gone. Can you please have me with this issue?

This is the keydb-cli dbsize when the loading process was executing:

127.0.0.1:6379> dbsize. (integer) 4991004. 127.0.0.1:6379> dbsize. (integer) 5136504. 127.0.0.1:6379> dbsize. (integer) 5136504. 127.0.0.1:6379> dbsize. (integer) 5136504. 127.0.0.1:6379> dbsize. (integer) 5136504.

This is the config about dir and dbiflename: `127.0.0.1:6379> config get dir

  1. "dir"
  2. "/mnt/md0"`

127.0.0.1:6379> config get dbfilename

  1. "dbfilename"
  2. "rdb.dump"

And the log in /var/log/keydb/keydb-server.log:

8830:8987:M 23 Jun 2023 10:52:26.539 * 10000 changes in 60 seconds. Saving... 8830:8987:M 23 Jun 2023 10:52:26.591 * Background saving started by pid 9063. 8830:8987:M 23 Jun 2023 10:52:26.591 * Background saving started 9063:8987:C 23 Jun 2023 10:53:15.419 * DB saved on disk 9063:8987:C 23 Jun 2023 10:53:15.490 * RDB: 1143 MB of memory used by copy-on-write 8830:8987:M 23 Jun 2023 10:53:15.776 * Background saving terminated with success 8830:8987:M 23 Jun 2023 10:54:16.097 * 10000 changes in 60 seconds. Saving... 8830:8987:M 23 Jun 2023 10:54:16.148 * Background saving started by pid 9074 8830:8987:M 23 Jun 2023 10:54:16.148 * Background saving started 9074:8987:C 23 Jun 2023 10:55:13.253 * DB saved on disk 9074:8987:C 23 Jun 2023 10:55:13.339 * RDB: 972 MB of memory used by copy-on-write 8830:8987:M 23 Jun 2023 10:55:13.688 * Background saving terminated with success 8830:8987:M 23 Jun 2023 10:56:14.057 * 10000 changes in 60 seconds. Saving... 8830:8987:M 23 Jun 2023 10:56:14.112 * Background saving started by pid 9144 8830:8987:M 23 Jun 2023 10:56:14.112 * Background saving started 9144:8987:C 23 Jun 2023 10:57:19.428 * DB saved on disk 9144:8987:C 23 Jun 2023 10:57:19.511 * RDB: 1462 MB of memory used by copy-on-write 8830:8987:M 23 Jun 2023 10:57:19.974 * Background saving terminated with success 8830:8987:M 23 Jun 2023 10:58:00.948 * Starting automatic rewriting of AOF on 100% growth 8830:8987:M 23 Jun 2023 10:58:01.006 * Background append only file rewriting started by pid 9164

/dev/md0 2.9T 17G 2.8T 1% /mnt/md0

And when all the data was in the database, after some minutes, I experienced that the data was gone, also the data in the disk is gone. I check the log every time at that moment and this is the weird log:

127.0.0.1:6379> dbsize. (integer) 4 127.0.0.1:6379> dbsize. (integer) 4

And the log is so weird: root@keydb-1:/mnt/md0# tail -n 50 /var/log/keydb/keydb-server.log 8830:8987:M 23 Jun 2023 12:06:45.050 * 10 changes in 300 seconds. Saving... 8830:8987:M 23 Jun 2023 12:06:45.057 * Background saving started by pid 9396 8830:8987:M 23 Jun 2023 12:06:45.057 * Background saving started 9396:8987:C 23 Jun 2023 12:06:45.057 # Failed opening the RDB file crontab (in server root dir /etc) for saving: Read-only file system 8830:8987:M 23 Jun 2023 12:06:45.157 # Background saving error 8830:8987:M 23 Jun 2023 12:06:51.074 * 10 changes in 300 seconds. Saving... 8830:8987:M 23 Jun 2023 12:06:51.081 * Background saving started by pid 9397 8830:8987:M 23 Jun 2023 12:06:51.081 * Background saving started 9397:8987:C 23 Jun 2023 12:06:51.082 # Failed opening the RDB file crontab (in server root dir /etc) for saving: Read-only file system 8830:8987:M 23 Jun 2023 12:06:51.182 # Background saving error 8830:8987:M 23 Jun 2023 12:06:57.000 * 10 changes in 300 seconds. Saving... 8830:8987:M 23 Jun 2023 12:06:57.007 * Background saving started by pid 9398 8830:8987:M 23 Jun 2023 12:06:57.007 * Background saving started 9398:8987:C 23 Jun 2023 12:06:57.007 # Failed opening the RDB file crontab (in server root dir /etc) for saving: Read-only file system 8830:8987:M 23 Jun 2023 12:06:57.107 # Background saving error 8830:8987:M 23 Jun 2023 12:07:02.022 * 10 changes in 300 seconds. Saving... 8830:8987:M 23 Jun 2023 12:07:02.029 * Background saving started by pid 9399 8830:8987:M 23 Jun 2023 12:07:02.029 * Background saving started 9399:8987:C 23 Jun 2023 12:07:02.029 # Failed opening the RDB file crontab (in server root dir /etc) for saving: Read-only file system

And after that, there was a changing in config file (automatically): 127.0.0.1:6379> config get dir

  1. "dir"
  2. "/etc"

127.0.0.1:6379> config get dbfilename

  1. "dbfilename"
  2. "crontab"

It was differenet with the initial setup.

Can you help me with this issue? It's very urgent with my team. Thanks.

keydb.conf.txt

tronglamitmo142 avatar Jun 23 '23 12:06 tronglamitmo142