redis
redis copied to clipboard
Published
20 hours ago •
tporadowski
tporadowski
Crash with message `zmalloc: Out of memory trying to allocate (too many) bytes`
Describe the bug/issue
Occasionally redis crashes with an Out of memory message, the memory allocation size it says it failed to allocate is unreasonably high, and seems like a bug. In my example the message was:
zmalloc: Out of memory trying to allocate 2466055263228213039 bytes
To Reproduce Please provide more information about:
- version of Redis for Windows you are using: 5.0.9
- configuration of your instance if it's not out-of-the-box one (i.e. persistence type): Out of the box
- typical workload handled by your instance: The dump file size is about 2-3 GB in size
Crash report
zmalloc: Out of memory trying to allocate 2466055263228213039 bytes
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9
[13084] 16 Jun 07:51:43.759 # --- ABORT
[13084] 16 Jun 07:51:43.760 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x00000016, 0x7EF855E64649, 0x1400DD360, 0x1401B5690)
redis-server.exe!AbortHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:209)(0x1401B5690, 0x1400F87BF, 0x0014ED48, 0x0014ED20)
redis-server.exe!raise(minkernel\crts\ucrt\src\appcrt\misc\signal.cpp:547)(0x0014ED01, 0x7EF800000000, 0x00000000, 0x223930323933372F)
redis-server.exe!abort(minkernel\crts\ucrt\src\appcrt\startup\abort.cpp:71)(0x2239303239330000, 0x1401A9210, 0x223930323933372F, 0x00000057)
redis-server.exe!zmalloc_default_oom(D:\dev\GitHub\redis\src\zmalloc.c:104)(0x0FC00000, 0x223930323933372F, 0x0014F220, 0x14016EDB0)
redis-server.exe!zmalloc(D:\dev\GitHub\redis\src\zmalloc.c:114)(0x2239303239333732, 0x06400008, 0x04C17488, 0x140086C5B)
redis-server.exe!rdbSaveLzfStringObject(D:\dev\GitHub\redis\src\rdb.c:362)(0x2239303239333732, 0x7EF855E64649, 0x0014F220, 0x0014F220)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:436)(0x06400008, 0x7EF8330FFCC1, 0x0014F220, 0x7EF6AA830A90)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x00000000, 0x7EF606C00800, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x08C00000, 0xD45A2100)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x005309E0, 0x02210000, 0x00000000, 0x0000331C)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x03030A08, 0x02210000, 0x02210000, 0x00000A3E)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000178, 0x00000000, 0x00000178, 0x00479E50)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x0047B810, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[13084] 16 Jun 07:51:45.922 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Event Log Please check your system's Event Log to see if there are additional details logged related to this issue. There are none
Additional context Please provide Windows version you are using, mode in which you are starting Redis (standalone, cluster, Sentinel), etc.
- OS Version: Windows Server 2016 (No service pack)
- Windows version: Version 1607 (OS Build 14393.3750)
- Redis mode: standalone
- System memory: 195083MB
- System type:12 Core 2.60 GHz Intel Xeon(R) Gold 6126 (Hyper-Threaded)
Thanks @paviad for your detailed report - it seems like an overflow issue, I will investigate that soon.
Thanks, I wanted to say that it happens about 5-6 times a day. Redis recovers from it automatically but it takes about 2 minutes for the recovery during which requests are answered with MISCONF. I had to change my code to take that into account and if I get MISCONF I go into a 1-minute-delay retry loop.
You mentioned 2-3GB dataset, is there anything specific about that data? (very long values, mostly values of one particular type - if so which one?). Could you please provide output of INFO command as well?
There are very long values, they are strings with json values so no particular type.
INFO command output:
# Server
redis_version:5.0.9
redis_git_sha1:9414ab9b
redis_git_dirty:0
redis_build_id:25845e7feb545d77
redis_mode:standalone
os:Windows
arch_bits:64
multiplexing_api:WinSock_IOCP
atomicvar_api:pthread-mutex
process_id:7408
run_id:b2671fd6746306909c7e71874b8b9e0cf4b6eb32
tcp_port:6379
uptime_in_seconds:169395
uptime_in_days:1
hz:10
configured_hz:10
lru_clock:15753462
executable:d:\redis5\redis-server.exe
config_file:D:\Redis5\redis.conf
# Clients
connected_clients:7
client_recent_max_input_buffer:63105320
client_recent_max_output_buffer:0
blocked_clients:0
# Memory
used_memory:10116798336
used_memory_human:9.42G
used_memory_rss:10116715376
used_memory_rss_human:9.42G
used_memory_peak:10183762464
used_memory_peak_human:9.48G
used_memory_peak_perc:99.34%
used_memory_overhead:210455326
used_memory_startup:665080
used_memory_dataset:9906343010
used_memory_dataset_perc:97.93%
allocator_allocated:10197678624
allocator_active:11106516992
allocator_resident:11626610688
total_system_memory:0
total_system_memory_human:0B
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.09
allocator_frag_bytes:908838368
allocator_rss_ratio:1.05
allocator_rss_bytes:520093696
rss_overhead_ratio:0.87
rss_overhead_bytes:-1509895312
mem_fragmentation_ratio:1.00
mem_fragmentation_bytes:0
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:185790
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:3498
rdb_bgsave_in_progress:0
rdb_last_save_time:1592811650
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:112
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
# Stats
total_connections_received:125122
total_commands_processed:6913760
instantaneous_ops_per_sec:33
total_net_input_bytes:134798356016
total_net_output_bytes:21750345126
instantaneous_input_kbps:2532.89
instantaneous_output_kbps:371.84
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:87653
expired_stale_perc:0.25
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:677726
keyspace_misses:296917
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:426493
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:master
connected_slaves:0
master_replid:25a564c9793f35877889f1617093585472d23325
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:2320.234375
used_cpu_user:791.890625
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=2226509,expires=2226468,avg_ttl=3486490337
Does client_recent_max_input_buffer:63105320 mean that there was a really long string value set? It is expected from my particular work load though, it is not indicative of a problem.
Yes, it shows maximum memory amount that was allocated for input data within like last 8 seconds, if you are curious here is the description how it is tracked: https://github.com/tporadowski/redis/blob/develop/src/server.c#L907-L919
Hey take a look at this, this is another type of crash we've been having and this looks weird:
[7408] 23 Jun 23:04:38.269 * Background saving terminated with success
[7408] 23 Jun 23:09:39.031 * 10 changes in 300 seconds. Saving...
[7408] 23 Jun 23:09:39.644 * Background saving started by pid 14440
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9
[14440] 23 Jun 23:10:32.663 # --- EXCEPTION_ACCESS_VIOLATION
[14440] 23 Jun 23:10:32.664 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x14018CBB0, 0x0014FF90, 0x00000001, 0x000CD810)
redis-server.exe!UnhandledExceptiontHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x000CD810, 0x00000001, 0x00000000, 0x00000001)
KERNELBASE.dll!UnhandledExceptionFilter(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x1401A0F88, 0x7FFFD3251E08, 0x00000000, 0x000CD930)
ntdll.dll!memset(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x7FFFD30184D4, 0x0014FF20)
ntdll.dll!_C_specific_handler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x000CD980, 0x000CDF80, 0x00000000)
ntdll.dll!_chkstk(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x7FFFD30B4498, 0x000CDF80, 0x00000000, 0x00000000)
ntdll.dll!RtlImageNtHeaderEx(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x00000108, 0x00000000)
ntdll.dll!KiUserExceptionDispatcher(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x35BF4180, 0x00000000, 0x7EF8B43FFFCE, 0x7EF8AFE555C9)
redis-server.exe!lzf_compress(D:\dev\GitHub\redis\src\lzf_c.c:206)(0x3231222C, 0x3231222C, 0x05017488, 0x140086C5B)
redis-server.exe!rdbSaveLzfStringObject(D:\dev\GitHub\redis\src\rdb.c:363)(0x3231222C, 0x7EF893779E09, 0x0014F220, 0x0014F220)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:436)(0x06800008, 0x7EF6603E2981, 0x0014F220, 0x7EF7FACED830)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x00000000, 0x7EF606C00800, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x09000000, 0x1F2D2200)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x006035A0, 0x022A0000, 0x00000000, 0x00003868)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x030C1468, 0x022A0000, 0x022A0000, 0x00000AFD)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000178, 0x00000000, 0x00000178, 0x0054AA90)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x0054C1A0, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[14440] 23 Jun 23:10:32.695 # --- INFO OUTPUT
[7408] 23 Jun 23:10:37.044 # fork operation failed
[7408] 23 Jun 23:10:38.449 # Background saving terminated by signal 1
[7408] 23 Jun 23:10:38.878 * 10 changes in 300 seconds. Saving...
[7408] 23 Jun 23:10:39.298 * Background saving started by pid 10120
[7408] 23 Jun 23:12:14.735 # fork operation complete
Also, there is a === REDIS BUG REPORT START but no matching END
Note that in both cases the crash is in rdbSaveLzfStringObject I think it sees a bad string prefix, in one case it results in an allocation attempt of too many bytes, and in the other when the header is corrupt but the length that happens to be there is not too large, then it results in a crash while trying to compress it (lzf_compress)
Yes, thanks for additional details - yesterday I had some time to play around with and wrote a small tool to simulate some workload like yours - and got similar crashes a couple of times, but randomly... For the moment you could try disabling compression with rdbcompression no in your config file, but certainly I will try to fix it :)
The paging file size on the drive where redis runs is 20GB, system memory is ~195GB... could that be the problem?
Yes, RDB/AOF are saved in forked process. While testing this I also observed some staleness between fork completes its job and result is returned to parent, so another place to take a look at (my "feeding" tool was getting timeouts in that period). As for the memory and paging file size - please refer to Wiki/Memory Configuration (taken from original microsoftarchive/redis), which recommends... a safe minimum requirement for the system paging file is 2 times the size of physical memory :). During my tests yesterday my paging file grew to 22GB (vs 32GB RAM), but I don't have a hard limit on it and was running into this issue, too.
I cannot say for sure that it is a MUST as obviously Redis for Windows works fine with much less paging space, but since this is recommended by people who wrote that port of Redis (far more knowledgeable than me) - I would say the more the better :)
I did not change any settings yet, I will do it after next backup
On Sun, Jun 28, 2020 at 5:11 PM Tomasz Poradowski [email protected] wrote:
With or without rdbcompression?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tporadowski/redis/issues/64#issuecomment-650766084, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJNV2FOV5XRKS26GPBRSWDRY5FSLANCNFSM4ODILBIA .
Question, won't my dump file size jump to 5-10 times its current size without compression?
EDIT Also, would redis start properly with rdcompression off but a compressed dump file?
Turning compression off shifted the problem somewhere else (two crashes below):
[7672] 30 Jun 08:12:55.519 * Background saving started by pid 10236
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9
[10236] 30 Jun 08:15:34.828 # --- EXCEPTION_ACCESS_VIOLATION
[10236] 30 Jun 08:15:34.829 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x14018CBB0, 0x0014FF90, 0x00000001, 0x0014D810)
redis-server.exe!UnhandledExceptiontHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x0014D810, 0x00000001, 0x00000000, 0x00000001)
KERNELBASE.dll!UnhandledExceptionFilter(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x1401A0F88, 0x7FFFD3251E08, 0x00000000, 0x0014D930)
ntdll.dll!memset(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x7FFFD30184D4, 0x0014FF20)
ntdll.dll!_C_specific_handler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x0014D980, 0x0014DF80, 0x00000000)
ntdll.dll!_chkstk(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x7FFFD30B4498, 0x0014DF80, 0x00000000, 0x00000000)
ntdll.dll!RtlImageNtHeaderEx(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x0014F220, 0x00000000)
ntdll.dll!KiUserExceptionDispatcher(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x140052DF2, 0x7EF802442200, 0x00000008, 0x00000008)
redis-server.exe!crc64(D:\dev\GitHub\redis\src\crc64.c:178)(0x7EF802442200, 0x00000008, 0x00000008, 0x006108C0)
redis-server.exe!rioGenericUpdateChecksum(D:\dev\GitHub\redis\src\rio.c:296)(0x00000008, 0x7EF6A8020009, 0x0014F220, 0x00000000)
redis-server.exe!rioWrite(D:\dev\GitHub\redis\src\rio.h:100)(0x5365676172657641, 0x7EF6A8020009, 0x0014F220, 0x00001000)
redis-server.exe!rdbWriteRaw(D:\dev\GitHub\redis\src\rdb.c:87)(0x5365676172657641, 0x4176657261676553, 0x00002181, 0x10E4C0000)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:445)(0x7EF60474A132, 0x00000001, 0x00001000, 0x7EF67F4396E0)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x1739EC7E800, 0x7EF606C00C00, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x09000000, 0xC9D51800)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x006108C0, 0x02280000, 0x00000000, 0x000027FC)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x030B0AA0, 0x02280000, 0x02280000, 0x00000A0F)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000190, 0x00000000, 0x00000190, 0x00559C10)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x0055B810, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[10236] 30 Jun 08:15:35.669 # --- INFO OUTPUT
[7672] 30 Jun 08:15:40.125 # fork operation failed
[7672] 30 Jun 08:15:41.640 # Background saving terminated by signal 1
[7672] 30 Jun 08:15:42.859 * 10000 changes in 60 seconds. Saving...
[7672] 30 Jun 08:15:43.233 * Background saving started by pid 22328
[7672] 30 Jun 08:19:35.675 # fork operation complete
[7672] 30 Jun 08:19:37.583 * Background saving terminated with success
[7672] 30 Jun 08:21:52.932 * 10000 changes in 60 seconds. Saving...
[7672] 30 Jun 08:21:53.396 * Background saving started by pid 2752
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9
[2752] 30 Jun 08:22:44.759 # --- EXCEPTION_ACCESS_VIOLATION
[2752] 30 Jun 08:22:44.760 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x14018CBB0, 0x0014FF90, 0x00000001, 0x0014D810)
redis-server.exe!UnhandledExceptiontHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x0014D810, 0x00000001, 0x00000000, 0x00000001)
KERNELBASE.dll!UnhandledExceptionFilter(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x1401A0F88, 0x7FFFD3251E08, 0x00000000, 0x0014D930)
ntdll.dll!memset(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x7FFFD30184D4, 0x0014FF20)
ntdll.dll!_C_specific_handler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x0014D980, 0x0014DF80, 0x00000000)
ntdll.dll!_chkstk(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x7FFFD30B4498, 0x0014DF80, 0x00000000, 0x00000000)
ntdll.dll!RtlImageNtHeaderEx(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x0014F220, 0x00000000)
ntdll.dll!KiUserExceptionDispatcher(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x140052DF2, 0x7EF672507D00, 0x00000008, 0x00000008)
redis-server.exe!crc64(D:\dev\GitHub\redis\src\crc64.c:178)(0x7EF672507D00, 0x00000008, 0x00000008, 0x004F0B90)
redis-server.exe!rioGenericUpdateChecksum(D:\dev\GitHub\redis\src\rio.c:296)(0x00000008, 0x7EF6A9048009, 0x0014F220, 0x00000000)
redis-server.exe!rioWrite(D:\dev\GitHub\redis\src\rio.h:100)(0x4853222C6C6C756E, 0x7EF6A9048009, 0x0014F220, 0x00001000)
redis-server.exe!rdbWriteRaw(D:\dev\GitHub\redis\src\rdb.c:87)(0x4853222C6C6C756E, 0x6E756C6C2C225348, 0x00002981, 0x189430000)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:445)(0x7EF6CBF78AAF, 0x00000001, 0x00001000, 0x7EF67F4EC950)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x1730459B900, 0x7EF606C00C00, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x08C00000, 0x9BD61800)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x004F0B90, 0x021E0000, 0x00000000, 0x00000AC0)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x03000C08, 0x021E0000, 0x021E0000, 0x00000A0F)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000190, 0x00000000, 0x00000190, 0x00439970)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x004354E0, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\agent\_work\4\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[2752] 30 Jun 08:22:44.820 # --- INFO OUTPUT
[7672] 30 Jun 08:22:47.708 # fork operation failed
[7672] 30 Jun 08:22:48.853 # Background saving terminated by signal 1
[7672] 30 Jun 08:22:49.327 * 10000 changes in 60 seconds. Saving...
[7672] 30 Jun 08:22:49.674 * Background saving started by pid 19692
[7672] 30 Jun 08:25:22.876 # fork operation complete
[7672] 30 Jun 08:25:24.632 * Background saving terminated with success
Also, would redis start properly with rdcompression off but a compressed dump file?
This flag is used when saving and RDB provides information if a value was compressed or not (only values longer than 20 bytes are considered for compression, so even if you have rdbcompresion on - not all of values will get compressed).
Question, won't my dump file size jump to 5-10 times its current size without compression?
I believe it could - it solely depends on how "compressable" your data is. On the other hand - it should be less CPU-intensive without compression.
Reporting situation after testing the alpha build you've sent me:
I can say the frequency of crashes has dropped significantly. I ran the alpha build at 9:00 AM (my time) and the first crash occurred at 19:42 - this is pretty good.
Here's the crash log that happened at 19:42 (and yes, I've reenabled rdb compression):
[13480] 02 Jul 16:42:03.469 * Background saving started by pid 21228
zmalloc: Out of memory trying to allocate 8319683857054704183 bytes
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9.1
[21228] 02 Jul 16:42:45.414 # --- ABORT
[21228] 02 Jul 16:42:45.415 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x00000016, 0x7EF83A162F49, 0x1400DF070, 0x1401B8690)
redis-server.exe!AbortHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:209)(0x1401B8690, 0x1400FA48F, 0x0014ED48, 0x0014ED20)
redis-server.exe!raise(minkernel\crts\ucrt\src\appcrt\misc\signal.cpp:547)(0x0014ED01, 0x7EF700000000, 0x00000000, 0x737574636F4C2237)
redis-server.exe!abort(minkernel\crts\ucrt\src\appcrt\startup\abort.cpp:71)(0x737574636F4C0000, 0x1401AC210, 0x737574636F4C2237, 0x0014F220)
redis-server.exe!zmalloc_default_oom(D:\dev\GitHub\redis\src\zmalloc.c:104)(0x10400000, 0x737574636F4C2237, 0x0014F220, 0x140170EC0)
redis-server.exe!zmalloc(D:\dev\GitHub\redis\src\zmalloc.c:114)(0x737574636F4C223A, 0x06400008, 0x04C17488, 0x14008905B)
redis-server.exe!rdbSaveLzfStringObject(D:\dev\GitHub\redis\src\rdb.c:362)(0x737574636F4C223A, 0x7EF83A162F49, 0x0014F220, 0x0014F220)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:436)(0x06400008, 0x7EF79FFE9F41, 0x0014F220, 0x7EF6FD509B00)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x00000000, 0x7EF606C00800, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x08C00000, 0x0EF51200)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x02A20670, 0x02200000, 0x00000000, 0x000052EC)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x03BC7E08, 0x02200000, 0x02200000, 0x000009CC)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000178, 0x00000000, 0x00000178, 0x0047A460)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x0047D180, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[21228] 02 Jul 16:42:47.597 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report this bug by following the instructions at:
https://github.com/tporadowski/redis/wiki/Submitting-an-Issue
Suspect RAM error? Use redis-server --test-memory to verify it.
[13480] 02 Jul 16:42:50.922 # fork operation failed
[13480] 02 Jul 16:42:52.065 # Background saving terminated by signal 1
[13480] 02 Jul 16:42:52.428 * 10000 changes in 60 seconds. Saving...
[13480] 02 Jul 16:42:52.827 * Background saving started by pid 13956
[13480] 02 Jul 16:44:07.846 # fork operation complete
[13480] 02 Jul 16:44:09.583 * Background saving terminated with success
Thanks for testing @paviad! This is much better, but still needs fixing. Changes are now committed in cfc8ac1 (also merged to win-5.0 branch).
Thank you for working on it, we had 3 relatively consecutive crashes about 3 hours ago:
[13480] 03 Jul 04:24:53.139 * Background saving started by pid 19036
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9.1
[19036] 03 Jul 04:25:41.483 # --- EXCEPTION_ACCESS_VIOLATION
[19036] 03 Jul 04:25:41.484 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x14018EF60, 0x0014FF90, 0x00000001, 0x000CD810)
redis-server.exe!UnhandledExceptiontHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x000CD810, 0x00000001, 0x00000000, 0x00000001)
KERNELBASE.dll!UnhandledExceptionFilter(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x1401A34D8, 0x7FFFD3251E08, 0x00000000, 0x000CD930)
ntdll.dll!memset(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x7FFFD30184D4, 0x0014FF20)
ntdll.dll!_C_specific_handler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x000CD980, 0x000CDF80, 0x00000000)
ntdll.dll!_chkstk(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x7FFFD30B4498, 0x000CDF80, 0x00000000, 0x00000000)
ntdll.dll!RtlImageNtHeaderEx(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x00000108, 0x00000000)
ntdll.dll!KiUserExceptionDispatcher(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x803F4180, 0x00000000, 0x7EF8563FFEFA, 0x7EF855EE4159)
redis-server.exe!lzf_compress(D:\dev\GitHub\redis\src\lzf_c.c:206)(0x53223A22, 0x53223A22, 0x05017488, 0x14008905B)
redis-server.exe!rdbSaveLzfStringObject(D:\dev\GitHub\redis\src\rdb.c:363)(0x53223A22, 0x7EF84C367BC9, 0x0014F220, 0x0014F220)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:436)(0x06800008, 0x7EF79FFE9F41, 0x0014F220, 0x7EF7D6919BA0)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x00000000, 0x7EF606C00800, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x09000000, 0x98F01200)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x00589960, 0x022C0000, 0x00000000, 0x00004A5C)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x03DA7028, 0x022C0000, 0x022C0000, 0x000009CC)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000178, 0x00000000, 0x00000178, 0x0049ECF0)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x0049F0B0, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[19036] 03 Jul 04:25:41.632 # --- INFO OUTPUT
[13480] 03 Jul 04:25:45.985 # fork operation failed
[13480] 03 Jul 04:25:47.063 # Background saving terminated by signal 1
[13480] 03 Jul 04:25:47.475 * 10000 changes in 60 seconds. Saving...
[13480] 03 Jul 04:25:47.846 * Background saving started by pid 9268
[13480] 03 Jul 04:27:12.762 # fork operation complete
Followed by:
[13480] 03 Jul 04:30:03.983 * Background saving started by pid 11512
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9.1
[11512] 03 Jul 04:30:51.353 # --- EXCEPTION_ACCESS_VIOLATION
[11512] 03 Jul 04:30:51.354 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x14018EF60, 0x0014FF90, 0x00000001, 0x000CD810)
redis-server.exe!UnhandledExceptiontHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x000CD810, 0x00000001, 0x00000000, 0x00000001)
KERNELBASE.dll!UnhandledExceptionFilter(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x1401A34D8, 0x7FFFD3251E08, 0x00000000, 0x000CD930)
ntdll.dll!memset(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x7FFFD30184D4, 0x0014FF20)
ntdll.dll!_C_specific_handler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x000CD980, 0x000CDF80, 0x00000000)
ntdll.dll!_chkstk(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x7FFFD30B4498, 0x000CDF80, 0x00000000, 0x00000000)
ntdll.dll!RtlImageNtHeaderEx(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x00000000, 0x00000000, 0x00000108, 0x00000000)
ntdll.dll!KiUserExceptionDispatcher(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:186)(0x803F4180, 0x00000000, 0x7EF8401FFF3F, 0x7EF83FD5A09E)
redis-server.exe!lzf_compress(D:\dev\GitHub\redis\src\lzf_c.c:245)(0x6148223A, 0x6148223A, 0x05017488, 0x14008905B)
redis-server.exe!rdbSaveLzfStringObject(D:\dev\GitHub\redis\src\rdb.c:363)(0x6148223A, 0x7EF8395B3049, 0x0014F220, 0x0014F220)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:436)(0x06800008, 0x7EF79FFE9F41, 0x0014F220, 0x7EF603D5DC20)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x00000000, 0x7EF606C00800, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x09000000, 0xEBEF1200)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x02AF0790, 0x022D0000, 0x00000000, 0x00002CF8)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x03DB7408, 0x022D0000, 0x022D0000, 0x000009CC)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000178, 0x00000000, 0x00000178, 0x00599DE0)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x0059D170, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[11512] 03 Jul 04:30:51.392 # --- INFO OUTPUT
[13480] 03 Jul 04:30:55.674 # fork operation failed
[13480] 03 Jul 04:30:56.889 # Background saving terminated by signal 1
[13480] 03 Jul 04:30:57.243 * 10000 changes in 60 seconds. Saving...
[13480] 03 Jul 04:30:57.554 * Background saving started by pid 5252
[13480] 03 Jul 04:32:17.677 # fork operation complete
[13480] 03 Jul 04:32:19.787 * Background saving terminated with success
Followed by:
[13480] 03 Jul 04:37:20.340 * Background saving started by pid 14656
zmalloc: Out of memory trying to allocate 15032400584770769054 bytes
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 5.0.9.1
[14656] 03 Jul 04:37:56.988 # --- ABORT
[14656] 03 Jul 04:37:56.989 # --- STACK TRACE
redis-server.exe!StackTraceInfo(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:153)(0x00000016, 0x7EF84EA80009, 0x1400DF070, 0x1401B8690)
redis-server.exe!AbortHandler(D:\dev\GitHub\redis\src\Win32_Interop\Win32_StackTrace.cpp:209)(0x1401B8690, 0x1400FA48F, 0x0014ED48, 0x0014ED20)
redis-server.exe!raise(minkernel\crts\ucrt\src\appcrt\misc\signal.cpp:547)(0x0014ED01, 0x00000000, 0x00000000, 0xD09DD0AFD0A2D09E)
redis-server.exe!abort(minkernel\crts\ucrt\src\appcrt\startup\abort.cpp:71)(0xD09DD0AFD0A20000, 0x1401AC210, 0xD09DD0AFD0A2D09E, 0x0014F220)
redis-server.exe!zmalloc_default_oom(D:\dev\GitHub\redis\src\zmalloc.c:104)(0x16000001, 0xD09DD0AFD0A2D09E, 0x0014F220, 0x140170EC0)
redis-server.exe!zmalloc(D:\dev\GitHub\redis\src\zmalloc.c:114)(0xD09DD0AFD0A2D0A1, 0x06800008, 0x05017C48, 0x14008905B)
redis-server.exe!rdbSaveLzfStringObject(D:\dev\GitHub\redis\src\rdb.c:362)(0xD09DD0AFD0A2D0A1, 0x7EF84EA80009, 0x0014F220, 0x0014F220)
redis-server.exe!rdbSaveRawString(D:\dev\GitHub\redis\src\rdb.c:436)(0x06800008, 0x7EF62EC018C3, 0x0014F220, 0x7EF76820B920)
redis-server.exe!rdbSaveObject(D:\dev\GitHub\redis\src\rdb.c:788)(0x0014F148, 0x17312FBE100, 0x7EF606C00800, 0x0014F220)
redis-server.exe!rdbSaveKeyValuePair(D:\dev\GitHub\redis\src\rdb.c:1072)(0x0014F220, 0x00000000, 0x09000000, 0x30EF1200)
redis-server.exe!rdbSaveRio(D:\dev\GitHub\redis\src\rdb.c:1216)(0x00501170, 0x02270000, 0x00000000, 0x00003940)
redis-server.exe!rdbSave(D:\dev\GitHub\redis\src\rdb.c:1318)(0x03C3DCA8, 0x02270000, 0x02270000, 0x000009CC)
redis-server.exe!QForkChildInit(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:297)(0x00000178, 0x00000000, 0x00000178, 0x0048B6F0)
redis-server.exe!main(D:\dev\GitHub\redis\src\Win32_Interop\Win32_QFork.cpp:1136)(0x00000000, 0x00000000, 0x004874E0, 0x00000000)
redis-server.exe!__scrt_common_main_seh(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[14656] 03 Jul 04:37:57.039 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report this bug by following the instructions at:
https://github.com/tporadowski/redis/wiki/Submitting-an-Issue
Suspect RAM error? Use redis-server --test-memory to verify it.
[13480] 03 Jul 04:38:01.113 # fork operation failed
[13480] 03 Jul 04:38:02.997 # Background saving terminated by signal 1
[13480] 03 Jul 04:38:03.332 * 10 changes in 300 seconds. Saving...
[13480] 03 Jul 04:38:03.673 * Background saving started by pid 19604
[13480] 03 Jul 04:39:31.696 # fork operation complete
[13480] 03 Jul 04:39:33.366 * Background saving terminated with success
Situation has not really improved, it was an illusion, been getting crashes every 30 minutes for the past couple of hours.
A new observation - after I restart Redis there's a long period with no crashes at all. So far since the latest restart 13 hours have passed with no crash.
Very bad day today, 15 crashes so far today.
And 4 more in the past 20 minutes... Our site downtime is starting to become substantial...
Totally understood. I will be on holiday for next 1,5 weeks, but then hope to find some time for it and find the root cause.
Totally understood. I will be on holiday for next 1,5 weeks, but then hope to find some time for it and find the root cause.
So, this 1.5 week vacation is really long and we need you so much to keep maintaining this warehouse! Looking forward to your return!
