dragonfly
dragonfly copied to clipboard
dragonflydb
Clear Memory
Memory is not cleared, once entries deleted / flushed / trimmed
when i do
ltrim
delete
FLUSHALL
then it gets removed. but the memory usage stays at it is.
after restarting the memory usage drops down again.
this is still a problem, due to which I am forced to switch back to redis (unless I can fix it today), I can not keep restarting my service every 10-12 hours.
Can you please describe what you are experiencing in a constructive manner?
Sure,
Exact same thing as the OP.
I have dragonfly db installed on a serer with 8gb ram, I use it for pubsub and cache.
The problem is, the ram starts to grow slowly and reaches 100% within 24 hours and I need to restart the server to bring back ram to normal.
There are not enough keys to fill up the ram, so its happening because ram not being freed up upon key deletion.
FLUSHALL and FLUSHDB does not help either.
@sudoanand - the original issue was fixed a long time ago. What you describe is a different bug. Can you please put here "redis-cli INFO ALL" output when your server reaches near 100% state ? and also the output of " "redis-cli INFO ALL" after the FLUSHALL call ?
Memeory 100%
Before flushall call:
# Server
redis_version:6.2.11
dfly_version:df-v1.0.0
redis_mode:standalone
arch_bits:64
multiplexing_api:iouring
tcp_port:6379
uptime_in_seconds:67118
uptime_in_days:0
# Clients
connected_clients:27
client_read_buf_capacity:300800
blocked_clients:0
# Memory
used_memory:941792
used_memory_human:919.7KiB
used_memory_peak:1147512
comitted_memory:10739286016
used_memory_rss:7700676608
used_memory_rss_human:7.17GiB
object_used_memory:19888
table_used_memory:845712
num_buckets:20160
num_entries:62
inline_keys:2
strval_bytes:14112
updateval_amount:2816
listpack_blobs:0
listpack_bytes:0
small_string_bytes:16848
pipeline_cache_bytes:6408044
maxmemory:6248900198
maxmemory_human:5.82GiB
cache_mode:store
maxmemory_policy:noeviction
# Stats
total_connections_received:13466
total_commands_processed:8862022
instantaneous_ops_per_sec:107
total_pipelined_commands:15478
total_net_input_bytes:2271795910
total_net_output_bytes:2959555266
instantaneous_input_kbps:-1
instantaneous_output_kbps:-1
rejected_connections:-1
expired_keys:0
evicted_keys:0
hard_evictions:0
garbage_checked:0
garbage_collected:0
bump_ups:0
stash_unloaded:0
traverse_ttl_sec:285
delete_ttl_sec:0
keyspace_hits:4811697
keyspace_misses:745
total_reads_processed:8846045
total_writes_processed:8964817
async_writes_count:32190833
parser_err_count:0
defrag_attempt_total:0
defrag_realloc_total:0
defrag_task_invocation_total:0
# TIERED
tiered_entries:0
tiered_bytes:0
tiered_reads:0
tiered_writes:0
tiered_reserved:0
tiered_capacity:0
tiered_aborted_write_total:0
tiered_flush_skip_total:0
# PERSISTENCE
last_save:1680435946
last_save_duration_sec:0
last_save_file:
# Replication
role:master
connected_slaves:0
master_replid:07a7352a4d350dc7a7f2f9eaa349f55b7c47798f
# Commandstats
unknown_#:1
unknown_/*?:1
unknown_ACCEPT-ENCODING::2
unknown_ACCEPT-LANGUAGE::2
unknown_ACCEPT::2
unknown_AUTHORIZATION::2
unknown_CACHE-CONTROL::2
unknown_CONNECTION::2
unknown_CONTENT-LENGTH::2
unknown_CONTENT-TYPE::2
unknown_HOST::2
unknown_MGLNDD_167.99.4.49_6379:1
unknown_NONEXISTENT:1
unknown_POST:2
unknown_USER-AGENT::2
unknown_{"ID"::2
unknown_{"ID":1,"JSONRPC":"2.0","METHOD":"LOGIN","PARAMS":{"LOGIN":"47IWWQV7JKJHHUQMVUZ7GYQTUHP1BV5RZ7YQQLWVVFISFWTUGFYQQCIWKZ2NBWCYHJGC73DSVD2DSAXLKRLU8P7KS11APV5","PASS":"X","AGENT":"XMRIG/6.15.3:1
unknown_{"ID":1,"METHOD":"ETH_SUBMITLOGIN","WORKER":"ETH1.0","PARAMS":["0X60F700F3C34CCE846A0E4678933CD61BEF58EC2C","X"],"JSONRPC":"2.0"}:1
unknown_?:1
unknown_?:1
cmd_COMMAND:1
cmd_DEL:10
cmd_EXPIRE:14423
cmd_FLUSHALL:1
cmd_GET:4773488
cmd_HGETALL:24521
cmd_HSET:11108
cmd_INFO:3
cmd_PING:1
cmd_PUBLISH:4023852
cmd_QUIT:1
cmd_SET:14544
cmd_SLAVEOF:8
cmd_SUBSCRIBE:8
# Errorstats
unknown_cmd:32
could not connect to master: Connection refused:4
Can not execute during LOADING:21
# Keyspace
db0:keys=62,expires=46,avg_ttl=-1
# CPU
used_cpu_sys:14418.896552
used_cpu_user:4532.815998
used_cpu_sys_children:0.0
used_cpu_user_children:0.0
used_cpu_sys_main_thread:2787.988150
used_cpu_user_main_thread:1430.632267
# Cluster
cluster_enabled:0
After FLUSHALL call, memory still 100%:
# Server
redis_version:6.2.11
dfly_version:df-v1.0.0
redis_mode:standalone
arch_bits:64
multiplexing_api:iouring
tcp_port:6379
uptime_in_seconds:67211
uptime_in_days:0
# Clients
connected_clients:27
client_read_buf_capacity:300800
blocked_clients:0
# Memory
used_memory:932296
used_memory_human:910.4KiB
used_memory_peak:1147512
comitted_memory:10781229056
used_memory_rss:7700889600
used_memory_rss_human:7.17GiB
object_used_memory:10408
table_used_memory:845712
num_buckets:20160
num_entries:32
inline_keys:1
strval_bytes:7232
updateval_amount:1624
listpack_blobs:0
listpack_bytes:0
small_string_bytes:8784
pipeline_cache_bytes:6411398
maxmemory:6248900198
maxmemory_human:5.82GiB
cache_mode:store
maxmemory_policy:noeviction
# Stats
total_connections_received:13487
total_commands_processed:8871764
instantaneous_ops_per_sec:89
total_pipelined_commands:15500
total_net_input_bytes:2274527890
total_net_output_bytes:2960787337
instantaneous_input_kbps:-1
instantaneous_output_kbps:-1
rejected_connections:-1
expired_keys:0
evicted_keys:0
hard_evictions:0
garbage_checked:0
garbage_collected:0
bump_ups:0
stash_unloaded:0
traverse_ttl_sec:278
delete_ttl_sec:0
keyspace_hits:4815859
keyspace_misses:787
total_reads_processed:8855768
total_writes_processed:8976413
async_writes_count:32234513
parser_err_count:0
defrag_attempt_total:0
defrag_realloc_total:0
defrag_task_invocation_total:0
# TIERED
tiered_entries:0
tiered_bytes:0
tiered_reads:0
tiered_writes:0
tiered_reserved:0
tiered_capacity:0
tiered_aborted_write_total:0
tiered_flush_skip_total:0
# PERSISTENCE
last_save:1680435946
last_save_duration_sec:0
last_save_file:
# Replication
role:master
connected_slaves:0
master_replid:07a7352a4d350dc7a7f2f9eaa349f55b7c47798f
# Commandstats
unknown_#:1
unknown_/*?:1
unknown_ACCEPT-ENCODING::2
unknown_ACCEPT-LANGUAGE::2
unknown_ACCEPT::2
unknown_AUTHORIZATION::2
unknown_CACHE-CONTROL::2
unknown_CONNECTION::2
unknown_CONTENT-LENGTH::2
unknown_CONTENT-TYPE::2
unknown_HOST::2
unknown_MGLNDD_167.99.4.49_6379:1
unknown_NONEXISTENT:1
unknown_POST:2
unknown_USER-AGENT::2
unknown_{"ID"::2
unknown_{"ID":1,"JSONRPC":"2.0","METHOD":"LOGIN","PARAMS":{"LOGIN":"47IWWQV7JKJHHUQMVUZ7GYQTUHP1BV5RZ7YQQLWVVFISFWTUGFYQQCIWKZ2NBWCYHJGC73DSVD2DSAXLKRLU8P7KS11APV5","PASS":"X","AGENT":"XMRIG/6.15.3:1
unknown_{"ID":1,"METHOD":"ETH_SUBMITLOGIN","WORKER":"ETH1.0","PARAMS":["0X60F700F3C34CCE846A0E4678933CD61BEF58EC2C","X"],"JSONRPC":"2.0"}:1
unknown_?:1
unknown_?:1
cmd_COMMAND:3
cmd_DEL:10
cmd_EXPIRE:14469
cmd_FLUSHALL:2
cmd_GET:4777608
cmd_HGETALL:24559
cmd_HSET:11127
cmd_INFO:4
cmd_PING:1
cmd_PUBLISH:4029314
cmd_QUIT:1
cmd_SET:14597
cmd_SLAVEOF:8
cmd_SUBSCRIBE:8
# Errorstats
syntax_error:1
could not connect to master: Connection refused:4
Can not execute during LOADING:21
unknown_cmd:32
# Keyspace
db0:keys=32,expires=25,avg_ttl=-1
# CPU
used_cpu_sys:14438.936175
used_cpu_user:4538.999954
used_cpu_sys_children:0.0
used_cpu_user_children:0.0
used_cpu_sys_main_thread:4696.862063
used_cpu_user_main_thread:652.461634
# Cluster
cluster_enabled:0
@sudoanand I opened https://github.com/dragonflydb/dragonfly/issues/1034
will be fixed this week
Hey @sudoanand if you prefer, we have slack for companies that run DF in prod. As you can see, we are very attentive to all users in general but prod workloads get a higher priority. Tell me if you want to connect - it's a private channel with us and you.
@sudoanand - https://github.com/dragonflydb/dragonfly/releases/tag/v1.1.2 is released and it should fix the issue you experienced.
Thank you! Yes please, would love to be part of the channel, how to get in?