dragonfly icon indicating copy to clipboard operation
dragonfly copied to clipboard
verified publisher icon dragonflydb

Broken TLS replication due to preemptions in Async writes

Open enzo-pauvy opened this issue 1 year ago • 9 comments

Hello there,

I'm in the process of deploying Dragonfly 1.25.4 as a replacement for Redis and I'm encountering a few errors, I was wondering if anyone could help or if this is expected. All the errors occurs in the first indexing phase, but once the total items is stable (balance between expiring and new keys), the master has changed on its own and then it seems more stable. I have 3 servers with 1 dfly and 1 sentinel service on each server.

  • OS: Ubuntu 24.04.1 LTS
  • Kernel: 6.8.0-45-generic
  • Containerized: Systemd
  • Dragonfly Version: 1.25.4

I can't figure out how to reproduce the error, but here's what I have in the dragonfly logs.

Error entry not found in expire table

  • Many errors in db_slice.cc
E20241204 13:46:07.394174 867126 db_slice.cc:1139] Internal error, entry ...:... not found in expire table, db_index: 0, expire table size: 11417, prime table size: 114690x59194f2bafa1  dfly::(anonymous namespace)::ScanCb()
  • Following by SIGSEGV signal in DbSlice::PostUpdate
*** SIGSEGV received at time=1733320149 on cpu 8 ***
PC: @     0x59194f5be1d7  (unknown)  dfly::DbSlice::PostUpdate()
  • Or
F20241205 05:18:09.021003 1063853 db_slice.cc:847] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second
	...
*** SIGABRT received at time=1733375889 on cpu 9 ***
PC: @     0x72b3a2e9eb1c  (unknown)  pthread_kill
  • Or
F20241205 02:49:47.255090 1054284 db_slice.cc:1239] Check failed: !prime_it.is_done()
	...
*** SIGABRT received at time=1733366987 on cpu 5 ***
PC: @     0x713361a9eb1c  (unknown)  pthread_kill

Error Unsupported tag

  • Unsupported tag xxx in compact_object.cc (68, 223, 254, ...)
F20241204 15:06:41.331504 889887 compact_object.cc:1129] Unsupported tag 254
	...
*** SIGABRT received at time=1733324801 on cpu 11 ***
PC: @     0x76b222c9eb1c  (unknown)  pthread_kill

Storage Here, I use S3 storage every 5 minutes to compensate for the crash. With cron, because the backup is not upload after a crash. The errors are the same without the backup. Screenshot from 2024-12-11 10-03-40

enzo-pauvy avatar Dec 12 '24 14:12 enzo-pauvy

@adiholden looks like another preemption bug.

romange avatar Dec 12 '24 15:12 romange

@enzo-pauvy what's an indexing phase?

romange avatar Dec 12 '24 15:12 romange

I call “indexing phase” the period (here from 15:00 to ~06:00) during which keys are added to an empty dragonfly database. After 06:00, the number of elements doesn't change much and there are no more crashes. I haven't tested much more after that yet.

enzo-pauvy avatar Dec 12 '24 16:12 enzo-pauvy

Is there a way for us to host Dragonfly server so we could debug this? Are you on the public cloud? Could you DM me on the discord please?

romange avatar Dec 12 '24 17:12 romange

@enzo-pauvy how do you run dragonfly? Can you please provide all its runtime flags?

romange avatar Dec 13 '24 06:12 romange

Here is my flags.txt file, I am running dragonfly locally with systemd

# https://www.dragonflydb.io/docs/managing-dragonfly/flags
--bind=0.0.0.0
--port=<dfly_port>
--log_dir=/var/log/dragonfly
--aclfile=/var/lib/dragonfly/preprod.acl
--requirepass=<pass>
--masteruser=<user>
--masterauth=<pass>
--maxmemory=4gb
--cache_mode=True
--dir=s3://<bucket>/...
--dbfilename=dfly_...
--snapshot_cron=*/5 * * * *

# https://github.com/dragonflydb/dragonfly/pull/3615
--replica_announce_ip=<dns>

# https://www.dragonflydb.io/docs/managing-dragonfly/using-tls
--tls
--tls_replication
--tls_key_file=<key>
--tls_cert_file=<cert>

And the sentinel.conf file

port <sentinel_port>

requirepass "<pass>"

sentinel deny-scripts-reconfig yes

sentinel monitor default <dns> <dfly_port> 2

sentinel down-after-milliseconds default 5000
sentinel parallel-syncs default 2
sentinel auth-user default <user>
sentinel auth-pass default <pass>
sentinel failover-timeout default 6000

sentinel resolve-hostnames yes
sentinel announce-hostnames yes

tls-replication yes
tls-auth-clients no
tls-port <sentinel_tls_port>
tls-cert-file "<cert>"
tls-key-file "<key>"
tls-ca-cert-file "<cert>"

# Generated by CONFIG REWRITE
...

enzo-pauvy avatar Dec 13 '24 07:12 enzo-pauvy

Is it possible for you to test with cache_mode=False and tell us if it still crashes?

romange avatar Dec 13 '24 08:12 romange

It looks like I still have the same errors with --cache_mode=False. Several Internal error followed by a SIGSEGV signal.

dragonfly_v1.25.4[3737700]: E20241213 09:51:46.731494 3737706 db_slice.cc:1139] Internal error, entry ...:... not found in expire table, db_index: 0, expire table size: 5646, prime table size: 57510x5fcf9282cfa1  dfly::(anonymous namespace)::ScanCb()
dragonfly_v1.25.4[3737700]: 0x5fcf9282d95a  dfly::(anonymous namespace)::OpScan()
dragonfly_v1.25.4[3737700]: 0x5fcf9282f52c  std::_Function_handler<>::_M_invoke()
dragonfly_v1.25.4[3737700]: 0x5fcf930c1d65  util::fb2::FiberQueue::Run()
dragonfly_v1.25.4[3737700]: *** SIGSEGV received at time=1734084495 on cpu 3 ***
dragonfly_v1.25.4[3737700]: PC: @     0x5fcf92b301d7  (unknown)  dfly::DbSlice::PostUpdate()
systemd[1]: dragonfly_preprod.service: Main process exited, code=dumped, status=11/SEGV
systemd[1]: dragonfly_preprod.service: Failed with result 'core-dump'.
systemd[1]: dragonfly_preprod.service: Consumed 21min 20.299s CPU time, 275.1M memory peak, 0B memory swap peak.
systemd[1]: dragonfly_preprod.service: Scheduled restart job, restart counter is at 1.
systemd[1]: Started dragonfly_preprod.service - Dragonfly Service.

enzo-pauvy avatar Dec 13 '24 11:12 enzo-pauvy

The bug is due to TLS replication being broken right now.

romange avatar Feb 12 '25 08:02 romange

Hello, we've recently started getting hit by this bug multiple times a day. We have one primary with many replicas doing TLS replication. This is how we run the primary:

/usr/local/bin/dragonfly \
    --logtostderr \
    --dir /mnt/disks/backups \
    --snapshot_cron "*/5 * * * *" \
    --no_tls_on_admin_port \
    --admin_bind 10.10.0.4 \
    --admin_port 8080 \
    --cache_mode \
    --tls \
    --tls_cert_file /etc/ssl/certs/client-cert.pem \
    --tls_key_file /etc/ssl/certs/client-key.pem \
    --tls_ca_cert_file /etc/ssl/certs/dragonfly-ca.pem \
    --replication_timeout 500000

Crash usually looks like this:

Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]: F20250624 17:36:14.893257 1383171 compact_object.cc:1239] Unsupported tag 90
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]: *** Check failure stack trace: ***
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f319ae923  google::LogMessage::SendToLog()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f319a70e7  google::LogMessage::Flush()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f319a8a6f  google::LogMessageFatal::~LogMessageFatal()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f3125a3be  dfly::CompactObj::Free()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f3125a63d  dfly::CompactObj::operator=()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f311fe376  dfly::detail::Segment<>::InsertUniq<>()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f311f3ea4  dfly::DbSlice::AddOrFindInternal()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f311f5b65  dfly::DbSlice::AddOrFind()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f30ef5f55  dfly::(anonymous namespace)::OpSet()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f30ef66fc  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly11Transaction18ScheduleSingleHopTIZNS3_10HSetFamily4HSetENS0_4SpanIKSt17basic_string_viewIcSt11char_traitsIcEEEERKNS3_14CommandContextEEUlPS4_PNS3_11EngineShardEE_EEDTclfp_fpTLDn0EEEOT_EUlSH_SJ_E_NS4_14RunnableResultEJSH>
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f312254af  dfly::Transaction::RunSquashedMultiCb()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f3122a015  dfly::Transaction::Execute()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f3122a09a  dfly::Transaction::ScheduleSingleHop()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f30eebb45  dfly::HSetFamily::HSet()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f3120ac5f  dfly::CommandId::Invoke()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f30f0ca53  dfly::Service::InvokeCmd()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f31047d39  dfly::MultiCommandSquasher::SquashedHopCb()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f31048a7a  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly20MultiCommandSquasher15ExecuteSquashedEPN6facade17RedisReplyBuilderEEUlPT_PT0_E0_NS3_11Transaction14RunnableResultEJPSD_PNS3_11EngineShardEEEESA_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f312277b9  dfly::Transaction::RunCallback()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f3122a59b  dfly::Transaction::RunInShard()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f31168790  dfly::EngineShard::PollExecution()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f312233a1  _ZNSt17_Function_handlerIFvvEZN4dfly11Transaction11DispatchHopEvEUlvE1_E9_M_invokeERKSt9_Any_data
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f31794f55  util::fb2::FiberQueue::Run()
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f31275730  _ZN5boost7context6detail11fiber_entryINS1_12fiber_recordINS0_5fiberEN4util3fb219FixedStackAllocatorEZNS6_6detail15WorkerFiberImplIZN4dfly9TaskQueue5StartESt17basic_string_viewIcSt11char_traitsIcEEEUlvE_JEEC4IS7_EESF_RKNS0_12preallocatedEOT_OSG_EUlOS4_E_EEEEvNS1_10transfer_tE
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]:     @     0x560f317b425f  make_fcontext
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]: *** SIGABRT received at time=1750786574 on cpu 6 ***
Jun 24 17:36:14 dragonfly-artifact-prod dragonfly[1383164]: PC: @     0x7fe6ff81aeec  (unknown)  (unknown)
Jun 24 17:36:14 dragonfly-artifact-prod systemd[1]: dragonflydb.service: Main process exited, code=killed, status=6/ABRT
Jun 24 17:36:14 dragonfly-artifact-prod systemd[1]: dragonflydb.service: Failed with result 'signal'.

Has there been any progress on this issue?

gravelg avatar Jun 24 '25 18:06 gravelg 

`*** Check failure stack trace: ***
F20250807 22:03:18.494117 18154 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494143 18215 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494122 18170 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494103 18115 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494208 18122 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494099 18113 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494102 18197 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494232 18230 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494284 18103 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second F20250807 22:03:18.494271 18112 db_slice.cc:820] Check failed: db.expire.Insert(main_it->first.AsRef(), ExpirePeriod(delta)).second 
*** Check failure stack trace: ***
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409c05923  google::LogMessage::SendToLog()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bfe0e7  google::LogMessage::Flush()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409bffa6f  google::LogMessageFatal::~LogMessageFatal()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a409447338  dfly::DbSlice::AddExpire()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092679c9  dfly::(anonymous namespace)::SetCmd::SetExisting()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092680bb  dfly::(anonymous namespace)::SetCmd::Set()
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0x61a4092687cf  _ZN4absl12lts_2024011619functional_internal12InvokeObjectIZN4dfly12_GLOBAL__N_110SetGenericERKNS4_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bPNS3_11TransactionEEUlSE_PNS3_11EngineShardEE_NSD_14RunnableResultEJSE_SG_EEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE`

Getting this without TLS as well. Correct me if this is not the similar issue which is being discussed! Configuration is like below /home/ubuntu/dragonfly --logtostderr --cache_mode=true --dbnum=1 --port=6379 --logbuflevel=-1 --conn_use_incoming_cpu=true --maxmemory=${DF_MAX_MEMORY} --masterauth=${DF_PASSWORD} --requirepass=${DF_PASSWORD} --break_replication_on_master_restart=true --eviction_memory_budget_threshold=0.25 --tiered_offload_threshold=0.25 --tiered_prefix /mnt/localDiskSSD/dfssd --dir=/mnt/localDiskSSD/backup --cluster_mode=emulated --lock_on_hashtags --interpreter_per_thread=128 DF version is: v1.26.3

shahyash2609 avatar Aug 07 '25 22:08 shahyash2609

@shahyash2609 Hello, you are using a very old version of df. I would suggest to use a newer one as we have made a lot of fixes since 1.26 (TLS replication is still not fixed though)

kostasrim avatar Aug 08 '25 07:08 kostasrim