yugabyte-db icon indicating copy to clipboard operation
yugabyte-db copied to clipboard

[DocDB] Master incorrectly complains if master_addresses does not contain its own rpc_bind_addresses

Open iSignal opened this issue 2 years ago • 1 comments

Jira Link: [DB-309](https://yugabyte.atlassian.net/browse/DB-309)

Description

  1. Create an entry in /etc/hosts of the form 127.0.0.2 localhost2
  2. Forward 127.0.0.2:7100 -> 127.0.0.1:7100 using something like ssh -L 127.0.0.2:7100:127.0.0.1:7100 localhost bash
  3. Start a yb-master with the cmd line /home/sanketh/yugabyte-2.9.0.0/bin/yb-master --master_addresses=localhost2 --server_broadcast_addresses=localhost2 --rpc_bind_addresses=127.0.0.1 --fs_data_dirs=/tmp/foo

Result: Fatal error of the form

F0511 22:37:35.656201 139843 master_main.cc:138] Illegal state (yb/master/catalog_manager.cc:1530): Unable to initialize catalog manager: Failed to initialize sys tables async: None of the local addresses are present in master_addresses localhost2.
Fatal failure details written to /tmp/foo/yb-data/master/logs/yb-master.FATAL.details.2022-05-11T22_37_35.pid139843.txt
F20220511 22:37:35 ../../src/yb/master/master_main.cc:138] Illegal state (yb/master/catalog_manager.cc:1530): Unable to initialize catalog manager: Failed to initialize sys tables async: None of the local addresses are present in master_addresses localhost2.
    @     0x7f050b4a1a9c  yb::LogFatalHandlerSink::send()
    @     0x7f050a8828ee  google::LogMessage::SendToLog()
    @     0x7f050a87fa7a  google::LogMessage::Flush()
    @     0x7f050a883169  google::LogMessageFatal::~LogMessageFatal()
    @           0x40998b  yb::master::MasterMain()
    @     0x7f0506473825  __libc_start_main
    @           0x408999  _start
    @              (nil)  (unknown)

*** Check failure stack trace: ***
    @     0x7f050b49fe81  yb::(anonymous namespace)::DumpStackTraceAndExit()
    @     0x7f050a88001d  google::LogMessage::Fail()
    @     0x7f050a882546  google::LogMessage::SendToLog()
    @     0x7f050a87fa7a  google::LogMessage::Flush()
    @     0x7f050a883169  google::LogMessageFatal::~LogMessageFatal()
    @           0x40998b  yb::master::MasterMain()
    @     0x7f0506473825  __libc_start_main
    @           0x408999  _start
    @              (nil)  (unknown)

Expected:

In this case, the check doesn't make sense because master_addresses is including the local server_broadcast_addresses entry. This reflects how a master would potentially be set up with a public IP address while rpc bind address would be a private IP.

We could also potentially remove this check completely because I am not sure the misconfiguration it is trying to protect against is actually common.

@sanketkedia @bmatican

iSignal avatar May 11 '22 22:05 iSignal

@hulien22 , Can we backport this to 2.14 using the backport tool?

rthallamko3 avatar Aug 11 '22 17:08 rthallamko3

Backported to 2.14 in 5f59a1540b656c05e9b978200c57073c8261aa87

hulien22 avatar Aug 12 '22 15:08 hulien22