Netbox fails to connect to read-only postgresql

[fumped]

Deployment Type

Self-hosted

NetBox Version

v4.3.1

Python Version

3.10

Steps to Reproduce

Netbox is supposed to be able to run towards read only replicas of Postgres. https://github.com/netbox-community/netbox/pull/12490 https://github.com/netbox-community/netbox/issues/11233

It fails to start.

We're running the official helm-chart but after input from Benjamin Reed it seems like the issue is in core netbox and not specific to the containerized version.

Expected Behavior

Netbox should successfully launch.

Observed Behavior

"django.db.utils.OperationalError: connection failed: connection to server at "ip-address-here", port 5432 failed: session is read-only
[ Use DB_WAIT_DEBUG=1 in netbox.env to print full traceback for errors here ]
:hourglass_flowing_sand: Waiting on DB... (3s / 30s)"

[arthanson]

@fumped Thank you for opening a bug report. Unfortunately, the information you have provided is not sufficient for someone else to attempt to reproduce the reported behavior. Remember, each bug report must include detailed steps that someone else can follow on a clean, empty NetBox installation to reproduce the exact problem you're experiencing. These instructions should include the creation of any involved objects, any configuration changes, and complete accounting of the actions being taken. Also be sure that your report does not reference data on the public NetBox demo, as that is subject to change at any time by an outside party and cannot be relied upon for bug reports.

Can you please add repro steps, you don't put anything there about how you are configuring anything - are you setting DATABASES to something or?

[fumped]

Hi,

We have two completely separate kubernetes klusters in different countries. Lets call the main country C1 and the other site C2.

Netbox and the workers + Redis lives in kubernetes as containers and the Postgres databases are installed on RHEL machines. C1 has 2 DBs, one Primary RW and one Replica in RO. C2 Just has a Replica in RO.

The idea is to have C1 running all the time and have C2 read-only towards the replica. In a DR scenario we can promote the RO replicas to RW.

C1 is up and running and is initialized. I want C2 to launch towards the same, populated databases. So Yes, the Database is set to the Replica in C2 and to the VIP in C1.

Values:

# Chart netbox C1
netbox:
  externalDatabase:
    host: IP-FOR-C1-VIP
    port: 5432
    database: netbox
    
# Chart netbox C2
netbox:
  externalDatabase:
    host: IP-FOR-C2-IP
    port: 5432
    database: netbox    

If this is not supported, please let me know :).

[pycolas]

Hi, do you set MAINTENANCE_MODE on C2 ?

[fumped]

Hi, do you set MAINTENANCE_MODE on C2 ?

Yes, it's set to maintenance mode, using this as value to the chart.

netbox:
  maintenanceMode: true

I tried setting this in C1 too and it got applied correctly with the banner.

[github-actions[bot]]

This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.

[fumped]

Got notice it will be auto-closed, I updates the main post. Hopefully that was it :)

[RangerRick]

Looking at the original links about RO mode, the work was done in NetBox 3.x. I wonder if the move to Django 5 in nb4.x broke something?

The pod startup is running ./manage.py showmigrations to check if the DB is available, and that's where the above error comes from. Just the act of showing migrations shouldn't, in theory, require read-write access to the database.

[cruse1977]

@fumped - is this relevant ? - https://github.com/netbox-community/netbox/discussions/17026

[github-actions[bot]]

This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.

[fumped]

@fumped - is this relevant ? - #17026

Hi, thanks for the link. It does not seem to be the same issue, i don't see this user trying to connect to a read-only replica and my pods don't get to the traceback stage. It just keeps sending the same messages. Seems like the solution for this ticket is not in the chart anymore but I tried setting it anyway, it made no difference.

externalDatabase: targetSessionAttrs: any