netbox-community
Lots of page not found errors while all pages exists after enabling Sentry
Deployment Type
Self-hosted
NetBox Version
v3.7.1
Python Version
3.10
Steps to Reproduce
- Enable Sentry on the Netbox instance
- Create an IP address, device or custom field (so far we only noticed these models having this issue).
- Try to use the API to retrieve it:
curl -X GET -L \
-H "Authorization: Token <Token>" \
https://netbox.example.net/api/extras/custom-fields/<ID here>
- Notice how Sentry adds a "Page not found" error log for this call while you got an 200 OK response.
Expected Behavior
Since the response is a 200 OK, it should not report to Sentry that the page is not found.
Observed Behavior
We are getting thousands of "page not found" errors in Sentry of our Netbox for pages (all related to API calls) that do exists just fine.
Here is a JSON dump of one of these errors in Sentry:
{"event_id":"447c3796541241f2a4bfb5bc0606f394","project":26,"release":"3.7.1","dist":null,"platform":"python","message":"Page not found","datetime":"2024-02-08T13:28:26+00:00","tags":[["environment","production"],["level","error"],["runtime","CPython 3.10.12"],["runtime.name","CPython"],["release","3.7.1"],["user","ip:::ffff:x.x.x.x"],["server_name","netbox-live-371"],["transaction","/api/dcim/devices/5710"],["url","https://netbox.redacted.net/api/dcim/devices/5710"]],"_meta":{"request":{"headers":{"2":{"1":{"":{"rem":[["@password:filter","s",0,10]],"len":46}}}}}},"_metrics":{"bytes.ingested.event":5495,"bytes.stored.event":6515},"breadcrumbs":{"values":[{"timestamp":1707398906.294575,"type":"redis","category":"redis","level":"info","message":"GET ':1:config'","data":{"db.operation":"GET","redis.command":"GET","redis.is_cluster":false,"redis.key":":1:config"}},{"timestamp":1707398906.294856,"type":"redis","category":"redis","level":"info","message":"GET ':1:config_version'","data":{"db.operation":"GET","redis.command":"GET","redis.is_cluster":false,"redis.key":":1:config_version"}},{"timestamp":1707398906.295473,"type":"default","category":"query","level":"info","message":"SELECT \"core_configrevision\".\"id\", \"core_configrevision\".\"created\", \"core_configrevision\".\"comment\", \"core_configrevision\".\"data\" FROM \"core_configrevision\" ORDER BY \"core_configrevision\".\"created\" ASC LIMIT 1"}]},"contexts":{"runtime":{"name":"CPython","version":"3.10.12","build":"3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]","type":"runtime"},"trace":{"trace_id":"52bf9c69cc9c43d7b0e395a84d4e14f5","span_id":"a5ef9987469bb720","parent_span_id":"b8a150a1f93a8a22","op":"middleware.django","status":"unknown","client_sample_rate":1.0,"description":"django_prometheus.middleware.PrometheusAfterMiddleware.__call__","type":"trace"}},"culprit":"/api/dcim/devices/5710","environment":"production","extra":{"sys.argv":["/opt/netbox/venv/bin/gunicorn","--pid","/var/tmp/netbox.pid","--pythonpath","/opt/netbox/netbox","--config","/opt/netbox/gunicorn.py","netbox.wsgi"]},"fingerprint":["{{ default }}"],"grouping_config":{"enhancements":"eJybzDRxc15qeXFJZU6qlZGBkbGugaGuoeEEAHJMCAM","id":"newstyle:2023-01-11"},"hashes":["d0fbda9855d118740f1105334305c126"],"key_id":"31","level":"error","logentry":{"formatted":"Page not found"},"logger":"","metadata":{"title":"Page not found"},"modules":{"aniso8601":"9.0.1","asgiref":"3.7.2","async-timeout":"4.0.3","attrs":"23.2.0","babel":"2.14.0","bcrypt":"4.1.2","bleach":"6.1.0","certifi":"2023.11.17","cffi":"1.16.0","charset-normalizer":"3.3.2","click":"8.1.7","colorama":"0.4.6","cryptography":"42.0.1","defusedxml":"0.8.0rc2","django":"4.2.9","django-cors-headers":"4.3.1","django-debug-toolbar":"4.2.0","django-filter":"23.5","django-graphiql-debug-toolbar":"0.2.0","django-js-asset":"2.2.0","django-mptt":"0.14.0","django-pglocks":"1.0.4","django-prometheus":"2.3.1","django-redis":"5.4.0","django-rich":"1.8.0","django-rq":"2.10.1","django-tables2":"2.7.0","django-taggit":"5.0.1","django-timezone-field":"6.1.0","djangorestframework":"3.14.0","drf-spectacular":"0.27.0","drf-spectacular-sidecar":"2024.1.1","feedparser":"6.0.11","future":"0.18.3","ghp-import":"2.1.0","graphene":"3.3","graphene-django":"3.0.0","graphql-core":"3.2.3","graphql-relay":"3.2.0","gunicorn":"20.1.0","idna":"3.6","inflection":"0.5.1","jinja2":"3.1.3","jsonschema":"4.21.1","jsonschema-specifications":"2023.12.1","junos-eznc":"2.7.0","lxml":"5.1.0","markdown":"3.5.2","markdown-it-py":"3.0.0","markupsafe":"2.1.4","mdurl":"0.1.2","mergedeep":"1.3.4","mkdocs":"1.5.3","mkdocs-autorefs":"0.5.0","mkdocs-material":"9.5.4","mkdocs-material-extensions":"1.3.1","mkdocstrings":"0.24.0","mkdocstrings-python-legacy":"0.2.3","napalm":"4.1.0","napalm-ce":"0.2.0","napalm-huawei-vrp":"1.1.0","ncclient":"0.6.15","netaddr":"0.10.1","netmiko":"4.3.0","netutils":"1.6.0","ntc_templates":"4.2.0","oauthlib":"3.2.2","packaging":"23.2","paginate":"0.5.6","paramiko":"3.4.0","pathspec":"0.12.1","pillow":"10.2.0","pip":"23.3.2","platformdirs":"4.1.0","prometheus-client":"0.19.0","promise":"2.3","psycopg":"3.1.17","psycopg-binary":"3.1.17","psycopg-pool":"3.2.1","pycparser":"2.21","pyeapi":"1.0.2","pygments":"2.17.2","pyjwt":"2.8.0","pymdown-extensions":"10.7","pynacl":"1.5.0","pyparsing":"3.1.1","pyserial":"3.5","python-dateutil":"2.8.2","python3-openid":"3.2.0","pytkdocs":"0.16.1","pytz":"2023.4","pyyaml":"6.0.1","pyyaml_env_tag":"0.1","redis":"5.0.1","referencing":"0.33.0","regex":"2023.12.25","requests":"2.31.0","requests-oauthlib":"1.3.1","rich":"13.7.0","rpds-py":"0.17.1","rq":"1.15.1","scp":"0.14.5","sentry-sdk":"1.40.2","setuptools":"59.6.0","sgmllib3k":"1.0.0","six":"1.16.0","social-auth-app-django":"5.4.0","social-authr-core":"4.5.1","sqlparse":"0.4.4","svgwrite":"1.4.3","tablib":"3.5.0","text-unidecode":"1.3","textfsm":"1.1.3","transitions":"0.9.0","ttp":"0.9.5","ttp-templates":"0.3.6","typing_extensions":"4.9.0","tzdata":"2023.4","uritemplate":"4.1.1","urllib3":"2.1.0","watchdog":"3.0.0","webencodings":"0.5.1","wheel":"0.42.0","yamlordereddictloader":"0.4.2"},"nodestore_insert":1707398984.177547,"received":1707398906.31036,"request":{"url":"https://netbox.redacted.net/api/dcim/devices/5710","method":"GET","headers":[["Accept","application/json"],["Accept-Encoding","gzip"],["Authorization","[Filtered]"],["Baggage","sentry-trace_id=52bf9c69cc9c43d7b0e395a84d4e14f5,sentry-sample_rate=1,sentry-transaction=GET%20app_redacted_supportrequest_adminsupportrequest_index,sentry-public_key=8f0f78b02f3e4b86a86ac51f1db040ae,sentry-release=dev-master%407866a59,sentry-environment=prod,sentry-sampled=true"],["Connection","close"],["Host","127.0.0.1:8001"],["Sentry-Trace","52bf9c69cc9c43d7b0e395a84d4e14f5-9fa4a9b996084d92-1"],["User-Agent","Symfony HttpClient/Curl"],["X-Forwarded-Host","netbox.redacted.net"],["X-Forwarded-Proto","https"],["X-Real-Ip","::ffff:x.x.x.x"]],"env":{"REMOTE_ADDR":"127.0.0.1","SERVER_NAME":"127.0.0.1","SERVER_PORT":"8001"}},"sdk":{"name":"sentry.python.django","version":"1.40.2","integrations":["argv","atexit","dedupe","django","excepthook","logging","modules","redis","rq","stdlib","threading"],"packages":[{"name":"pypi:sentry-sdk","version":"1.40.2"}]},"timestamp":1707398906.297148,"title":"Page not found","transaction":"/api/dcim/devices/5710","transaction_info":{"source":"url"},"type":"default","user":{"ip_address":"::ffff:x.x.x.x"},"version":"7","location":null}
In your examples you are missing the trailing slash, both in your curl call and in the JSON dump. Can you retest with the correct call?
$ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \
http://netbox-test.lein.io/api/extras/custom-fields/11 -i
HTTP/1.1 301 Moved Permanently
...
$ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \
http://netbox-test.lein.io/api/extras/custom-fields/11/ -i
HTTP/1.1 200 OK
...
<JSON output from NetBox API>
In your examples you are missing the trailing slash, both in your
curlcall and in the JSON dump. Can you retest with the correct call?$ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \ http://netbox-test.lein.io/api/extras/custom-fields/11 -i HTTP/1.1 301 Moved Permanently ... $ curl -H "Authorization: Token 176d4c04ccc8f2a549ea6fd393567d9da5a796ff" \ http://netbox-test.lein.io/api/extras/custom-fields/11/ -i HTTP/1.1 200 OK ... <JSON output from NetBox API>
Ah I see, so without the trailing slash you first get a 301 and then a 200. This 301 is then reported as a 404 in Sentry. I do wonder if this should be the case?
I have now added trailing slashes to all our API calls and the problem is fixed. Thanks markkuleinio