Cannot get data to display on NGINX dashboard

[Buckeyes1995]

Describe the bug I've installed the NGINX template and dashboard. I've verified (via TCPDUMP) that the NGINX logs are being received. I get no NGINX data on the dashboard; I have no issue with the Firewall or Suricata dashboards.

To Reproduce Steps to reproduce the behavior:

1. Install NGINX template
2. Install NGINX dashboard
3. Go to dashboard.

Screenshots

Firewall System (please complete the following information):

• OPNsense
• 24.1.6

Operating System (please complete the following information):

• OS Linux 5.15.0-107-generic x86_64 PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy

Installation method (manual, ansible-playbook, docker, script): Script

Elasticsearch, Logstash, Kibana (please complete the following information):

• Latest as downloaded by script, today.

Elasticsearch, Logstash, Kibana logs:

• [2024-05-17T21:16:23,173][WARN ][logstash.outputs.elasticsearch][pfelk][3ee1112ba72cf1114fa76f95359ea7b9209ad0c04ac6f2f018d266410bd988b7] Could not index event to Elasticsearch. {:status=>404, :action=>["create", {:_id=>nil, :_index=>"logs-pfelk-nginx", :routing=>nil}, {"@timestamp"=>2024-05-17T21:16:23.066155517Z, "data_stream"=>{"namespace"=>"nginx", "type"=>"logs", "dataset"=>"pfelk"}, "tags"=>["pfelk", "nginx", "GeoIP_Source"], "type"=>"firewall", "host"=>{"ip"=>"192.168.1.1"}, "log"=>{"syslog"=>{"hostname"=>"opnsense.buckeyes1995.com", "facility"=>{"code"=>16, "name"=>"local0"}, "priority"=>134, "appname"=>"nginx", "severity"=>{"code"=>6, "name"=>"Informational"}}}, "service"=>{"type"=>"system"}, "timestamp"=>"17/May/2024:16:16:23 -0500", "nginx"=>{"access"=>{"body_sent"=>{"bytes"=>"0"}, "url"=>"/api/server-info/features", "method"=>"GET", "referrer"=>"https://immich.buckeyes1995.com/auth/login", "http_version"=>"2.0", "agent"=>"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1", "response_code"=>"304", "user_name"=>"-"}}, "event"=>{"created"=>2024-05-17T16:16:23.000Z, "dataset"=>"pfelk.nginx", "original"=>"<134>May 17 16:16:23 opnsense.buckeyes1995.com nginx: 172.69.67.148 - - [17/May/2024:16:16:23 -0500] "GET /api/server-info/features HTTP/2.0" 304 0 "https://immich.buckeyes1995.com/auth/login" "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1" "2600:387:f:e1b::6" "immich.buckeyes1995.com" sn="immich.buckeyes1995.com" rt=0.024 ua="192.168.40.5:2283" us="304" ut="0.024" ul="0" cs=-"}, "@version"=>"1", "client"=>{"ip"=>"172.69.67.148", "geo"=>{"country_iso_code"=>"US", "postal_code"=>"75270", "country_name"=>"United States", "region_iso_code"=>"US-TX", "timezone"=>"America/Chicago", "location"=>{"lon"=>-96.8022, "lat"=>32.7797}, "continent_code"=>"NA", "region_name"=>"Texas", "city_name"=>"Dallas"}, "as"=>{"number"=>13335, "organization"=>{"name"=>"CLOUDFLARENET"}}, "mmdb"=>{"dma_code"=>623}}}], :response=>{"create"=>{"status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [logs-pfelk-nginx] and composable template [-pfelk-nginx] forbids index auto creation", "index_uuid"=>"na", "index"=>"logs-pfelk-nginx"}}}}

Additional context Logstash log above indicates the issue.. just not sure what it means.

[a3ilson]

@Buckeyes1995

Thanks for providing the log above! It appears that the index template is not installed, based on the error, {"create"=>{"status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [logs-pfelk-nginx] and composable template [-pfelk-nginx] forbids index auto creation", "index_uuid"=>"na", "index"=>"logs-pfelk-nginx"}}}}

To confirm:

1. Navigate to Index Templates and filter by typing "pfelk"

• 

2. If the template is not installed, navigate to the templates portion, copy the nginx template and apply via "Dev Tools"

• 

[Buckeyes1995]

Thanks for the reply.. it is installed :(

[a3ilson]

Within OPNsense do you have RFC5424 ticked? I took the provided log (above) and tested manually which parsed it out correctly when RFC5424 was enabled.

Original Message:

<134>May 17 16:16:23 opnsense.buckeyes1995.com nginx: 172.69.67.148 - - [17/May/2024:16:16:23 -0500] "GET /api/server-info/features HTTP/2.0" 304 0 "https://immich.buckeyes1995.com/auth/login" "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1" "2600:387:f:e1b::6" "immich.buckeyes1995.com" sn="immich.buckeyes1995.com" rt=0.024 ua="192.168.40.5:2283" us="304" ut="0.024" ul="0" cs=-"}, "@Version"=>"1"

Original Message w/RFC5424:

<134>2024-05-18T13:35:49+00:00 opnsense.buckeyes1995.com nginx: 172.69.67.148 - - [17/May/2024:16:16:23 -0500] "GET /api/server-info/features HTTP/2.0" 304 0 "https://immich.buckeyes1995.com/auth/login" "Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1" "2600:387:f:e1b::6" "immich.buckeyes1995.com" sn="immich.buckeyes1995.com" rt=0.024 ua="192.168.40.5:2283" us="304" ut="0.024" ul="0" cs=-"}, "@Version"=>"1"

Parsed Message (RFC5424):

{
  "[event][created]": "24-05-18T13:35:49+00:00",
  "[log][syslog][version]": "20",
  "[log][syslog][hostname]": "opnsense.buckeyes1995.com",
  "filter_message": "172.69.67.148 - - [17/May/2024:16:16:23 -0500] \"GET /api/server-info/features HTTP/2.0\" 304 0 \"https://immich.buckeyes1995.com/auth/login\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1\" \"2600:387:f:e1b::6\" \"immich.buckeyes1995.com\" sn=\"immich.buckeyes1995.com\" rt=0.024 ua=\"192.168.40.5:2283\" us=\"304\" ut=\"0.024\" ul=\"0\" cs=-\"}, \"@Version\"=>\"1\"",
  "[log][syslog][appname]": "nginx:"
}

Alternatively, if you elect not to use RFC5424 and/or RFC5424 doesn't work you could amend the pfelk.grok file line 15 to:

OPNSENSE %{SYSLOGTIMESTAMP:[event][created]}\s%{SYSLOGHOST:[log][syslog][hostname]}\s%{PROG:[log][syslog][appname]}(\[%{POSINT:[log][syslog][procid]}\]\:)?\s%{GREEDYDATA:filter_message}

Give RFC5424 a try first. If elected to try the alternate method, once the change is applied, you'll need to restart Logstash.

RFC5424:

[Buckeyes1995]

Hmm.. either of those worked for me.. I enabled RFC5424 and got the same error below.

Same thing with changing line 15 of the .grok file.

Could not index event to Elasticsearch. {:status=>404, :action=>["create", {:_id=>nil, :_index=>"logs-pfelk-nginx", :routing=>nil}, {"@timestamp"=>2024-05-19T19:27:37.090273440Z, "data_stream"=>{"namespace"=>"nginx", "type"=>"logs", "dataset"=>"pfelk"}, "tags"=>["pfelk", "nginx", "GeoIP_Source"], "type"=>"firewall", "host"=>{"ip"=>"192.168.1.1"}, "log"=>{"syslog"=>{"hostname"=>"opnsense.buckeyes1995.com", "facility"=>{"code"=>16, "name"=>"local0"}, "priority"=>134, "appname"=>"nginx", "severity"=>{"code"=>6, "name"=>"Informational"}}}, "service"=>{"type"=>"system"}, "timestamp"=>"19/May/2024:14:27:37 -0500", "nginx"=>{"access"=>{"body_sent"=>{"bytes"=>"0"}, "url"=>"/api/server-info/features", "method"=>"GET", "referrer"=>"https://immich.buckeyes1995.com/auth/login", "http_version"=>"2.0", "agent"=>"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1", "response_code"=>"304", "user_name"=>"-"}}, "event"=>{"created"=>2024-05-19T14:27:37.000Z, "dataset"=>"pfelk.nginx", "original"=>"<134>May 19 14:27:37 opnsense.buckeyes1995.com nginx: 172.68.27.103 - - [19/May/2024:14:27:37 -0500] \"GET /api/server-info/features HTTP/2.0\" 304 0 \"https://immich.buckeyes1995.com/auth/login\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1\" \"2600:387:15:3313::4\" \"immich.buckeyes1995.com\" sn=\"immich.buckeyes1995.com\" rt=0.008 ua=\"192.168.40.5:2283\" us=\"304\" ut=\"0.007\" ul=\"0\" cs=-"}, "@version"=>"1", "client"=>{"ip"=>"172.68.27.103", "geo"=>{"country_iso_code"=>"US", "postal_code"=>"75270", "country_name"=>"United States", "region_iso_code"=>"US-TX", "timezone"=>"America/Chicago", "location"=>{"lon"=>-96.8022, "lat"=>32.7797}, "continent_code"=>"NA", "region_name"=>"Texas", "city_name"=>"Dallas"}, "as"=>{"number"=>13335, "organization"=>{"name"=>"CLOUDFLARENET"}}, "mmdb"=>{"dma_code"=>623}}}], :response=>{"create"=>{"status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index [logs-pfelk-nginx] and composable template [*-pfelk-nginx*] forbids index auto creation", "index_uuid"=>"_na_", "index"=>"logs-pfelk-nginx"}}}}

[Buckeyes1995]

I reverted back the .grok file. Doing some google searching, I set the 'Auto Create' option in the NGINX template to true, but still having issues.. all of the dashboard windows state 'failed to find geo field [client.geo.location.'

[a3ilson]

Are you able to see any logs via Discover? If so can you provide the 'event.original' value(s)?

[Buckeyes1995]

I’m not sure what fixed it.. I reinstalled the template and it didn’t work.. comeback the next day and it’s working.. very odd. The only window that isn’t showing anything is the map.. but that may be due to lack of access to the server? I see the whole world map, but no statistics or sources on it.

[Buckeyes1995]

Sorry to use this older reply, but noticed that my pfelk filled up all my disk space.. turned out it was an 8 GB indice… I removed it, and now the firewall log isn’t working.. do I just need to reinstall the template?

Any idea why it would be filing up so much?

thanks.

On May 21, 2024, at 6:28 PM, a3ilson @.***> wrote:

Are you able to see any logs via Discover? If so can you provide the 'event.original' value(s)?

— Reply to this email directly, view it on GitHub https://github.com/pfelk/pfelk/issues/537#issuecomment-2123589061, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIMEBXDQ6VQ7SRT42KBVQV3ZDPKCDAVCNFSM6AAAAABH42CQPKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGU4DSMBWGE. You are receiving this because you were mentioned.

[a3ilson]

Updated the index template...navigate to Stack Management>> Index Management>>Templates and delete the pfelk-nginx template. Next, reinstall the pfelk-nginx index via step 2, Templates here

However, I doubt it'll fix/resolve the map visualization and will need someone willing to update the dashboard as I do not have haproxy logs.

Regarding the space filling up...I fixed one contributor to the excessive storage sizes within #527 but the amount of storage will vary based on logs and your system(s). I'd recommend updating the ILM to best meet your needs (e.g., Stack Management>>Index Lifecycle Management>>pfelk

[BeNeDeLuX]

@Buckeyes1995 Maybe I can help out here :-) I created the Nginx dashboard a few years ago. Looks like it needs a few tweaks. As for your main problem, I see a filter in your screenshot that is causing this error. Please remove this filter, otherwise you won't see the data in the other fields.

I will do a fresh installation with latest pfelk version and recheck the dashboard and the incoming data. Just give me some time ;-)

[BeNeDeLuX]

I can confirm the problem with the missing data inside the World Map. All other Dashboard-Fields are fine so far by removing that filter mention in the last screenshot.

I did a quick recheck and it looks like that there is no more a GeoPoint field filled with data. GeoData out of the IP is generated fine, but there is no GeoPoint field that i can select for a map overlay.

There are also many fields empty. May it has something todo with the renaming of fields in that change -> https://github.com/pfelk/pfelk/commit/ed2100d7542f73c0bf5668e4532db00ad4853a56?diff=split&w=0

[a3ilson]

@BeNeDeLuX & @Buckeyes1995

Update to the latest nginx mappings here which adds mappings to the client. fields

[BeNeDeLuX]

Hi @a3ilson,

thanks for your quick adjustment :+1: I've imported the new mapping and deleted the old nginx indexes.

Here are the current fields:

There exists still two GeoPoint Fields, but both are empty:

Is there anything that i can check or change on my end ? Thank you very much!

[a3ilson]

@BeNeDeLuX - I apologize...I had an oversight with the last revision and omitted the *.geo.*nested object. Please update the "index_template_pfelk-nginx" and try again.

Based you the provided screenshots (thank you), the geo points should be stored correctly for mapping.

[BeNeDeLuX]

@a3ilson The new mapping is working fine now, thank you. :+1:

I updated the NGINX dashboard. Pull request with new filename exists.

@Buckeyes1995 Please update to latest index template for the nginx data and import the new NGINX Dashboard.

[a3ilson]

@BeNeDeLuX - Thank you! 💥