ytti
Fortigate Backups Timing Out
Hi All
Sorry if this isn't the right place to post something I am struggling with. I will promptly remove it if it isn't.
I have a few questions as I must be missing something.
I struggled with this in my test deployment but eventually got it to work after a load of fiddling where I increased the timeout and tried different hostnames for the device in the router.db file and on the gui.
However since I have redeployed the system to a production variant, I am once again getting the timeouts. I understand this is due to the size of the fortigates configs but I am not sure what to try as all the fiddling I have done to try get it to work hasn’t helped.
Nov 23 11:00:17 oxidized oxidized[1919200]: W, [2023-11-23T11:00:17.304632 #1919200] WARN – : /FORTIGATE status no_connection, retry attempt 1 Nov 23 11:00:17 oxidized oxidized[1919200]: W, [2023-11-23T11:00:17.304787 #1919200] WARN – : /FORTIGATE status no_connection, retry attempt 1 Nov 23 11:10:20 oxidized oxidized[1919200]: W, [2023-11-23T11:10:20.359146 #1919200] WARN – : FORTIGATE raised Timeout::Error with msg “execution expired” Nov 23 11:10:20 oxidized oxidized[1919200]: W, [2023-11-23T11:10:20.359361 #1919200] WARN – : FORTIGATE raised Timeout::Error with msg “execution expired” Nov 23 11:10:20 oxidized oxidized[1919200]: W, [2023-11-23T11:10:20.656281 #1919200] WARN – : /FORTIGATE status no_connection, retries exhausted, giving up Nov 23 11:10:20 oxidized oxidized[1919200]: W, [2023-11-23T11:10:20.656635 #1919200] WARN – : /FORTIGATE status no_connection, retries exhausted, giving up
oxidized@dc1-oxidized:~$ cat .config/oxidized/config username: username password: password resolve_dns: true interval: 3600 use_syslog: false debug: false threads: 30 use_max_threads: true timeout: 600 retries: 1 prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/ rest: 0.0.0.0:8888 next_adds_job: false
The above is the top part of the config file. I know the fortigate backups work as I have smaller fortigates that backup just fine with no issues at all.
Any help would be great.
Regards, Josh
Hi Josh!
Have you tried applying the oxidized debug configuration?
debug: true
This way we can better analyze what is happening when oxidized tries to connect to your Fortigate.
Thanks for the quick response!
I have done so, I can see that the SSH appears to get in and sends through the commands to get the configs out. But after that there are just loads of "2 jobs running in parallel", obviously thats the two fortigates that haven't yet succeeded.
Nov 23 13:24:53 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:53.396631 #2017283] DEBUG -- : resolving DNS for FORTIGATE... Nov 23 13:24:57 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:57.251800 #2017283] DEBUG -- : lib/oxidized/model/model.rb Executing show inventory | no-more Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.765683 #2017283] DEBUG -- : lib/oxidized/worker.rb: Added /FORTIGATE to the job queue Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.766136 #2017283] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for FORTIGATE at 2023-11-23 13:24:59 UTC Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.767265 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to FORTIGATE Nov 23 13:24:59 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:24:59.912108 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/] at FORTIGATE Nov 23 13:25:00 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:00.730706 #2017283] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login commands at FORTIGATE Nov 23 13:25:00 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:00.730926 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb get system status @ FORTIGATE with expect: /^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/ Nov 23 13:25:00 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:00.731222 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/] at FORTIGATE Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.148704 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb config global @ FORTIGATE with expect: /^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/ Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.148895 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/] at FORTIGATE Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.551642 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb get system ha status @ FORTIGATE with expect: /^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/ Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.551898 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/] at FORTIGATE Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.957126 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb get hardware status @ FORTIGATE with expect: /^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/ Nov 23 13:25:01 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:01.957359 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/] at FORTIGATE Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.362177 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb end @ FORTIGATE with expect: /^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/ Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.362567 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/] at FORTIGATE Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.796538 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb show full-configuration | grep . @ FORTIGATE with expect: /^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/ Nov 23 13:25:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:25:02.796691 #2017283] DEBUG -- : lib/oxidized/input/ssh.rb: expecting [/^([-\w.~]+(\s[(\w-.)]+)?~?\s?[#>$]\s?)$/] at FORTIGATE
Is there perhaps a better spot in the config to put the debug that would give me a better idea of what is failing, because this seems to suggest the login is working and its simply not reading the returned data.
I have also tested the regex and it seems to be fine as well.
Nov 23 13:35:02 dc1-oxidized oxidized[2017283]: D, [2023-11-23T13:35:02.972196 #2017283] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH failed for FORTIGATE
The above would seem to suggest that its SSH related. But I have tested SSH using the creds in the configs as well as the fortigates that are working have the same creds configured. Is it timing out perhaps because it cannot get the expected data back in time?
EDIT: in the .config/oxidized/logs/ dir I can see the IP of the fortigates with configs in them. Is this perhaps something else?
Thanks in advance.
If you open the file located at .config/oxidized/logs/{Fortigate-IP}-ssh you can see where exactly oxidized is "hanging". In my opinion, it presents a better visualization than the oxidized service logs.
If possible, could you share the result here? Of course, not displaying your Fortigate's settings, only the commands executed, and error messages, if any.
So I have actually had this in my first deployment as well, in that case I removed the file, removed the device in Libre and readded everything and thats when it started to work (in my testing I mean). So I removed the files this time again to try that again, unfortunately no such luck this time.
I did get an output of the files before I removed them though. It appears that its getting to the last command and exiting.
Last entry in the log file is:
FORTIGATE $ exit
Currently trying to get it to regenerate the files, but yet to have any luck. Do you think its worth making the timeout something silly high like 20-30 mins?
So I suspect I have semi figured out the issue but some clarification for it would be nice.
Essentially what seems to be the issue is that each of the larger config dumps takes roughly 6 minutes to be processed.
I have been testing with two Fortigates configured and when I set it down to just the one it work and the configs are taken (as I mentioned the completion for the Fortigate is normally 6 minutes into the run).
I increased the overall timeout for run to 960 seconds and added the second Fortigate back. As expected at roughly 12 minutes, both Fortigates configs complete successfully. I tried changing the threads to 60 from 30 assuming that perhaps it was taking to long to run each scan. Is there another setting I should be looking to change to get this down? I have more Fortigates that have about the same amount of lines in config, so making the timeout close to 30 minutes seems a little high. Unless this is expected behaviour?
Any info to clarify would be great!
