Fortigate config backup failing

[LoveSkylark]

I am trying to backup my FortiGate Firewalls and I keep getting the error:

│ W, [2024-07-05T11:01:20.253418 #33] WARN -- : x.x.x.x raised Timeout::Error with msg "execution expired"

I'm successfully backing up 100+ other devices from different vendors. It is large config and it takes few minutes to scroll though the CLI if I run 'show | grep .' manually, I have seen previous posts about similar issues and Have tried those solutions.

I changed the global timeout to 600 and I have also tried:

models:
  fortios:
    timeout: 600
    vars:
      fullconfig: false

Problem is I see this time out error 1-2 minutes after the server has started so there is no chance that 10 minutes have passed.

This is how my config file looks like:

    model: cisco
    resolve_dns: true
    interval: 72000
    use_syslog: false
    threads: 30
    use_max_threads: false
    timeout: 600
    retries: 3
    prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
    rest: 0.0.0.0:8888
    next_adds_job: false
    vars: {}
    groups:
      core:
        username: xxxx
        password: yyyy
    models:
      fortios:
        timeout: 600
        vars:
          fullconfig: false
    model_map:
      juniper: junos
      cisco: ios
      cisco: iosxe
      mikrotik: routeros
      ubiquiti: airos
      fortigate: fortios
      fortiswitch: fortios
    pid: "/home/oxidized/.config/oxidized/pid"
    crash:
      directory: "/home/oxidized/.config/oxidized/crashes"
      hostnames: false
    stats:
      history_size: 10
    input:
      default: ssh, telnet
      ssh:
        secure: false
      ftp:
        passive: true
      utf8_encoded: true
    output:
      default: git
      git:
          user: Oxidized
          email: [email protected]
          repo: "/home/oxidized/.config/oxidized/configs"
    source:
      default: http

Any help/suggestion on what I may be doing wrong is greatly appreciated.

[robertcheramy]

Which version of Oxidized are you using? 0.30.1 introduced a fix for this - not doing a full config (with all defaults).

[LoveSkylark]

I'm using the latest docker image oxidized/oxidized:latest, downloaded 3 days ago witch would make it: 0.30.1-113-g1269d04.

And yes I think I read your ticket about the "nofull config" and I looked through the "fortios.rb" file and ran each command individual over SSH from the same server, I tried both 'show full-configuration | grep .' and 'show | grep .' over the SSH without any problems, but the screen ran for a long time (have 40+ vDoms) but everyhting ran fine in SSH.

As you can see from above I even tried to disable full backup manually in the config but I suspect this is not the issue because the error I get is happening around 2 minutes after the server comes up, long before the "timeout 600" is in effect (unless my config file is somehow incorrect).

[robertcheramy]

I'm sorry to read this. I've no way to reproduce the behavior as I don't have a fortigate with 40+ vDoms. => Setting the flag help_wanted.

[KevinTang8888]

what's the lg message with following CLI?

journalctl -u oxidized.service

[iRomanyshyn]

Hi guys! I think it's not good to do only show instead of show full-configuration as it makes a much smaller file and drops all users configured including any external users. And if someone restores that config the router will be inaccessible on any port due to missing "secrets" I think. I mean it cannot enable HTTPS, SSH, etc. on any interface as there are no certificates in the configuration imported.

[KevinTang8888]

Hi guys! I think it's not good to do only show instead of show full-configuration as it makes a much smaller file and drops all users configured including any external users. And if someone restores that config the router will be inaccessible on any port due to missing "secrets" I think. I mean it cannot enable HTTPS, SSH, etc. on any interface as there are no certificates in the configuration imported.

Have you tried to restore the oxidized backuped config to a new FortiGate device?

[robertcheramy]

Hi guys! I think it's not good to do only show instead of show full-configuration as it makes a much smaller file and drops all users configured including any external users. And if someone restores that config the router will be inaccessible on any port due to missing "secrets" I think. I mean it cannot enable HTTPS, SSH, etc. on any interface as there are no certificates in the configuration imported.

This has been fixed and will be included in Oxidized 0.31: https://github.com/ytti/oxidized/blob/master/docs/Model-Notes/FortiOS.md#config-vs-full-config

The issue here is that oxidized gets timeouts on fortigate, even when the getting the configuration without defaults. If you have an unrelated issue, please open a separate issue.

[iRomanyshyn]

Hi guys! I think it's not good to do only show instead of show full-configuration as it makes a much smaller file and drops all users configured including any external users. And if someone restores that config the router will be inaccessible on any port due to missing "secrets" I think. I mean it cannot enable HTTPS, SSH, etc. on any interface as there are no certificates in the configuration imported.

Have you tried to restore the oxidized backuped config to a new FortiGate device?

Yes, I did.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity.

[YoucefYousfi]

I think adding a custom timeout for FortiOS specifically on the config file is required. In my case, it times out even with the short configuration on Fortigates due to the size of the configuration with a small timeout value of 60. If I increase it, I am able to fetch the configuration but at the cost of slowing down everything else when any other devices are not properly backed up.

[robertcheramy]

Do you get timeouts when running device2yaml.rb (from the latest git) on the device?

extra/device2yaml.rb user@host -o fortigate.yaml -c "get system status
get system ha status
get hardware status
show | grep .
"

If not, submit your yaml simulation file, so that we can analyse the problem.

[LoveSkylark]

looking at this from the FGT side it looks like the execution just halts after running the:

cfg << cmd('config global') if @vdom_enabled
cfg << cmd('end') if @vdom_enabled

FortiGate diag:

SSH: lastlog_openseek: Couldn't stat /var/log/lastlog: No such file or directory
cmd=config global
end

SSH: channel 0: rcvd adjust 131072
SSH: channel 0: request [email protected] confirm 1
SSH: channel 0: rcvd adjust 131072
SSH: channel 0: rcvd adjust 131072
SSH: channel 0: rcvd adjust 131072
SSH: channel 0: rcvd adjust 131072

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity.