Issue with snmp-generator with 2 APC Mibs

[33Fraise33]

Host operating system: output of uname -a

Linux billy 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

snmp_exporter version: output of snmp_exporter -version

latest docker version (0.21.0)

What device/snmpwalk OID are you using?

APC PDU: https://www.se.com/us/en/product/APDU10250SM OID: pduInputPhaseStatusTable # 1.3.6.1.4.1.318.1.1.32.3.4

If this is a new device, please link to the MIB(s).

apc_cpdu_v1_3.mib.txt (only availble from the device itself: https://www.se.com/us/en/product/APDU10250SM) powernet446.mib.txt (available for download here: https://www.se.com/be/nl/download/document/APC_POWERNETMIB_446_EN/)

What did you do that produced an error?

Running a generator.yml with the following content gives an error:

  apc:
    walk:
      -  sPDUMasterConfigPDUName #  .1.3.6.1.4.1.318.1.1.4.3.3 => the name of the PDU
      -  sPDUIdentModelNumber #  .1.3.6.1.4.1.318.1.1.4.1.4 => the model number
      #-  sPDUIdentSerialNumber #  .1.3.6.1.4.1.318.1.1.4.1.5 => serial number
      #-  sPDUIdentHardwareRev #  .1.3.6.1.4.1.318.1.1.4.1.1 => hardware revision
      #-  sPDUIdentFirmwareRev #  .1.3.6.1.4.1.318.1.1.4.1.2 => firmware revision
      -  sPDUMasterState #  .1.3.6.1.4.1.318.1.1.4.2.2 => status of all outlets
      -  sPDUOutletCtl #  .1.3.6.1.4.1.318.1.1.4.4.2.1.3 => the outlet state
        # => 1:On; 2:Off; 3:Reboot; 4:Unknown; 5:OnWithDelay; 6:OffWithDelay; 7:RebootWithDelay
      -  sPDUOutletCtlName #  .1.3.6.1.4.1.318.1.1.4.4.2.1.4 => outlet name
      -  rPDULoadDevMaxBankLoad #  .1.3.6.1.4.1.318.1.1.12.2.1.3 => Bank maximum rated power
      -  rPDULoadDevNumBanks #  .1.3.6.1.4.1.318.1.1.12.2.1.4 => number of Banks
      -  rPDULoadDevBankMaxLoad #  .1.3.6.1.4.1.318.1.1.12.2.1.6.1.3 => maximum rated power for each Bank
      -  rPDUStatusBankState #  .1.3.6.1.4.1.318.1.1.12.5.2.1.3 => bank load state
        # => 1:BankLoadNormal; 2:BankLoadLow; 3:BankLoadNearOverload; 4:BankLoadOverload
      -  rPDULoadStatusLoad #  .1.3.6.1.4.1.318.1.1.12.2.3.1.1.2 => phase/bank load measured in tenths of Amps
      -  rPDULoadStatusBankNumber #  .1.3.6.1.4.1.318.1.1.12.2.3.1.1.5
        # => The bank number to which this record refers.
        # A value of 0 will be returned if any bank is not present or if it is phase related
      -  rPDUIdentDevicePowerWatts #  .1.3.6.1.4.1.318.1.1.12.1.16 => power in Watts
      -  rPDUIdentDevicePowerVA #  .1.3.6.1.4.1.318.1.1.12.1.18 => power in VA
    version: 1
    auth:
      community: my-community
  
  apc_10000:
    walk:
      -  pduInputPhaseStatusTable # 1.3.6.1.4.1.318.1.1.32.3.4
    version: 2
    auth:
      community: my-community

What did you expect to see?

Correct run of generator

What did you see instead?

level=info ts=2023-04-13T13:33:44.624Z caller=main.go:52 msg="Generating config for module" module=apc_10000
level=error ts=2023-04-13T13:33:44.650Z caller=main.go:130 msg="Error generating config netsnmp" err="cannot find oid 'pduInputPhaseStatusTable' to walk"

[dswarbrick]

I see pduInputPhaseStatusTable defined in your attached apc_cpdu_v1_3.mib.txt, so the most likely reason you are getting that error is that you have not configured netsnmp how to find that MIB on your system.

pduInputPhaseStatusTable  OBJECT-TYPE
        SYNTAX SEQUENCE OF PduInputPhaseStatusEntry
        MAX-ACCESS      not-accessible
        STATUS  current
        DESCRIPTION 
                "A table of PDU Input Phase status data"
        ::= { pduInputPhase 4 }

Are you able to walk that table with snmpwalk?

[33Fraise33]

I see pduInputPhaseStatusTable defined in your attached apc_cpdu_v1_3.mib.txt, so the most likely reason you are getting that error is that you have not configured netsnmp how to find that MIB on your system.

pduInputPhaseStatusTable  OBJECT-TYPE
        SYNTAX SEQUENCE OF PduInputPhaseStatusEntry
        MAX-ACCESS      not-accessible
        STATUS  current
        DESCRIPTION 
                "A table of PDU Input Phase status data"
        ::= { pduInputPhase 4 }

Are you able to walk that table with snmpwalk?

It is able to reach the mibs directory as other mibs are being read without issue:

snmp_generator:
    image: prom/snmp-generator:latest
    volumes:
       - $PWD/snmp-exporter:/opt

$ tree snmp-exporter
snmp-exporter
├── generator.yml
└── mibs
    ├── AGENTX-MIB.txt
    ├── BGP4-MIB.txt
    ├── BRIDGE-MIB.txt
    ├── CELLULAR
    ├── CUMULUS-BGPUN-MIB.txt
    ├── CUMULUS-SNMP-MIB.txt
    ├── DEVICE
    ..........
    ├── apc_cpdu_v1_3.mib.txt
    └── powernet446.mib.txt
2 directories, 109 files

[33Fraise33]

I created a minimal testcase here: https://github.com/33Fraise33/snmp-generator-issue

this gives the following error:

$ docker-compose up
[+] Running 1/0
 ⠿ Container test-snmp_generator-1  Created                                                                                                                                           0.0s
Attaching to test-snmp_generator-1
test-snmp_generator-1  | ts=2023-04-14T08:47:02.414Z caller=net_snmp.go:161 level=info msg="Loading MIBs" from=mibs
test-snmp_generator-1  | ts=2023-04-14T08:47:02.566Z caller=main.go:119 level=warn msg="NetSNMP reported parse error(s)" errors=1
test-snmp_generator-1  | ts=2023-04-14T08:47:02.681Z caller=main.go:51 level=info msg="Generating config for module" module=apc
test-snmp_generator-1  | ts=2023-04-14T08:47:02.703Z caller=main.go:66 level=info msg="Generated metrics" module=apc metrics=1
test-snmp_generator-1  | ts=2023-04-14T08:47:02.704Z caller=main.go:51 level=info msg="Generating config for module" module=apc_10000
test-snmp_generator-1  | ts=2023-04-14T08:47:02.724Z caller=main.go:129 level=error msg="Error generating config netsnmp" err="cannot find oid 'pduInputPhaseStatusTable' to walk"
test-snmp_generator-1 exited with code 1

$ docker run --platform linux/amd64 --restart unless-stopped -v $PWD:/opt prom/snmp-generator parse_errors
ts=2023-04-14T08:47:16.185Z caller=net_snmp.go:161 level=info msg="Loading MIBs" from=mibs
ts=2023-04-14T08:47:16.353Z caller=main.go:119 level=warn msg="NetSNMP reported parse error(s)" errors=1
MIB search path: mibs

[dswarbrick]

I am able to reproduce the generator error with an extremely bare-bones generator.yml:

modules:
  apc_10000:
    walk:
      - pduInputPhaseStatusTable

The error occurs when both apc_cpdu_v1_3.mib.txt and powernet446.mib.txt are present in my mibs dir. If I remove the powernet446.mib.txt mib however, the generator succeeds:

ts=2023-04-14T12:59:56.477Z caller=net_snmp.go:161 level=info msg="Loading MIBs" from=$HOME/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf
ts=2023-04-14T12:59:56.720Z caller=main.go:51 level=info msg="Generating config for module" module=apc_10000
ts=2023-04-14T12:59:56.739Z caller=main.go:66 level=info msg="Generated metrics" module=apc_10000 metrics=9
ts=2023-04-14T12:59:56.739Z caller=main.go:91 level=info msg="Config written" file=/tmp/snmp/snmp.yml

This obviously smells like some kind of conflict between those two MIBs - possibly an object name clash, or even re-use of OIDs. I haven't investigated deeper yet.

[SuperQ]

Yea, it's a common conflict when two MIBs define the same OID names.

The only workaround is to specify the number OID instead of the name.

[dswarbrick]

Aha... well, it's as simple as this: both MIBs are trying to declare the same PowerNet-MIB module. They happily coexist and the generator succeeds if I change the module name of one of them. But this is obviously a hack, and one has to ask oneself which MIB is the "official" PowerNet-MIB.

Both MIBs define objects up to cPDU (1.3.6.1.4.1.318.1.1.32), but that's where the branch ends in powernet446.mib.txt. Only apc_cpdu_v1_3.mib.txt defines further child objects of that node, hence why you need that additional MIB.

You could try performing some MIB surgery to graft extra object definitions into powernet446.mib.txt, but this is really probably something that you should report to the vendor and get them to fix it properly.

[33Fraise33]

Yes the powernet mib module declares items for rpdu, spdu,... and it includes the toplevel of cpdu. apc_cpdu_v1_3.mib.txt defines the subtree of cpdu. What changes should be made to cpdu or powernet for it to work completely? I think it will be faster for me to edit it myself as getting SE/APC to edit the mib will be slow I guess.

changing the mib definition from PowerNet-MIB DEFINITIONS ::= BEGIN to cPDU-MIB DEFINITIONS ::= BEGIN fixes it.

Thanks all!

[dswarbrick]

Believe me when I say that hand-editing MIBs is seriously not fun.

The only objects which technically clash (redefined) in apc_cpdu_v1_3.mib.txt are:

apc					OBJECT IDENTIFIER 		::= { enterprises 318 }

products				OBJECT IDENTIFIER 		::= { apc 1 }

hardware				OBJECT IDENTIFIER 		::= { products 1 }

cPDU					OBJECT IDENTIFIER 		::= { hardware 32 }

Since these don't diverge from the OIDs in powernet446.mib.txt, you should be fine with just changing the module name in apc_cpdu_v1_3.mib.txt. Everything below that (i.e., below the cPDU branch) is unique.

I think it would still be worth emailing the vendor to ask them why these are omitted from the main PowerNet-MIB.

[aaron-asdf]

Yea, it's a common conflict when two MIBs define the same OID names.

The only workaround is to specify the number OID instead of the name.

Is this really the only workaround? I ran into an issue with duplicate MIB names for different OIDs. I was hoping that I could simply reference the MIB module name within the generator file to remove the ambiguity (something like MIB-MODULE::SpecificMIB), which is the standard for mibs but doesn't seem to work for the generator. Is there a format that is supported for the generator to do this, or only solution is to either edit the MIB file (to keep human readable names in the prometheus data) or to put in an OID (if you are ok with hard to read data)?

[SuperQ]

The other workaround is to have separate generator files and combinations of MIBDIRS. I filed an issue to support MIBDIRS as part of the generator config, but for now it needs to be separate files.

We also just released the ability to pass multiple snmp.yml config files to the exporter, so it's less cumbersome to generate and use configs.

[bang-uin]

I have a similar problem over here.

snmpwalk is able to walk the table:

snmpwalk -v1 -M mibs -c public 10.0.0.1  1.3.6.1.4.1.2254.2.4.1
iso.3.6.1.4.1.2254.2.4.1.1.0 = STRING: "Delta"
iso.3.6.1.4.1.2254.2.4.1.2.0 = STRING: "UPS103R2RT2N035 "
iso.3.6.1.4.1.2254.2.4.1.3.0 = STRING: "0F0011BR00.02.03"
iso.3.6.1.4.1.2254.2.4.1.4.0 = STRING: "02.01.15"
iso.3.6.1.4.1.2254.2.4.1.5.0 = STRING: "InsightPower"
iso.3.6.1.4.1.2254.2.4.1.6.0 = ""
iso.3.6.1.4.1.2254.2.4.1.7.0 = INTEGER: 10000
iso.3.6.1.4.1.2254.2.4.1.8.0 = INTEGER: 230
iso.3.6.1.4.1.2254.2.4.1.9.0 = INTEGER: 500
iso.3.6.1.4.1.2254.2.4.1.10.0 = INTEGER: 230
iso.3.6.1.4.1.2254.2.4.1.11.0 = INTEGER: 500
iso.3.6.1.4.1.2254.2.4.1.12.0 = INTEGER: 240
iso.3.6.1.4.1.2254.2.4.1.13.0 = INTEGER: 0
iso.3.6.1.4.1.2254.2.4.1.14.0 = INTEGER: 0
iso.3.6.1.4.1.2254.2.4.1.15.0 = INTEGER: 0
iso.3.6.1.4.1.2254.2.4.1.16.0 = INTEGER: 0
iso.3.6.1.4.1.2254.2.4.1.17.0 = INTEGER: 0
iso.3.6.1.4.1.2254.2.4.1.18.0 = INTEGER: 1
iso.3.6.1.4.1.2254.2.4.1.19.0 = INTEGER: 1

But the generator fails to find the OID:

---
auths:
  public_v1:
    version: 1
  public_v2:
    version: 2

modules:
  delta_ups:
    walk:
      - 1.3.6.1.4.1.2254.2.4.1       # dupsIdent
      - 1.3.6.1.4.1.2254.2.4.4       # dupsInput
      - 1.3.6.1.4.1.2254.2.4.5       # dupsOutput
      - 1.3.6.1.4.1.2254.2.4.6       # dupsBypass
      - 1.3.6.1.4.1.2254.2.4.7       # dupsBattery
      - 1.3.6.1.4.1.2254.2.4.8       # dupsTest
      - 1.3.6.1.4.1.2254.2.4.9       # dupsAlarm

# make generate
MIBDIRS='mibs' ./generator --fail-on-parse-errors generate
ts=2023-09-05T14:42:59.918Z caller=net_snmp.go:162 level=info msg="Loading MIBs" from=mibs
ts=2023-09-05T14:43:00.049Z caller=main.go:53 level=info msg="Generating config for module" module=delta_ups
ts=2023-09-05T14:43:00.063Z caller=main.go:132 level=error msg="Error generating config netsnmp" err="cannot find oid '1.3.6.1.4.1.2254.2.4.1' to walk"
make: *** [Makefile:75: generate] Error 1

The mib is in snmp_exporter/generator/mibs alongside the automatically downloaded mibs.

UPSv4.mib.zip

[lanrat]

For anyone else who runs into this issue. I was able to craft a generator.yml that works with the mibs linked above. And to save you some time, I also included the snmp-apcpdu.yml generated output as well.

https://gist.github.com/lanrat/8d10c164c0aa9d78ef73d4b317ed054c

[ym]

I have a similar problem but the cause is different, net-snmp wasn't able to import RFC1155-SMI when we use a custom MIB directory. After downloading https://github.com/haad/net-snmp/tree/master/mibs and putting it in the mibs directory, it works.

[SuperQ]

This seems solved.