OpnSense 22.1 Issue

[dkowis]

https://opnsense.org/opnsense-22-1-released/

Has anyone had experience with it yet? I haven't gotten around to it and probably won't have time for a while. Starting an issue thread to keep track of it.

EDIT: reported working well with wpa_supplicant

EDIT MORE: https://github.com/MonkWho/pfatt/issues/65#issuecomment-1043984610 seems to be the victorious solution that covers both WPA and Tethered operating modes.

[ehassett]

I upgraded to 22.1 without even recalling that I had this script running. Haven't been able to get connection through the gateway since the upgrade. I've verified that the upgrade didn't overwrite/remove the script from /usr/local/etc/rc.syshook.d/early/. The opnatt log is also showing as everything as having been setup properly. Investigating more now as I'm not sure what would have caused the break with the update.

[ehassett]

Looks like DHCP traffic is not being bridged between ngeth0 and $ONT_IF now. Will update as I troubleshoot more...

[smplman]

I tried my hand at this over the weekend with trying to use supplicant mode. When I reboot the system hangs at waiting EAP for authorization... which is in this while true loop here https://github.com/MonkWho/pfatt/blob/supplicant_OPNsense_testing/bin/opnatt.sh#L212. I'm testing on the supplicant_OPNsense_testing branch. I'm trying to get some insight and logging to see if I can figure out what's going on.

For some reason none of my logs are being output to /var/log/opnatt.log or /var/log/pfatt.log. I'm no BSD expert, but /usr/bin/logger -st "pfatt" "starting pfatt..." seems like it would log straight to STDOUT.

Edit: I was able to get some better logging through the web ui. It appears something with dhclient is causing issues. I'm going to try running through the commands in the script one by one to see if I can track down what is going on.

2022-01-31T01:50:02	Error	opnsense	/usr/local/etc/rc.linkup: DEVD: Ethernet attached event for dynamic wan(ngeth0)	
2022-01-31T01:50:02	Notice	pfatt	waiting EAP for authorization...	
2022-01-31T01:50:02	Notice	pfatt	setting wpa_supplicant network configuration...	
2022-01-31T01:50:02	Notice	pfatt	wpa_supplicant running on PID 35917...	
2022-01-31T01:50:02	Notice	pfatt	starting wpa_supplicant...	
2022-01-31T01:50:02	Notice	pfatt	enabling promisc for em1...	
2022-01-31T01:50:02	Notice	pfatt	creating vlan node and ngeth0 interface...	
2022-01-31T01:50:02	Notice	pfatt	ONT---[] [em1]	
2022-01-31T01:50:02	Notice	pfatt	cabling should look like this:	
2022-01-31T01:50:02	Notice	pfatt	configuring EAP environment for supplicant mode...	
2022-01-31T01:50:00	Error	dhcp6c	invalid interface(ngeth0): Device not configured	
2022-01-31T01:50:00	Notice	opnsense	plugins_configure dhcp (execute task : dhcpd_dhcp_configure(,inet6,Array))	
2022-01-31T01:50:00	Notice	opnsense	plugins_configure dhcp (,inet6,Array)	
2022-01-31T01:50:00	Error	opnsense	/usr/local/etc/rc.linkup: DEVD: Ethernet detached event for dynamic wan(ngeth0)	
2022-01-31T01:50:00	Critical	dhclient	exiting.	
2022-01-31T01:50:00	Error	dhclient	connection closed	
2022-01-31T01:50:00	Critical	dhclient	exiting.	
2022-01-31T01:50:00	Error	dhclient	No live interfaces to poll on - exiting.	
2022-01-31T01:50:00	Error	dhclient	Interface ngeth0 no longer appears valid.	
2022-01-31T01:50:00	Error	dhclient	ioctl(SIOCGIFFLAGS) on ngeth0: Operation not permitted	
2022-01-31T01:50:00	Error	dhclient	receive_packet failed on ngeth0: Device not configured	
2022-01-31T01:50:00	Notice	pfatt	resetting netgraph...	
2022-01-31T01:50:00	Notice	pfatt	EAP_BRIDGE_5268AC = 0	
2022-01-31T01:50:00	Notice	pfatt	EAP_BRIDGE_IF = xx1	
2022-01-31T01:50:00	Notice	pfatt	EAP_SUPPLICANT_IDENTITY = (redacted)
2022-01-31T01:50:00	Notice	pfatt	EAP_MODE = supplicant	
2022-01-31T01:50:00	Notice	pfatt	RG_ETHER_ADDR = (redacted)	
2022-01-31T01:50:00	Notice	pfatt	ONT_IF = em1	
2022-01-31T01:50:00	Notice	pfatt	configuration:	
2022-01-31T01:50:00	Notice	pfatt	starting pfatt...	
2022-01-31T01:49:59	Notice	sudo	smplman : TTY=pts/0 ; PWD=/home/smplman ; USER=root ; COMMAND=/usr/local/etc/rc.syshook.d/early/99-opnatt

Edit Edit: I was able to get this working with the steps below. To me it seems like there is something going on with the order of operations of the script. After a reboot I still get stuck on waiting EAP for authorization.. and have to repeat the steps below. My /etc/wpa_supplicant.conf is the one that got generated when I extracted my certs from the device locksmith. So it seems like running wpa_supplicant directly instead of with wpa_cli is making some sort of difference?

Move WAN from ngeth0 to em1 in the web gui sudo /usr/sbin/wpa_supplicant -Dwired -iem1 -c/etc/wpa_supplicant.conf Move WAN from em1 to ngeth0 in the web gui stop wpa_supplicant with CTRL + C sudo /usr/sbin/wpa_supplicant -Dwired -ingeth0 -c/etc/wpa_supplicant.conf stop wpa_supplicant with CTRL + C

sudo /usr/local/etc/rc.syshook.d/early/99-opnatt IP4 and IP6 addresses acquired and speed is okay. I'm getting 400/400, but I think it's because my box is running on an old Intel Atom SuperMicro Intel(R) Atom(TM) CPU D525 @ 1.80GHz (2 cores, 4 threads). Might need to test out some better hardware. Nothing special was done for IP6, it was acquired about a minuet after the IP4

[zombielinux]

I'll have to give this a go. Glad to see its working.

If I recall, when I made my PR to go from wpa_cli to wpa_supplicant, there was some poor escaping or bash interpretation happening in the arguments list. Switching to the .conf file is far more supported anyways in the *nix community at large.

[dkowis]

I've got a non-wpa_supplicant setup, so I'm not sure those steps will apply to me. May be a reconfiguration of the interface did something, however.

[ehassett]

I have not been able to get any more figured out for non-wpa_supplicant setup, has anyone else made any progress?

[MrCaturdayNight]

I have not been able to get any more figured out for non-wpa_supplicant setup, has anyone else made any progress?

Not yet but I'm working on it too. I also tried converting to the wpa supplicant method and can't get it going either but that's true for me on 21.7. If I figure either out I'll report back.

[owenthewizard]

21.7 works for me with supplicant but 22.1 doesn't.

[Hou-dev]

I am also having problems with the latest version of OPNsense, I had to revert to 21.7.8

[dkowis]

Since we've got multiple methods of PFATT in this issue, could you please be specific as to which method you're using?

To summarize:

• 22.1 does not work with the "tethered" method, where the router remains on and connected.
• At least one report of 22.1 working with the wpa_supplicant method

[Hou-dev]

Since we've got multiple methods of PFATT in this issue, could you please be specific as to which method you're using?

To summarize:

• 22.1 does not work with the "tethered" method, where the router remains on and connected.
• At least one report of 22.1 working with the wpa_supplicant method

I am using the supplicant method. After the update the scripts hangs on "waiting on EAP for authorization". On occasion running the script manually worked. I decided to move back to the older release of OPNsense since if I had to reboot for any reason the script would not terminate unless it gets a IP address.

[zombielinux]

Are you using my pull request? Or the script in the repository? I had mine hang at the same place, but it was the wpa-cli code block that was failing. There was some bash interpretation and escaping that failed.

Try my pull request script and see if it works.

On Mon, Feb 7, 2022 at 19:19 Hou-dev @.***> wrote:

Since we've got multiple methods of PFATT in this issue, could you please be specific as to which method you're using?

To summarize:

• 22.1 does not work with the "tethered" method, where the router remains on and connected.
• At least one report of 22.1 working with the wpa_supplicant method

I am using the supplicant method. After the update the scripts hands on "waiting on EAP for authorization". On occasion running the script manually worked. I decided to move back to the older release of OPNsense since if I had to reboot for any reason the script would not terminate unless it gets a IP address.

— Reply to this email directly, view it on GitHub https://github.com/MonkWho/pfatt/issues/65#issuecomment-1032078001, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABN74JWQQTNWVJXMG26OALU2BOQ5ANCNFSM5NDL4CRQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

-- No trees were harmed in the sending of this message, but a rather large number of electrons were terribly inconvenienced.

[owenthewizard]

I tried the zombielinux fork and it had the same issue.

[smplman]

I'm running in supplicant mode with zombilunix's branch.

@zombielinux I also tried your script and it still hangs at waiting EAP for authorization.. I still have to do the dance of reassigning the WAN port and running wpa_supplicant manually. I did have a thought though that maybe the script is running before the interfaces are fully initialized? But that doesn't line up because if I run the script manually after breaking out and letting the system fully boot it hangs at the same spot.

I want to try updating the script to manually set the WAN back to em1, run wpa_supplicant, then once authenticated set the WAN back to ngeth0. I have had good luck getting wpa_supplicant to auth against em1 instead of ngeth0. Or maybe it's the other way around? I'm getting inconsistent results in my testing.

[dangeist]

I was able to get this working with @zombielinux's updates at the encouragement of @MrCaturdayNight I'm on OPNsense 21.7.8 on protectli hardware ( uses Intel(R) I210 gig copper ports for reference ). I tried running all the commands on the CLI one by one and kept getting erroneous behavior with ngctl, then I tried inserting the kernel module prior to that point in the script explicitly and it worked. So, this is NEARLY identical to the pfatt pull request above. Perhaps the new(er) kernels were removing unused modules (or similar). Interestingly, I noticed the dhcpclient had to iterate over a few retries before success. Not sure if that's relevant or if I simply hadn't seen it run before. Files included / attached for reference:

# cat /conf/pfatt/wpa/wpa_supplicant.conf
eapol_version=1
ap_scan=0
fast_reauth=1
network={
        ca_cert="/conf/pfatt/wpa/CA.pem"
        client_cert="/conf/pfatt/wpa/Client.pem"
        eap=TLS
        eapol_flags=0
        identity="C0:A0:0D:**:**:**"
        key_mgmt=IEEE8021X
        phase1="allow_canned_success=1"
        private_key="/conf/pfatt/wpa/PrivateKey.pem"

# cat /usr/local/etc/rc.syshook.d/early/99-opnatt-supplicant
#!/usr/bin/env sh
#Required Config
# ===============
ONT_IF="igb0"
RG_ETHER_ADDR="C0:A0:0D:**:**:**"
EAP_MODE="supplicant"

# Supplicant Config
# =================
EAP_SUPPLICANT_IDENTITY="C0:A0:0D:**:**:**"

##### DO NOT EDIT BELOW #################################################################################

/sbin/kldload -nq ng_ether

/usr/bin/logger -st "pfatt" "starting pfatt..."
/usr/bin/logger -st "pfatt" "configuration:"
/usr/bin/logger -st "pfatt" "  ONT_IF = $ONT_IF"
/usr/bin/logger -st "pfatt" "  RG_ETHER_ADDR = $RG_ETHER_ADDR"
/usr/bin/logger -st "pfatt" "  EAP_MODE = $EAP_MODE"
/usr/bin/logger -st "pfatt" "  EAP_SUPPLICANT_IDENTITY = $EAP_SUPPLICANT_IDENTITY"

/usr/bin/logger -st "pfatt" "resetting netgraph..."
/sbin/kldload -nq ng_ether
/usr/sbin/ngctl shutdown waneapfilter:
/usr/sbin/ngctl shutdown laneapfilter:
/usr/sbin/ngctl shutdown $ONT_IF:
/usr/sbin/ngctl shutdown o2m:
/usr/sbin/ngctl shutdown vlan0:
/usr/sbin/ngctl shutdown ngeth0:

/usr/bin/logger -st "pfatt" "configuring EAP environment for $EAP_MODE mode..."
/usr/bin/logger -st "pfatt" "cabling should look like this:"
/usr/bin/logger -st "pfatt" "  ONT---[] [$ONT_IF]$HOST"
/usr/bin/logger -st "pfatt" "creating vlan node and ngeth0 interface..."

/usr/sbin/ngctl mkpeer $ONT_IF: vlan lower downstream

/usr/sbin/ngctl name $ONT_IF:lower vlan0

/usr/sbin/ngctl mkpeer vlan0: eiface vlan0 ether
/usr/sbin/ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'

/usr/sbin/ngctl msg ngeth0: set $RG_ETHER_ADDR

/usr/bin/logger -st "pfatt" "enabling promisc for $ONT_IF..."

/sbin/ifconfig $ONT_IF ether $RG_ETHER_ADDR
/sbin/ifconfig $ONT_IF up

/sbin/ifconfig $ONT_IF promisc

/usr/bin/logger -st "pfatt" "starting wpa_supplicant..."

WPA_DAEMON_CMD="/usr/sbin/wpa_supplicant -Dwired -i$ONT_IF -B -C /var/run/wpa_supplicant -c /conf/pfatt/wpa/wpa_supplicant.conf"

# kill any existing wpa_supplicant process
PID=$(pgrep -f "wpa_supplicant.*ngeth0")
if [ ${PID} > 0 ];
then
  /usr/bin/logger -st "pfatt" "terminating existing wpa_supplicant on PID ${PID}..."
  RES=$(kill ${PID})
fi

# start wpa_supplicant daemon
RES=$(${WPA_DAEMON_CMD})
PID=$(pgrep -f "wpa_supplicant.*ngeth0")
/usr/bin/logger -st "pfatt" "wpa_supplicant running on PID ${PID}..."

# Set WPA configuration parameters.
/usr/bin/logger -st "pfatt" "setting wpa_supplicant network configuration..."
IFS=","
for STR in ${WPA_PARAMS};
do
  echo $STR
  STR="$(echo -e "${STR}" | sed -e 's/^[[:space:]]*//')"
  RES=$(eval wpa_cli ${STR})
done

# wait until wpa_cli has authenticated.
WPA_STATUS_CMD="wpa_cli status | grep 'suppPortStatus' | cut -d= -f2"
IP_STATUS_CMD="ifconfig ngeth0 | grep 'inet\ ' | cut -d' ' -f2"

/usr/bin/logger -st "pfatt" "waiting EAP for authorization..."

# TODO: blocking for bootup
while true;
do
  WPA_STATUS=$(eval ${WPA_STATUS_CMD})
  if [ X${WPA_STATUS} = X"Authorized" ];
  then
    /usr/bin/logger -st "pfatt" "EAP authorization completed..."
     IP_STATUS=$(eval ${IP_STATUS_CMD})
     if [ -z ${IP_STATUS} ] || [ ${IP_STATUS} = "0.0.0.0" ];
    then
      /usr/bin/logger -st "pfatt" "no IP address assigned, force restarting DHCP..."
      RES=$(eval /etc/rc.d/dhclient forcerestart ngeth0)
      IP_STATUS=$(eval ${IP_STATUS_CMD})
    fi
    /usr/bin/logger -st "pfatt" "IP address is ${IP_STATUS}..."
    break
  else
    sleep 1
  fi
done
/usr/bin/logger -st "pfatt" "ngeth0 should now be available to configure as your WAN..."
/usr/bin/logger -st "pfatt" "done!"

[zombielinux]

@dangeist Thanks for the excellent work!

I've updated the pull request with the explicit module loading.

Also, the DHCP taking a few attempts was observed over here too, I'd say its normal behavior.

[dangeist]

Purely selfish reasons. I'm moving all my home network gear over to battery-backed DC power and the RGW was a non-standard voltage and power pigtail :)

[smplman]

I tried the updates from @dangeist and @zombielinux and i'm still getting hungup on boot. I should note that i'm running on 22.1 and not 21.x.

After boot i'm able to get authenticated by running sudo wpa_supplicant -Dwired -ingeth0 /conf/pfatt/wpa/wpa_supplicant.conf I'm not sure why it won't auth with em1, my WAN port, but it will with ngeth0

A big thank you to all of those who are putting work into this. If I find a work around I will report back.

[jasonsansone]

pfatt.sh also breaks on pfSense 2.6.0. I wasn't able to get it to work and I had to downgrade to 2.5.2.

[ChronicledMonocle]

I can also confirm this is broken on pfSense Plus 22.01 and pfSense 2.6.0 CE. Seems something broke in the newer FreeBSD kernel, I'd imagine.

[Hou-dev]

I did some testing last night and it seems with Opnsense 22.1 there are some problems with my setup (Intel nic) with the updated kernel. There is a option to update the kernel only using opnsense-update -kr 22.1 I was able to get authenticated when using my other Broadcom nic but it had issues with its driver dropping packets. I had to revert to use my Intel 350 nic with the older 21.7.8 kernel.

Edit 0: I also tried ZombieLinux's version but the same issue with the latest version 22.1 with my Intel nic.

Edit 1: There might be some change in FreeBSD 12.3 that broke compatibility with the script since I am seeing reports that pfsense 2.6 is broken. Opnsense skipped 12.3 and used FreeBSD 13's kernel.

[bigjohns97]

Had the issue this morning when I moved over to 2.6 and ended up trying out OPNsense but haven't tried implementing this, watching closely to look for a solution.

Silver lining is this gave me an opportunity to take a good look at OPNsense and it's pretty nice.

[neydah700]

Sounds like a similar issue with us pfSense people. I am a bit lost thought. Has anyone gotten @zombielinux fork working with Opnsense 22.1? Looking at his code it looks like the only major difference between what I was using was is adding "/sbin/kldload -nq ng_ether" at the start of the script. If someone was successful with 22.1 I'll upgrade again and try it on pfSense 22.01.

[bigjohns97]

Sounds like a similar issue with us pfSense people. I am a bit lost thought. Has anyone gotten @zombielinux fork working with Opnsense 22.1? Looking at his code it looks like the only major difference between what I was using was is adding "/sbin/kldload -nq ng_ether" at the start of the script. If someone was successful with 22.1 I'll upgrade again and try it on pfSense 22.01.

I can try this later on once the fam goes to bed.

[bigjohns97]

Well this is quite different than with pfsense which I am used to, only been running opnsense for a couple of days now.

When I boot up it says it can't find the file so I don't know what the issue is, when I try and run the file manually using ./filename it says the same thing so IDK.

[bigjohns97]

root@OPNsense:/usr/local/etc/rc.syshook.d/early # ./99-opnatt-supplicant : No such file or directory root@OPNsense:/usr/local/etc/rc.syshook.d/early # ls -l total 15 -rwxr-xr-x 1 root wheel 135 Jan 25 03:35 05-upgrade -rwxr-xr-x 1 root wheel 63 Jan 25 03:35 10-configd -rwxr-xr-x 1 root wheel 77 Jan 25 03:35 15-templates -rwxr-xr-x 1 root wheel 93 Jan 25 03:35 20-backup -rwxr-xr-x 1 root wheel 631 Jan 25 03:35 90-carp -rwxr-xr-x 1 root wheel 3563 Feb 15 17:33 99-opnatt-supplicant

[neydah700]

I know this is focused on OpnSense but I was able to do a lot of troubleshooting on the possibly related pfSense issue and may have found a solution. If not a solution for you all hopefully it adds some insight.

https://github.com/MonkWho/pfatt/issues/67#issuecomment-1043358822

And more info on possibly the root issue in FreeBSD

https://github.com/MonkWho/pfatt/issues/67#issuecomment-1043433763

[lnxsrt]

I really do think this is related...

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260068

Unfortunately, it appears than OPNsense has if_em compiled into the kernel. I can confirm their 22.1 branch of the FreeBSD-13 kernel does not have the patch applied...

https://github.com/opnsense/src/tree/stable/22.1/sys/dev/e1000

However, master does...

https://github.com/opnsense/src/tree/stable/13/sys/dev/e1000

[neydah700]

I really do think this is related...

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260068

Unfortunately, it appears than OPNsense has if_em compiled into the kernel. I can confirm their 22.1 branch of the FreeBSD-13 kernel does not have the patch applied...

https://github.com/opnsense/src/tree/stable/22.1/sys/dev/e1000

However, master does...

https://github.com/opnsense/src/tree/stable/13/sys/dev/e1000

I agree. I feel very strongly this is part, if not all of the issues, that have cropped up on opn/pfsense.

[lnxsrt]

Victory!

No custom module or kernel required. Just change this... /sbin/ifconfig $ONF_IF promisc

To this... /sbin/ifconfig $ONF_IF promisc -vlanhwtag -vlanhwfilter -vlanhwtso