Migration from PCRE to PCRE2

[BKPepe]

Due to recent findings described in https://github.com/openwrt/packages/pull/21800, we should track somewhere (= here) packages that are still using PCRE instead of PCRE2 to propose their developers to look into PCRE2 and prepare the support (Assignee: Here maintainers of relevant packages).

---

Reasons to switch to PCRE - Perl Compatible Regular Expressions 2:

• PCRE2 was first released in 2015 to replace the API in the original PCRE library, which is now obsolete and no longer maintained. ^1 ^2
• Since OpenWrt targets to be minimalistic, there is no reason to have installed these two libraries side by side on the device, which results in being bigger and it may not fit devices with small storage.  ^3

---

Packages, which needs to be taken care of:

• [x] aircrack-ng Maintainer: @ZeroChaos- even though I thought this package is rather not maintained Steps to be done:

1. Ask aircrack-ng developer to release a new version with PCRE2 as those changes were merged to the master branch (See PR: https://github.com/aircrack-ng/aircrack-ng/pull/2391 )
2. Seek a new maintainer for this package in OpenWrt or if not found, we should decide if we want to keep this package here or not.
3. Backporting patch is simple enough, while backporting an additional patch proposed upstream is now needed. https://github.com/aircrack-ng/aircrack-ng/pull/2580
4. Pending https://github.com/openwrt/packages/pull/22256

• [x] atftp Maintainer: @dddaniel Steps to be done:

1. Backport patch from https://sourceforge.net/p/atftp/support-requests/11/
2. Someone should step it and take maintainership of this package here.
3. Pending: https://github.com/openwrt/packages/pull/22228

• [x] net-snmp Maintainer: @stintel Steps to be done:

1. ~~Waiting for https://github.com/net-snmp/net-snmp/issues/420~~ Support merged but still not in latest release https://github.com/net-snmp/net-snmp/commit/d3e95c87b32397815f6d5bcfc844259f2552697a
2. Pending https://github.com/openwrt/packages/pull/22254

• [x] shadowsocks-libev Maintainer: @yousong Steps to be done:

1. Remove this package because there were efforts to port it to PCRE2 and it is not merged and as well, this package is getting to be removed from Debian. See: https://github.com/shadowsocks/shadowsocks-libev/issues/1788
2. Pending and under consideration: https://github.com/openwrt/packages/pull/22356

• [x] uwsgi Maintainer: @Ansuel Steps to be done: Oh, this time, its different! There is a pending pull request: https://github.com/unbit/uwsgi/pull/2543 Pending https://github.com/openwrt/packages/pull/22184
• [ ] nmap Maintainer: @nunojpg Steps to be done: There is support since the commit https://github.com/nmap/nmap/commit/828ab48764b82d0226e860c73c5dac5b11f77385 , no release was tagged since that time Pending: https://github.com/openwrt/packages/pull/22226
• [x] apache Maintainer: @heil Steps to be done:

1. Is anyone interested to take maintainership of this package?
2. Support is already there https://svn.apache.org/viewvc?view=revision&revision=1898399 , so we need to bump this package and switch the library in the Makefile
3. Pending: https://github.com/openwrt/packages/pull/22225

• [x] kismet Maintainer: @padre-lacroix and @sourceindex Steps to be done: This package is not maintained in our repository since it was introduced here even though upstream are releasing new versions This package is going to be removed in https://github.com/openwrt/packages/pull/22236
• [x] mg Maintainer: @nxhack Steps to be done:

1. Needs to be investigated further

• [x] bandwidthd Maintainer: @padre-lacroix Steps to be done: Upstream repository has been archived and read-only since February 2022. See: https://github.com/NethServer/bandwidthd Removed this package. (https://github.com/openwrt/packages/pull/22008)
• [x] zsh Maintainer: @msva Steps to be done: There is support in the upstream repository based on https://sourceforge.net/p/zsh/code/ci/10bdbd8b5b0b43445aff23dcd412f25cf6aa328a/ ~~Ansuel inspection: Backporting the patch is doable but results in the package not correctly generating the libs...~~ Thank god someone manage to find the libs problem. Pending: https://github.com/openwrt/packages/pull/22197
• [x] slang2 Maintainer: @jefferyto Steps to be done:

1. Needs to be investigated further

• [x] czmq (#22007) Maintainer: @ja-pa Steps to be done:

1. Needs to be investigated further

• [x] micropython-lib Maintainer: @jefferyto Steps to be done:

1. Needs to be investigated further
2. Pending https://github.com/openwrt/packages/pull/22243

• [x] nginx Maintainer: @Ansuel and @heil Steps to be done:

1. ~~Since version 1.12.5 is using pcre2 by default according to https://nginx.org/en/CHANGES , but still in the Makefile, there is legacy pcre being used.~~ Pending https://github.com/openwrt/packages/pull/21018

• [x] nginx-utils Maintainer: @peter-stadler Steps to be done:

1. ~~Adapt source to pcre2.~~ Pending https://github.com/openwrt/packages/pull/22185

• [x] postfix Maintainer: @Shulyaka Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22533

• [x] zabbix Maintainer: @champtar Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22534

• [x] fdm Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22536

• [x] freeradius3 Maintainer: @odmdas / @neheb Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22532

• [x] haproxy Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22537

• [x] libndpi Maintainer: Steps to be done:

1. Need further investigation.
2. Pending: https://github.com/openwrt/packages/pull/22570

• [x] privoxy Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22539

• [x] tvheadend Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22540

• [x] snort3 Maintainer: Steps to be done:

1. ~~Planned support for last year... Still not done...~~ Took care of manually converting it
2. Pending https://github.com/openwrt/packages/pull/22610

• [x] snort Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/packages/pull/22600

Packages in other feeds, which needs to be taken care of:

• [ ] freeswitch Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/telephony/pull/841

• [x] sipgrep Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/telephony/pull/838

• [x] rtpengine Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/telephony/pull/839

• [x] kamailio Maintainer: Steps to be done:

1. Pending https://github.com/openwrt/telephony/pull/840

I ask anyone to be responsible and look into it, sooner than later. Otherwise, if there is no response to the package, which you are maintaining, the package will be removed at the beginning of the next month - 1st October 2023, so we have a soft deadline. I am fully aware that if there is no deadline, no further actions will be taken. You are not doing it just for me but also for others, including OpenWrt users.

Thank you for your cooperation, guys and make sure that it is run-tested on the router! ;)

[1715173329]

shadowsocks-libev

I guess it's time to switch to shadowsocks-rust, which has much better performance (10x faster on aarch64 platform).

[stintel]

https://github.com/net-snmp/net-snmp/issues/420

[BKPepe]

Thanks, @stintel, I included your referenced issue to OP.

[padre-lacroix]

For the package kismet, I originally become the maintainer to prevent this package to be abandoned. But, I do not wish to maintain it anymore, mostly because this is a package that I do not use and also because I have no way to test it. If somebody wants to become the maintainer, then I am OK with that, otherwise it can be dropped.

[jefferyto]

slang2 is done in #22020.

[neheb]

I looked into kismet before. seems to now be a convoluted project with protocol buffers and a bunch of other dependencies.

edit: also if I had to guess, kismet being C++ would mean pcre is not used.

[Ansuel]

@BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

[nxhack]

@BKPepe mg editor done.

[BKPepe]

I looked into kismet before.

@neheb The question is do we care about quality over quantity? Many packages here are not maintained for various reasons. Almost everyone can add a new package to this repository, no matter what. No votes, if it has purpose or if it makes any sense. I know it will sound bad, but I would rather remove those packages, which could be updated by upstream but not here. This should hopefully result in end users or someone else complaining that the package is missing within OpenWrt and that they can help us. I'm not saying that there are not enough maintainers. We (existing ones) are doing it for some purpose. We care about it. We are trying to update it, we want to have the latest versions on our routers, etc.

I might be strict, but I would simply remove those packages. You know, it is very pleasant of you that you will take into kismet, but how many users are using it?

I fail to see such unmaintained packages here: https://downloads.openwrt.org/stats/awstats.downloads.openwrt.org.allextra2.html

Is anyone really using them? If we are doing that for a minority <5-10 users then it is probably not worth all the efforts we are putting into it. They can compile themselves. They can use GitHub Actions (simply by forking this repository), if they are not familiar with the OpenWrt build system and from the CI, download the package and install it or even prepare their own feed.

With the removing these packages, we will as someone from OpenWrt core team is saying: "Less using compute resources, faster build time, happy users"

[BKPepe]

@BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

@Ansuel: You have there two packages, where you are listed as maintainer. We can probably backport uwsgi pending pull request as it seems that for 3 weeks, there has been no activity. Regarding nginx, it can be done immediately because PCRE2 support is there for many weeks.

[Ansuel]

Was talking about other packages since the deadline is 1 October. For nginx I will check today what are the required thing to do. The blocker was lua support

Il Ven 22 Set 2023, 09:45 Josef Schlehofer @.***> ha scritto:

@BKPepe https://github.com/BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

You have there two packages, where you are listed as maintainer. We can probably backport uwsgi pending pull request as it seems that for 3 weeks, there has been no activity. Regarding nginx, it can be done immediately because PCRE2 support is there for many weeks.

— Reply to this email directly, view it on GitHub https://github.com/openwrt/packages/issues/22006#issuecomment-1730959891, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2ZMQQ24PDSTHEQ6SCQD43X3U623ANCNFSM6AAAAAA4JHKF2M . You are receiving this because you were mentioned.Message ID: @.***>

[BKPepe]

Having a deadline is great, but I think we need to put these packages to some categories:

1. OpenWrt packages maintainer is not active
2. The package is not maintained anymore.
3. Many people do not use the package, whereas 1st point applies.
4. PCRE2 support is on its way and it would be too pity to remove it
5. PCRE2 is already there, but 1st point apply.

So... it is not easy as it seems. :-)

I don't currently have the answer what are you looking for. It's open discussion, so any comment is appreciated. Time will tell us.

[Ansuel]

@BKPepe I like deadlines... force things to move and honestly we should add them all around our project. For me it's ok to apply patch and then drop them on release bump... (assuming they are mature enough and just stalled)

[BKPepe]

Yeah, I like deadlines as well that's why I gave it in the first place. Exactly for the reasons what are you saying. It moves things to move.

[neheb]

kismet can go away. I don’t use it.

[padre-lacroix]

I am the maintainer of kismet and I do not use it myself: as I mentioned in a previous message, I took it to prevent at the time (2015-2016) that it will disappear.

So, kismet can disappear as far as I am concerned: I cannot and will not maintain it as I cannot even test it or use it myself.

[oskarirauta]

@msva

Proposal for zsh libpcre2 usage: #22197

[BKPepe]

It seems that we do have a lot of progress here! Glad to see it, and thanks anyone, who looked into it.

• Unfortunately, I am not sure what to do with Aircrack-ng.
  • While looking at the link, which I already put here (https://downloads.openwrt.org/stats/awstats.downloads.openwrt.org.allextra2.html), it seems that Aircrack-ng was downloaded 146 times. It wouldn't be great to remove it. I asked the developers if they could release a new version because it would help us instead of backporting patches, using the master branch, and giving it to users, etc.
    • We still need to seek for a new maintainer here.

[trippleflux]

@BKPepe

Trying my best about Aircrack-ng, hmm not to be sound rude but the bad news is I don't yet feel the like or knowledgeable enough in aircrack-ng to maintain or use it.

[Ansuel]

@BKPepe https://github.com/micropython/micropython-lib/pull/737 wasted way too much time on this but eheheheh

[Ansuel]

Well tomorrow is the deadline. I would drop pcre library and mark the remaining package as @BROKEN while we wait for the remaining package to be update. For the one where we don't have any info I would move them to package-abbandoned https://github.com/openwrt/packages-abandoned

[stintel]

Well tomorrow is the deadline. I would drop pcre library and mark the remaining package as @BROKEN while we wait for the remaining package to be update. For the one where we don't have any info I would move them to package-abbandoned https://github.com/openwrt/packages-abandoned

Please don't. You can't seriously consider dropping net-snmp from a distribution that mainly targets network devices.

[Ansuel]

@stintel if you want I can waste also today on migrating another package to pcre2.

[stintel]

@stintel if you want I can waste also today on migrating another package to pcre2.

The upstream issue hasn't seen any updates since January. Don't see any references to a branch where this might be worked on, and no idea how much work it's going to be. I currently don't have the time to dig in myself.

But dropping net-snmp would be a bad thing. I would actually consider finding an alternative to OpenWrt if that happens.

[Ansuel]

@stintel well this is main/master so it's expected to have some package broken for some time. These changes won't affect stable...

Aside from this... I'm checking the package with a quick grep and I can see there is actual support for pcre2... https://github.com/net-snmp/net-snmp/commit/d3e95c87b32397815f6d5bcfc844259f2552697a

Let me prepare a pr and see how it goes.

[stintel]

Aside from this... I'm checking the package with a quick grep and I can see there is actual support for pcre2... net-snmp/net-snmp@d3e95c8

Nice find! Tag me in a PR and I can make some time to do some testing. Thanks for looking at this!

[BKPepe]

As you can see, I put in a few extra days over the deadline. I'm just joking. I didn't look at it recently. I see that everyone did a great job over here. Mostly @Ansuel, who was working on it. Appreciated! 👏

Maybe some of you wouldn't like me, that's understandable, but most of those switches to PCRE2 were backported to the stable branches. You know, it is quite simple on this one. We would not get rid of the deprecated PCRE really fast, and it would be quite boring once the master is ready, but stable releases are not. This helps us that users can report it to us and we can really fast investigate it and fix it. OpenWrt does not release quite often as I was used to do with Turris OS (yep... they are still based on OpenWrt 21.02 so far with Turris OS 6.x. Thus some fixes landed there as well)

I am not using legacy PCRE on my router anymore, but some of you might. We are getting to the end. 🤞 There are just a few packages remaining to be moved/migrated to PCRE2.

I haven't heard anything about shadowsocks-libev, I will create the pull request to remove it from the master branch very soon.

[Ansuel]

@BKPepe

• micropython is getting investigated (why I can't repro errors other reports...)
• zsh is ready
• nmap we need to wait for a new point release since adding a 4mb patch is too much

with shadowsocks-libev getting dropped i think pcre can be dropped.

Also consider that uwsgi is currently broken due to a problem with python and PATH but another member is working on fixing that so that should be handled indirectly. (also wasn't catch by CI test as they don't compile every sub package)

[yousong]

@BKPepe @Ansuel Though not a user of the package myself, I tend to believe that shadowsocks-libev seems to be still used by quite a few people (stats).

Before an alternative is available, I will try to keep it here by modifying the source code to drop dependency on pcre. Hopefully it's just acl support only recently enabled by @fertkir in #20647

[Ansuel]

@yousong or... https://github.com/openwrt/packages/pull/22356