tracker
tracker copied to clipboard
turnkeylinux
We should remove redundant Webmin modules from Promxox/LXC builds
We currently convert the ISO build to the other build types and only make changes that are required. We could probably trim a little off the LXC/Proxmox builds if we remove any redundant Webmin module.
Off the top of my head, both the RAID and LVM modules could go, but there's possibly others.
Is there someplace I can see which appliances are targeted for debian12/ tkl18?
was specifically looking for the observium container; as py2 support (deb11) is now deprecated in observium, meaning anyone downloading the existing deb11 based observium appliance will have to sort out their own way to deb12/tlk18
Is this planned anytime soon? I looked at the http image list here, but don't see a d12/tkl18 observium image....
I looked on the observium tkl splash page and the linked github issues but saw no mention of an updated container image... Should I be looking someplace else for this information?
TIA! ❤️🐺W
Hi @wolfspyre,
Thanks for your interest.
Is there someplace I can see which appliances are targeted for debian12/ tkl18?
Unfortunately not at the moment. We are currently using a Mattermost board to track the app updates but at this point it's only available internally. My intention is to make that public, but there are a few things that need to be done before that is possible and I've been trying to focus all my energy on updating the appliances.
FWIW part of the hold up on updates is that we're simultaneously implementing some (semi-)automated smoke testing, with the intention of making future updates quicker/easier and thus make doing more regular builds much less labor intensive in the future. (We probably should have done this years ago, but anyway better late than never I hope...).
was specifically looking for the observium container; as py2 support (deb11) is now deprecated in observium, meaning anyone downloading the existing deb11 based observium appliance will have to sort out their own way to deb12/tlk18
Could you please remind me why py2 will be a problem? TKLBAM does require it but that's another story (FWIW for v18.x we've packaged it ourselves for now), but Observium shouldn't?! IIRC the only py2 requirement was that the poller script called by the cron job only had a /usr/bin/python shebang. However the code is actually py3 compatible, so in v17.2, it was adjusted to explicitly be run with py3.
But perhaps there is something I'm missing?
Is this planned anytime soon?
Short answer; yes - unless there is a decent reason (e.g. upstream software abandoned/insecure/broken) all v17.x apps will be released as v18.0. How soon "soon" is though, is another question... In a perfect world, we'd already be finished the v18.0 release, but it's really dragging... I'd hope that we will be done within the next month or 2.
However, now that I know you are hanging out for an updated Observium, I'm happy to prioritise it. What that will actually mean for release date though is still unclear. My hope is to have another batch of apps ready to publish by weekend after next (so by mid Jan) and from then, provide a batch of updated apps every ~2 weeks until they're all done.
So now you've requested it, my aim is to include a v18.0 Observium build in the next batch of updates. Assuming that the Observium update isn't that hard (my suspicion is that it will "just work"), then it should make it into the next batch.
In the meantime, you could assist by trying to build it yourself if you want? The buildcode is here on GitHub. You'll need a TKLDev instance. Officially we only support running that in a "proper" VM (or bare metal - i.e. not a container) but one our devs has created a script to generate docker container from the ISO if that's your thing. The repo is called tkldev-docker if you want to test that out, but as noted in the readme, it's not "production ready" yet (although a couple of devs are using it internally for app updates and it seems to be working well, we are also using it for our semi-automated smoke tests. Whilst the TKLDev docs are quite extensive, they're not well organized and probably include some out of date info and likely also have some gaps.
If you do want to help out and/or you have any specific questions re TKLDev, please ask. Probably the best place for that is on our forums although due to spammers, new accounts need to be manually approved. So if you do sign up there, please shoot me an email to support AT turnkeylinux.org noting that we've had this chat and I'll approve your account. Somewhere here on GH is probably also an option, although I'm not sure where would be best as we try to keep GH issues for actionable bugs/feature requests, rather than support/discussion.
Hi @wolfspyre,
Thanks for your interest.
Howdy Jeremy!
I've already gotten an account spun up on the forums... you approved me a few weeks ago.
I'm really sorry that the spammers have become SO burdensome that this is what y'all have had to resort to.... it's really saddening.
Y'all do great work. Thanks for your efforts.
I appreciate you.
Is there someplace I can see which appliances are targeted for debian12/ tkl18?
Unfortunately not at the moment. We are currently using a Mattermost board to track the app updates but at this point it's only available internally. My intention is to make that public, but there are a few things that need to be done before that is possible and I've been trying to focus all my energy on updating the appliances.
Yeah. I get that. NBD... MOSTLY I was just trying to demonstrate that I did, in fact, attempt to solve my own problem before bumping up to request it here.
FWIW part of the hold up on updates is that we're simultaneously implementing some (semi-)automated smoke testing, with the intention of making future updates quicker/easier and thus make doing more regular builds much less labor intensive in the future. (We probably should have done this years ago, but anyway better late than never I hope...).
scaffolding is always a PITA. TRUST ME I KNOW. ;)
was specifically looking for the observium container; as py2 support (deb11) is now deprecated in observium, meaning anyone downloading the existing deb11 based observium appliance will have to sort out their own way to deb12/tlk18
Could you please remind me why py2 will be a problem? TKLBAM does require it but that's another story (FWIW for v18.x we've packaged it ourselves for now), but Observium shouldn't?! IIRC the only py2 requirement was that the poller script called by the cron job only had a
/usr/bin/pythonshebang. However the code is actually py3 compatible, so in v17.2, it was adjusted to explicitly be run with py3.But perhaps there is something I'm missing?
ABSOLUTELY. here's a scrollshot from my tkl observium rig (16.1 ) showing observium's update-log-warning:
loiosh@whatsup ~$ sudo bash
root@whatsup /home/loiosh# turnkey-version
turnkey-observium-16.1-buster-amd64
root@whatsup /home/loiosh# su - observium
$ bash
observium@whatsup:~$ ./wplUpdateObservium.sh
Last Date: 20231230
Last Rev: 13240
20231230 - 13239
20231230 - 13240
successfully updated Observium to rev 13244 on 20240104: 20240104 - 13244
updating /opt/observium/wpl_version to include 20240104 - 13244
___ _ _
/ _ \ | |__ ___ ___ _ __ __ __(_) _ _ _ __ ___
| | | || '_ \ / __| / _ \| '__|\ \ / /| || | | || '_ ` _ \
| |_| || |_) |\__ \| __/| | \ V / | || |_| || | | | | |
\___/ |_.__/ |___/ \___||_| \_/ |_| \__,_||_| |_| |_|
Observium Professional 24.1.13244
https://www.observium.org
+---------------------------------------------------------+
| |
| DANGER! ACHTUNG! BHUMAHUE! |
| |
| Python 2.x unsupported. |
| Update to Python 3.x supported by your distribution. |
| Currently recommended version(s): >= 3.6.6 |
| |
| See additional information here: |
| https://docs.observium.org/software_requirements/ |
| |
+---------------------------------------------------------+
-- Database is up to date.
-- Observium is up to date.
Fetching changelog
------------------------------------------------------------------------
r13238 | mike | 2023-12-29 06:50:03 -0600 (Fri, 29 Dec 2023) | 2 lines
[ICONS] Added more os/vendor svg logos.
------------------------------------------------------------------------
r13239 | mike | 2023-12-30 07:11:07 -0600 (Sat, 30 Dec 2023) | 2 lines
[TRIVIAL] Fixed dbShowVariables() with where clause.
------------------------------------------------------------------------
r13240 | mike | 2023-12-30 08:16:23 -0600 (Sat, 30 Dec 2023) | 2 lines
[MINOR] Added Configuration Ages sensors by CISCO-CONFIG-MAN-MIB. Added sensor age class.
------------------------------------------------------------------------
r13241 | adama | 2023-12-31 11:27:36 -0600 (Sun, 31 Dec 2023) | 1 line
[FIX] Fix role assignment on user page.
------------------------------------------------------------------------
r13243 | adama | 2024-01-02 20:14:46 -0600 (Tue, 02 Jan 2024) | 1 line
[IMPROVE] Remove generate_query_values_ng()
------------------------------------------------------------------------
r13244 | mike | 2024-01-03 12:15:50 -0600 (Wed, 03 Jan 2024) | 1 line
[TRIVIAL] Fixed ajax query for bgp peers.
------------------------------------------------------------------------
observium@whatsup:~$
the wplUpdateObservium.sh script is just a shell script I wrote:
observium@whatsup:~$ cat wplUpdateObservium.sh
#!/usr/bin/env bash
cd /opt/observium;
getObserviumCreds
if [[ $? != 0 ]]; then
#something went wrong
echo "Something is wrong, getting observium repo creds errored. OBU: ${OBU} "
else
_LDATE=`tail -1 /opt/observium/wpl_version |awk '{print $1}'`;
_LREV=`tail -1 /opt/observium/wpl_version |awk '{print $3}'`;
echo "Last Date: ${_LDATE}"
echo "Last Rev: ${_LREV}"
tail -2 /opt/observium/wpl_version
svn up --username ${OBU} --password ${OBP}> /tmp/update_observium.log&&
if [[ `tail -1 /tmp/update_observium.log |grep -c revision` == 1 ]]; then
_REV=`tail -1 /tmp/update_observium.log|sed -e 's/.*revision //' -e 's/\.$//'`;
_D=`date +%Y%m%d`;
_S="${_D} - ${_REV}";
echo "successfully updated Observium to rev ${_REV} on ${_D}: ${_S} "&&
echo "updating /opt/observium/wpl_version to include ${_S}" ;
echo "${_S}" >> /opt/observium/wpl_version ;
else
echo "something failed to happen right. /tmp/update_observium.log doesn't match 'revision'. ";
fi &&
./discovery.php -uv
if [[ ${_D} == ${_LDATE} ]]; then
echo "last update was today. Not emitting changelog"
else
echo "Fetching changelog"
svn log -r {${_LDATE}}:{${_D}}
fi
fi
Jus sharing so's its obvious that its nothing terribly out of the ordinary there.
Is this planned anytime soon?
Short answer; yes - unless there is a decent reason (e.g. upstream software abandoned/insecure/broken) all v17.x apps will be released as v18.0. How soon "soon" is though, is another question... In a perfect world, we'd already be finished the v18.0 release, but it's really dragging... I'd hope that we will be done within the next month or 2.
However, now that I know you are hanging out for an updated Observium, I'm happy to prioritise it. What that will actually mean for release date though is still unclear. My hope is to have another batch of apps ready to publish by weekend after next (so by mid Jan) and from then, provide a batch of updated apps every ~2 weeks until they're all done.
So now you've requested it, my aim is to include a v18.0 Observium build in the next batch of updates. Assuming that the Observium update isn't that hard (my suspicion is that it will "just work"), then it should make it into the next batch.
In the meantime, you could assist by trying to build it yourself if you want? The buildcode is here on GitHub. You'll need a TKLDev instance. Officially we only support running that in a "proper" VM (or bare metal - i.e. not a container) but one our devs has created a script to generate docker container from the ISO if that's your thing. The repo is called tkldev-docker if you want to test that out, but as noted in the readme, it's not "production ready" yet (although a couple of devs are using it internally for app updates and it seems to be working well, we are also using it for our semi-automated smoke tests. Whilst the TKLDev docs are quite extensive, they're not well organized and probably include some out of date info and likely also have some gaps.
wilco!
If you do want to help out and/or you have any specific questions re TKLDev, please ask. Probably the best place for that is on our forums although due to spammers, new accounts need to be manually approved. So if you do sign up there, please shoot me an email to support AT turnkeylinux.org noting that we've had this chat and I'll approve your account. Somewhere here on GH is probably also an option, although I'm not sure where would be best as we try to keep GH issues for actionable bugs/feature requests, rather than support/discussion.
I'll check it out for sure!! thanks again! W
I've already gotten an account spun up on the forums... you approved me a few weeks ago.
Doh! :rofl:
I'm really sorry that the spammers have become SO burdensome that this is what y'all have had to resort to.... it's really saddening.
Yeah it's been an issue for a while, but about a year ago it ramped up to the point that there were literally hundreds of spam posts per day (either a team of human spammers or perhaps just a really smart bot). I tried blocking IPs and they just changed to a new IP so I threw my hands in the air and locked things down. The software we currently use is old and clunky and I've been wanting to update it for a long time now, so that will hopefully be properly fixed at some point in the future (argh - so much to do and so few hours in the day...).
Y'all do great work. Thanks for your efforts.
I appreciate you.
Thank you for your kind words. TBH, I generally I find (constructive) critique more useful - because that's something that helps TurnKey get better. Regardless, it's still really nice to get some positive feedback too and be clearly reminded that many people find TurnKey useful and valuable too (not just the ones that are super active and tell us all the time). So thanks! :grin: :+1:
Yeah. I get that. NBD... MOSTLY I was just trying to demonstrate that I did, in fact, attempt to solve my own problem before bumping up to request it here.
Nice. I missed that, but quite a cool way of doing it. I could learn something there... (I tend to have the verbosity turned up to 11).
scaffolding is always a PITA. TRUST ME I KNOW. ;)
Say no more... :grin:
ABSOLUTELY. here's a scrollshot from my tkl observium rig (16.1 ) showing observium's update-log-warning:
I'm assuming you're referring to this:
+---------------------------------------------------------+ | | | DANGER! ACHTUNG! BHUMAHUE! | | | | Python 2.x unsupported. | | Update to Python 3.x supported by your distribution. | | Currently recommended version(s): >= 3.6.6 | | | | See additional information here: | | https://docs.observium.org/software_requirements/ | | | +---------------------------------------------------------+
Interestingly enough, python isn't mentioned (at all) on that page?!
My guess is that it's checking the PATH for python and getting /usr/bin/python (which is python2). Python3 should be installed, but will be found at /usr/bin/python3, which in v16.x should be v3.7.3.
Python2 is still in v17.x - packaged by Debian. That is despite it being EOL. The Debian security team note that there is no security support for py2 in 11/Bullseye. Although I also note that there has been at least one security update since they said they wouldn't maintain it. With one exception (TKLBAM) in v17.x all the code found in appliances (with the exception of TKLDev) should be py3 only, but at least some of our build infrastructure (i.e. TKLDev components and other related stuff) was still py2. In v18.x we (mostly) finished the py3 transition (again with the exception of TKLBAM - but all other TKL stuff is py3). In v18.x we packaged py2 ourselves (as it's all gone from Debian now) and the py3 upgrade for TKLBAM is still in progress (I really need to complete that within the lifetime of v18.x - as IMO packaging py2 in v19.x shouldn't be an option). FWIW we've audited TKLBAM and all the python libraries that it leverages and for the purposes of TKLBAM we're confident that it's secure in the context that TKLBAM uses it.
So whilst the Observium warning is legit (in as much as py2 being available but EOL and not recommended for general use), it seems to me like it expects python to be python3. AFAIK Arch Linux is the only distro that does that. Python themselves recommend using python3 (which Debian has followed).
Having said that, starting in 11/Bullseye (v17.x) python2 only provides /usr/bin/python2 and there is a python-is-python3 package (which just includes a symlink from /usr/bin/python to /usr/bin/python3 - FWIW there is also a corresponding (conflicting) python-is-python2 package too). It's not installable on v16.x (the package doesn't exist) but you could manually just create the symlink if you wanted (although I would NOT advise that - there is a good chance that it would break lots of other stuff). IIRC in v17.x installing that package (python-is-python3) should be ok (I'm pretty sure that the remaining py2 code we provided in v17.x had the shebang updated to explicitly use /usr/bin/python2).
Also, as I noted previously, AFAIK the only python code that Observium provides is the cron job, which we explicitly launch with python3 now. Perhaps there is more somewhere that I've missed?
the
wplUpdateObservium.shscript is just a shell script I wrote
That script looks pretty nice!
FWIW we do provide an update script (turnkey-update-observium - and a dependency) OOTB. Although I'm not sure if it's in the appliance version you have. I like the changelog display from your script, that's very cool! If you'd like to modify/improve our script, that would warmly be welcomed. I'm not an Observium user myself, but IIRC the basis of that/those scripts was/were supplied by a user, then I massaged it into something that I was relatively happy with, but it could certainly be further improved.
I'll check it out for sure!!
Sounds good! Please start a thread on the forums with any questions/feedback/etc. Please feel free to open PRs for any docs that you find that are broken/incorrect. Same applies to the Observium buildcode itself (and/or anything else) - although it's probably best to discuss first if you have ideas that are more significant that tweaks. It's not compulsory, but it's nice if PRs have matching issues here on the tracker (don't worry for an appliance update and minor tweaks - but other code changes, especially bug fixes, are best with a corresponding bug report/feature request).
FWIW, building an appliance is actually pretty straight forward. This is probably a good place to start (it's a bit dated, but from a quick glance it should be still relevant).
As another aside, now that Debian's Live system is maturing, we're considering rewriting all our appliance build code to leverage that instead of our own custom build system. But that won't happen within the life of v18.x. It will happen in v19.x at the soonest and even that seems a bit doubtful if I look at my todo list...