Request for comment - Binary packages and apt repo

[linuxkidd]

Hi all, It has always bugged me that the installation method sets up a full dev environment and compiles on the fly against the HEAD of the repo.

I'd like to propose a version tag release method (possibly date based? v2021.09.23 for example), and accompanying compiled package releases, preferably even hosted in an apt compatible repo directory. My initial research on the packaging and repo setup looks very straight forward and it can easily be hosted on github. I'm happy to spearhead the PR's necessary.

What I'd like to discuss, is a move to a more standardized config file format and minimizing the need or complexity of the 'allsky.sh' wrapper script, where all configuration lives in /etc/allsky, and the binaries / scripts live in /usr/local/bin. I know there are others working on getopt() and possibly other modifications that could streamline this effort.

Please comment here with references to PRs and or forks that would help move in this direction, and/or any questions, concerns, or general insights about this path. I would love to minimize duplicating work on multiple similar paths.

My ultimate goal would be to have a sub-directory off of this, or another repo which hosts the compiled packages along with all the necessary meta files. This would allow anyone who wanted to install allsky, the allsky-portal and/or allsky-website, to simply add a repo to their /etc/apt/sources.list.d folder, and apt update && apt install $package.

Thanks!

[ckuethe]

Speaking only as a random person on the internet, go for it. I think you've seen plenty of struggles here as people test new versions of code to debug issues or upgrade their systems and find unexpected behavior. Try some experiments and let us know what you find. :)

I am indeed working on getopt() to make argument handling easier, and I've been muttering here and there about my fondness for JSON as a configuration format. I think it would be quite possible to put all the setting into a single file that could be used to fully represent the state of your system.

@EricClaeys recently did a diff that retrieves the serial number from ZWO cameras. In a world with a unified JSON config, we could potentially have a config file that could store global system parameters, per-camera settings, and customized scripts.

[EricClaeys]

@linuxkidd Michael, the version I am working on adds an ALLSKY_CONFIG variable that will point to the single directory that holds all the config file(s). It could easily point to /etc/allsky. It also adds ALLSKY_SCRIPTS that points to where the scripts are.

[linuxkidd]

A few thoughts on config stuffs.

• Current Method:
  • settings_(ZWO|RPiHQ).json for camera settings
  • config.sh for post-image processing config ( upload, timelapse, delete old, etc )
  • scripts/filename.sh for naming convention of the output files
  • scripts/ftp-settings.sh for Upload config
  • The above list is parsed by allsky.sh as a wrapper around capture or capture_RPiHQ, and/or various other handlers in the scripts/ folder.

As I see it, there are a few (better?) alternatives to this:

• JSON all the things, then parse the JSON into capture or capture_RPiHQ, as well as the other supporting scripts/*.sh files.
• YAML all the things, then parse the YAML .... ^^ same as above.
• ENV file all the things, then.. yadda yadda....
• BOOST parsed ini style config file
• Some other database of key/value pairs

From the above possible alternatives, the easiest for an end-user to manage is an ENV file. It's a pretty common config management technique in the *nix world. The boost parsed ini file is similarly easy to understand and very forgiving on formatting. JSON and YAML add a bit of overhead in formatting rigidity which new users may or may not be confused by.

From my perspective, the boost parsed ini has the best bang for buck. The ability to use different [section] headers to isolate settings specific to a given sub-process makes a lot of sense. The BoostPropertyTree style seems quite handy from the C++ side of things. For shell scripts, awk or sed could be used to extract specific section / param values. Other languages have an easy ability to parse them as well ( php and python for example ).

Whatever is opted for, the command line arguments for capture could be drastically reduced to simply a config file spec, with everything else being configured within. Another one or two options for debug, or verbosity maybe?

Thoughts?

[linuxkidd]

I chatted with @thomasjacquin on this topic... highlights below:

• Unified INI file seems a good fit ( ease of manual editing, ease of programmatic editing, centralized config )
• Refactoring code to split different functional parts into separate header / source files for easier future maintenance and collaboration sounds like a good idea.
• New repo created for binary apt repository.

I know there's a lot of work going on by @EricClaeys and @ckuethe -- I'm going to try and get this refactor done ASAP so it doesn't block other progress.

Please update this thread if there are any blockers anyone has to these changes.

[ckuethe]

I would prefer to JSON all the things. I find YAML hard to read/write correctly (eg. in complicated HomeAssistant/ESPhome configurations). If I do have to YAML, I have a couple of nasty python one-liners to transcode between JSON and YAML. Another thing I like about json is that there are some nice small JSON libraries for C (https://github.com/DaveGamble/cJSON) and python/php/ruby/perl all have good JSON modules too. But if you're writing it, all I ask is that you pick a format with good interface library support in multiple languages so that you don't have to do a lot of painful hand parsing and string comparisons.

Capture's configuration and argument parsing could indeed be drastically simplified. I would very happy if most of the arguments except --help, --verbose, and --config-file went away. One consideration would be a signal handler (SIGUSR1, SIGHUP) to reload the config file and start using the new settings.

For a backup file, I guess you could encode all of the state as base64 blobs in an INI file.

[allsky_config_ini]
checksum = <sha256 of contents>
payload = <base64 of allsky_config.ini>

[end_of_night_additional]
checksum = <sha256 of contents>
payload = <base64 of end_of_night_additional_steps.sh>

[upload_settings]
checksum = <sha256 of contents>
payload = <base64 of nightly_uploader.sh>
...

And then we can write a CLI tool and a web UI section to work with the backups.

[ckuethe]

I'll start my work on refactoring the configuration state of keogram and startrails first. Those should be pretty small and they'll give you something to run your unified config loader against.

[linuxkidd]

@ckuethe check out my first blush of the allsky.conf unified config.

[ckuethe]

Cool. I have some comments but it seems pretty clear.

• why does scriptspath have to end with '/' ? Both /home/pi/allsky/scripts/endOfNight.sh and /home/pi/allsky/scripts//endOfNight.sh are valid and resolve to the same file
• up to you, but maybe put units in the property names if applicable, eg. exposure_us or delay_ms
• bin -> binning, since bin might also mean binary.
• maybe break out the text parameters so that both capture and keogram can use them (font, size, color)?

@linuxkidd @thomasjacquin - thoughts on using this little INI parser - https://github.com/benhoyt/inih? Ubuntu, dietpi, and raspbian have libinih-dev

[linuxkidd]

• Point taken on trailing /, removed as requirement
• Added units to property names -- I may have missed some, open to suggestions.
• Renamed bin to binning, lemme know if there are others I missed.
• Removed white space in property names since there's variation in how that's handled across languages / libraries.
• Started switching to camel case in property names for ease of visual parsing.
• inih looks good for this use case. Light weight and complete. I don't see any reason the multi-line handling is an issue for allsky.

Keep the feedback come'n!

[ckuethe]

I'd suggest snake_case for property names for consistency with the unit-suffixed properties, but that's just me.

Also, I'm not clear on the differences between keogram.upload, startrails.upload, timelaps.upload and upload.endOfNight are. would it be better to move them to upload.keogram, upload.startrails, upload.timelapse?

[linuxkidd]

snake_case is fine with me.

I teetered back and forth on putting a main [upload] section with each flag, or putting the flag into each section. Ultimately, I opted for moving those under their respective section so the user can toggle generation and upload in one place. Then leave the [upload] section for configuring how/where/etc...

endOfNight didn't seem to fit anywhere else.. but, I'm happy to reposition it.

[ckuethe]

OK, that's reasonable. maybe put some example URLs in there then like:

[upload]
# method = rsync
# dest = user@server:/path/to/allsky/dest
# becomes `rsync -r --partial --inplace $NIGHTLY_DIR user@server:/path/to/allsky/dest/
#
# method = scp
# dest = user@server:/path/to/allsky/dest
# becomes `scp -r $NIGHTLY_DIR user@server:/path/to/allsky/dest/
#
# method = ftp
# dest =user:password@server/path/to/allsky/dest/
# becomes lftp ftp://user:password@server/path/to/allsky/dest/ "put ..."

method =
dest =

[linuxkidd]

I like it... added.

Also, re: Text params (and others) I don't intend for options from a given section to be strictly enforced to only that application. However, the text params are all under the [overlay] section, so it isn't enforcing indicative in the name anyway.

[ckuethe]

Indeed there is nothing preventing one config consumer from using a different section, I was just thinking about it from the perspective of making it clear that this parameter might be shared and what it might be used for

[EricClaeys]

Chris. Could you do me a favor and send Michael the version 2 of the text file I sent out with all the possible enhancements? I am out of town and wont have access to it for a while.

@linuxkidd Michael, I have forks of allsky-portal and allsky in github/EricClaeys that includes everthing I have in progress.

Thanks - Eric

[ckuethe]

Will do

[EricClaeys]

The camra_options_*.json files have sections as does the Camera Settings GUI page. Daytime, nighttime, overlay, etc.

---

From: Chris Kuethe @.> Sent: Monday, September 27, 2021 1:02:08 AM To: thomasjacquin/allsky @.> Cc: EricClaeys @.>; Mention @.> Subject: Re: [thomasjacquin/allsky] Request for comment - Binary packages and apt repo (#506)

Will do

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/thomasjacquin/allsky/issues/506#issuecomment-927531700, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AT2PYK4F3RLZVPDX6OTFQU3UD73FBANCNFSM5EVDHMFA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[EricClaeys]

@linuxkidd Overall I really like it. Some suggestions:

• combine image_name and image_extension so user just specifies the file name, e.g., image.jpg. I think that's easier for users rather than having to split into name and extension. It's also easier to to explain with just one variable.
• We'll need sections for different cameras. Right now just ZWO and RPiHQ, but soon we'll be able to handle ZWO cameras by serial number or ID. Many of the settings don't apply to RPiHQ. There are probably only a handful of options that would always apply to every camera - location, path_*, locale, current_alias, auto_delete, night_to_keep, [thumbnail] are the only that I can think of. uhubctl would apply to all ZWO cameras.
• Default Image type should be "auto". That will work for almost everyone.
• Have users spell out the Flip type: None, Horizontal, Vertical, Both. Our code could just check for the 1st character.
• I'd like to propose we do away with 1 and 0 for boolean argument the user enters, and always user either "true" and "false", or "yes" and "no" (but our code could accept any of them). I believe most non-programmers think in terms of "true" and "false", not 0 and 1.
• Somewhere we should state that "_ms" is microseconds (1000 ms = 1 second) and "_s" is seconds.
• Have user enter 1x1, 2x2, or 4x4 for binning. I think that's more intuitive. Our code could just check the 1st character.
• gain_transition_time_s should be gain_transition_time_min since it's in minutes.
• I'd really like to see the GUI become part of the base allsky release so we can more-or-less force people to use it to configure everything. We can then enforce basic type checking and all the variables will be commented. Chris and I talked about having values like "gain" be percents, where 1 = minimum gain supported by the camera and 100 = the max camera gain. This way users don't have to know that model xyz's gain is from -100 to 100 and model abc's is from 0 to 100. We can hide all that complexity behind the GUI.
• [upload] probably should go in the individual post-processing section since it's for uploading the current picture. Ditto for "end_of_night" which is only for data.json.
• Based on the problem a lot of people have with lftp, I think it might be best to have separate variables for lftp's user, password, host, etc. as is currently done. Us experienced people would have no problem filling in only a "method" and "dest", but for a lot of people this will probably be over their head.
• @ckuethe the rsync example - isn't the rsync command executed once (per boot?), rather than for every image upload? If so, should we include that example in the config file?

[linuxkidd]

A lot of good things in your update...

One thing jumped out at me... _ms == milliseconds ( 1000 ms == 1 s ) _us == microseconds ( 1000 * 1000 us == 1 s )

:)

[linuxkidd]

Say... is there a real reason to allow selection of file name? Shouldn't it always either be image.jpg or image.png ? Maybe just a selection of output type ( JPG or PNG ) is better?

Re: Multi-camera

• since I'm switching from ini to json, we can add tags in the JSON for which camera types a given option applies to.
• I plan to abstract the image acquisition to a separate included files, where the calling functions only request an image with parameters. The library bits handle the direct camera interaction. This should allow us to expand to other cameras by just adding another library with that model's specifics.

Re: Tweaks to how the user specifies certain settings

• Agreed on all fronts, the JSON config file will give better flexibility around this, same as the camera_options_ZWO.json does for the WebUI.
• Actually, the biggest difference between that existing camera_options file and what I'm putting together.. is that we also store the current value in the same file.

Re: [upload] in each post-processing section

• I'm not sure I follow... Wouldn't the overall upload parameters be expected to be identical for all post-processed outputs?
• Well, plus or minus whether a specific post-processed bit gets uploaded, but those options are under each other section.
• Maybe I'm missing something.. :)
• Also, imo... the rsync option should be modified to include each image output, as well as endOfNight, etc

[ckuethe]

So we should probably define what kind of upload we mean. You could scp/sftp/ftp/rsync every image as it's created, or you could do it at the end of night. I was thinking about the end of night use case, since my sky cam is behind an nginx proxy, and also has access to network shares.

And since we now have serial number retrieval in capture.cpp we can use that to control per-camera tuning. I don't know, for example, if 50% gain or 33% gamma is right for a 183 vs a 120.

[linuxkidd]

Re: every image vs endOfNight -- I think I see what you mean. Someone may not want to upload each exposure, instead just the Startrails & Keogram or what not..

Sure, we can certainly store per-camera settings in the JSON file. Are there people who run multiple cameras on a single Pi? Or is this more for having sensible defaults per camera model?

[EricClaeys]

Isn't the [upload] for whether or not the user wants the current image.jpg to be uploaded? If so, that's a single file so I thought it should go with the other single file options, not in the multi-file section.

Is there something wrong with the _ms and _us? Those dang things always confuse me.

I think we should continue allowing the user to specify the filename with "image" the default. I don't know how often it's modified, although I am pretty sure one user did per his Issue. BTW, the code currently only supports .jpg and .png, and only .jpg has hardware support so .png files take about 10 seconds to save on a Pi 4.

If I understand correctly, the new single config file will be similar to the current camera_option_.json file but will also contain the value (from the settings_.json file as well as a field stating what camera it applies to? If so, we will also want to add the min and max value for the capture* programs to use.

Some people do run multiple cameras on a single Pi (not just me). Not at the same time since the software doesn't support it.

The user already has the ability to, for example, upload the startrails but not the image.jpg files. I started implementing the ability to upload the image.jpg files every Nth time. I added a variable to config.sh and will put the logic in the saveImage.sh script.

[ckuethe]

I've been tempted to run alternate cameras on my Pi, just to monitor a particular patch of sky, but I haven't seen anyone doing it in production. If ZWO wanted to donate me one of every camera I'd happily come up with sensible defaults for every camera. :)

[EricClaeys]

I posted a request to the Facebook Allsky group for a really cheap used camera for testing, or a donated one, and someone gave me $300 to buy a new one. That's how I got the 290. You never know until you ask. I then bought an extra Pi and RPiHQ for testing.

[linuxkidd]

FYI, latest unified config version -- JSON all the things... https://github.com/linuxkidd/allsky/blob/packaging/config/allsky_config.json.repo

[ckuethe]

Part of me wonders whether or not to draw some line in the sand, fix all the bugs we know about, and tag a release. Then we can hack with wild abandon in a dev branch which brave folks cand play with too, and once we're happy with it merge it back.

Or we can keep developing in main/master and give the advice to check out the 'current' or 'latest tag if you want something that just works. Basically, I'm looking for a low friction way to allow us to develop rapidly, without creating too many sudden breaking changes.

[linuxkidd]

Ya, @ckuethe ... that's very much a good idea I think. Maybe cap off / tag 0.8.5 and get all the known issues resolved there ( I think it's probably very close at this point. )

Then, we update the README so that people use that tagged release by default ( or just leave master at that level until we're ready ).

Next iteration we move to the dated release versions with the breaking changes.

Thoughts?

[EricClaeys]

I like the idea of having a stable release people get by default and another release people can test with. At one point Thomas and I talked about releasing things every couple weeks.

I would like to fix the dark subtraction bug before we lock the release. It impacts darkCapture.sh and darkSubtract.sh.

Eric

---

From: Michael J. Kidd @.> Sent: Friday, October 1, 2021 3:17:33 PM To: thomasjacquin/allsky @.> Cc: EricClaeys @.>; Mention @.> Subject: Re: [thomasjacquin/allsky] Request for comment - Binary packages and apt repo (#506)

Ya, @ckuethehttps://github.com/ckuethe ... that's very much a good idea I think. Maybe cap off / tag 0.8.5 and get all the known issues resolved there ( I think it's probably very close at this point. )

Then, we update the README so that people use that tagged release by default ( or just leave master at that level until we're ready ).

Next iteration we move to the dated release versions with the breaking changes.

Thoughts?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/thomasjacquin/allsky/issues/506#issuecomment-932489481, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AT2PYK52EVNQAOVEQ77VD43UEYCM3ANCNFSM5EVDHMFA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[ckuethe]

Yeah, no worries. I'm glad we agree on tagging a stable point release. I'll save multi_out and qhy and other stuff for after we cut a new release.