socallinuxexpo
Dealing with repo artifacts in a uniform manner
Description
The tech team has several artifacts that come out of this repo. To name a few:
- Switch configs
- Switch postscript port maps and pdfs
- Pi images
- Openwrt images
- Openwrt template outputs
We need a way to collect and version these artifacts
Acceptance Criteria
- Inventory all artifacts that are generated by this repo
- Establish a process for collecting artifacts
- Establish a process for publish those artifacts for the team to consume
Perhaps I am confused…
This sounds like attempting to create a parallel repo for object code versions of a product.
Can’t we reproduce any version of the “artifacts” simply by checking out the correct version of the repo and building them?
I agree it would be good to inventory all such items and ensure that the repo contains instructions for how to build them.
We should also make updating the build instructions whenever relevant part of our checkin review responsibilities.
It would probably also be a good idea to tag the as-built repository before each show with a SCaLENNx “tag” of some form for easy recall. Then, if we end up applying any sort of patches or updates during the show, those would get incremental SCaLENNx.y tags. Finally, the base at the end of the show would get tagged “SCaLENNx.final”
Thoughts?
Owen
On Feb 21, 2020, at 08:40 , Robert James Hernandez [email protected] wrote:
Description
The tech team has several artifacts that come out of this repo. To name a few:
Switch configs Switch postscript port maps and pdfs Pi images Openwrt images Openwrt template outputs We need a way to collect and version these artifacts
Acceptance Criteria
Inventory all artifacts that are generated by this repo Establish a process for collecting artifacts Establish a process for publish those artifacts for the team to consume — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/286?email_source=notifications&email_token=AAK6GTWP5K4UOPR3IAMB4ZDRD77WZA5CNFSM4KZGOADKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPLGSVA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTQTKNMDNRK4SFKUPC3RD77WZANCNFSM4KZGOADA.
we really do need the ability to have people get the latest version of the artifacts and use them without having to have a full build environement (or having to wait for things to build)
examples:
your pdf switch charts, which are trival to build on linux/mac, but hard on a windows machine
AP images take a lot of resources (including time) to compile. People should be able to get an image and flash it quickly.
David Lang
On Fri, 21 Feb 2020, owendelong wrote:
Date: Fri, 21 Feb 2020 14:15:28 -0800 From: owendelong [email protected] Reply-To: socallinuxexpo/scale-network [email protected] To: socallinuxexpo/scale-network [email protected] Cc: Subscribed [email protected] Subject: Re: [socallinuxexpo/scale-network] Dealing with repo artifacts in a uniform manner (#286)
Perhaps I am confused…
This sounds like attempting to create a parallel repo for object code versions of a product.
Can’t we reproduce any version of the “artifacts” simply by checking out the correct version of the repo and building them?
I agree it would be good to inventory all such items and ensure that the repo contains instructions for how to build them.
We should also make updating the build instructions whenever relevant part of our checkin review responsibilities.
It would probably also be a good idea to tag the as-built repository before each show with a SCaLENNx “tag” of some form for easy recall. Then, if we end up applying any sort of patches or updates during the show, those would get incremental SCaLENNx.y tags. Finally, the base at the end of the show would get tagged “SCaLENNx.final”
Thoughts?
Owen
On Feb 21, 2020, at 08:40 , Robert James Hernandez [email protected] wrote:
Description
The tech team has several artifacts that come out of this repo. To name a few:
Switch configs Switch postscript port maps and pdfs Pi images Openwrt images Openwrt template outputs We need a way to collect and version these artifacts
Acceptance Criteria
Inventory all artifacts that are generated by this repo Establish a process for collecting artifacts Establish a process for publish those artifacts for the team to consume — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/286?email_source=notifications&email_token=AAK6GTWP5K4UOPR3IAMB4ZDRD77WZA5CNFSM4KZGOADKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPLGSVA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTQTKNMDNRK4SFKUPC3RD77WZANCNFSM4KZGOADA.
-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/socallinuxexpo/scale-network/issues/286#issuecomment-589859992
On Feb 21, 2020, at 15:15 , David Lang [email protected] wrote:
we really do need the ability to have people get the latest version of the artifacts and use them without having to have a full build environement (or having to wait for things to build)
examples:
your pdf switch charts, which are trival to build on linux/mac, but hard on a windows machine
It’s hard to run a perl script on a windows machine? When did this happen?
(Their only dependency is PERL and a few CPAN modules).
Literally it parses some text files and produces other text files containing PostScript.
I suppose there might be some issues with the fact that it uses “/“ as a directory separator and Windows doesn’t.
AP images take a lot of resources (including time) to compile. People should be able to get an image and flash it quickly.
Perhaps we should add a VM to the year-round SCaLE server and have it be a web site that hosts releases of the compiled tech repository. We could create a hierarchy where the root would contain a list of versions (e.g. SCaLENNx.RCvv, SCaLENNx, SCaLENNx.Pvv, SCaLENNx.final, etc.). These should correspond to the aforementioned tags I proposed in the repo. I believe the directories in .gitignore are a good place to start looking for the inventory of what goes into the release. Obviously not all of them are object-code, but I think most, if not all of the object-code does end up there. There should be symlinks “Production” and a link “Latest” in the top level of the hierarchy that are curated to point to the most recent appropriate versions.
Owen
David Lang
On Fri, 21 Feb 2020, owendelong wrote:
Date: Fri, 21 Feb 2020 14:15:28 -0800 From: owendelong <[email protected] mailto:[email protected]> Reply-To: socallinuxexpo/scale-network <[email protected] mailto:[email protected]> To: socallinuxexpo/scale-network <[email protected] mailto:[email protected]> Cc: Subscribed <[email protected] mailto:[email protected]> Subject: Re: [socallinuxexpo/scale-network] Dealing with repo artifacts in a uniform manner (#286)
Perhaps I am confused…
This sounds like attempting to create a parallel repo for object code versions of a product.
Can’t we reproduce any version of the “artifacts” simply by checking out the correct version of the repo and building them?
I agree it would be good to inventory all such items and ensure that the repo contains instructions for how to build them.
We should also make updating the build instructions whenever relevant part of our checkin review responsibilities.
It would probably also be a good idea to tag the as-built repository before each show with a SCaLENNx “tag” of some form for easy recall. Then, if we end up applying any sort of patches or updates during the show, those would get incremental SCaLENNx.y tags. Finally, the base at the end of the show would get tagged “SCaLENNx.final”
Thoughts?
Owen
On Feb 21, 2020, at 08:40 , Robert James Hernandez <[email protected] mailto:[email protected]> wrote:
Description
The tech team has several artifacts that come out of this repo. To name a few:
Switch configs Switch postscript port maps and pdfs Pi images Openwrt images Openwrt template outputs We need a way to collect and version these artifacts
Acceptance Criteria
Inventory all artifacts that are generated by this repo Establish a process for collecting artifacts Establish a process for publish those artifacts for the team to consume — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/socallinuxexpo/scale-network/issues/286?email_source=notifications&email_token=AAK6GTWP5K4UOPR3IAMB4ZDRD77WZA5CNFSM4KZGOADKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPLGSVA https://github.com/socallinuxexpo/scale-network/issues/286?email_source=notifications&email_token=AAK6GTWP5K4UOPR3IAMB4ZDRD77WZA5CNFSM4KZGOADKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPLGSVA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK6GTQTKNMDNRK4SFKUPC3RD77WZANCNFSM4KZGOADA https://github.com/notifications/unsubscribe-auth/AAK6GTQTKNMDNRK4SFKUPC3RD77WZANCNFSM4KZGOADA>.
-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/socallinuxexpo/scale-network/issues/286#issuecomment-589859992 https://github.com/socallinuxexpo/scale-network/issues/286#issuecomment-589859992 — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/286?email_source=notifications&email_token=AAK6GTSUANMC4YYDQJTVAY3REBN7PA5CNFSM4KZGOADKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMUM2PI#issuecomment-589876541, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTUWD42ZGB45BMDFSZ3REBN7PANCNFSM4KZGOADA.
This sounds like attempting to create a parallel repo for object code versions of a product.
No I dont think we would need to do that. We have a couple of options:
- github has the ability to store artifacts with a given release (they all them assets but same thing): https://developer.github.com/v3/repos/releases/#upload-a-release-asset
- s3 bucket since github does have a 2GB limit which might not work for some things (pi images)
Can’t we reproduce any version of the “artifacts” simply by checking out the correct version of the repo and building them?
Yes we can, but I dont think the expectation should be that everyone has to produce their own artifacts or figure out where they might be stored (currently its not obvious). Having a single source of truth would be beneficial.
I agree it would be good to inventory all such items and ensure that the repo contains instructions for how to build them. We should also make updating the build instructions whenever relevant part of our checkin review responsibilities.
Since source of truth would fix these issues and re-enforce the instructions. I think generally we've been ok on the updating instructions part.
It would probably also be a good idea to tag the as-built repository before each show with a SCaLENNx “tag” of some form for easy recall. Then, if we end up applying any sort of patches or updates during the show, those would get incremental SCaLENNx.y tags. Finally, the base at the end of the show would get tagged “SCaLENNx.final”
I tend to agree with you here. We had a scenario last year https://github.com/socallinuxexpo/scale-network/pull/203 that we held off merging due to the fact that currently we operate directly against master and couldnt risk this effecting the show. I dont mind creating a branch off of master at the time of the show and then cherry-picking things into it that are patches/updates for the show and bumping the tag.
In fact, Ive already been tagging post show master so we have tags for each post-scale since this repos been in use: https://github.com/socallinuxexpo/scale-network/releases
It’s hard to run a perl script on a windows machine? When did this happen?
I havent run windows in 10+ years but Id rather not have to help someone troubleshoot this. Artifacts sound so much better.
Perhaps we should add a VM to the year-round SCaLE server and have it be a web site that hosts releases of the compiled tech repository. We could create a hierarchy where the root would contain a list of versions (e.g. SCaLENNx.RCvv, SCaLENNx, SCaLENNx.Pvv, SCaLENNx.final, etc.). These should correspond to the aforementioned tags I proposed in the repo. I believe the directories in .gitignore are a good place to start looking for the inventory of what goes into the release. Obviously not all of them are object-code, but I think most, if not all of the object-code does end up there. There should be symlinks “Production” and a link “Latest” in the top level of the hierarchy that are curated to point to the most recent appropriate versions.
@owendelong I agree, Ill make it a topic of discussion tomorrow once we have things rolling for the core team to discuss this in more depth. But the webserver seems like a solid way to go about this during the show.
On Fri, 21 Feb 2020, Robert James Hernandez wrote:
It’s hard to run a perl script on a windows machine? When did this happen?
would you like to try and walk Stephan though installing Perl on his laptop? (no offense to Stephan, that's just not his area of expertise)
if we were to start using ps2pdf to convert those diagrams to postscript, that's not a tool that's readily available on windows.
David Lang
I agree, Ill make it a topic of discussion tomorrow once we have things rolling for the core team to discuss this in more depth. But the webserver seems like a solid way to go about this during the show.
How about we discuss Sunday so I can participate in the discussion remotely when I’m not driving a train?
Owen
On Feb 21, 2020, at 17:48 , David Lang [email protected] wrote:
On Fri, 21 Feb 2020, Robert James Hernandez wrote:
It’s hard to run a perl script on a windows machine? When did this happen?
would you like to try and walk Stephan though installing Perl on his laptop? (no offense to Stephan, that's just not his area of expertise)
I’m probably not the ideal person to do so, but yeah, I could muddle through it if necessary.
From here: https://learn.perl.org/installing/windows.html it looks pretty straight forward and I suspect Steven wouldn’t have much difficulty following the process there.
if we were to start using ps2pdf to convert those diagrams to postscript, that's not a tool that's readily available on windows.
Not a tool I use on the Mac, either. On the Mac, it’s utterly unnecessary as you can open a PS in Preview the same as a PDF. Preview can then print to PDF or save as PDF.
Apparently Ghostscript is available for windows with a printer driver for PDF output.
Looks like Adobe has a “Distiller Service” for windows that will also do that.
Honestly, since Mac has been able to do this for more than 15 years, I assumed Micr0$0ft must have caught up on that feature by now.
Owen
Despite my limited involvement with the core team going forward I’ll throw out an opinion. I support the view that artifacts should be distributed. I think s3 is the way to go. The barriers to entry for helping the team should be as low as possible so more people can jump in and also to support the “transient team” model. I think this a no brainer for AP images but generally like it even for the smaller items.
On Feb 21, 2020, at 18:22 , kylerisse [email protected] wrote:
Despite my limited involvement with the core team going forward I’ll throw out an opinion. I support the view that artifacts should be distributed. I think s3 is the way to go. The barriers to entry for helping the team should be as low as possible so more people can jump in and also to support the “transient team” model. I think this a no brainer for AP images but generally like it even for the smaller items.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/286?email_source=notifications&email_token=AAK6GTRUTWJ5TKGF5RDDSYTRECD75A5CNFSM4KZGOADKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMUUMRQ#issuecomment-589907526, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTWN2K7SFASLMF4VXCTRECD75ANCNFSM4KZGOADA.
I’d rather we used the VMs that we’re already maintaining for the SCaLE web site or new VMs on the same hardware.
Here’s the main reason why:
kiev:owen (350) ~/SCALE/networking/scale_repo/scale-network/switch-configuration/config % host robhernandez.s3.com robhernandez.s3.com has address 104.199.117.31
kiev:owen (351) ~/SCALE/networking/scale_repo/scale-network/switch-configuration/config % host www.socallinuxexpo.org www.socallinuxexpo.org has address 151.101.66.217 www.socallinuxexpo.org has address 151.101.2.217 www.socallinuxexpo.org has address 151.101.130.217 www.socallinuxexpo.org has address 151.101.194.217 www.socallinuxexpo.org has IPv6 address 2a04:4e42::729 www.socallinuxexpo.org has IPv6 address 2a04:4e42:600::729 www.socallinuxexpo.org has IPv6 address 2a04:4e42:200::729 www.socallinuxexpo.org has IPv6 address 2a04:4e42:400::729
Owen
S3 supports IPv6. I imagine @sarcasticadmin would be willing to setup a AAAA record for his bucket. https://aws.amazon.com/blogs/aws/now-available-ipv6-support-for-amazon-s3/
when we build the AP images that have credentials, they should be put in a password protected https accessable location, and a tarball should be created that includes the ap images and the flash script so they can be added to the offline flashing stations after the pi image is frozen.
my suggestion to @davidelang was that our team specifically have a VPS available year round for things like this and storing logs/configs online between shows. Action item for next team call.