website
website copied to clipboard
datenanfragen
Optimising deploy script to use GNU parallel
Thank you for making your first PR on website!
We are glad about every contribution!
Feel free to add yourself to our AUTHORS file to appear as a contributor on our websites.
![welcome[bot] avatar](https://avatars.githubusercontent.com/in/4141?v=4 "Published by a pub.dev verified publisher"){kind=small}
I need some help fixing this Netlify thing. It says it can't find apk. @mal-tee
I think we have to install that manually. ~~@baltpeter Can you do us the honors?~~
But in general we want to move the installation outside of the deploy.sh script. Mohit, can you please add parallel as a requirement to the README and add it to our CI config: https://github.com/datenanfragen/website/blob/master/.circleci/config.yml
Can you try it again without the install-step? It might be preinstalled in netlify.
Can you try it again without the install-step? It might be preinstalled in netlify.
Looks like it doesn't.
Sad. CI stuff. :( That means we have to check if we are in netlify and if we are, we should obtain a binary from somewhere and just use that. Or build it ourselves in the script.
I've just tested the improved script and I am quite happy:
# old:
./deploy.sh 218,96s user 28,39s system 191% cpu 2:09,35 total
# this PR:
./deploy.sh 221,53s user 19,38s system 250% cpu 1:36,34 total
I've just tested the improved script and I am quite happy:
# old: ./deploy.sh 218,96s user 28,39s system 191% cpu 2:09,35 total # this PR: ./deploy.sh 221,53s user 19,38s system 250% cpu 1:36,34 total
Not sure whether this will yield a noticeable improvement, but it would be nice if we could benchmark whether also parallelizing the copying of the companies and supervisory authorities is worth it.
Thanks! Other than the comments below, if you install
parallelin Netlify, this is good to go as far as I'm concerned.
Resolved the comments. How can I install parallel in Netlify?
That's unfortunately not so easy. You don't have root in the container (https://answers.netlify.com/t/apt-get-install-build-dependency/3095/2) and there doesn't seem to be a binary distribution of parallel (https://www.gnu.org/software/parallel/).
And compiling it each time would likely negate the benefits of using parallel. But you could try this:
https://git.savannah.gnu.org/cgit/parallel.git/tree/README#n84
That's unfortunately not so easy. You don't have root in the container (https://answers.netlify.com/t/apt-get-install-build-dependency/3095/2) and there doesn't seem to be a binary distribution of
parallel(https://www.gnu.org/software/parallel/).And compiling it each time would likely negate the benefits of using
parallel. But you could try this:https://git.savannah.gnu.org/cgit/parallel.git/tree/README#n84
Is it not possible to install it on your Netlify runner from the backend?
Is it not possible to install it on your Netlify runner from the backend?
No.
@baltpeter So I've added the --embed solution but now we are stuck at apk: command not found. @mal-tee You said this needs to be installed manually?
I was slightly mistaken. We need to install it manually as in getting it in the deploy-script, cause we can not install things. Did you try to acquire a binary, as @baltpeter proposed?
https://git.savannah.gnu.org/cgit/parallel.git/tree/README#n84
I was slightly mistaken. We need to install it manually as in getting it in the deploy-script, cause we can not install things. Did you try to acquire a binary, as @baltpeter proposed?
https://git.savannah.gnu.org/cgit/parallel.git/tree/README#n84
That part is done! But to install the binary we need dpkg installed and for that, we need to have apk on Netlify.
Sorry, I can't follow you. I've checked the install script at pi.dk/3/ and the source code's configure and makefile and couldn't find a reference to dpkg.
Sorry, I can't follow you. I've checked the install script at pi.dk/3/ and the source code's configure and makefile and couldn't find a reference to
dpkg.
This is what I am trying to do:
wget "https://download.opensuse.org/repositories/home:/tange/xUbuntu_14.04/all/parallel_20210922_all.deb"
apk add dpkg
dpkg -x parallel_20210922_all.deb ~
I downloaded its .deb and was trying to install it using dpkg.
A couple of notes:
- The
--embedwon't work here. I was referring to the "= Minimal installation =" heading, that's the line I linked to. - As I said, you don't have root in Netlify. Thus, it isn't possible to run
dpkg,apt,apkor any of the common package managers. I know that's annoying but there is nothing we can do about that. - In this case, this isn't really a problem, though. The instructions I have linked show that you can just download
parallelas a script, make that executable and add it to the$PATH. - You'll need to make sure to run these installation steps only for Netlify. See the Netlify docs on how to detect that, pretty sure they set an environment variable.
A couple of notes:
- The
--embedwon't work here. I was referring to the "= Minimal installation =" heading, that's the line I linked to.- As I said, you don't have root in Netlify. Thus, it isn't possible to run
dpkg,apt,apkor any of the common package managers. I know that's annoying but there is nothing we can do about that.- In this case, this isn't really a problem, though. The instructions I have linked show that you can just download
parallelas a script, make that executable and add it to the$PATH.- You'll need to make sure to run these installation steps only for Netlify. See the Netlify docs on how to detect that, pretty sure they set an environment variable.
Thanks for the clarification! I've done it and tested it on my Ubuntu 20.04 and it works. However, it still fails to find parallel on Netlify. Maybe it is using some other version of Ubuntu that is following a different structure. Which Ubuntu container is it using? @mal-tee @baltpeter
Ok so after adding exit to the loop it is now missing the public folder which is being ignored by the .gitignore. I think we are close!
Ok so after adding
exitto the loop it is now missing thepublicfolder which is being ignored by the.gitignore. Is it supposed to fail at this point guys? I think we are close!
Which Ubuntu container is it using? @mal-tee @baltpeter
Seems like netlify are using 20.04 (https://github.com/netlify/build-image/blob/focal/Dockerfile)
Ok so after adding exit to the loop it is now missing the public folder which is being ignored by the .gitignore. Is it supposed to fail at this point guys?
No.
It was missing the public folder due to this branch filter: if [ "$NETLIFY" = "true" ] && [ "$BRANCH" = "master" ];
All checks are passing now! You can re-run your performance tests again @mal-tee @baltpeter
As @mal-tee said, disabling the deploy previews with Netlify is obviously not an option.
I will look at solving the Netlify problem myself on your branch. But I can't make any promises as to when that is going to happen, there's a lot of other stuff I have to do first.
As @mal-tee said, disabling the deploy previews with Netlify is obviously not an option.
I will look at solving the Netlify problem myself on your branch. But I can't make any promises as to when that is going to happen, there's a lot of other stuff I have to do first.
Hey, were you guys able to fix it?
Hey, were you guys able to fix it?
No, unfortunately, I didn't have time for that yet.
Seems like parallel is (or was?) a single perl script: https://www.gnu.org/software/parallel/parallel_design.html