Replace ol-www1 with docker-based ol-www0

[cdrini]

ol-www1 hosts static files as well as the nginx that serves the main site.

---

• [x] Provision the server
  • [x] @mekarpeles to ask for an replacement server to be provisioned for ol-www1 called ol-www0
  • [x] Install Docker and create docker group w/ nagios,cclauss,drini,mek as members (run scripts/setup_olserver.sh)
    • [x] olsystem, openlibrary, infogami is to be synced
    • [x] #4719 openlibrary will live inside the Docker container - Need to update the static stuff (as a part of volume mounts)
    • [x] #4719 Might need i18n, etc. if we do not have volume mounts.
  • [x] Provision to have docker/etc
    • [x] Add HTTPS certs for openlibrary.org

---

• [x] #4725 Create docker images/docker-compose services
  • [x] #4566 Create web_nginx service in docker-compose.production.yml
    • [x] Copy nginx confs from olsystem into openlibrary
  • [x] #4566 Create web_haproxy service in docker-compose.production.yml
    • [x] Copy haproxy confs from olsystem into openlibrary
  • [x] Launch on ol-www0
  • [x] Create a Dockerfile for ol-www1 which includes services for nginx
  • [x] Switch covers to use the IA-specific nginx
  • [x] @cdrini Add www0 to deploy.sh/friends #5954

---

• [ ] #6000

---

• [ ] #6001

---

To Investigate:

• ol-www1 crons: Sitemaps, ipstats
• ol-www1 has some relation to sitemaps; make sure they're still generated. Might need to copy them over.
• ol-www1's nginx logs are read to determine some of the openlibrary.org/stats . Confirm they all still work.
• ipstats crons jobs
• Log strategy: log rotation, where to store; need them to live a long time.

Considerations

If/when we remove volume mounts, our cron jobs may need to be re-architected for commands which ssh to hosts (as the code will now be inside docker and we'll need some way to trigger these commands in docker from ol-home0 + ssh)

[cclauss]

Q: What is the upside of doing all this work? What will www0 give us that www1 does not currently deliver?

A: Being able to build all this static content less often and store those results in an easy-to-serve Docker container.

[cdrini]

@mekarpeles Note: Logs might need to be longer-lived on www1, and regularly uploaded to IA.

[cdrini]

@cclauss One purpose is being able to build static files at all :) Currently we have to build the static files on ol-home outside of docker; this is going to be deprecated as we work towards ~#4060~ (merged) .

Another important purpose is that we make our nginx confs available to the community, so they can make changes which would currently be impossible for them to do (e.g. add HTTP/2 support).

[cdrini]

Note: Might need GeoMind on this :/

[mekarpeles]

@cclauss please check-in with @cdrini first to coordinate but I see this as a potential opportunity for you to help out with the initial step of provisioning ol-www0 like we we have some of the other VMs (e.g. to get Docker installed, etc)

[cclauss]

• [x] Install Docker and create docker group w/ nagios,cclauss,drini,mek as members (run scripts/setup_olserver.sh)
  • Modified scripts/setup_olserver.sh to not install olsystem, openlibrary, infogami, etc.

ol-www0% docker version

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Fri Dec 18 12:15:19 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Fri Dec  4 23:02:49 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

ol-www0% cat /etc/group | grep docker

docker:x:116:nagios,cclauss,drini,me

[cclauss]

Q: Which repos (olsystem, openlibrary, infogami, etc.) need to be installed on ol-www0? I assume we want a minimal set.

[cdrini]

Here's @traceypooh nginx binary: https://archive.org/serve/nginx

[cdrini]

Potential problems:

• ol-www1 crons: Sitemaps, ipstats
• Log strategy: log rotation, where to store; need them to live a long time.
• Messy rebase

[mekarpeles]

What we learned today...

Log Rotation

In order for anonymized log rotation and backups to occur, we rely on a petabox script called /opt/petabox/sw/bin/archive-weblogs.py.

Some notes, (yikes):

1. ol-www1's version of petabox is from 2014 and not version controlled
2. archive-weblogs.py has since been migrated to python3 in petabox
3. ol-www0 will need this script and ol-www1's log rotation + upload cron: /etc/cron.d/archive-webserver-logs + the latest archive-weblogs.py (which we should add to olsystem)

On the last episode of Dragon Ball ol-www0, Goku:

1. [x] Set up volume mounts for /var/log to live in /1/var/log/openlibrary
2. [x] Copy the python file + cron into olsystem
3. [x] We need to create a symlink for: /etc/logrotate.d/nginx -> /olsystem/etc/logrotate.d/nginx
4. [x] Wire up the cron somehow
5. [x] Test: Kick off a real log upload on ol-www0 (this should work fine because our logs will contain host names like /ia_webserverlogs_20210601/openlibrary/20210601-ol-www1-access.log.gz and won't interfere w/ ol-www1 production logs)

Next Steps

In the next exciting episode (session) of Dragon Ball ol-www0 we will:

1. [x] IP anonymization
2. [x] Verify log rotation is happening
3. [x] Merge what we have
4. [x] Verify that ol-www0 actually uploaded via cron to archive.org
5. [x] Add volume mount for a new file openlibrary.ol_www which takes the ol-www* cronts out of crontab openlibrary.allnodes
6. [x] verify Sitemaps :sob:
7. [ ] verify ipstats :sob:

Open Questions:

• [x] How does the log rotation cron run? Can we add it to ol-home0 cron w/ ssh since docker + cron is annoying? A: We put it directly on "ol-www0" docker-compose profile (web_nginx) which is probably where we'll also add sitemaps.

Other Details:

On ol-www1 we ran sudo find /etc -type l -exec readlink -f {} + | grep -F /olsystem to see (we think) where olsystem/etc files were referenced from /etc and we see:

• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/rsyncd.conf N1
• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/logrotate.d/nginx !!!
• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/nginx/sites-available/default
• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/nginx/sites-available/openlibrary.conf
• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/nginx/nginx.conf
• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/nginx/sites-available
• [ ] /opt/openlibrary/deploys/olsystem/54c55f8/etc/nginx/hash_ip_for_logs.conf !!! (done by @traceypooh nginx?)
• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/haproxy/haproxy.cfg
• [x] /opt/openlibrary/deploys/olsystem/54c55f8/etc/default/rsync N1
• [ ] /opt/openlibrary/deploys/olsystem/54c55f8/etc/cron.d/openlibrary.allnodes

Note 1: This was needed for ol-home0 because to enable other VMs to pull data from ol-home0. ol-www* shouldn't need it, to the best of our knowledge unless somehow the python log rotation uses rsync

We believe the relevant files which we'll have to wire up are...

• /etc/logrotate.d/nginx

[cdrini]

Note: not uploaded to https://archive.org/download/ia_webserverlogs_20210609/openlibrary/ :/

[mekarpeles]

re: sitemaps, be warned, ol-www1 is still (to this day) trying to pull from ol-home in allnodes and so I wouldn't be surprised if this is completely borked and we have no sitemaps rolling through. This code should almost certainly be pulling from ol-home0 and also, be added to ol-www0's cron (accordingly)

[cclauss]

• [x] #6000
• [ ] #6001

---

• [ ] #4252
  • [x] #7544
  • [x] Fix IP anonymization #4252 (comment)
  • [x] Fix log rotation #4252 (comment)
  • [x] #7622
  • [x] Migrate ipstats from cron to graphite #7745
  • [x] #5144
  • [x] Fix monthly sitemap rsync #4252 (comment) --> #7781