Testing HTTPS

[tmanternach]

I saw on Dave's Drummer change notes that he posted about HTTPS today. Here is my experience with making this work.

Preface: I am going to assume you already have (or know how to create) an SSL certificate for your domain. If you need a nudge, Google your web server (Apache/nginx) and "Lets Encrypt" for tons of instructions on how to deploy a free one. All of my experience is with a DigitalOcean VPN running Ubuntu and nginx. You can visit my working site here.

I'd love to help anyone else work through this!

1. You need to host the 3 template files listed in the Drummer docs yourself. These are index.html, styles.css, and code.js. They are discussed in the Drummer docs here.
2. These files need to be accessible on the internet via https. I did this using a sub-domain (so you technically would need 2 SSL certificates). I couldn't figure out a clean way to server these from the same domain I am using for Drummer.
3. You need to edit your index.html template file and replace all the references to fargo.io and scripting.com with the s3.amazonaws.com/fargo.io and s3.amazonaws.com/scripting.com equivalents. Here is what my index.html file looks like after those edits. The reason for this is so these files get downloaded via https in your browser. If not, modern browsers don't allow for "mixed content" (the main page loaded via https but script/resources loaded via http).
4. Once your index.html is accessible via https, you need to add a "urlTemplate" OPML header to your blog.opml. This is done by clicking File->Edit OPML Headers in Drummer and clicking the plus button. The value of mine, for example, is "https://static.trevormanternach.com/index.html"
5. You need to configure your web server properly to 1) proxy your main blog to your oldschool.script.com url and 2) serve your 3 template files via https. Here is my nginx config for doing all of this.

Cheers; have a good weekend everyone!

[scripting]

I added a link to this item from the change note about HTTPS.

[sabre23t]

You can visit my working site here.

I just reviewed your blog, tmanternach. It doesn't have the twitter sign on icons, and no retweet buttons for the post. Was that intentional or limitation in https mode?

[tmanternach]

@sabre23t, good catch! Not intentional, no. I'll take a look at this tomorrow and report back.

[am1t]

@tmanternach This is brilliant. I was able to set my blog behind HTTPS too. I also managed to make the same domain serve the static files. Here's the relevant nginx.conf file. And just to document, I also see the Twitter sign on icon and retweet buttons too. So, it is not related to HTTPS for sure.

Thank you for attempting and documenting this! This is my working blog now.

[tmanternach]

@am1t Looks great. Your Twitter sign on icon is even working. I need to see what I messed up on my site for that and the reblog buttons.

Good work with the aliases. I had tried that the other day, but didn't try with a subfolder like you did and I ran into issues.

[jackbaty]

I've just added https support to my blog as well, so thanks for this writeup! A couple of notes from my experience...

I am using a copy of the most recent template from Dave's repo here: https://github.com/scripting/drummerCms. Your example in the gist looks somewhat different (other than the s3 link replacements, of course). I wonder if that's part of why you're not seeing the twitter and version bits at the top.

I was getting mixed-content warnings until I realized my header image attribute was using http://. It's a good idea to check that all referenced files and images use https://.

I changed my urlBlogWebsite attribute to use the https:// version. It works without that change, but all the links will still point to the non-ssl version and this adds an unnecessary redirect when clicking links.

Otherwise, this is awesome, so thanks again. The key for me was learning that there were https:// equivalent URLs in S3 for all of the references (scripting.com, fargo.io, etc.)

Update: It seems the internal links are still using http:// even though I have urlBlogWebsite set as "https://rudimentarylathe.org/". Not sure why, yet.

[tmanternach]

@jackbaty and @am1t, I think I figured out my issues! I had somehow managed to use some really different template files. I'm not even going to pretend to understand how I did that, but I grabbed new ones from that drummerCms repo and everything looks happier now.

I've updated my index.html gist from above to reflect my now corrected template.

Lastly, @jackbaty, to comment on your update about links not being https, I think that isn't a huge deal. Modern browsers try pretty heavily to prefer https, and your web server looks to be properly redirecting a true http:// request to https://

trevormanternach@Trevors-MacBook-Pro ~> curl "http://rudimentarylathe.org/rss.xml" -I
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Date: Sat, 16 Oct 2021 13:16:45 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://rudimentarylathe.org/rss.xml

[jackbaty]

Another note. I added a new link in Radio3 (after moving the blog to https) and the links stopped showing in the LInks tab (the tab was blank). Suspecting an http/https thing, I changed my urlLinkblogJson attribute from http://radio3.io/users/jackbaty/linkblog.json to https://s3.amazonaws.com/radio3.io/users/jackbaty/linkblog.json and the radio3 links are now showing again. I can't be 100% sure that was the cause and fix (it could have been just coincidence and timing, and I did try doing a hard reload first) but seems likely. I mention it in case anyone runs into the same thing. Worth a shot.

@tmanternach You're right, the links will be redirected properly. It's just a loose end I'd like tied up if possible.

[scripting]

BTW, the template files are already accessible over HTTPS.

If you use these URLs, I will be able to fix any bugs that come up in them and have those changes go through to the HTTPS sites.

1. https://s3.amazonaws.com/scripting.com/code/drummercms/templates/minimal/styles.css

2. https://s3.amazonaws.com/scripting.com/code/drummercms/templates/minimal/code.js

3. https://s3.amazonaws.com/scripting.com/code/drummercms/templates/minimal/index.html

[tmanternach]

@scripting, the main issue is with all of the scripts and stylesheets included in index.html. If you leave that file as-is and try to use HTTPS, it doesn't work, because farg.io and scripting.com aren't available via https directly.

I'm not sure if you'd want to or not, but if you updated index.html to reference//s3.amazonaws.com/scripting.com instead of //scripting.com and //s3.amazonaws.com/fargo.io in place of //fargo.io, then I believe we wouldn't have to host (and edit index.html) these template files ourselves. ;-)

EDIT: There is also a radio.io stylesheet that would need to get updated to its s3 equivalent.

[scripting]

@tmanternach -- of course -- i'll go back into lurk mode. ;-)

I'm juggling too much stuff right now to take this on.

Make minimal changes to make it work, and at some point hopefully i'll be able to pop up for air and review them. If they work and look good i can possible merge them back into the default versions.

And thank you for taking charge here. So far it's a prototype for other collaborative projects I hope to start. :smile:

[scotthansonde]

Since all the cool kids are doing HTTPS these days, I'm doing it too.... https://drumbeat.scotthanson.eu (Don't bookmark that, it's just for testing. 😄 )

I'm using Caddy to reverse proxy, here is the Caddyfile.

drumbeat.scotthanson.eu {
rewrite * /ScottHansonDE{path}
reverse_proxy http://oldschool.scripting.com {
      header_up Host {upstream_hostport}
      header_up X-Forwarded-Host {host}
  }
}

tmpl.scotthanson.eu {
reverse_proxy localhost:1339
}

That's it. The drumbeat.scotthanson.eu with no port specified handles both HTTPS and redirecting HTTP to HTTPS. It also automatically gets the certs from Let's Encrypt. The tmpl.scotthanson.eu is just where I threw the templates in PagePark.

[jackbaty]

At the moment, the About tabs are not loading content on any of the blogs from this thread that are otherwise working well under https, mine included (https://rudimentarylathe.org/?tab=about). See also Amit's at https://ol.amitgawande.com/?tab=about.

I'm posting this in the https thread, because it feels https related due to this error that appears on the console (in latest Safari, macOS):

XMLHttpRequest cannot load http://httpproxy.scripting.com/httpReadUrl?url=http%3A%2F%2Fdrummer.scripting.com%2Fjackbaty%2Fabout.opml&type=text%2Fplain due to access control checks.

I've done the usual hard reload, multiple browser testing with the same result.

[dcoletta]

@papascott I did not know about Caddy but clearly it is very cool. Thanks for that recommendation. However, I tried setting up Caddy the way you described, and ran into mixed-content errors on all the files that are loaded from http://scripting.com. I looked at your site to see how you worked around that, and I can see that it's loading those files from AWS over https instead. Guessing you made your own template which loads them from there? I can see you still have maybe one mixed-content error left?

I guess that copying the files hosted at http://scripting.com to somewhere else where you can host them over https is necessary, but do you have a thought about how to keep them up to date?

[scotthansonde]

@dcoletta I was following @tmanternach's advice at the top of this issue to change the template to link to Dave's files directly on S3. Those are Dave's actual S3 buckets, so we'll catch any changes he makes.

There are a couple of http requests inside the client-side scripts (that's why the About pages aren't working), but that's a task for another day. 😄

[jackbaty]

@papascott You're saying that the failing request is generated from somewhere in the template script (code.js)? I didn't see it there, but if this is something we (as users) can modify ourselves I'm all ears :).

[dcoletta]

@papascott I should really learn to read from the top down instead of the bottom up. The good news is now I know what to do. Thanks!

[dcoletta]

With that extra clue I've got https://cannabiscto.com working. Thanks!

[tmanternach]

At the moment, the About tabs are not loading content on any of the blogs from this thread that are otherwise working well under https, mine included (https://rudimentarylathe.org/?tab=about). See also Amit's at https://ol.amitgawande.com/?tab=about.

@jackbaty, you are on to something here. I think the console message you posted is unrelated, though (the one you posted is probably a difference problem, btw). When I browse to my About tab via https, I get a slightly different message about failing to load a request to http://httpproxy.scripting.com/httpReadUrl.

It looks like Dave has open-sourced the software running at that url in this repo. Just as a proof of concept, I am going to try to run this node program behind an HTTPS endpoint and edit my code.js file (starting at line 426 for those following along) to replace the httpproxy.scripting.com url with my own. This wouldn't be any sort of production fix, just me mostly trying to see what is possible here.

[dcoletta]

I added a favicon to my https-via-Caddy setup. It was non-trivial, because I decided to put the icon resources into PagePark, so they had to be served from the tmpl subdomain. But since the site manifest was also in the tmpl subdomain, I had to enable CORS in order to <link> to it.

This worked, but I'm probably missing something. If there is support for favicons in Drummer, then there's probably a way to do this that doesn't involve hosting my own site manifest.

Or maybe there is an option in Caddy's reverse_proxy command that allows me to write a regular expression for URLs not to proxy.

[jackbaty]

@tmanternach You're absolutely right, I pasted the wrong error from the console, making my report nearly useless, sorry. I've edited the comment with the correct error. Thanks for pointing that out.

[tmanternach]

@jackbaty, I got this to work (proof here), but it is a huge mess that I don't think is sustainable.

I am not actually running the httpproxy node app myself. I am just proxying it (much like the blog itself) through an HTTPS URL (https://static.trevormanternach.com/httpReadUrl proxies to http://httpproxy.scripting.com/httpReadUrl).

I then had to download (and host myself) a separate code.js file. My version of that file is here. I replaced the script src in index.html template (line 19 on mine) to replace "my" version with the original. I then edited "my" basic-code.js file and replaced line 426 with my proxy URL for httpReadUrl.

TLDR: Giant kludge, would not recommend.

[jackbaty]

@tmanternach Nice work, but yeah, seems like a bit too many moving parts :). I'm probably better off waiting to see if Dave knows of a simpler solution.

[scripting]

what's the question, as concisely as possilble?

[dcoletta]

As a Drummer blogger, I'd like my https-hosted Drummer blog to be able to load the content of a tab without a mixed-content error.

[tmanternach]

what's the question, as concisely as possilble?

@scripting, Would it be possible to serve httpproxy.scripting.com via http and https? ;-)

[scripting]

Okay I have no idea why that's something you need me to answer??

I am not the HTTPS dude here, you are. :smile:

[scripting]

okay i think what you mean is would i be willing to host it, somehow with some additional support for https.

is that correct?

it's something that you need to happen on my server?

[tmanternach]

I just answered your question. :-D I'm not actually asking for anything myself right now.

[scripting]

ok cool! that's the best thing possible. I already have far too many things to do! :-)