Move PNGs with QR codes out of DB

[vitvly]

Description

Type: Feature

PNGs containing QR codes visible in bot comments on GitHub are stored in DB in the issue_comment table. Images in bot comments link directly to the OpenBounty server, and so are fetched from DB each time link is accessed.

Currently, PNG generation/storage is handled by commiteth.util.png-rendering namespace. Image request handling resides in commiteth.routes.qrcodes namespace, it contains a route for qr.png links contained in bot comments.

Relevant queries from queries.sql are save-issue-comment-image! and get-issue-comment-image! .

Solution

• Devise a way of storing PNG binary data outside of DB (either filesystem or external service like S3/IPFS/Swarm/etc.). DB should only store image hashes, or URLs if images will be stored on an external service.
• Implement the solution after gathering feedback from other OpenBounty contributors.
• Provide a script of some kind so that existing assets can be migrated.

[status-open-bounty]

Current balance: 0.0 ETH Tokens: SNT: 5500.00 Contract address: 0xde940f7d50f286319a989e72b20f254a6b455ab0 Network: Mainnet To claim this bounty sign up at https://openbounty.status.im and make sure to update your Ethereum address in My Payment Details so that the bounty is correctly allocated. To fund it, send ETH or ERC20/ERC223 tokens to the contract address.

[msuess]

Taking a stab at this :) I'll do a little bit of research first as to which service to use and post my findings here. @siphiuel do you have any preferences or experience regarding that?

[msuess]

swarm-gateways.net seems very slow.

I posted some IPFS gateway benchmarks here.

[martinklepsch]

This is great stuff @msuess! Seems like Infura might be the best option perf-wise? Did any of the other stand out for other reasons?

[jwagener]

How about simply rendering the images on the fly? A CDN could be used in front of the web server, if load / rendering times are an issue. This would avoid the headache of interacting with external services to store and update the images.

[martinklepsch]

@jwagener that's a great idea to consider. Would be interesting to investigate how long one image-rendering takes given that we currently launch a bunch of processes for it. But yeah I agree, this could be significantly easier :+1:

@jwagener btw are you in Riot as well?

@msuess what do you think about the caching approach?

[martinklepsch]

Completely different approach could be using an API like Figma’s: https://blog.figma.com/introducing-figmas-platform-ee681bf861e7

Not even sure if possible but could be a nice opportunity to test drive some of this pretty exciting (imho), new stuff.

Figma Design

Introducing: Figma’s Platform – Figma Design

The first step toward an open design ecosystem

[jwagener]

I think the Figma API would be a bit overkill, but other image composition APIs could do the job.

I also just realised that SVG images can be used as well. Looks like it is well supported and used by other badges: https://github.com/boennemann/badges

[msuess]

@jwagener @martinklepsch I think that generating SVGs/images is a great idea! Maybe we can find a way to do it in-process, without wkhtmltopdf (so that's what it's used for, hah). @jwagener did you have any specific caching provider/CDN in mind?

[martinklepsch]

Just to double check: if we go with SVG we probably don't need a CDN at all? (for now at least)

[msuess]

I'd say probalby not, especially if we can do it in-process.

[jwagener]

Had no specific CDN in mind. Also agree that it’s probably not necessary with the lightweight SVG solution.

Things to consider:

• providing a legacy support for the existing embedded Png images. Maybe a redirect to SVG works, then png support can be dropped completely.
• Embedding the Roboto font.

I guess a good next step is a proof of concept to check if it works fine to render the current style in SVG.

Do you want to do that? If not I’m happy to jump in.

[msuess]

I can look into rendering the QRs as SVGs, if you want :)

I just realized that I'm not sure what we're trying to fix here @martinklepsch @siphiuel @jwagener. Is this just about reducing the size of the DB or is it about reducing traffic to the OB server? The load?

[martinklepsch]

It's about removing image/binary data from DB. Server load is a non-issue at this point :)

[msuess]

@denis-sharypin could you please provide the status logo (the one with the "S" in the blue circle) as svg so I can embed it within the QR code? Thank you! :)

[denis-sharypin]

@msuess you can take it here — https://www.dropbox.com/s/djrlvrobuk0u98w/status_logo_blue.svg?dl=0

Dropbox

status_logo_blue.svg

Shared with Dropbox

[msuess]

It looks like the embedding of the font might be a bit problematic. I was able to import Roboto as a web font through css, but that only works when embedding as <svg>. The CSS won't be evaluated by the browser if the SVG is loaded through an img tag. Another solution might be to use SVG Fonts, as described here. I'll investigate further.

[martinklepsch]

Thanks for keeping us posted @msuess :+1:

[msuess]

I embedded the font using a css @font-face declaration with a data URI. This works well in Firefox and Chrome, however it blows up the size of the SVG to 400k, which is about 10x the size of the PNGs.

Using some fontforge magic and my awesome StackOverflow copypaste skills I was able to strip the TTF of all non-ASCII characters with a script lifted from here.

This brings down the size of the output SVG to 28KB, which still only saves us about 6KB (and is the same size as the gzipped PNG).

Gzipping the SVG brings down the size to 12K, which is about a third of the size of the PNGs.

Keep in mind that this output file still contains some Inkscape stuff that could be stripped, but I don't think it will save that much.

~~I think the best (as in most space saving) solution at this point might be to programmatically render the SVG, convert the text to paths and stripping the @font-face directive. I am still trying to find some Java libraries that can do this. Basically the same functionality as Inkscape's "Convert to Path".~~

Disregard what I wrote about the paths, I just tried to confirm it with inkscape and it actually makes the output SVG larger (52K) and it's the same as with the stripped font when gzipped (12k).

@martinklepsch @jwagener In light of my findings, do you guys still feel that the SVG solution is lightweight enough?

[martinklepsch]

If we can do SVG with 12k gzipped and it works on major browsers that seems reasonable to me 👍 (did I misunderstand anything?)

[msuess]

Nope, that's what I said. I'll test in Safari, IE, Mobile as well.

[martinklepsch]

Hey @msuess — hope you had a nice weekend! 🙂 Just wanted to check if there has been any further progress on this? With the GitHub proxy stuff you mentioned in #412 I'm wondering if we could maybe use GitHub as CDN? 😛

Darn. It looks like the github proxy is stripping the src attributes from inline SVG CSS definitions. The same thing happens if I host the image somewhere else.

[endenwer]

@siphiuel @martinklepsch Can I just save png to filesystem when creating issue_comment and store image hash in db. If that's ok I can start working on it.

[endenwer]

Hey @andytudhope. Is there anyone who can review my pr?

[pablanopete]

Sorry for the long wait @endenwer -- @siphiuel may be able to review but we're putting a resource hold on Open Bounty for now so this bounty would be considered out of commission for the time being.

cc @andytudhope for revoking