[FEATURE] Allow a HO user to securely submit an encrypted database to the HO developers

[tychobrailleur]

Is your feature request related to a problem? Please describe. Sometimes when a user encounters a problem with HO, developers need to be able to see their database to be able to debug the problem. In the past, users have been sharing their db by uploading it to GitHub, which allows anyone to access their team details.

Describe the solution you'd like HO should provide a feature that automatically encrypts the database and uploads it to GitHub, with only the HO developers able to decrypt, so the data is fully protected.

Additional context N/A

[tychobrailleur]

I have started work on this, and my current thinking on this one is as follows:

• First create a zip file containing the content of the db (and logs?) of an HO instance (facilities already exist in core.db.backup)
• Generate a random 16-byte secret used to encrypt the zip file (using AES-256, CBC mode)
• Encrypt secret with a pre-defined public key (EC, with curve P-256)
• Create a new package with encrypted file + encrypted secret
• Submit that package as part of a new issue to Github using REST API (I think this requires a new Github App with OAuth)

We would then have to have a small utility to decrypt the DB on our side, and agree on how to manage the secret key to decrypt. Thoughts?

[wsbrenk]

@tychobrailleur What would you find if we used pgp for example and built a public key into HO from each developer, which the user can deselect individually when creating the backup. I don't know if it is possible, but a nice feature would be, if the encryption keys will expire after a defined timespan.

[tychobrailleur]

@tychobrailleur What would you find if we used pgp for example and built a public key into HO from each developer, which the user can deselect individually when creating the backup. I don't know if it is possible, but a nice feature would be, if the encryption keys will expire after a defined timespan.

The algorithm I described is pretty much pgp, except the public key is not a signed cert, rather a key we would embed in HO. I however like the idea of a cert, because we could retire it if we need to -- it would have to be self-signed though because I can't afford a proper cert. Also I believe pgp is RSA-based rather than ECC.

I was also thinking one key for all (therefore devs would share the same private key), one key per developer might complicate things further, although there are probably 1-out-of-n schemes that are possible... I can research further. I don't want to roll out my own crypto, though.

Thanks for your comments.

[tychobrailleur]

Also I believe pgp is RSA-based rather than ECC.

Interestingly ECC is not supported yet in plain JDK 17, so I may need to revert to RSA. I knew that, having had that issue in the past, but had forgotten – and the solution would be to use BouncyCastle, but if we can avoid adding that dependency, all the better. Current POC encrypts the zip with a random secret with AES, then encrypts that secret with a hard-coded RSA public key.

[tychobrailleur]

Current POC encrypts the zip with a random secret with AES, then encrypts that secret with a hard-coded RSA public key.

POC now also decrypts key, and successfully decrypts the zip to create a valid HO DB backup. Now onto making this production code. :)

[tychobrailleur]

Some good progress here albeit slow as I have been busy at work: started to clean up the code, and removed the “decryption” code (probably a separate app in the future?), and started looking into Github Apps. Shaping up nicely, getting excited by this!

[tychobrailleur]

I have implemented a successful “on-behalf” authentication for the Github App. The next step is to store the access token for the github user in the database, make the UI for the OAuth flow, and create the GH issue along with the encrypted DB.

[tychobrailleur]

I managed to automatically create a Github issue using my new HOReport Github app: https://api.github.com/repos/tychobrailleur/NotHO/issues/1

I haven't been able to create one under ho-dev/HattrickOrganizer ("Resource not accessible by integration") despite permissions being set correctly. I think it's either due to where the Github app is created, or the owner of the token.

[tychobrailleur]

I have transferred the Github app to the ho-dev organisation, but still getting the same permission issue:

{"message":"Resource not accessible by integration","documentation_url":"https://docs.github.com/rest/issues/issues#create-an-issue","status":"403"}

I tried to add additional permissions, such as workflow, but no luck. There must be some other permission required due to the fact that the repo belongs to an organisation, or that there are templates and workflows associated with our issues... Still digging, but this is quite frustrating...

[tychobrailleur]

Finally got this to work by installing the Github App on the organisation itself. Duh.

Next steps:

• Build the UI, incl. to handle the OAuth flow
• Upload the encrypted attachment, and link to the issue
• Build the decrypting application,
• Generate a Production key pair.

@wsbrenk for decryption, I am hesitating between building a separate standalone app, or making it part of HO itself. Only thing is that this would only be used by HO developers, and would create useless bloat for all remaining users. WDYT?

I still haven't explored the multi-key aspect, but I suspect this might be quite complex to implement. That will be for a later date, I think.

[wsbrenk]

@tychobrailleur if decryption becomes part of the HO, it could be perceived as a security problem. An app that is only available to selected developers should be less of a concern. Starting with the simple solution is a good idea.

[tychobrailleur]

@wsbrenk Question for you: I have taken the route of adding the capability of creating Github issues from HO!, but I am having some doubt as to whether this is the right approach, as this doesn't necessarily get in sync with the templates we have for submtting bugs. What would you be your preference:

• Being able to create an issue from an HO, and having the ability to optionally add the encrypted DB (current implementation),
• Or specify an existing Github issue in HO, and request the encrypted to be added to that issue?

After a bit of thinking, I am now leaning more on the latter. Any thoughts?

[wsbrenk]

@tychobrailleur first reflex is to choose the first point, because most of our users are not familiar with github and it's usage. Delux solution would be that on follow-up issues the user would get an option to add another report to a previously created ticket.

Logs are also transferred? Is there a new error message dialog, which allows the creation of tickets?

[wsbrenk]

@tychobrailleur I'm thinking about releasing the first 9 beta in the next few weeks

• what do you want to finish?
• What do you think of that?

[tychobrailleur]

Sounds good. There is still a fair bit of work to get this issue done, but I can try to push to get it finished for 9.0... No pressure :-)

[wsbrenk]

Of course no pressure... If it takes months instead of weeks, it takes months!

[tychobrailleur]

@wsbrenk I think I am going to pull back my ambitious plans a little bit, as I have been trying to figure out how to upload files to Github issues for a while, without success. One alternative route I explored was also upload files to a separate ho branch, but I am not comfortable giving the Github App write permissions to the repo.

For now my thinking is that I will create the encrypted file, direct the user to it on their own machine, and redirect them to github issues for manual upload through the browser, and can figure out a more automated way to solve this! I just need to move on from this to be able to contribute other things to HO!

[wsbrenk]

I'm fine with this. Delay this ticket to Milestone 10, if you want.

And rest for a while with easier tickets...