hicetnunc2000
Increase site reliability proposal
As the platform continues to grow in users, we see more and more scaling issues mostly tight to API's fails. But we're also testing new code in production, and all of that reduces trust in the platform and increases stress and frustration with the users.
I'm proposing a couple of actions we need to take to ensure we can test code in a reliable way without compromising the uptime of the production website or the user experience and trust in the platform.
1) Create separate instances for development. For the frontend and for the API. Running the site locally should point to the test instances, and not to the production URL. If this needs to have another blockchain app associated with this, then we should create it as have a few minted pieces for testing. It's important to be able to test new things without pushing to production and if anything fails to revert the code to an old version and get all the new features lost. This has happened twice already and it's very hard to manage and fix.
2) the production API should only accept requests from the production URL. Several people have been building their own apps and data scrappers by accessing our API directly. So far there hasn't been a repercussion, but having the API exposed to external domains means that anyone can write malicious scrips, and flood the API, taking the API and the website down. It's very important that only hicetnunc.xyz is able to access the production API URL.
3) create automatic deployments directly from Github.
If anything fails on the website, we're currently relying on @crzypatchwork to wake up and re-deploy any urgent fixes a collaborator might have done. We should implement continuous integration. It should automatically deploy to production any time someone with privileges deploys to the main branch. In the same way, we should implement continuous integration also on the develop branch, in which any code being pushed will trigger a deployment to a password-protected development URL (proposed on point 1). At the moment as far as I'm aware only myself and @crzypatchwork are able to directly commit code to the repository, anyone else has to create a fork and create a pull request to add new code (which is ideal).
4) Implement error handling messages The current API implementation needs to return a human-readable error message that we can then display on the website explaining what failed and why. At the moment only a user that knows about "dev tools" is able to open the browser's console and inspect for errors. Now that we're running a stable copy of the frontend, most of those errors are always related to the API. If the API fails for some reason, we get a cors issue, and that's a false positive. We're not getting cors issues, we're just not handling the correct errors which right now are caused by our dependency on "Better Call Dev" services being down.
5) implementing a cache database This is probably the most important thing in the long term to implement. At the moment, we're currently relying on other API's in order to get us the data we require. Things like "what NFT's does this user own", "who are the owners", etc. We need to be able to get this data by ourselves without relying on 3rd party API's.
Implementing this ourselves will also allow us to implement several features:
- search functionality on the website, where we search by wallet, or objkt, or description or tags, etc.
- a reporting system where the community can report art that shouldn't be on the platform and art that was stolen.
At the moment we have a very rudimentary filtering system where we filter by objkt and wallets that are on a banlist and this is a manual process where someone needs to do a pull request and get someone else to merge it before it gets updated. This should be done on the database level. So all requests from the database are already filtered. The website's role is to display information not to filter and sanitize data.
Also having this data in the database will allow us to increase better cache. Which will accelerate the response from the API and increase the overall site speed.
6) Unit testing
As suggested on Twitter we should implement unit testing. on the components, but more importantly, on the pages and API calls on src/data/.
There are a lot of little improvements we can do to our current process, some of them I'll be implementing as time permits, like improving the CONTRIBUTING document, implementing hooks on the commits to run prettier and unit tests (point 6) . All of the above points are things that I do not have permissions to do myself and ultimately rely on @crzypatchwork to approve or implement.
Adding this here will give the community time to suggest any other improvement process they might think is appropriate. I'm not a security or blockchain expert, so I'm sure I might have miss something very obvious to someone else. In that case, please give us your feedback below.
How can you help? Reach out on our discord channel, get in touch with @crzypatchwork I'm sure he can create a private channel dedicated to this topic. I don't have knowledge of all the stack, so for example on the topic of continuous integration I don't even know where to start. I have experience in the past from working with things like Heroku, but I'm not sure what hosting platform we're currently using.
I'm thinking about ways we can implement a staging site workflow on staging.hicetnunc.xyz. The idea is obviously to first push to the staging site, test it, and then if its good to go we push it live. Can the person who has access to the domain make a subdomain for staging.hicetnunc.xyz?
That would be @crzypatchwork and also We need another instance of the API. So they both can be tested at the same time and deployed at the same time.
This is list is what will make hicetnunc evolve from an experiment to an actual production ready platform.
Is still a false positive the cors issue? Because we're in October and still get a Cors Policy error from "https://cloudflare-ipfs.com/ipfs/undefined". The undefined part seem pretty smelly to me 👽 Hope this helps you, also because I really love this project.
