splits-io
splits-io copied to clipboard
glacials
Parse 100% in Lambda
We currently have an AWS Lambda job that "parses" runs on upload, but it does so by hitting a webhook in Rails that downloads the file and parses it, so still technically on the web servers.
We want to move this parsing to the Lambda job itself, which would let us budget way less memory on the Rails servers, way less disk space, would not consume CPU for parsing giant runs causing web requests to fail, etc. etc.
The Lambda job already has livesplit-core included in it correctly, it just isn't using it right now. We should make it start using it, and somehow insert the parsed data into the database, then notify Rails and/or the user that the run is ready to be loaded.
Is that still going to use the Ruby bindings or should we use the direct Rust + Lambda support? I could look more into that: https://aws.amazon.com/de/blogs/opensource/rust-runtime-for-aws-lambda/
Undecided atm, but the current setup uses the JS bindings. I think in general I'd prefer to use a set of bindings than a language not officially supported by Lambda, though.
Oh oops I didn't click your link, I didn't realize there was a Rust runtime! I'd be willing to try it out.
Yeah if those JS bindings are the asm.js ones (and not the node.js ones) then the native rust runtime would likely be much faster. The node.js ones are probably close.
Oh, and apparently pricing is based on memory usage and cpu time. Both of these would be much lower with the Rust version. So this should absolutely be attempted imo.