chitin3 Roadmap

[SamStudio8]

trafficstars

In case you hadn't noticed, I trashed the entire chitin repo to make chitin2. It's around half as much less garbage as the last time and moves a little away from the idea of replacing your shell, but instead wrapping a script to keep track of what happens inside. I got overexcited in the last version, and made chitin a clever, parallelised shell that sent commands to a remote machine and allowed any chitin-capable shell to download and process the jobs. At this point, I realised I'd made a grid engine, so I've nuked the code base and started over: this time trying to remember the goal of chitin was to be a watchful guardian of your filesystem. For a reminder of my November 2016 tirade that caused chitin to come into existence, check my blog.

Pretty much all the cool features of chitin1 are missing, but I plan to bring them back:

• [ ] filetype handlers (eg. catching the number of alignments in a BAM)
• [ ] command handlers (eg. remembering the alignment rate from bowtie2 stderr)
• [ ] fetch or put a resource (file) to any other of your machines if a command needs it

I've finally made a business decision about the metadata storage part of chitin. I don't like sqlite, the database gets big, slow and locked. Originally the metadata was to be presented in the terminal (and it was), but we've outgrown this by necessity (commands and resources are linked together and I want you to click on them to find out stuff). Thus we're in your browser. The current version of chitin2 has an integrated webserver using Flask and SQLalchemy but this is troublesome for migration, and it was never my intention to bundle the shell-part and web-part together. Thus my roadmap includes:

• [x] Extracting all the web crap and deploying it to django instead; I'll be making a chitin-server repo soon. Django is definitely OTT for this, it's also wonderfully crafted, extendable, well-supported and has an excellent database migration system. chitin-server

Additional ideas of things that are to come:

• [ ] An arbitrary resource monitor can ping the server with CPU/RAM info every minute or so, we'll then graph these between the start and end of a command on its detail page; alternatively, we could keep track of the PID for a command and try to keep more specific numbers.
• [ ] Leverage all of your hashes and sell chitin-coins
• [ ] Associate ENA ids with the chitin:\ resource locator
• [ ] Dumping generated graphs and some text data for an experiment to the server

Early 2019 Stories

• [ ] User should be able to see most recent commands run on a node
• [ ] User should be able to see most recent resource changes on a node
• [ ] User should be able to see a list of over-arching top-level "projects"
• [ ] User should be able to see a list of "experiments" that belong to a project (eg. all the assemblies), ideally an API would be able to generate tables to present parameters/results

Late 2019 Stories

• [x] ditch chitin-server and plug chitin into majora (https://github.com/SamStudio8/majora)
• [ ] rename chitin to ocarina

[SamStudio8]

Server repo established, functionality has basically been fully ported from the current web UI to a Django one instead. https://github.com/SamStudio8/chitin-server

[SamStudio8]

The API is pretty much in the same condition as it was before, but is now Django based. I've got it talking with the "old" chitin API test script with little modification.

[SamStudio8]

I've killed the old code that hosts the Flask/SQLAlc chitin.core API and website from the chitin repo, and updated the API calls in chitin.client to coincide with some very small changes in the new Django-based API. So we're effectively where we started, but the client and server components are now properly separate.

[SamStudio8]

Additionally I've nuked the develop branch with the chitin2 branch, just to make sure nobody gets the wrong idea about where there is going

[SamStudio8]

Filetype and command handlers are back

[crazyhottommy]

I am waiting for a tutorial on chitin2! Thx!

[SamStudio8]

@crazyhottommy Hey Ming, this is somewhat intentional at the moment, as I'm still working on some core features that I want to pin down before letting people actually use chitin! The database schema and API are high-risk at the moment, which is no good for real production users...

[crazyhottommy]

@SamStudio8 waiting patiently :)

[SamStudio8]

we're at it again

[SamStudio8]

chitin3 now talks to https://github.com/SamStudio8/majora, which is certainly something functionality wise, it does less than ever, so that will have to be rectified soon

[SamStudio8]

same but different