The Great GitBook Rewrite

[Trashtalk217]

GitBook takes the content and converts it to the HTML code webbrowsers can read. However, it became apparent that GitBook is no longer a great fit for the AAA.

Reasons

The version of GitBook we are using is no longer supported. This means that security flaws and bugs are no longer fixed by the dev team.

These problems are the main reasons for a migration away from GitBook, but there are also other potential benefits we could enjoy by switching.

Other Benefits

• A Faster Build The build process is currently unbearably slow. I timed it very roughly with my phone and on my machine it takes about 26.7 seconds before you can open the website locally on my machine. This is way too long if you consider that you have to rebuild the website every time you make an edit to an .md file. Debugging becomes especially annoying and time consuming. Switching from GitBook or at least optimizing the build process could mitigate this downtime.

• An Easier Time Adding Features Adding features like the language picker can be a bit of a problem with GitBook. You have to work around GitBook instead of working with it and that can sometimes be a real headache.

• Faster Load Times This one is not a problem yet. Using a bit of Chrome analytics gives the first meaningful paint at around 250 milliseconds after page startup. This is however with a good wifi connection. It would be worth finding out how fast the page loads with airplane internet. It's also worth seeing how extra chapters affect the load times. When we get more chapters, will it slow down the loading as it loads all the content in one go. Again, this isn't a problem yet, but worth looking into when going for a rewrite.

There are probably other advantages to switching I forgot to mention, but these are the main ones. I think I've made my case pretty clear and most of the archivists already agree with me that switching is a good move. This is not my idea, I'm just stating what has already been said before.

Now the question becomes: 'What do we switch to?"

Options

• The New GitBook We are using an old gitbook which is no longer supported, but there is a newer version out there. Sadly, this version isn't open source and monitized. While this version would solve some problems it would create some new ones and is therefore an unattractive candidate.

• Nev's Python Version There already have been made some private attempts at improving the build system of the archive. One of these attempts comes from community member Nev (or Neverik). While I'm not sure if it is totally operational it can be found here.

• MdBook This is a program @berquist has been looking into. It's written in rust and presumably quite fast, but adding the features we need is going to be difficult.

• Just Fork It Another way to fix the problems we have is to just fork the version of GitBook we currently use, rename it AAABook and develop it further to our wishes. The license might get in the way however.

• Alternatives To GitBook There are some other out-of-the-box solutions available, which is probably worth checking out first.

• A Tabula Rasa (blank slate) We could always start from scratch. This way you can make it exactly like you want and use whatever language you want. I would personally advise against this since it takes a long time and you end up reinventing the wheel a lot.

Final Advice

My final advice is to just start. The current system works, but when something works it's all to easy to become complacent. There current setup can improve and if we don't do anything, it won't improve. So I recommend Leios to open up a separate chat in the Discord server under projects called something along the lines of 'AAABook'. It's also important that we have an official place to work on this projects so I recommend Leios to open up a separate project on GitHub. And while I know that it will still take a while to properly get started, I hope that this can be a start.

[leios]

Thanks for the write-up, it is thorough and clear!

I think @berquist was working on an mdbook solution, but couldn't replicate the language switching behaviour. I am also not sure how @Neverik's python build is.

@c252 has also started working on a script to translate the AAA into LaTeX, which might be worth looking at too.

[berquist]

Here is the Rust "conversion" with https://github.com/rust-lang-nursery/mdBook:

• https://github.com/berquist/algorithm-archive/tree/mdbook
• https://github.com/berquist/algorithm-archive/commit/08cf6fe605ddae7fbcf94a223dd17c5771321f22 (the diff showing what I changed)

It looks like I renamed contents/ to src/, nuked the JS config, and added a minimal mdbook config. Most notably,

• there is no language picker (which was a JS addon anyway),
• I think MathJax is broken,
• and it was not clear how to extend mdbook, and if you need to do it in Rust or if you can do it in JS.

To try and build it,

• Install Rust (https://www.rust-lang.org/tools/install) and do cargo install mdbook
• Inside the repo, mdbook build and mdbook serve work just like gitbook.

[berquist]

Maybe relevant for syntax highlighting, it indicates you can add HTML/JS and have it picked up: https://github.com/rust-lang-nursery/mdBook/issues/657

[june128]

I created a project board which holds requirements and optional requirements for the AAAbook solution: https://github.com/orgs/algorithm-archivists/projects/2 Please feel free to discuss and extend the requirements, so that we end up with a good understanding of our needs.

[Trashtalk217]

Thanks to the work of @Neverik, we have a working alternative to gitbook. This has been posted in the discord chat but I feel that documenting it here is important.

Minimum viable alternative: https://aaapy.herokuapp.com/

[neverix]

Just as an example of what aaabook can look like, here's an AAA-Py screenshot

[leios]

Yeah, I have no doubt the aaa-py can work for this. I'm excited to see where it goes. I'm open to other ideas from the community as well.

[Trashtalk217]

We have to make a decision between mdbook and aaa-py someday and I feel like providing some stats might help with that. Below is listed a hassel comparison.

Hassel

Hassel is defined as the difficulty in installing and using the software. To stay open towards newer programmers trying to contribute to open source software, the hassel required getting aaa to build should be as low as possible.

Mdbook

I'll describe the setup required to set up the program in these sections.

Linux setup

The Linux setup is relatively simple although it's not all out of the box.

• Download the special version of the aaa
  • Go to berquist's repo
  • Go to the download button and dowload the zip or clone the repo
  • extract the zip and put it where you want
• Download Rust
  • $ curl https://sh.rustup.rs -sSf | sh
  • $ sudo apt install gcc-multilib (if you don't do this, the next step will not work!)
• Dowload mdbook
  • $ cargo install mdbook

Windows setup

• Copy the modified aaa.
  • I took the zip for this perticular setup, since the cloning gave me a weird error
  • Take it from here
• Install Rust
  • go here
  • press the button with "RUSTUP-INIT.EXE"
  • in command prompt type “y” and “1” and then enter to do a normal installation
  • install additional software
    • go to this site
    • press "Build Tools for Visual Studio 2019"
    • only select "Visual C++ build tools" in the wizard and then install
• Install mdbook
  • Open the command prompt
  • > cargo install mdbook

Build process (both Linux and Windows)

• Go into the algorithm-archive-mdbook folder (with a shell or a terminal)
• mdbook serve

AAA-py

Linux setup

• Clone or download the aaa-py repo
• Python is already pre-installed with most Linux distributions
• Open up a shell
  • $ sudo apt install python3-pip (this may not be necessary)
  • $ pip3 install jinja2
  • $ pip3 install markdown
  • $ pip3 install requests
  • $ pip3 install pybtex
  • $ pip3 install pygments

Windows setup

• Clone or download the zip form nev's repo
• Download Python (not pre-installed on Windows, turns out)
  • Do this the normal way
  • Or via the Windows app store (IT HAS USE!)
  • Make sure you download the latest version (3.x)
• Open up a command prompt
  • > pip install jinja2
  • > pip install markdown
  • > pip install requests
  • > pip install pybtex
  • > pip install pygments

Build process Linux

• Open up a shell
  • cd into the aaa-py folder
  • $ python3 build.py
  • $ cd _book
  • $ python3 -m http.server 8080
• Go to http://localhost:8080/ in your browser

Build process Windows

• Open up a command prompt
  • cd into the aaa-py folder
  • > python build.py
  • > cd _book
  • > python -m http.server 8080
• Go to http://localhost:8080/ in your browser

Conclusion

The Rust for Windows installation was definitely the most painfull, since I had to figure a lot out on the fly. The "Build Tools" download was definitely the most annoying. I had to install about 5 GB of stuff to make an error go away. Mdbook had a definite advantage over aaa-py in the build department. Just one command and it just works. While you have to type like three to get the aaa-py online. AAA-py has a major advantage over mdbook in that we can in fact do something about the current setup and build steps. The pip downloads are probably easy to change.

P.S. These instructions give you a working algorithm archive, but this doesn't mean this is also the shortest way to do things. The instructions are unoptimized and some corners could potentially be cut.

[Trashtalk217]

Now a short speed comparison.

Speed

All of the statistics come from a test done on Linux using the time command.

Mdbook

$ time mdbook load

Gives the following output:

real	0m0,007s
user	0m0,007s
sys	0m0,000s

I tried this several times, but the main takeaway here is that it's ridiculously small. Instant for all practical purposes.

AAA-py

$ time python3 build.py

Gives the following output in normal operations:

real	0m1,133s
user	0m2,750s
sys	0m0,144s

The same command gives the following time when the contents folder is not present (it needs to be cloned). But this barely happens since you only really need to get the full folder once. This is the worst case scenario.

real	0m4,645s
user	0m3,009s
sys	0m0,278s

Conclusion

Mdbook is instant while aaa-py is just very fast. The main thing is that they are both infinitely faster than GitBook.

[Trashtalk217]

Feature comparison

Compare for yourself! Mdbook vs aaa-py

[neverix]

The commands you ran do different things: build.py builds the entire project from scratch and mdbook load... Shouldn't even exist? This comparison isn't really fair because mdbook load just fails. I believe you used the wrong command. But even then, aaa-py still doesn't have live reload - which is why it may sometimes be slower. I would suggest someone to try to comprehend its source and maybe add live reload. aaa-py doesn't have an integrated web server either - that would be an interesting thing to add too.

[Trashtalk217]

Revision

A slight revision to what I just said. Got you @neverix !

AAA-py

AAA-py is a pipenv project so there is a slight different setup then what I described. Instead of installing all dependencies with pip install dependency-name, the following command can be used:

• $ pip3 install pipenv
• $ pipenv install (maybe not necessary)

This also changes the build process, which now turns in this:

• $ pipenv run python3 build.py (or > pipenv run python build.py for Windows)
• $ cd _book
• $ python -m http.server 8080
• Go to http://localhost:8080/ in your browser

This also changes the speed of the build process.

`$ time pipenv run python3 build.py`

Gives the following output:

real	0m1,571s
user	0m3,062s
sys	0m0,189s

Mdbook

mdbook load doesn't actually exists, turns out. So to get accurate measurements I had to use mdbook build.

$ time mdbook build

Gives he following output:

real	0m0,271s
user	0m0,217s
sys	0m0,053s

[leios]

I am not convinced that the mdbook build is actually that much faster. After all, it is lacking core functionality to switch languages, which takes up a majority of the compilation time. If we add this in, it will probably be much slower.

Honestly, even if we don't get a huge performance benefit from changing to a new markdown renderer, that's ok. We are ultimately trying to find a platform that will be maintainable in the long-run.

I am leaning towards the aaa-py repo right now because it seems more maintainable and is fast enough for now. If we need to update it, we can do so because it's python and everyone knows python. On the other hand, I thought the rust build process was incredibly easy and fast.

I did a strawpoll on stream, and it was somewhat split: https://www.strawpoll.me/18273691/r

As another note: eventually, we would like to find a custom way to output the aaa into a pdf file / book. One viewer (teko2k) suggested potentially looking into orgmode. Here is an example book made in this way: https://github.com/jkitchin/pycse

[leios]

After a lot of discussion amongst the community, it seems like we are more or less settled on the python version. That said, there was some discussion on discord not too long ago about another js method, but right now, I think we should emphasize moving away from gitbook.

I think we need to get the aaa-py repo to have all the requirements on the project board (on which, I slightly modified some of the notes): https://github.com/orgs/algorithm-archivists/projects/2

We had a discussion about one note on stream, particularly the one that says, Be a standalone executable that can be downloaded reasonably quickly and have minimal dependencies. I believe this is useful, but others feel that it might be alright to couple the solution with the AAA, itself.

Assuming we are moving forward with the aaa-py, should we modify the project board and add a column for competed requirements?

[Amaras]

I feel like since we settled for Honkit instead of AAA-py, this issue is now irrelevant. I won't close it until I have confirmation from @leios, though

[leios]

Right, I think we can probably close this one, but I want to be clear and thank @Neverik for all the work put into this and @Trashtalk217 for leading the conversation here!

[neverix]

Feels like the end of an era :cry:

[leios]

I know, I 100% agree.

[leios]

Reopening this issue as we are now thinking about switching off of honkit as well.

All of the same alternative options apply here, except now I would also like to consider Franklin.jl. Right now, I see 3 viable alternatives:

1. mdbook
2. aaa-py
3. franklin.jl

None of these will have language swapping and such immediately, so we are looking for the thing that is most robust and easiest to maintain in the future.

[leios]

So I just built both the AAA-py and mdbook versions. Here are the positives of both:

Mdbook positives:

1. Build is super fast
2. Dynamic search is nice

Mdbook negatives:

1. Missing core features (language swapping and math)
2. Built on rust, which only a few community members know

AAA-py positives:

1. Most things work. There are a few small rendering issues, but markdown, bibliographies, math, language swapping are all basically working
2. Home-built, which means it will be "easier" to customize

AAA-py negatives:

1. No dynamic search
2. It's home-built, so we need to maintain it

Franklin will obviously take more work. So I will look at that later.

[ShadowMitia]

Mdbook has some mathjax support: https://rust-lang.github.io/mdBook/format/mathjax.html