jor1k
jor1k copied to clipboard
s-macke
improve modularisation/API
I am a big fan of this project! I am thinking of integrating the emulator with an interaktive learning environment. Going through the code base, I noticed several opportunities to improve re-usability and cleaning up the overall project structure. I would volunteer to work on this, but thought, I start a discussion first.
Here are the steps I would take first
- use browserify (require) to build one js file from various modules, designing a clean interface between modules (module.exports, like you would do in NodeJS)
- replace grunt by browserify and a simple Makefile (to uglify the result)
- implement more unit tests
- split off terminal and terminal_input from the main repo and put them into their own npm package, allowing integrators to use alternative terminal emulators and using the jor1k terminal emulator in other projects
- split off utils like UTF8Stream, or look for pre-existing npm modules providing the same functionality
- improve the emulator's API, for example provide Node-like streams for serial input/output using stream-browserify
What would you think of this? Would you accept pull requests that take the project into this direction?
Thanks for creating this awesome, lightning-fast emulator! It will serve as a superb learning tool!
There is so much to say for each little point. But Ok, let's start.
Just some common things. I started this project without any clue about Javascript and without any real goal. The source code is a directly result of this. The problem is, that when you start to implement something, you don't know if the interface design is sufficient or not. Of course, most often it is not. Therefore like you, I see tons of improvements of the code. Your list is by far not complete.
The other point is, that I had three primary goals. And that were 1. speed, 2. speed, 3. speed. Some strange decisions (like the prevention of normal Javascript arrays) of the code are a direct result of this. The most craziest thing of the code are the asm.js CPUs, in which I was forced to make my own memory layout like in the old days of the C64. A consequence of the priority for speed and out of curiosity, how everything works in detail I implemented everything myself.
For example, there were several bottlenecks in the code (like subtleties in the terminal), which prevented a fast "ls -la /usr/bin". I know there are other terminals out there, which might have more features and are more compatible. But I am afraid, that they could throttle down everything because of subtleties.
Together with the complexity of the hardware specifications and the severe limitations of Javascript (just look at the "execute" ping pong) to write really clean, but powerful code, this is were I ended.
Ok, but let's start with the points.
- Grunt was never used. I know it is written in the Wiki. It's there, because Gerard Braad, wanted to use it. But so far, nothing has been done. The code runs without any additional compilation procedures.
However sometimes I wished to have a preprocessor. A lot of things in the CPU directory are identical.
I have never used minimization algorithms to tighten the code. The original idea of minimization amd compaction is, to reduce latency when loading websites. Because of the 3MB binary blob, which has to be loaded anyhow, this latency does not play a big role in the end.
Next year, all browsers and hopefully also most of the servers will support http/2 and this will solve such problems anyhow.
If you think of using jor1k as sort of a library, you are of course fully right. Including twenty files in a complex directory structure is not an option.
So, in the end I am fine with browserify, and some sort of Makefiles, as long as for the user the option remains to use the code out of the box. That means, "git clone" and then double click on the first html file in the main directory.
- Unit tests are great, and it is a shame, that I never included one.
- I am not glad with the keyboard handling in the current code. The main problem is, that I implement up to four different input types. A tty keyboard, a hardware keyboard for the framebuffer, a keyboard input for the editor and a special input for the small textfield for pasting. All of them should be well separated in modules and there should be a clean interface.
The terminal is one of those rare things of the emulator, that can be indeed useful outside of jor1k. So a separation makes sense. However, don't complain, if other terminal implementation slow down the whole thing when you do fancy stuff with it.
- The UTF-8 code was a fast compromise and no big deal. I found some code, but nothing seemed to fit. Especially the streaming capability. And I don't want to implement a 100kB fancy library, which has tons of features. At the moment I am fine with those few functions. Of course one reason was, that I was curious of how it works :)
- The first time I read the last point I said probably not. But then I looked on the website and it sounds very promising. There are features included, which I want since a long time. Especially useful for the filesystem. Wonder, If there is an extension, which include also .xz streams.
However at the moment I don't understand, what you exactly mean with serial input/output. But it seems to be Ok.
Ok, to summarize: Clean interface: yes Module exports: yes Unit tests: Yes UTF8: Depends on the alternative. Terminal splitting: yes, but we must be a little bit careful with the terminal-input. Stream: Have to look closer. But sounds promising Browserify: yes, if it is just a Makefile which produces additional files. I am not sure about the features of browserify and how dependent I will be in the end on this tool. I like independence.
Thanks for the detailed answer! Speed of course is vital in this project and I entirely agree that any refactoring that would decrease speed is unacceptable. My main goal here would be to de-couple the major building blocks you created, so that they can be recombined in different ways. One example is the terminal. Or to put it differently: the goal would be to harvest the power of npm for this project. It is great for client-side JavaScript, too.
I also totally agree that no bloated libraries should be pulled into the project. But there are real gems out there on npm, very well designed microlibraries that do one thing right (don't we all love the unix philosophy?) , are battle-hardened, fast and tiny. Good examples are substack's streaming libraries. These things can make life significantly easier.
My main point about browserify is that it gives you an implementation of require(). This has various advantages. For example it makes dependencies between source files explicit, the ordering problem when loading the various js files goes away, every module is nicely kept in its own scope and does not pollute the global namespace (unless you explicitly want that). This of course means, that require() calls would appear at the beginning of every source file (that's the point), so they won't run as-is in a browser (because essentially they are now CommonJS modules), so the browserify build step is needed to produce the one js file that the browser can load. If you want to keep the git clone && open index.html way of "installation", you would have to check-in the file that browserifying outputs. Otherwise it would be npm install jor1k && open index.html. I would split the jor1k library from the jor1k demo (the demo would be one application of the library and the terminal), and in the demo repo you could check-in the browserified version for convenience (I'd rather put it in the gh-pages branch only though). You would still have the clone and click install (just clone the gh-pages branch), but you would also have a library that is a nice citizen of the npm universe by being a collection of CommonJS modules with clean APIs (some of which even would be usable in NodeJS)
Yes, I would like to use jor1k as a library, I would like to have a nice, versatile jor1k API that uses familiar concepts like streams. The idea is to be able to connect one or more UARTs to code elsewhere on the page that can use a terminal session over that serial link to investigate the state of the emulated machine. This would be of great value for an interactive learning environment. A text next to the terminal could send students on a quest like "now edit /etc/sshd_conf to disable password auth" and it could check periodically, if the file has the correct content by issuing shell commands and give a hint that sshd needs to be restarted when it sees that the file has the right content but sshd's pid didn't change. The student would be free to use whatever editor they want (well, nano or vi) to finish the task. That's a much more flexible learning environment than most.
Or that same stream could be connected to a terminal of another user via a WebRTC data channel, directly peer-to-peer, without a relay. Using tmux in the emulator would then allow for sharing a session and hence pair programming. There are many great small libraries on npm that have a streaming interface, by combining all these small building blocks, you can send everything over everything. I would really like to see jor1k to become part of this ecosystem. See substack's streaming handbook.
Thanks for pointing out the subtle problems a terminal implementation can cause. I was actually thinking that it might be possible to implement an even faster terminal using a div with monospaced font and one span per segment of a line with same-colored characters. So, in a best-case scenario, the terminal would simply be 25 spans in a div. I'd like to experiment with that. But maybe you already did?
If you are generally okay with this direction, I would put some initial work into a small pull request and then we can see where it goes from there.
Thanks again for creating this awesome piece of software!
Oh, I don't care for uglify as well. I think it is a decision of whoever deploys application code, not a decision of a library author. So you could use it to accelerate loading of the demo, but I agree, it probably won't even be noticeable.
This means, we don't even need a Makefile. And yes, browserify will help with getting rid of code redundancies.
not really, it seems to be using AMD instead of CommonJS modules and the only npm module they actually use is a command line parser for the use of the emulator as a command line tool. Internally, it is modularised using AMD instead of npm packages. More importantly, jemul8 is too slow for most practical purposes.
Yes, I generally agree with the points. I will give you a more detailed answer later.
First a comment to the terminal.problem. I started indeed with a div and a table and removed it maybe around commit 50-100. This was also before I started to work with web workers. The problem was, that some web browsers started a whole rerendering of the window after each update. Well, it was just slow sometimes and I couldn't do anything at that time. Now, with more experience and understanding of the inner workings of the browsers I wanted to test the DOM solution again but haven't done it so far. The question is, if Javascript is slowed down, when I change the DOM. In case of a canvas it renders immediately and stops Javascript during this rendering. So, the current solution might be indeed bad.
The problem with Javascript is staying responsive while emulating a CPU, which can use 100% of the processing power, but interrupts everywhere. Javascript is not well designed to allow this. Therefore I have implemented the execute ping pong hack. But that means that master and worker have to be idle at least 100 times a second. If the master has something to do, e. g. rendering the whole terminal, it can take more than 50ms and slows down the CPU too, because it is waiting for the execute message. At the moment I solve this problem with a requestAnimationFrame and a long timeout if the whole terminal is to be rerendered. With my own terminal implementation I can easily experiment with those subtleties. By the way. If the message queue would have a feature, that would give me the number of messages in the queue, everything would be fine. But for some reason, such a one line feature was omitted from Javascript.
I have thought about all this and I am fine with all those changes and I would also do some of the changes myself, because they make absolutely sense. Maybe you have seen it already, I compiled git and vim already. Both work fine.
That's awesome, Sebastian, thanks a lot! I will get started with preparing the first PR in the next couple of days then. To not duplicate work: what will you be working on first? Do you have an idea already?
Well, the first step would be, that you point me to an example of how the interfaces of the objects should look like (with exports). I would try to apply this to the worker/dev directory, because that's the easiest one.
It is probably easiest, if you start here: https://github.com/substack/browserify-handbook
Do you have any more hints on compiling for jor1k? I would like to try to get tmux running. Are you compiling in qemu or directly in jor1k?
(currently running git clone https://github.com/ThomasAdam/tmux.git inside the emulator. I had to create /etc/gitconfig to include
[http]
sslVerify = false
to make it work. I did that using vim :)
Here is a description of methods available https://github.com/s-macke/jor1k/wiki/How-to-develop-for-jor1k
I use the fastest method, running those cross compiling scripts: http://www.jor1k.com/or1k-toolchain.tar.bz2 Unfortunately, cross compiling is an annoying task, and is dependent on the distro you're using. Those scripts will help, but without a deep knowledge about the internals about configure, make and make install and all kind of Unix stuff, most people will fail.
Yes, git with SSL doesn't work for some reason. But you can use git://github.com/ThomasAdam/tmux.git for git clone. That's good enough for now. Note that, I removed tons of stuff from jor1k-filesystem to keep it small for github. That means, that a lot of header files are not available, as well as language and documentation files. However, for tmux it is probably enough.
automake is a Perl script. And perl is one of the most hostile programs when cross compiling. I spent already several hours on that thing. Why do you use git anyhow for this? Just download http://downloads.sourceforge.net/tmux/tmux-1.9a.tar.gz . I guess then you don't need automake for this.
I just wanted to test git. I actually used the tarball then, ./configure fails with 'unable to guess build type' which seems to be an automake related problem. I'll investigate further tomorrow.
This is a common problem. not every distribution has updated autoconf with support for the or1k CPU. I use something like this to get rid of the problem: find . -name "config.sub" -exec sed -i 's/or32/or1k/g' {} ;; Maybe you have to the same with config.guess. The problem will solve itself in a few years. New architectures like arm64 have the same problems.
tmux needs also libevent. I included that. Unfortunately automake is still needed.
On Sat, Nov 29, 2014 at 11:23 AM, Sebastian Macke [email protected] wrote:
Grunt was never used. I know it is written in the Wiki. It's there, because Gerard Braad, wanted to use it. But so far, nothing has been done. The code runs without any additional compilation procedures.
Grunt could be dropped, however if you are using a flow with all javascript related tooling, such as uglify, browserify, mocha, etc. it would still be preferred over a Makefile.
The goal I had was to modularize and allow it to run outside of the browser, for instance on Node. Although I never came around to reserve enough time to do this.
The good thing is, moving to CommonJS modules is even a good first step towards running inside Node. I think right now, we neither need grunt nor make. browserify alone will be sufficient to build the project.
Any kind of build tool can help in easing the pain with repetitive tasks, such as pushing to the github pages, etc.
BTW, since it is git just dump it now if unused... and when we would need it we can always get it back from history or re-implement it in a better way. ;-)
I compiled perl as well as automake. But I had to remove a lot of libraries from perl. Not sure, if I miss anything important.
I did the first step. Take a look. https://github.com/s-macke/jor1k/tree/browserify https://github.com/s-macke/jor1k/commit/bd139ba4a0ae4040ad95f8c639af99a479f0cfe7 Maybe I misunderstood a little bit the usage of global objects. For example I give each device the same messagehandler object in a function. Now it looks like, that this is not necessary. A "require" always exports the same object.
Yes, CommonJS modules are singletons that you access with require(), which is pretty handy.
That is a huge commit :) I left some comments on it. I think this is a good step towards modularity all in all!
Thanks. Yes, the require and exports statements fill a huge gap in Javascript and I hope, that such things will become a standard soon in the browser. Sometimes I feel, that I am back in 90s when I write Javascript. Anyhow, the huge patch just marks the first working version using browserify. Not something final.
Huge +1 to this effort. If I had any time at all I'd likely be working on this.
