cosmopolitan

cosmopolitan copied to clipboard

Thank you @jart for this amazing project and thanks for encouraging me to open an issue. I believe many people would be amazed to see cosmopolitan 3.0+ be able to compile https://bellard.org/quickjs to APE to run on bare metal.

Something fails with qemu-system-x86_64 -fda o//third_party/quickjs/qjs.com

This already exists ... download https://github.com/jart/cosmopolitan/releases/download/3.1.3/cosmocc-3.1.3.zip and you'll find it in the bin directory.

I reached out to @tkchia earlier on Discord who's our bare metal expert. Another one of our experts is @ghaerr. Would either of you like to help our friend use Bellard's fabulous JavaScript REPL on bare metal? Thanks!

@mikeptweet i downloaded the zip file and while the README explains how the toolchain can be used to compile executables that run cross platform, nothing seems to be specifically about quickjs.

I'm a javascript developer and C is quite challenging to get into. I found a few articles titled "C for JavaScript Devs", but cosmopolitan is yet another beast. I have done C many many years ago during my studies, but haven't touched it since. I would like to get into cosmopolitan by trying to play with quickjs, but some guidance would be appreciated.

What tools do i need? How do I start writing, debugging and generally set up a feedback loop to progress getting to know cosmopolitan?

I read through https://justine.lol/cosmopolitan/ and the documentation is not comparable to MDN and the tutorials aren't helpful for somebody with my background.

Is there some pragmatic way to learn to set up a cosmopolitan project in combination with e.g. virtualbox or (i do have an unused spare HP laptop) to experiment with getting a "hello world" program and at some point a quickjs based "hello javascript" program work on "bare metal" via e.g. virtualbox or via e.g. my actual HP laptop using a bootable USB stick and/or ipxe or something?

I would love to go through recommended learning materials to learn more about C and how to setup and work on C projects with cosmopolitan as well, but I just have no idea where or how to start towards the goal of using cosmopolitan with quickjs this way.

I agree that Cosmopolitan ought to have MDN-quality documentation. You could help blaze that trail for us. Documentation is oftentimes written by people documenting their getting started learning experience. Our wiki is open for public editing. Please help us do better!

Hey @serapath ... the bin directory in the zip file I mentioned contains a universal binary - qjs that is quickjs.

btw, there is a much more actively maintained quickjs fork with new es2023 features, bug fixes, and perf improvements here: https://github.com/quickjs-ng/quickjs

they cherry pick any bugfixes/changes that land upstream, too.

Hello @serapath,

The business of getting QJS to run on bare metal is a bit complicated, but certainly could be done. The way bare metal is supported in Cosmo is literally booting the qjs.com file in real mode, which then internally sets up the CPU and system to run in (x86) 64-bit long mode, then starts running the compiled program. The statically-linked Cosmo library then special cases each of the what-otherwise-would-be system or C library calls, and performs them on bare hardware.

The work involved depends on which system calls and C library routines QJS might use, as many system calls aren't yet implemented on bare metal. I can only imagine the REPL uses some variation of read and write to the "console", along with possibly quite a few more. At the moment, we support a VESA graphics-mode output mechanism, which allows for screen display and ANSI terminal emulation. Keyboard input is complicated, as it requires the installation of a keyboard interrupt handler and scan code to unicode conversion tables based on keyboard type. While keyboard interrupt handlers and scan code converters have been written many times for x86 real mode, running in long mode can't use the BIOS, as that only works in real mode.

IIRC @tkchia may have a long mode interrupt handler working, but has gotten bogged down in the complex UEFI specification, which requires full implementation to guarantee working on any (that is "most newer") 64-bit PCs.

Sorry for the complicated explanation; but getting a javascript interpreter running could require much more, depending on its set of external C library and system call dependencies. We should probably investigate QJS a bit to see what it requires. QJS is actually pretty large for a JS interpreter, as its advertised as being quite complete and fast. I know it has lots of code very fast handling of heap vs stack upvalue conversions for proper closures, etc. Perhaps a much smaller interpreter, like MuJS, might be interesting, with much fewer depencies (and JS external functions).

In summary, getting this done depends on the sorts of things you need to run from the REPL... and thus which interpreter might be best used to keep complexities minimized.

Thank you!

Thank you for the detailed explanation. My head is spinning now. Much of what you shared is new to me and i was unaware. When you mentioned uefi/bios i somehow was reminded of the coreboot or libreboot, but probably not really relevant or any simpler.

I guess it would be interesting to have quickjs work on bare metal and at least one additional feature, which would be fetch or alternatively something more low level and raw. maybe udp/dgram access, or if bare metal means thats tough, then something even lower level?

i do feel a project like this would open an entire new world or new universe to so many developers, just like nodejs/npm created an enormously large new ecosystem out of seemingly nowhere.

@jart i would love to work towards an MDN, but i think you might underestimate how little i comprehend at the moment. I dont feel like I found a door that would even allow me to enter and do any first steps at all yet. It is difficult to explain, but right now it feel completely inaccessible. There is nothing around cosmos that relates to anything i am familiar with.

To start anything, i would need to be at the shores of cosmopolitan, but currently i feel i'm on another continent and there is an ocean in between where i am and the shores of cosmopolitan land if that makes any sense

@serapath If you're looking for a gentle introduction to Cosmopolitan, our bare metal support isn't it. We haven't elevated it yet to a level of quality that it's ready to be consumed by people who aren't able to help us work on improving it.

@ghaerr Here's the system calls for non-interactive mode quickjs

master jart@nightmare:/opt/cosmo$ o//third_party/quickjs/qjs.com -e 'console.log("hi")' --strace
SYS      0      0             14'354 bell system five system call support 329 magnums loaded on gnu/systemd
SYS  18433  18433             29'359 getenv("COSMOPOLITAN_DISABLE_ZIPOS") → NULL
SYS  18433  18433             33'339 getenv("COSMOPOLITAN_INIT_ZIPOS") → NULL
SYS  18433  18433             52'786 getcwd([u"/home/jart/cosmo"], 1'019) → 17
SYS  18433  18433             58'238 openat(AT_FDCWD, "/home/jart/cosmo/o//third_party/quickjs/qjs.com", O_RDONLY) → 3
SYS  18433  18433             75'891 fstat(3, [{.st_size=1'292'999, .st_blocks=1'294'336/512, .st_mode=0100755, .st_uid=1000, .st_gid=1000, .st_dev=0x31, .st_ino=0x910d}]) → 0
SYS  18433  18433             79'705 mmap(0, 1'292'999, PROT_READ, MAP_SHARED, 3, 0) → 0x100080100000 (1'376'256 bytes total)
SYS  18433  18433            106'892 munmap(0x100080100000, 1'179'648) → 0 (196'608 bytes total)
SYS  18433  18433            112'080 madvise(0x10008022b000, 65'536, 4) → 0
SYS  18433  18433            114'042 mmap(0, 136, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080040000 (262'144 bytes total)
SYS  18433  18433            120'494 close(3) → 0
SYS  18433  18433            121'745 __zipos_get("/home/jart/cosmo/o//third_party/quickjs/qjs.com") → 0
SYS  18433  18433            125'334 mmap(0, 196'608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080060000 (458'752 bytes total)
SYS  18433  18433          1'367'281 inflate([u"SYMT☺   !☼        ♥       ♥       @     "...], 196'608, u"∞▌y|‼┼π└²)≈▌Ç (*♣E♫♣J╣Eí▄-gÑ@╣$Mô┤♪MôÉú•"..., 51'509) → 0
SYS  18433  18433          1'370'760 GetSymbolTableFromZip() → 0x100080060000
SYS  18433  18433          1'372'640 getenv("TERM") → "xterm-256color"
SYS  18433  18433          1'379'233 getenv("HOME") → "/home/jart"
SYS  18433  18433          1'380'612 getenv("TMPDIR") → NULL
SYS  18433  18433          1'404'278 openat(AT_FDCWD, "/zip/.args", O_RDONLY) → -1 ENOENT
SYS  18433  18433          1'413'033 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080050000 (524'288 bytes total)
SYS  18433  18433          1'628'338 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080090000 (589'824 bytes total)
hi
SYS  18433  18433          1'709'485 write(1, u"hi◙", 3) → 3
SYS  18433  18433          1'808'852 exit(0)
SYS  18433  18433          1'810'176 __cxa_finalize(&fflush_unlocked, 0)
SYS  18433  18433          1'812'033 __cxa_finalize(&__gdtoa_Bclear, 0)
SYS  18433  18433          1'813'109 __cxa_finalize(&TeardownGc, 0)
SYS  18433  18433          1'814'415 _Exit(0)

Here's what it takes to bring up the prompt in interactive mode:

master jart@nightmare:/opt/cosmo$ o//third_party/quickjs/qjs.com --strace
SYS      0      0             14'250 bell system five system call support 329 magnums loaded on gnu/systemd
SYS  18493  18493             28'312 getenv("COSMOPOLITAN_DISABLE_ZIPOS") → NULL
SYS  18493  18493             32'062 getenv("COSMOPOLITAN_INIT_ZIPOS") → NULL
SYS  18493  18493             35'515 getcwd([u"/home/jart/cosmo"], 1'019) → 17
SYS  18493  18493             40'772 openat(AT_FDCWD, "/home/jart/cosmo/o//third_party/quickjs/qjs.com", O_RDONLY) → 3
SYS  18493  18493             44'982 fstat(3, [{.st_size=1'292'999, .st_blocks=1'294'336/512, .st_mode=0100755, .st_uid=1000, .st_gid=1000, .st_dev=0x31, .st_ino=0x910d}]) → 0
SYS  18493  18493             48'283 mmap(0, 1'292'999, PROT_READ, MAP_SHARED, 3, 0) → 0x100080100000 (1'376'256 bytes total)
SYS  18493  18493             59'708 munmap(0x100080100000, 1'179'648) → 0 (196'608 bytes total)
SYS  18493  18493             64'248 madvise(0x10008022b000, 65'536, 4) → 0
SYS  18493  18493             84'923 mmap(0, 136, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080040000 (262'144 bytes total)
SYS  18493  18493             90'917 close(3) → 0
SYS  18493  18493             92'175 __zipos_get("/home/jart/cosmo/o//third_party/quickjs/qjs.com") → 0
SYS  18493  18493            109'103 mmap(0, 196'608, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080060000 (458'752 bytes total)
SYS  18493  18493          1'312'096 inflate([u"SYMT☺   !☼        ♥       ♥       @     "...], 196'608, u"∞▌y|‼┼π└²)≈▌Ç (*♣E♫♣J╣Eí▄-gÑ@╣$Mô┤♪MôÉú•"..., 51'509) → 0
SYS  18493  18493          1'315'267 GetSymbolTableFromZip() → 0x100080060000
SYS  18493  18493          1'316'462 getenv("TERM") → "xterm-256color"
SYS  18493  18493          1'322'876 getenv("HOME") → "/home/jart"
SYS  18493  18493          1'323'867 getenv("TMPDIR") → NULL
SYS  18493  18493          1'330'820 openat(AT_FDCWD, "/zip/.args", O_RDONLY) → -1 ENOENT
SYS  18493  18493          1'338'617 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080050000 (524'288 bytes total)
SYS  18493  18493          1'571'852 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x100080090000 (589'824 bytes total)
SYS  18493  18493          1'745'596 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x1000800a0000 (655'360 bytes total)
SYS  18493  18493          1'931'961 isatty(0) → true
SYS  18493  18493          1'935'043 tcgetwinsize(0, [{.ws_row=56, .ws_col=213}]) → 0
SYS  18493  18493          1'939'525 tcgetattr(0, [{.c_iflag=ICRNL|IXON|IUTF8, .c_oflag=OPOST|ONLCR, .c_cflag=CS8|CREAD|0xf, .c_lflag=ISIG|ICANON|ECHO|ECHOE|ECHOK|ECHOCTL|ECHOKE|IEXTEN, .c_cc[VMIN]=1, .c_cc[VTIME]=0, .c_cc[VINTR]=CTRL('C'), .c_cc[VQUIT]=CTRL('\\')}]) → 0
SYS  18493  18493          1'943'987 tcsetattr(0, TCSANOW, {.c_iflag=IUTF8, .c_oflag=OPOST|ONLCR, .c_cflag=CS8|CREAD|0xf, .c_lflag=ISIG|ECHOE|ECHOK|ECHOCTL|ECHOKE, .c_cc[VMIN]=1, .c_cc[VTIME]=0, .c_cc[VINTR]=CTRL('C'), .c_cc[VQUIT]=CTRL('\\')}) → 0
SYS  18493  18493          1'948'786 sigaction(SIGINT, {.sa_handler=&os_signal_handler, .sa_flags=SA_RESTART}, [{.sa_handler=SIG_DFL}]) → 0
QuickJS - Type "\h" for help
SYS  18493  18493          1'967'991 write(1, u"QuickJS - Type “\\h“ for help◙", 29) → 29
qjs >

So all you'd have to do is go down that burndown list of functions, e.g. tcgetwinsize(), madvise(), etc. and make sure they have an if (IsMetal()) { return enosys(); } so the Linux SYSCALL instruction doesn't cause it to crash. Then it'll likely start working.

Hello @jart, hello @ghaerr,

Thank you. I have been a bit busy with life lately — and expect to be so for at least the next few weeks. Hopefully I can get around to work on the bare metal implementation some time soon. In the meantime perhaps you could have a go at it.

Hello @ghaerr,

I have a semi-working IRQ 0 (legacy timer) interrupt handler working on a development branch (https://github.com/tkchia/cosmopolitan/commits/tkchia/20230827/) — it probably needs some effort to be rebased or merged into the mainline code though.

What the code tries to do is to put the interrupt controllers "back" into legacy BIOS mode — even if booted via UEFI! — and also reprogram the legacy interval timer (PIT 0) to fire at 18.2 Hz. There is also some fantastically complicated logic to (semi-correctly) parse ACPI tables, cribbed from SeaBIOS. This is mainly there to sanity-check whether the PC actually has a legacy-compatible PIT 0.

Probably the more "correct" thing to do is to ditch all the legacy stuff, and program the interrupt controllers as modern APICs and look for newer hardware timer devices (e.g. Intel HPET) rather than legacy PITs. But this likely takes a bit more time to do right.

Thank you!

Note: cosmopolitan is now natively supported by https://bellard.org/quickjs