ledger
Ledger-cli in WebAssembly
Perhaps a crazy idea but an interesting one, nonetheless. Is it possible to compile ledger-cli into wasm? Has anyone tried, what are the challenges? Opening this issue as a placeholder for the topic.
The benefits should be obvious, I guess. Having Ledger compiled to Wasm would allow running it natively in browsers, including PWAs, and also on Android. I've seen some people compiling Boost with Emscripten but I'm quite new to the topic at the moment.
~~Presumably, file reading would be one of the main issues~~
Seems there is also a virtual filesystem available, which is convenient for storing and reading the files.
- https://stackoverflow.com/questions/47313403/passing-client-files-to-webassembly-from-the-front-end
- https://kripken.github.io/emscripten-site/docs/porting/files/file_systems_overview.html
Files can be stored separately and preloaded into the virtual file system.
- http://www.jamesfmackenzie.com/2019/12/08/webassembly-loading-files/
Interesting idea! It would permit me to run Ledger directly on the Internet Computer too.
Boost 2 WebAssembly links
- emscripten
- emscripten and boost library: how to compile existing project for webassembly?, SO
- Using Boost with Emscripten, SO
There is already #2004 but nowadays it seems that a WASI target would be a perfect fit for Ledger-CLI.
In addition, OPFS is now available is all major browsers, effectively providing the conditions for Ledger to run in a browser, using wasm target.
There is also Chirp - https://github.com/leaningtech/cheerp-meta - compiler.
I just managed to get ledger compiled with emscripten following the steps stated in https://github.com/ledger/ledger/issues/2004#issuecomment-1878164915 with changes in https://github.com/gudzpoz/ledger/commit/3d131345ae86616b92c0e35013698e14a0089114 applied.
By the way, currently it seems to me that emscripten is the de facto compiler to port C++ code to WASM:
- The way cheerp binds to C++ classes seems too intrusive to me and might require tons of macros to integrate (compared to WebIDL Binder — Emscripten).
-
Wasi-sdk does not support exceptions.
- But ledger uses exceptions.
- As is discussed in C++ Exceptions Support — Emscripten, exception support in WASM may not be supported in all WebAssembly engines yet. And emscripten supports exceptions mainly via JavaScript, which means we may only be able to target JS platforms for now.
If we are to target JS platforms, should a JS/TypeScript wrapper be supplied (for, e.g., libledger)? (Or maybe it is to be put in a separate repository?) Working with WASM seems to involve all kinds of manual memory management and can be painful when one has no prior experiences. (I wrote a simple ledger-cli wrapper to test things out here: https://gist.github.com/gudzpoz/009fc2445ab717d2e049eb069040f9b4 )
That's great news, @gudzpoz, congratulations! Thanks for putting in the time and effort. I'll have a closer look at (your comment in #2004](https://github.com/ledger/ledger/issues/2004#issuecomment-1878164915) and will think about how this could be integrated into ledger's current CMake-based build process. In your mind what are good next steps to move forward with this?
I am not exactly sure though... Maybe a typical goal is to write a proper JS/TS wrapper and have ledger published in NPM so that ledger-cli can be readily installed with npm install -g ledger....
Also, maybe we should discuss ways to expose the libledger API as WASM functions? Personally I am satisfied with WASM ledger-cli, and implementing & maintaining & testing the WASM-libledger API would require a lot of work. But, anyway, it can be done theoretically :-P
And concerning testing, with a bit of tweaking (https://github.com/gudzpoz/ledger/commit/0ef943b6b813179b56548c99df745fc74da7896e) we can now run the tests against the compiled bundle and at least some of them are passing: 85% tests passed, 63 tests failed out of 421. I haven't looked into the failed ones yet, but many of them are likely to be due to my poorly written wrapper.
Edit: Now 97% tests passed, 13 tests failed out of 421:
The following tests FAILED:
1 - DocTestsTest_ledger3 (Failed) <-- No error when manually run?
45 - BaselineTest_dir-import_py (Failed)
47 - BaselineTest_dir-python_py (Failed)
57 - BaselineTest_feat-import_py (Failed)
58 - BaselineTest_feat-option_py (Failed)
60 - BaselineTest_feat-value_py (Failed)
91 - BaselineTest_opt-color (Failed)
309 - RegressTest_4D9288AE_py (Failed)
347 - RegressTest_78AB4B87_py (Failed)
360 - RegressTest_9188F587_py (Failed)
377 - RegressTest_B21BF389_py (Failed)
381 - RegressTest_BF3C1F82 (Failed)
421 - RegressTest_xact_code_py (Failed)
Most of them are about python, which is not supported anyway. BF3C1F82 may be due to a bug (or undefined behavior?) of strptime in emscripten: it converts "2012/02/30" into tm(tm_year=2012, tm_mon=2, tm_mday=1).
A bit OT but I'd like to try the Wasm version from a Blazor app. Apparently it can invoke Wasm methods directly so no js wrapper needed. It would be cool if this would become an official build. That would give ledger a much wider reach. Is emscripten using OPFS or it's own storage mechanism? Where is ledger wasm expecting to find files? Where can we have discussion on related topics? I wouldn't want to spam this thread outside the actual Wasm version. Once that is ticked off, I think I'd try Wasm and Wasi versions. Wasi would be a particularly interesting build for Windows users as there's no official version, as far as I can tell. Great job and thank you for the effort. I find this is a significant breakthrough. 2024 is expected to bring Wasm to the forefront, or so they say.
The problems is that the Wasm build itself will most likely require a js wrapper due to usage of C++ exceptions in ledger.
As is summarized in https://github.com/crystal-lang/crystal/issues/13130:
- Native exception handling in Wasm is still an ongoing proposal and not all Wasm runtimes support it. (Emscripten supports using Wasm native exception handling by setting
-fwasm-exceptions.) - When compiled with
-fexceptionsinstead in emscripten, "JS-based exceptions works by calling into JavaScript and having a try-catch block there, that in turn calls into Wasm. Throwing an exception works by invoking JavaScript again tothrow."
Taking the first approach (with flags like -fwasm-exceptions and -sSTANDALONE_WASM=1), it should be possible to use without a wrapper, and the Wasm bundle should hopefully comply with WASI standards - what FS it uses will depend on the runtime you use. But, again, runtimes supporting Wasm native exceptions are scarce and most of them are ... browsers, which don't seem to provide a WASI interface anyway.
With the current -fexceptions approach, since it requires a wrapper, then it needs JS after all. (The filesystems used by the JS wrapper is probably implemented by emscripten and provide their own API.)
Concerning python bindings, it seems that pyodide (Wasm port of CPython) has in-tree support for boost, libgmp and libmpfr, and it should be possible to build ledger using pyodide-build while supporting python bindings. But I have no idea how this can get integrated into CMake building process though...
My head hurts just trying to imagine what the output of that should be. How should Python integration work with respect to the Wasm constraints?
Could we summarize what the available options are, the level of difficulties and the steps required to get them to fruition, as well as some common use cases which this would be intended for?
For example, I wouldn't mind having support in browser only, for start. I'd like to try this and see how to pass the files from OPFS. This in itself would be a fantastic milestone.
BTW, maybe Wasm modules will make the python integration obsolete?
Edit: The primary use case would then be integration into a PWA, and providing a GUI in a browser which uses ledger directly instead of a Web server executable calling it in a shell.
Just built a ledger python wheel that runs in browsers following the steps to build an in-tree Pyodide package.
One minor change in packages/boost-cpp/meta.yaml was necessary to let pyodide produce some boost libraries:
diff --git a/packages/boost-cpp/meta.yaml b/packages/boost-cpp/meta.yaml
index 9106dd4a..d9c86e53 100644
--- a/packages/boost-cpp/meta.yaml
+++ b/packages/boost-cpp/meta.yaml
@@ -19,7 +19,7 @@ build:
printf "using clang : emscripten : emcc : <archiver>emar <ranlib>emranlib <linker>emlink ;" | tee -a ./project-config.jam
./b2 variant=release toolset=clang-emscripten link=static threading=single \
- --with-date_time --with-filesystem \
+ --with-date_time --with-filesystem --with-iostreams --with-nowide --with-python \
--with-system --with-regex --with-chrono --with-random --with-program_options --disable-icu \
cxxflags="$SIDE_MODULE_CXXFLAGS -fexceptions -DBOOST_SP_DISABLE_THREADS=1" \
cflags="$SIDE_MODULE_CFLAGS -fexceptions -DBOOST_SP_DISABLE_THREADS=1" \
The following was the contents of packages/ledger/meta.yaml:
package:
name: ledger
version: "3.3.2"
top-level:
- ledger
source:
path: https://github.com/gudzpoz/ledger/archive/refs/heads/master.zip
sha256: 0659cfb2812d14a72a6388c1f8acfcd0e39f279713e4e0a30937d3821682edce
requirements:
host:
- boost-cpp
- libgmp
- libmpfr
build:
script: |
export INSTALL_DIR=${WASM_LIBRARY_DIR}
sed -i.bak -E "s/COMPONENTS Interpreter Development/COMPONENTS Interpreter/" CMakeLists.txt
sed -i.bak -E "s/HAVE_GETPWUID|HAVE_GETPWNAM/0/" src/utils.cc
mkdir -p build
cd build
emcmake cmake -DBUILD_LIBRARY=ON -DUSE_PYTHON=ON \
-DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
-DEMSCRIPTEN=ON -DHAVE_BOOST_PYTHON=ON \
-DPython_INCLUDE_DIRS=$PYTHONINCLUDE \
-DBoost_ROOT=$WASM_LIBRARY_DIR \
-DBoost_INCLUDE_DIR=$WASM_LIBRARY_DIR/include \
-DBOOST_LIBRARYDIR=$WASM_LIBRARY_DIR/lib \
-DBoost_DATE_TIME_LIBRARY_RELEASE=$WASM_LIBRARY_DIR/lib/libboost_date_time.a \
-DBoost_FILESYSTEM_LIBRARY_RELEASE=$WASM_LIBRARY_DIR/lib/libboost_filesystem.a \
-DBoost_IOSTREAMS_LIBRARY_RELEASE=$WASM_LIBRARY_DIR/lib/libboost_iostreams.a \
-DBoost_NOWIDE_LIBRARY_RELEASE=$WASM_LIBRARY_DIR/lib/libboost_nowide.a \
-DBoost_PYTHON_LIBRARY_RELEASE=$WASM_LIBRARY_DIR/lib/libboost_python$PYMAJOR$PYMINOR.a \
-DBoost_REGEX_LIBRARY_RELEASE=$WASM_LIBRARY_DIR/lib/libboost_regex.a \
-DBoost_SYSTEM_LIBRARY_RELEASE=$WASM_LIBRARY_DIR/lib/libboost_system.a \
-DGMP_PATH=$WASM_LIBRARY_DIR/include \
-DGMP_LIB=$WASM_LIBRARY_DIR/lib/libgmp.a \
-DMPFR_PATH=$WASM_LIBRARY_DIR/include \
-DMPFR_LIB=$WASM_LIBRARY_DIR/lib/libmpfr.a \
-DUTFCPP_INCLUDE_DIR=$PWD/../lib/utfcpp/v4/source \
..
emmake make -j ${PYODIDE_JOBS:-3} libledger
cp ledger.so $DISTDIR
echo "from setuptools import setup" > ../setup.py
echo "setup(name='ledger', version='3.3.2', packages=[''], package_dir={'': '$DISTDIR'}," >> ../setup.py
echo " package_data={'': ['ledger.so']})" >> ../setup.py
about:
home: https://ledger-cli.org
summary: Double-entry accounting system with a command-line reporting interface
license: LGPL-3.0+
And, no, I don't think Wasm modules will make the python integration obsolete.
Concerning possible options, the first few questions that occur to me are:
-
Does the Ledger project plan to publish official builds for WASM builds / WASM-built python wheel?
Currently most ledger binaries are mostly maintained by package maintainers from different distros. However, ecosystems around JS/Python somehow promote a more centralized way of package distribution: developers are often expected to build & publish "official" binaries/bundles to a centralized store (NPM/PyPI). Doing this of course puts burdens on the project, like secret key management and maintenance of the JS/TS wrapper.
-
Is it ever possible to modify
acprepto set up environment for WASM builds? (Or is there a need for it in PRs?)It seems to me the
acprepprepares for local builds but WASM builds sound more like cross-compiling (requiring custom-builtboostlibgmplibmpfr...). Using pyodide simplifies the job but I have no idea if it works across platforms. Is this part expected in a PR? I am rather convinced that the pyodide approach can build in GitHub workflows but providing a cross-platform set-up script seems a bit overwhelming to me.