djinni implementation question

[sgerbino]

I want to use djinni (https://github.com/dropbox/djinni) to generate obj-C, Java, and C++ from an interface definition file. I would like to make a fipsified version of it.

My question is, how best to implement this?

• Do I create a fips verb that calls the djinni script and somehow allow other projects to call that verb with its own arguments as a dependency?
• Or do I create a fips djinni cmake macro that can somehow create targets with arguments that fips can understand?
• Another suggested implementation method?

[floooh]

From first glance it looks like a good scenario for the fips code-generator feature. A code generator is basically a python script which is called from cmake to create target files from source files. Here's the section from the docs:

http://floooh.github.io/fips/codegen.html

The advantage is that this is fully integrated into the build process. When the source files are edited, the code-generator will automatically be run.

Here are examples from Oryol and my emulator:

Sprite sheet generator:

This converts a PNG file and sprite-defintions into an embedded C source file:

• the source 'inplace' python script: https://github.com/floooh/oryol-samples/blob/master/src/Paclone/sprites.py
• the resulting C header/source pair: https://github.com/floooh/oryol-samples/blob/master/src/Paclone/sprites.h, https://github.com/floooh/oryol-samples/blob/master/src/Paclone/sprites.cc
• the fips-specific CMakeLists.txt call: https://github.com/floooh/oryol-samples/blob/master/src/Paclone/CMakeLists.txt#L12
• an additional helper python script imported by the above generator script: https://github.com/floooh/oryol/blob/master/fips-generators/SpriteSheet.py

Import MagicaVoxel files into embedded C header/source pair

• the python generator script: https://github.com/floooh/oryol-samples/blob/master/fips-generators/ImportVox.py
• the cmake file entry: https://github.com/floooh/oryol-samples/blob/master/src/KC85-3/CMakeLists.txt#L8, in this case the generated C header/source pair is generated 'out of source' and not committed into version control

Shader Code Generator

This is the most complex use-case so far, it parses a 'annotated GLSL' shader source files, translates them to various GLSL dialects, HLSL and MetalSL, checks for error by calling offline shader compiler tools, and translates the errors back to IDE format.

• a simple shader source file looks like this: https://github.com/floooh/oryol/blob/master/code/Samples/Triangle/shaders.shd
• it is added to the build with an Oryol specific cmake macro: https://github.com/floooh/oryol/blob/master/code/Samples/Triangle/CMakeLists.txt#L4
• the custom cmake macro looks like this: https://github.com/floooh/oryol/blob/master/fips-include.cmake#L189
• and here's the code-generator script (pretty big though): https://github.com/floooh/oryol/blob/master/fips-generators/Shader.py

Read emulator ROM dumps:

Here's another simple example which loads binary dumps and writes them into a C array:

• the generator script: https://github.com/floooh/yakc/blob/master/fips-generators/dump.py
• how it's added to the build: https://github.com/floooh/yakc/blob/master/src/yakc/CMakeLists.txt#L17
• a YAML file, which defines what to dump: https://github.com/floooh/yakc/blob/master/src/yakc/roms/roms.yml
• and what's generated: https://github.com/floooh/yakc/blob/master/src/yakc/roms/roms.h, https://github.com/floooh/yakc/blob/master/src/yakc/roms/roms.cc

Z80 instruction decoder

And finally here's a code-generator example which creates a Z80 instruction decoder source using an 'inplace' python script:

• the python script: https://github.com/floooh/yakc/blob/master/src/yakc/z80_opcodes.py
• the generated C++ source: https://github.com/floooh/yakc/blob/master/src/yakc/z80_opcodes.cc
• how it's added to the build: https://github.com/floooh/yakc/blob/master/src/yakc/CMakeLists.txt#L15

I hope this helps :) -Floh.

[sgerbino]

Thank you for this detailed response. It does seem to be a perfect fit for my scenario. I will go down this road for implementation, thank you! 👍

[sgerbino]

djinni generates many files when invoked, is it possible for fips_generate to support multiple HEADER and SOURCE arguments?

Perhaps it would make sense to implement HEADERS and SOURCES multivalve arguments?

Implementation reference: https://github.com/sgerbino/fips-djinni

[floooh]

TL;DR: not possible at the moment, but see bottom for a workaround using 'verbs'

Yes, that's a current limitation. I'll write a feature ticket that multiple target files should be supported, it should be doable but affects both the cmake as well as python side.

I can't promise a delivery date though. I also can't think of a simple workaround the at the moment, at least one that would fully integrate with the build process. The problem is the multiple generated .cpp files which need to be included in the build, if djinni would just generate headers it would be easy (those don't need to be included as input files to the build, only included somewhere else). Since cmake doesn't accept non-existing files as input, the code-generated writes empty dummy files when cmake is run to workaround that chicken-egg-situation.

I guess until this is fixed another workaround is to run djinni as a 'fips verb', verbs are python scripts in a project's 'fips-verb' directory which can then be run as './fips [verbname] [args]'. Have a look here: https://github.com/floooh/oryol/tree/master/fips-verbs

These scripts need to implement a 'help' and a 'run' function (check the 'testrunner.py' script which is fairly simple:

https://github.com/floooh/oryol/blob/master/fips-verbs/testrunner.py

The run() function takes 3 args: fips_dir, proj_dir and args. Args is an array of string following the verb name, and proj_dir ist the absolute path to the project.

For instance, if you have a file 'djinni.py' in your project's (or a dependency project's) fips-verbs directory, you can run './fips djinni' to invoke that script, this would need to happen when the djinni input files change (that's the drawback versus the generators, which would run automatically at build time).

You would also need to add the generated files manually to the CMakeLists.txt files where they are needed.

I hope this works as a first workaround until it is possible to define multiple target files for generators...

Cheers, -Floh.