Proper way to share generated container accross multiple .c files

[jens-f]

trafficstars

We are sharing a generated List across multiple .c files. The code generated for the list is making all the functions static. In order to call these, we are creating wrapper functions in the same .c file that generates the list and have these call the generated functions. In addition, we are exposing the structs for the generated List in the header file as anonymous pointer (typedef struct MyList_head_s *MyList_t;)

Is there a better way to achieve the same in a more automatic fashion?

[P-p-H-d]

There are currently no better way to do it. I have a few questions:

Do you need:

• a full encapsulation? (type + functions are encapsulated)
• function encapsulation? (type is exported, functions are encapsulated)
• partial function encapsulation? (type and basic functions are exported, huge functions are encapsulated)

What is the reason that makes you chose to avoid putting LIST_DEF in your header?

[jens-f]

We actually didn't try using the LIST_DEF in the header. Wouldn't that explode the code size since the same functions are generate multiple times in each .c file that consumes the header?

Regarding the other questions, options 1 or 2 sound like the preferable ones to keep code-size small and not to leak too much of the internals into other code-parts that want to just "use" the container. But either of them could probably work.

[P-p-H-d]

Wouldn't that explode the code size since the same functions are generate multiple times in each .c file that consumes the header?

Well it depends. First if you don't use a function in the associated .c file, it won't be present in the object file (so it will consume 0). Then, a list is a low overhead container. Typically, in release mode (NDEBUG=1 + optimization enabled), the low level functions of a list (pretty much everything except I/O & the O(N) functions) doesn't consume much object size IF the basic type has low overhead. And with optimization enabled they may even generated smaller code size. This behavior may be different for other containers. It is however pretty much the same than the C++ template library consumption.

Regarding the other questions, options 1 or 2 sound like the preferable ones to keep code-size small and not to leak too much of the internals into other code-parts that want to just "use" the container. But either of them could probably work.

I'll see what I can do. I think option 1 is the easier to do.

[jens-f]

Thanks for the explanation on the code size. We might give this a try to see the impact in our case. Our data types are a bit more complex - including lists of lists which in turn use memory pools.

[begriffs]

What if mlib provided xxx_DECL macros in addition to xxx_DEF?

Then a project could, for instance, have these files:

/* containers.h */

#include "m-array.h"

ARRAY_DECL(r_int, int)   /* just the declarations */

/* containers.c */

#include "containers.h"

ARRAY_DEF(r_int, int)    /* the full code */

Other C files in the project could #include "containers.h" and link against containers.o

[P-p-H-d]

This solution would do the job. Some C container libraries already do this. Note that the current macro _DEF could not be used for this (another one will be needed) to avoid breaking the interface.

However, I am working on another solution. It adds another container (currently named 'wrap') that will encapsulate the targeted container in a wrapper with one declaration macro (WRAP_DECL which only uses the oplist of the container) and one definition macro (WRAP_DEF that will use the _DEF macro of the container). I have a working prototype that seems to works for array & list (however not tested), but not for associative arrays (it needs more work).

[Mango0x45]

Has any progress been made on this?

[P-p-H-d]

Well, not much. I still struggle with the right interface. I have more or less abandoned the WRAP API (too much complex to make it right from a user point of view) that was explained above.

My preference is now more like what is proposed by begriffs (with the suffix _DECL and another suffix different than _DEF - since it is already reserved, and maybe more like \DEF_AS). I still wonder if it needs to add a parameter to define the level of inlining - the user may want still to have the very small functions still inline (like a _get_size function if it only reads a field in the structure). There is also the issue that generating the functions as classic functions may increase the code size since a lot of functions may be generated by the expansion but only few ones are really used by the code like the I/O functions or the emplace functions (and the problem will only get worse as more and more features are included in the library).

Do you need this evolution?

[Mango0x45]

I do, and in the meantime I'm solving it by just using my own datastructure library with two macro calls (as suggested here). It would be nice to be able to use this lib though.

[P-p-H-d]

I have implemented a prototype (on the branch feature/decl-gen) of such generations for the m-list header (only and excluding emplace generation). To do that, the parameter of the interface M_LIST_DEF_AS (LIST_DEF & LIST_DEF_AS remain compatible) has been updated by adding a new argument that defines how to generate (it can be INLINE - same as before -, DECL -generate declaration only- or BODY - generate body of functions, need declaration before-). Example:

#include "m-list.h"

M_LIST_DEF_AS(DECL, list_int, list_int_t, list_int_it_t, int)
//...
M_LIST_DEF_AS(BODY, list_int, list_int_t, list_int_it_t, int)

So the arguments of the macro are:

• level (INLINE, DECL or BODY)
• prefix of the functions
• name of the container type
• name of the container IT type
• type of the item in the container
• (optional) oplist of this type

This is not a final interface but it is for review.

[P-p-H-d]

Has someone tested the proposed prototype? Any comments on it?

[P-p-H-d]

I probably will change the interface so that it takes an oplist that parameterize the generation as second argument, like this:

M_LIST_DEF_AS(list_int, (INTERFACE(BODY), TYPE(list_int_t), IT_TYPE(list_int_it_t)), int, M_BASIC_OPLIST)

Rationale: There exists more and more parameters that influence the generation but are not part of the interface of the contained type. Using oplist to parameterize the generation enables much more flexibility and uniformity.

Branch not updated with this change.

[P-p-H-d]

I have found another way to implement this by using a finer grain "inline" semantics and combining it with optimization level requested to compiler. It was much easier to implement and supports all M*LIB functions (including strings)!

It is available in the branch feature/decl-gen-v2

Documentation:

• in all files except one, define M_USE_EXTERN_DECL before including any M*LIB header to requests to call M*LIB functions and don't inline then.
• in exactly one file, define M_USE_EXTERN_DEF before including any M*LIB headers to define the M*LIB functions. This translation unit shall contain all templated macro too.

There is no other changes. It works only with GCC or CLANG.

And don't forget to optimize for size ("-Os"), remove assertions ("-DNDEBUG") and drops unused functions ("-ffunction-sections -fdata-sections -Wl,--gc-sections") to optimize your build size.

If everything is ok for you, I'll integrate this change instead.

[P-p-H-d]

I have tested with flipperzero-firmware with its pre furistring state (which uses a lot m_string). Before the firmware ( dist/f7-D/flipper-z-f7-full-local.bin ) size is 827080 bytes. With M_USE_DECL and latest M*LIB, the firmware size is 765128 bytes (minus -7.5%)

[P-p-H-d]

Merged with master (see #110)