Flavour rework

[cerna]

Based on @zultron's work in single module RTAPI build branch, for which pull requests exist at machinekit/machinekit#1462 and at machinekit/machinekit-hal#227, I have been thinking, theorizing and working on flavour rework with the intended end-goal of making Linux based RTOSes other than Preempt-RT first class citizens in Machinekit too. So far that means the co-kernel idea based around Xenomai. (Xenomai version 2, Xenomai version 3 and EVL Core.) I have been lurking around the Linuxcnc-devel channel on IRC and at the moment there is interesting discussion about porting the LinuxCNC onto slightly hacked version of RTAI 5.2 done by @NTULINUX. They were able to reach sub 10μs latencies measured by the current HAL script with full-blown X server enabled desktop. That is nice, given the IPIPE is almost the same (or completely the same, I don't know) as in Xenomai project. I don't think that the same number are possible with user-space application code, however I would like to bring my numbers as close as possible.

So far, with Xenomai3 with 4.14 and 4.19 kernel versions, I was able to reach latencies about 50μs with X server running and latency-test 1000000 and with EVL Core on mainline 5.2 kernel latencies about 50μs with X server and under 8μs with X server turned off with latency-test 15000 1000000. Both on Clevo N131WU with i7-8550u and with Xenomai even on Core2Duo E8400 with similar results. (So far the EVL Core is not supported on older hardware, or better said Philippe Gerum is more interested in ARM than he is in x86, so x86 is on slow burner.) All tests done without CPU isolation, so chances are the results will be better.

Implementing new flavour is with the new API quite easy, or at least straightforward. Problematic is the actual HAL code which will run in the infinite loop of flavoured tasks/threads. So far I did not found the provision for this in the HAL code itself. Which is why I logged a token at machinekit/machinekit-hal#161. Every flavour is backed by some sort of general use library, for Xenomai it is libXenomai, for EVL Core it is libEVL and for Preempt-rt it is pretty much everything else. So to comply with real-time constraints, the code needs to use function calls from it for it's work. In other words, some critical parts of RT HAL components needs to be written flavour specific. Otherwise, the thread will context switch to secondary mode in Xenomai or in-band in EVL Core. And that is something what we don't want.

The change in code could be just different linking, as Xenomai implements its own variation on POSIX. (But probably it would be better to wrap the function call into COBALT(function call) to enforce the usage.) Or it could be completely different logic as will be needed for EVL Core as libEVL is very light-weight wrapper. (Thinking about it, what needs to be done is something Xenomai itself is trying to do: Create abstraction layer.)

So one way forward is to implement system for function hooking in HAL modules themselves similar how the API for flavours works and export the information about currently loaded flavour into HAL layer. The implementation of the actual functions could go either to the flavour code library (flavor-evl-core.so), the access library (libXenomai, libEVL) or the HAL modules themselves. Given the need for encapsulation I think the code should go directly to the HAL module code. HAL layer then should refuse to load modules on flavours for which the module explicitly does not state support. (With Preempt_rt or POSIX used as a default flavour on which all current modules should be able to run.) But this is only in thinking phase.

Then there is the issue of the single module RTAPI build branch itself, as currently it is not intended for plug-in loading of flavors which I think would make lives easier (at least mine). (Idea based on what @ArcEye said in #201.) The idea is to decouple the Machinekit-HAL from flavour API implementations. These API implementations would be shared libraries which would be runtime loadable from rtapi library. From packaging point of view the work-flow would be to install machinekit-hal which would be the Core package, then install arbitrary number (but for running condition minimally one) of machinekit-hal-flavor-posix, machinekit-hal-flavor-xenomai2, machinekit-hal-flavor-xenomai3, machinekit-hal-flavor-preempt-rt or machinekit-hal-flavor-evl-core. These packages would install into folder modules/flavors its own directory with predefined name {number}-flavor-{name}/' and into this directory minimally one file flavor-{name}.so, where the number would represent weight for automatic default loading (flavour with higher number will be loaded), the name would be for user to specify the flavour explicitly with command line argument or environment variable and the need for folder is because some flavours will need more than one shared libraries, as explained by @zultron in machinekit/machinekit-hal#227. That way on production machine user can chose to have only one flavour installed to minimize chances of loading unwanted flavour.

The loading itself of flavour library will be done by constructor function calling register_flavor(flavor_descriptor_ptr flavor) function in rtapi library. Loading of any other library or acting out any other logic would be sole responsibility of library plug-in module to reach fully encapsulated functioning. rtapi will be only interested if after dlopening the flavour library flavor-{name}.so the global flavor_descriptor will be populated. This I am currently working on.

The only thing I don't know what to do with is the ULAPI flavour. Instructions welcome.

The last but not least is the need for rework of latency-test program/script itself. Or maybe adding another script. One which would be able to distribute with Machinekit-HAL and run on system without X server running. I have been thinking about something written in my "favourite" python with option to generate dummy load and ncurses based TUI.

[cerna]

These packages would install into folder modules/flavors its own directory with predefined name {number}-flavor-{name}/' and into this directory minimally one file flavor-{name}.so (...) will be populated. This I am currently working on

Implementing search function for this idea I started to realize how messy it is. And quite un-Machinekit. Because as far as I can see, Machinekit is trying to put every single .so module library into one directory. Introducing tree structure only for one part of the system seems unreasonable. Also, putting weight into directory name introduces possible problems with some users playing around.

So, I was thinking what to do instead which would allow me to put the flavour shared libraries into the same folder as other libraries (like rtapi.so itself) and I recalled the EXPORT_SYMBOL() code. What about creating new meta section named mk-flavor in the ELF file .so with the weight integer and flavor name and then programmatically read it in the rtapi.so code which create list of available flavors on the system by use of libelf gelf.h?

The problem I can see is the fact that the code would have to read all libraries in the folder and filter only the flavour ones. But it isn't time-critical code, so it shouldn't be a problem.

EDIT: OK, tests show that finding libraries with specific sections by way of libelf in ldconfig is 9cs operation on my system. Not sure caching is worthwhile.

[zultron]

So the sound of crickets doesn't deafen here, I'm still trying to understand everything here and mulling over it all. Lots of interesting ideas. I'll get back ASAP.

[cerna]

I have created prototype - more to the point of model, really, as it is not working at the moment - of how I envision working of this process. rt-preempt.c, posix-nonrt.c and xenomai2.c illustrate how the flavour API libraries should be done. (BTW, I really don't understand the reasoning behind RTAPI and ULAPI defines as I see no pattern and quite frankly I don't see how it could work with ULAPI define as in xenomai.c there were function definition called nowhere. I tried to somehow insert the compile switches, but really have no idea.)

To the point of FLAVOR_STAMP("xenomai2", 2, 1) - this is pre-processor macro which will create 3 global constants of types: unsigned int, unsigned int and char array in ELF section named machinekit-flavor. One unsigned integer represents the API version (so far 1, intended for possible dissynchronization between flavour library and rtapi.so), other unsigned integer represents the weight of flavour (ordering in which flavours will be tested when autoinstalling into running rtapi), char array is null terminated string representing the name of the flavour. I am also thinking about adding third unsigned integer representing the flavor_id of the flavour.

I am hoping that reading on platform files created on this particular platform by way of universal GELF library will be enough to make this code universally portable.

This is then read in loop by search component. Searching whole /lib (every file in every folder under /lib "mountpoint") takes on my system about 3.5s, searching only the /lib directory without child-directories takes about 0.005s. So I don't think some form of caching will be strictly necessary. However, there is a need to reach conclusion where the machinekit-flavor-{name}.so (BTW, given this approach, they can be named mysupercode.extrafine and it should still work) files would reside for package install and for RIP build.

Given extreme invasiveness, again, comments welcome.

[NTULINUX]

I don't use MachineKit nor endorse any of mhaberler's work. Unsubscribing.

[cerna]

I have completely missed the @ArcEye's response https://github.com/machinekit/machinekit-hal/issues/161#issuecomment-510834523 in #161 . That changes things, I thought I was original in my idea. (Well, I implemented it by use of function library, but still.) Will have to think about that. That kind of took my wind.

I don't use MachineKit nor endorse any of mhaberler's work. Unsubscribing.

This is state secret level of information. Every head of department should have it on their table first thing in the morning.

[zultron]

@cerna,

(You should be able to get much better latencies on your i7 by limiting the cstates; see #226.)

Since we'll need two plugins to accommodate at least Xenomai, I think each flavor should ideally have two plugins just to make things uniform. Let's call the one containing metadata and prerequisite checks the "info" plugin and the other with the actual RTAPI routines (and any linked libs) the "business" plugin (feel free to update this terminology!).

Currently, the flavor_descriptor struct contains things that need to be split up between the info & business plugins. It sounds like you intend your FLAVOR_STAMP() to convey the metadata needed on the info side. That makes sense, and whatever we end up with should also deal with the flavor prerequisite check function as well, the other notable bit that belongs in the info plugin.

About the search path, see my recent related comments over at #104. I think there are important use cases to build in some means of loading modules from an alternate directory. Maybe a single additional directory is enough (MACHINEKIT_EXTRA_MODULE_DIR=~/git/my-project/modules), or maybe multiple (MACHINEKIT_EXTRA_MODULE_PATH=~/git/my-project/modules:~/git/other-project/modules).

I'm sorry I still haven't had time to look at your tree. There are too many commits in there to look at through the GitHub interface, so I'll check out your branch locally when I get time.

[cerna]

(You should be able to get much better latencies on your i7 by limiting the cstates; see #226.)

Truth be told, I wasn't really interested in making it the best it could be at the time. I really wanted just to test it if it can be used (which it can). This i7 is my main “personal” PC, not something I will (probably) ever use to seriously turn motors or do other automation stuff.

Testing of this idea is scheduled for second stage when I will prettify my code and — you know — make it generally usable. But I have just dirty-tested it on EVL Core and I actually didn't see much (if any) improvement. (However inconclusive it may be.)

Since we'll need two plugins to accommodate at least Xenomai, I think each flavour should ideally have two plugins just to make things uniform. Let's call the one containing metadata and prerequisite checks the “info” plugin and the other with the actual RTAPI routines (and any linked libs) the “business” plugin (feel free to update this terminology!).

Funny, I have spent time trying to investigate if the Xenomai 2 could be merged into one shared library (one .so file) by use of (for example) 'LD_PRELOAD' trick. And it's really only version 2 which make a problem, I have investigated and I think I come up with nice idea of Xenomai 3 initialization, which wouldn't need two separate libraries. (Maybe similar action is possible with Xenomai 2 [just not the exactly same] but I wasn't really interested in spending several hours/days looking into code of EOLed library.)

However, I don't think that we should try to make things uniform with number of required plug-ins or modules for each flavour API. I really see a quite attractive solution in one point of contact (I am not saying that it has to be one struct, it can be some logically structured tree) with 'rtapi.so' (which itself is connected to 'rtapi_app'. So if year in the future some new RTOS with requirement of ten modules comes along, it wouldn't matter. Because all of that will be encapsulated in the new — let's call it — module unit.

Currently, the flavor_descriptor struct contains things that need to be split up between the info & business plugins. It sounds like you intend your FLAVOR_STAMP() to convey the metadata needed on the info side. That makes sense, and whatever we end up with should also deal with the flavor prerequisite check function as well, the other notable bit that belongs in the info plugin.

Yes, you are right, I am interested in basically three types of information: compile-time metadata, run-time metadata and run-time functionality. The compile-time metadata (it doesn't really have to be baked-in at compile time, one can change it by some sort of post-processor program) or statically known data represents types of information like name of the module, versions, architectures and so on, everything that is known beforehand and is not going to change (or the change is discrete and not conditional on some variable) and should be known before the shared library is loaded. (That is - I think - the most important part, we do not want any part of that code to run because we are using this metadata for identification purposes.) The run-time metadata represents the conditional part which cannot be known beforehand, as it is dependent on some system condition and the Machinekit's 'rtapi.so' doesn't even know how to determine this (because it is again situation dependent). It has to be determined after the shared library is loaded by dlopen and by running some code. I am planning on doing this by way of constructor function automatically. (However, thinking about it you are right that it is some form of initialization which does not need to be performed only on flavour library module load. But I cannot think of situation where I would need to run initialization after the RTAPI is running for some time from the top of my head. Maybe some flavour dependant low-level module like the Xenomai's RTNET?) To this class we can include the can_run_flavor variety of functions. Or maybe tests like for your hardening techniques explained in #226. (Or — I don't know — d(Temperature)/d(Load) of CPU, it doesn't really matter at the moment.) And the last group is the run-time functionality. The aptly named “business” pointy end which mediate access to the RT core. But this last part will have to be probably split into the global “Machinekit” flavour implementation and “HAL modules” implementation as stated beforehand.

So, the filtering will be: Take a set of files in some directory, filter from it all ELF files, filter from it all files with section machinekit-flavor and create list of objects (each with name, weight, id). This way we could get the exact same information as is currently baked-in the #227 pull request. User could then specify flavour to load or just let system do it automatically for him. This way the shared library would load in rtapi and do the necessary load-time checks (run-time metadata). And only after this it could run the "business" part (run-time functionality).

About the search path, see my recent related comments over at #104. I think there are important use cases to build in some means of loading modules from an alternate directory. Maybe a single additional directory is enough (MACHINEKIT_EXTRA_MODULE_DIR=~/git/my-project/modules), or maybe multiple (MACHINEKIT_EXTRA_MODULE_PATH=~/git/my-project/modules:~/git/other-project/modules).

I have been thinking about using the standard 'ldconfig' routine. And looking into all paths registered with it. Problem is that it has to survive package installation, RIP mode (with probably multiple compiled and ready to run instances of Machinekit on one system at the same time) and whatever you are cooking at the moment with the new build system. It needs deliberation.

I'm sorry I still haven't had time to look at your tree. There are too many commits in there to look at through the GitHub interface, so I'll check out your branch locally when I get time.

Yes, sorry about that. I am not really used to this work-flow of publishing unfinished work to the public eye. But I am more interested in ideas than solid code implementation. After the dust settles on how it is supposed to work, the coding will be straightforward.

(I am going to update this comment after I though about it more.) (UPDATED once to make more sense.)

[cerna]

(...) That changes things, I thought I was original in my idea. (Well, I implemented it by use of function library, but still.) Will have to think about that. (...)

I have looked into Michael Haberler's implementation of ELF section crawler nad even if his idea is little bit different than mine, I think bending his code to encompass both of our needs should be possible. (OK, his code is basically Ctrl+C&Ctrl-V from https://stackoverflow.com/a/10865542/11837388 that fail on my need to signal that the file is not ELF and given that opening FD and all that checking is not exactly cheap on big number of files, I rewrote it so it suits me and do not use libelf-dev.)

However, what is the official policy on non-standard compiler dependent functionality? I need some implementation for closures in C and the GCC extension of nested function looks better then using void * or libffcall to support my callbacks filled code.

Also, is there any chance somebody would like to have the V2 HAL modules separated the same way? It is the same idea. But so far every HAL module shared library was compiled at the same time as a main Machinekit application and nobody to my knowledge was copying .so files on it's own.

I have been thinking about using the standard 'ldconfig' routine.

It makes the most sense to put the path to the rtapi.ini file, I think. With @zultron's rework there is unused line flavor=... which could be used.

[cerna]

@zultron, I have been thinking about your idea about reworking the flavor_descriptor_t struct and the rtapi_flavor_hook_wrappers.c source. Given that I have come again at the problem described in https://github.com/machinekit/machinekit-hal/pull/227#issuecomment-511242953 at which core is the fact that the Machinekit-HAL system does not (at compile time) know the flavor_descriptor_ptrs, I now think that you are right in your observation that the data and function graph needs reworking. I have tried to avoid this as much as possible, but it's inevitable.

To illustrate my world-view, there are few prerequisites and assumptions:

• There can be only one registered Flavour module at any given time with RTAPI.so

• The RTAPI.so creates list of known flavour modules in its start stage, then registers one runnable Flavour module from this list

• After registering concrete Flavour module it is registered until the RTAPI.so code itself is unloaded

• To run any flavour specific code (creating new threads etc.) the Flavour module has to be registered

• Flavour module, it's state and runnability, list of known flavour modules and so on is always specific to concrete RTAPI_APP instance

• Flavour module is keeping all additional data it needs for running tasks internally.

That out of the way, there is - I hope - clear reason why I see the function argument of flavor_descriptor_ptr in pretty much everything in rtapi_flavor_hook_wrappers.c as nonsensical. These functions should be points of liaison between lower flavour specific code and the higher RTAPI/HAL code. They should signal an error if somebody is trying to call them before there is a registered Flavour module (or after deregistering) in some controlled fashion (i.e. not exit(), however signal to RTAPI_APP).

Also, the name of the flavour, version, ID and whatnot should not be part of the flavor_descriptor_t but be just separate in the STAMP (given I am putting them to special section) and I end up having two copies in the ELF file. It is the compile-time metadata.

And also, pretty much every Flavour API module implements it's own internal array with task data to map to the externally referenced task_id, so I think passing the task_data pointer argument is unnecessary and could be misleading. What about passing the task_data pointer only to the task_new_hook and to all other task_ functions only task_id. That way, if the function takes in task_id, it signals that it can be run on (almost) any task, when it takes in void, it signals that it has to be run from context of task thread. (That way nobody can even think about spoofing the task_data* for hackish reasons.)

So, the API would look like:

1. compile-time metadata

• Name
• ID
• Magic
• Flags
• API Version

2. run-time metadata

• Constructor
• • Can_run_flavor
• • Registering the module in RTAPI
• Destructor
• • Deregistering the module from RTAPI
• Module_init_hook
• Module_exit_hook

3. run-time business logic

• Exception_handler_hook
• Task_update_stats_hook
• Task_print_thread_stats_hook
• Task_new_hook
• Task_delete_hook
• Task_start_hook
• Task_stop_hook
• Task_pause_hook
• Task_wait_hook
• Task_resume_hook
• Task_delay_hook
• Task_pll_get_reference_hook
• Task_pll_set_correction_hook
• Get_time_hook
• Get_clocks_hook
• Get_self_hook

BTW, is there still need for separate get_time_hook and get_clocks_hook?

Both run-time metadata and run-time business logic would be separate structs. run-time metadata would take point and after loading the .so file into memory will run constructor function which will determine based on can_run_flavor if it can run and auto register the module by registering the run-time metadata struct. Then the program will have to call the module_init_hook which can do other run-time initialization and by registering the run-time business logic struct will open the calling on the rest of Flavour API module code. (So error on flavor_task_new_hook before the Flavour API module is registered and before the flavor_module_init_hook is run.)

The compile-time metadata struct will be created and listed (there will be multiple instances) at start of RTAPI based on path in rtapi.ini and/or from environment variable (or maybe as a argument to rtapi_app, I might copy the logic from Xenomai 3 as I quite like it).

There is a possibility to leave something akin to flavor_descriptor_ptr f in rtapi_flavor_hook_wrappers.c based on the fact that there will be the list of compile-time metadata structs. But to run the function on some arbitrary flavour in known list, it would mean stopping and unloading the currently registered flavour, loading the new flavour, testing it, initializing it and then running the function. And I cannot see the use case for that. You could just run other instance or the RTAPI_APP.

(BTW, it may be a time to think if the initialization process would be in any way affected by other instances of RTAPI_APP.)

Sorry that it is such an overhaul. But is it at least little aligned to what you meant in Since we'll need two plugins to accommodate at least Xenomai, I think each flavor should ideally have two plugins just to make things uniform. Let's call the one containing metadata and prerequisite checks the "info" plugin and the other with the actual RTAPI routines (and any linked libs) the "business" plugin (feel free to update this terminology!)?

[zultron]

@cerna I've been taking a little break from this, but will get back to it shortly and think about this harder. First thoughts reading this latest post:

• Assumptions/prerequisites agree with my thinking
• It makes sense to issue graceful errors when calling RTAPI flavor functions before the flavor is loaded
• Adding flavor metadata to a loadable module sounds nifty
• API conceptual organization looks good, except
  • In the "run-time metadata" section, I'm having a hard time wrapping my head around calling executable functions "metadata", but that's OK

By "constructor", I assume you mean the loadable module's ELF .ctor function declared with __attribute__((constructor))?

What precisely does "registering" a flavor mean? I can't tell if it means adding it to the flavor management system's data structures as a candidate for selection, or if it means loading the .so module, or something else.

I'm totally fine with an overhaul if it makes sense to do so. You're definitely making some sense here and I'm looking forward to thinking about it more deeply.

The one big question in my mind: What are you thinking to address the Xenomai flavor loading question? It almost sounds like you're saying the "run-time metadata" will go into the same module with the "run-time business logic", but I can't tell if you have a way to deal with the Xenomai libraries' own .ctor function that exits the application at load-time when no Xenomai environment is detected. Do you have a way to work around this or is separating metadata and business logic into two plugins still the plan?

[cerna]

By "constructor", I assume you mean the loadable module's ELF .ctor function declared with __attribute__((constructor))?

Basically yes.

What precisely does "registering" a flavor mean? I can't tell if it means adding it to the flavor management system's data structures as a candidate for selection, or if it means loading the .so module, or something else.

It is intended to work like this:

1. Loading module (RTAPI.so) will look into rtapi.ini for value of FLAVOR_LIB_DIR (for example /lib)
2. Then scan every file in specified directory for:
   • Executable and Linkable Format files only
   • Files with specified section(s) (for example machinekit-flavor)
   • Parse the data of section(s) into structure compile-time metadata (so far that is done putting everything in block of data and then reading from it, using some serialization format thingy would be from long term support better, but I am pretty unhappy about the choice of Protocol buffers)

• This is the cold stage as nothing should run and nothing should be triggered

3. Based on some predicate one of the discovered Flavour modules will be selected for loading
4. The selected module .so will be dlopened and the constructor function will be run
5. In the constructor, it will be determined if the flavor can run (is able to run) and if so the constructor will call function exported from rtapi.so which takes as a parameter pointer to struct runtime-metadata and as so register the Flavour module

• This state is the hot stage, Flavour module (code) is executable, everything is almost ready to run, Machinekit does not expect any errors to occur

6. The call to Module_init_hook (by way of flavor_module_init_hook) will call rtapi.so exported function to which passes the run-time business logic struct
7. Everything is ready for task/thread creation

So the registration is the transformation from cold->hot and ability of RTAPI to call any function as everything in Flavour module is static, so invisible to dlsym.

(In functions and structures it should look something like https://github.com/cerna/machinekit-hal/blob/flavor-rework/src/rtapi/flavor/rtapi_flavor.h#L49-L144).

The one big question in my mind: What are you thinking to address the Xenomai flavor loading question? It almost sounds like you're saying the "run-time metadata" will go into the same module with the "run-time business logic", but I can't tell if you have a way to deal with the Xenomai libraries' own .ctor function that exits the application at load-time when no Xenomai environment is detected. Do you have a way to work around this or is separating metadata and business logic into two plugins still the plan?

I would love to say yes, but unfortunately I only have several ideas about how to do it. No functioning prototype. Just to be pedantic, it is problem with Xenomai 2, not Xenomai 3. And given the Xenomai 2 is going to be soon shot through X anyway, I did not spend on it much time. You can run the constructor function before others by way of sorting the .init_array function pointers, but you cannot stop the calling of others from one as this is done by simple for loop from glibc. You could completely rework the .init by post-processing the ELF, but it is not one ME job (man-evening).

In the "run-time metadata" section, I'm having a hard time wrapping my head around calling executable functions "metadata", but that's OK

I don't care one way or another, give me terminus technicus you are fine with and I will be using it from now on.

[cerna]

When we are talking about renaming there is one more instance I would like to discuss. There are two sets of function definitions in rtapi_flavor.h, one prefixed by rtapi_ and the other by flavor_. One is used for signature definition for the flavour module API definition, the other for access from the RTAPI side of Machinekit. Like https://github.com/cerna/machinekit-hal/blob/8e9ccc160f58999c9afaaf3b5cd1650afc2fae06/src/rtapi/flavor/rtapi_flavor.h#L78-L108 and https://github.com/cerna/machinekit-hal/blob/3ab22fb5d39229c75716b004cef945350eff9406/src/rtapi/flavor/rtapi_flavor.h#L173-L194. However, for me, it would make sense that the RTAPI side functions would be prefixed by rtapi_ and the signature definitions for flavour module implementing API would be prefixed by flavor_, so the exact opposite of how it is now. Any thoughts?

It makes sense to issue graceful errors when calling RTAPI flavor functions before the flavor is loaded

Thinking about it, there would be need to query some variable at every call of these functions. I have no idea about what performance hit it would cause in faster real-time loops. You know, if it is worth it? EDIT: Investigating, the natural alignment of int32 should be good enough. But maybe use RTAPI atomics for the state variable.

[cerna]

Working little more on the set of functions every FLAVOUR API module should support, which so far I have extracted from the flavor_descriptor_t structure to structs flavor_cold_metadata, flavor_hot_metadata and flavor_runtime_business, I am thinking about slashing the flavor_runtime_business into two structs based on operations which would and would not cause in-band context switch. (In other words, functions which are called from hard, time sensitive code in infinite loop and function, which are called from normal threads.) For example, functions running on the RT side - like the flavor_task_self_hook or flavor_get_time_hook or flavor_task_delay_hook - and the operation which cannot run from RT context like the flavor_task_new_hook and flavor_task_print_thread_stats_hook. Idea behind it is to allow for HAL modules to dynamically register (let's say export) FLAVOUR specific functions which they or other modules would need for functioning (as I mentioned in first post in this thread, this is not something I am trying to implement right now, it's something for second stage). The stellar example - I think - of this is the flavor_task_pll_set_correction_hook and the flavor_task_pll_get_reference_hook, which - as far as I can see - are only used by the pll_correction.comp. Basically, what I am thinking is broadening of FLAVOUR API modules based on HAL modules without main HAL functions (which is something M. Haberler in these annals talked about long time ago).

But, because this whole rework has a primary goal to allow Machinekit to encompass even yet non-existing RTOS, I think it makes a sense to tighten requirements about exactly what the set of functions of FLAVOUR module has to implement. And in addition, to move some code from Machinekit proper to module code (I am talking about the get_time for example, to reach fully enclosed codebase).

These functions will need to be mandatory:

• flavor_module_init_hook
• flavor_module_exit_hook
• flavor_task_update_stats_hook
• flavor_task_new_hook
• flavor_task_delete_hook
• flavor_task_start_hook
• flavor_task_stop_hook
• flavor_task_pause_hook
• flavor_task_wait_hook
• flavor_task_resume_hook
• flavor_task_delay_hook
• flavor_get_time_hook
• flavor_get_clocks_hook
• flavor_task_get_self_hook

With the understanding, that the function flavour so far does not implement, then it will raise exception (like the unused _resume_, _pause_ [which on the other hand would be nice to have]).

Also, all the functions within the rtapi_flavor_hook_wrappers.c will need to return something like integer to return error when they are not allowed to be called. I'm thinking about -EPERM. That is no problem, as most of these are double wrapped (looks like for historical reasons), so in the outside wrapper, there can be logged an error, or maybe raised exception.

Finally, all of these puts me at odds with the ULAPI flavour. Frankly, it is extremely hard for me to wrap my head around the ULAPI code concept. As far as I can tell, the RTAPI/ULAPI stems from prehistoric era when it differentiated between the kernel-space and user-space code. Given the kernel space HAL modules are pretty much dead and the only kernel space code currently in Machinekit is very much needed and in addition to user-space threads, it is now used to differentiate between real-time and non-realtime components. But only subset of this. Because as I understand it, you can have the non-realtime components by way of loadusr with it's forking, or you can have non-realtime components as RTAPI components running in the low priority, not-FIFO scheduled thread. Only the latter part would not be possible with the ULAPI flag. (Right?) So, what I don't understand is the whole ULAPI business with the flavouring part.

[cerna]

I decided that the easiest way how to accommodate Xenomai 3 and others is to allow multistage parsing of command line arguments. The current situation allows parsing only in the main function of rtapi_app executable with error on encountering unknown option. Then in mainloop() all parameters are deleted so only changed name of the process remains. I am going to change it so that there can be arbitrary parameters and the actual parsing can be done in multiple places as needed. At first, it looks little nuts, but it is all non-critical part of the code, so I am hoping it will be all right.

Given the choice of reading the /procFS or storing global variable, for rtapi_* using modules I think access though several globally accessible function is probably easiest.

And by moving the environ block elsewhere I was able on my test install to get free block of about 3510 chars. This can be dynamically used for exporting important information like --installed_flavor=preempt_rt which can be read by the ps command.

Anybody knows about reason why this is not good idea?

[cerna]

Somewhat lousy thing with this mode of operation is the fact, that because of the support for the 3.8 KERNEL version, I cannot use this for rtapi_app and msgd both. (More info - for those who want to investigate - is in the source code.)

However, this whole endeavour is done majorly for the rtapi_app benefit. The only problem is that the access to cmdline will have to be only through (yet not defined) get/set functions, otherwise it will crash and burn. (It's not unsolvable problem, but then it would take more time and I see no need for it at the moment.)

I am pushing the state export functionality in ps to the time I start seriously play with #230 as it is very much connected. (And even if FUSE is not good enough for real-time HAL portion, it should be good enough for non-realtime rtapi_app portion.)

I also discovered, that there used to be a /procFS export of information from rtapi, but it didn't survive @zultron's bloodbath. (Probably because it was RTAI specific, in LinuxCNC's rtai_rtapi_app it is still implemented.) Not sure why it eroded away. Anybody knows?