Sub-graph's YAML-Based definition and plugin-like extension

[drslebedev]

Enable the creation of new block functionalities by combining existing basic blocks (typically C++-based) within the same or other top-level (sub)graphs.

Proposed Actions:

1. Dedicated YAML-Based Nested Graph Definitions

   • Implement dedicated YAML files for defining nested graphs.
   • Store these YAML definitions in the block registry for reuse across multiple graphs.
   • Leverage the grc-loader and its YAML-parsing capabilities.

2. Plugin-Like Extensions for User-Defined Nested Graphs

   • Introduce a plugin-style extension allowing users to browse and load nested graph definitions from user-specified directories.
   • This enhances flexibility and modularity in defining and reusing nested graphs.

3. Plugin-Like Extensions for External Graph Sources

   • Extend the plugin architecture to support external sources beyond local files.
   • Include dynamic querying of graph definitions from sources such as databases or community-driven repositories via HTTP(S).
   • Enable nested graph YAML definitions to be fetched and loaded dynamically from external websites or HTTPS-based repositories.

Message-Based API for sub-graph management

Add support a message-based API for creation, duplication and editing sub-graphs.

Key Considerations for Messaging API Design:

• Minimalism: The API should include only the essential methods necessary for sub-graph management. Avoid introducing additional methods or functionalities unless they are absolutely required.
• Consistency: Ensure the API remains consistent and easy to use across different components of the system.
• Avoid UI-Specific API: Features that can be implemented at the UI level should not drive changes to the message API. New APIs should only be added if they are essential for core functionality and not just for UI convenience.
• Maintenance Overhead: Keep in mind that once an API is introduced, it is hard to remove or refactor without breaking backward compatibility. Minimizing the API surface area reduces long-term support and maintenance burdens.

Sub-Graph Modifications: All Instances vs. Single Instance

Sub-graphs are essentially templates that abstract common graph topologies into a reusable form. When a sub-graph is modified, all instances of that sub-graph should automatically reload and reflect the updated version.

Key Behavior:

1. Modifying All Instances:

   • Changes made to the sub-graph template will propagate to all instances of that sub-graph. It is considered acceptable that runtime modifications may cause a loss of state information e.g. buffer contents and/or block settings returning to their defaults if needed.

2. Modifying a Single Instance:

   • To modify only one instance of a sub-graph, the user can ungroup the sub-graph.
   • Once ungrouped, the connection to the original sub-graph template is removed, and the instance becomes independent.

3. Re-grouping Modified Sub-Graphs:

   • If the user groups the modified sub-graph again, a new sub-graph template is created based on the changes.

This approach ensures flexibility by allowing both global modifications (sub-graph template) and instance-specific changes.

This is a follow-up of the following issues/PRs:

• https://github.com/fair-acc/gnuradio4/pull/479
• https://github.com/fair-acc/opendigitizer/issues/184

[RalphSteinhagen]

see also: https://github.com/fair-acc/gnuradio4/issues/487

[ivan-cukic]

3SP so far

• https://github.com/fair-acc/gnuradio4/commit/47f3bef0ab19bea17ce569568f755c1eadc697e1
• https://github.com/fair-acc/gnuradio4/pull/602

[ivan-cukic]

0.5SP for schedulerRefactoring branch issues investigation

[iamsergio]

1SP: Rebasing pr #650 over ralph's refactorings. conflicts impossible to solve. Rolled back our changes and rewrote them on top of ralph's. Investigated tests crashing/regressions.

[iamsergio]

1SP:

• debugged reason of scheduler.exchange crash
• wrote 2 minimal unit-tests
• implemented move-assign ctors, investigated workarounds for const fields
• fixed crashes due to Block move-ctor assigning calling base class move twice
• debugged more crashes in qa_schedulerMessages caused by move-assign fixes, and fixed regressions
• fixed dtors of moved-from objects

[iamsergio]

2SP lifecycle:

• lifecycle propagation from root scheduler to nested scheduler
• fixed root scheduler being blocked due to nested schedulers blocking in start()
• Wrote tests and partially rewrote Ivan's qa_ManagedSubGraph
  • simplified by porting to testing::sendAndWaitForReply
  • tested less popular lifecycle states like paused and resuming
  • Debugged test failures regarding connecting ports of sub-scheduler (Ivan's wip had edge.at("destinationBlock") instead of edge.at("destination_block"))
• Fixed gracefully stopping scheduler with nested schedulers (sub-scheduler needs to join its thread after stopping)
• Worked-around Scheduler::init() behaving different than other lifecycle methods: it does not call it's children's init(). Means sub-scheduler needs to do it manually when starting (just like the root)
• Workaround gr::forEachBlock<TransparentBlockGroup> bug. This actually be patched and workaround removed.
• Fixed crash due to Graph's move operator not moving the plugin loader as well
• Fixed crash where plugin loader wasn't initialized in one of Graph's constructor in Ivan's initial WIP

TODO:

• Fix gr::forEachBlock<TransparentBlockGroup> and remove workaround, and add unit-tests for different traversal options
• Passing property_map into nested scheduler. I haven't tested or know if it's really needed (review)