How to Execute nnpackage with multiple models ?

[chunseoklee]

Current process : nnpkg with 1 model

• Load : 1 Model -> 1 Subgraphs(= n Graphs)
• Compile : Compiler has 1 _subgraphs to be compiled into n executor. 1 Executor for 1 Graph.

class Compiler
{
  Compiler(const std::shared_ptr<ir::Subgraphs> &subgs, util::TracingCtx *tracing_ctx);

  std::shared_ptr<exec::ExecutorMap> compile(void);

  std::shared_ptr<ir::Graph> &primary_subgraph() { return _subgraphs->at(ir::SubgraphIndex{0}); }

  std::shared_ptr<ir::Subgraphs> _subgraphs;
}

std::shared_ptr<exec::ExecutorMap> Compiler::compile()
{
  executors = std::make_shared<exec::ExecutorMap>();
  for (auto &pair : lowered_subgs)
  {
    const auto &subg_index = pair.first;
    auto &lowered_subg = pair.second;
    auto indexed_ranks = lowered_subg->indexed_ranks();

    ir::OperationDumper dumper("Executor generation of Subgraph " +
                               std::to_string(subg_index.value()));
    lowered_subg->graph().operations().iterate(
      [&](const ir::OperationIndex &, const ir::Operation &op) { op.accept(dumper); });
    auto executor = std::unique_ptr<exec::IExecutor>{
      ExecutorFactory::get().create(std::move(lowered_subg), _options, executors)};
    executor->setIndexedRanks(indexed_ranks);
    executors->insert(std::make_pair(subg_index, std::move(executor)));
  }
}


class Execution
{

 Execution(const std::shared_ptr<ExecutorMap> &executors);

 const ir::Graph &primary_subgraph() const { return primary_executor()->graph(); }
}

• Execution : No execution order. Execution starts with primary(0th) executor. Execution order is automatically resolved by While/If Kernels(using Subgraphs/Executor index)

void Execution::execute()
{
  VERBOSE(Execution) << "Start execution" << std::endl;

  primary_executor()->execute(_io_desc);
  finished = true;

  VERBOSE(Execution) << "Execution finished" << std::endl;
}

• No subgraph if no While/If op

To-be : nnpkg with multiple models

• Load : n models -> n Subgraphs(Let M be the # of total Graphs in n models)
• Compile :
  • how to compile n models(= n Subgraphs) ?
• Execution :
  • 1 Execution or n Executions or M Executions ?
  • How to generate execution order and Who take care of it ?

[chunseoklee]

As commented in https://github.com/Samsung/ONE/pull/9244#issuecomment-1154736748 about Frontend(Loader, IR ...), Compiler, Executor and Execution has similar issues like strong assumption and SubGraph(or ExecutorMap) dependency. Also, several flags are defined globally : e.g. compilationOptions.

I will update a draft PR(https://github.com/Samsung/ONE/pull/9266) by reflect #9244 's changes

[chunseoklee]

on 38988042a

1. New constructor for Compiler takes ModelGraph instead of Subgraphs

https://github.com/Samsung/ONE/blob/38988042a51d1855b08033079b85afd484ee6faa/runtime/onert/api/src/nnfw_api_internal.cc#L394-L395

2. New constructor of Execution takes compile()'s output

https://github.com/Samsung/ONE/blob/38988042a51d1855b08033079b85afd484ee6faa/runtime/onert/api/src/nnfw_api_internal.cc#L423-L425

[chunseoklee]

Requirement for multiple models

• Compiler takes ModelGraph as input
• Execution takes Compilation Result of the ModelGraph
• Execution generates "execution order" among Compilation Results
• Execution provides a way to setup ModelGraph's input / output(e.g io_desc)