[one-cmds] Provide visualization facility for memory allocation info

[ai-moiseev]

What?

There's a need for visualization utility for model developers, as stated in ONE-vscode issue (Memory measurement in compilation environment and visualization of measurement result). This feature requires support in ONE and codegen backends. It also requires unified documented format for backends to emit.

Why?

There's need in for model development and deployment visualization utilities (with ONE-vscode plugin support) for allocation tracking in memory-constrained environments.

Implementation notes

Data format draft

There's already CSV-based format is in use internally, but it was discussed that for more generic usage it would make more sense to use JSON-based format containing all requiring metadata and versioning info.

Proposed allocation data file example:

{
  "schema_version": 1,
  "model_name": "model.circle",
  "codegen_ts": 1648028141,
  
  "memory_segments": [
    {
      "segment": "SEGMENT1",
      "memory_allocations": [
        { "front_addr": 0, "back_addr": 256, "alive_from": 5, "alive_till": 120, "origin": "Node1" }
      ]
    }
  ]
}

Visualization utility

Proposed solution is to introduce one-cmd utility for visualization tasks (e.g. one-visualizer) with dependency on matplotlib and ability to convert memory allocation info JSON files into SVGs and JPEGs. It can be used not only for memory allocations, but for other visualization tasks as well in the future.

Links

• ONE-vscode Performance & Memory Management issue
• ONE-vscode UX support issue

[seanshpark]

Q) why does this tool should be developed in ONE, not inONE-vscode ?

[ai-moiseev]

@seanshpark This tool already existed and was in active use in one of backends as a CLI utility. The effort here is to provide cross-backend variant of this utility which has its own value and which can be used even without VSCode plugin (and after that the aim is to provide a GUI for it in VSCode...)

[seanshpark]

I actully don't understand what you are going to do. Can you please describe what you want to do? add a new tool? what does that tool do? are you going to modify exist codes? etc...

[ai-moiseev]

@seanshpark There's a memory measurement and allocation tracking utility which is currently used in private backend. This tool takes a CSV file with memory allocation info and creates SVG (using matplotlib) which shows memory blocks on a cycle timeline.

The idea is to make this utility a part of ONE's toolchain (by introducing a new tool, e.g. one-visualizer) so it can be used with other backends and also can be used in ONE-vscode (as part of the effort to provide performance & memory profiling tools).

Current idea is to share code with exsiting backend utility mostly, but provide a more flexible input format.

[seanshpark]

I assume this tool( one-visualizer ) is not related with ONE (frontend and runtime). I understand integration with ONE's toolchain may help something ( but particullay what? in view point of ONE?) But the tool itself is not used nor can be tested (or can it? I need this point) For this point I have little confidence of introduceing the tool in ONE.

I am not talking about the usability of this tool. I am talking about why this tool be in ONE.

[ai-moiseev]

@seanshpark I'm not entirely sure which place is right for this utility, but I'm afraid there's will be a duplication of effort if this tool will be maintained separately both in ONE-vscode and in backend (as there's a practical need for this tool in both places...), so this is my point for introduction of it in ONE (maybe as some optional component?)

Regarding testing, this tool can be unit tested by validating its input parsing capability and integrity-tested by comparing sample input files with reference outputs.

[seanshpark]

@ai-moiseev , I assume this tool already exist in backend and you want to utilize it also in ONE-vscode. If you move it to ONE-vscode, users cannot use the tool unless ONE-vscode is installed. (as currently the backend itself cannot be used alone, ONE must be installed)

So if you move the tool to ONE then user can use the tool with or without ONE-vscode. Am I correct?

Although this situation may be the case, (1) I assume you can/want to open the codes of the tool, but are you sure? no legal problems? (2) there maybe other similar cases like this. are we going to move thoses to ONE too?

[seanshpark]

Regarding testing, this tool can be unit tested by validating its input parsing capability and integrity-tested by comparing sample input files with reference outputs.

I'm not sure about this.

[ai-moiseev]

So if you move the tool to ONE then user can use the tool with or without ONE-vscode. Am I correct?

That's right, the idea is to keep ability of users to continue usage of this CLI utility separately of ONE-vscode.

Although this situation may be the case, (1) I assume you can/want to open the codes of the tool, but are you sure? no legal problems?

This is a very valid concern, I need to ask about this in repository directly. I'll ask and mention you in backend discussion.

(2) there maybe other similar cases like this. are we going to move thoses to ONE too?

AFAIK, in other cases exported profiling info is self-sufficient and doesn't require an additional transformation steps. But I may be mistaken here...

[hyunsik-yoon]

@ai-moiseev

IMHO there are a few thing to consider.

The idea is to make this utility a part of ONE's toolchain (by introducing a new tool, e.g. one-visualizer) so it can be used with other backends and also can be used in ONE-vscode (as part of the effort to provide performance & memory profiling tools).

I am not sure whether a memory tool for one backend can be used with other backends. If the tool is limited to one type of backend, maybe providing such info directly to ONE-vscode would be better.

One way around is

1. for now, how about providing memory info to ONE-vscode?
2. later, if it become certain that the tool can be used by more backends, we can start discussing to introduce such feature into ONE again.

How about this approach?

[ai-moiseev]

@hyunsik-yoon At first I though exactly of this approach :) In backend discussion it was stated that export of memory info outside of backend requires documented and extensible data format and I thought it would be a proper way to introduce this format in ONE.

But I think we can keep it backend-specific for now and implement separate ONE-vscode graphic viewer for it. As a bonus, in that case it doesn't have to be a static SVG image, but can be an interactive tool with cursor hover tooltips, convenient scaling/scrolling etc...

I'll focus on providing memory info from backend to ONE-vscode for now.

[lemmaa]

In backend discussion it was stated that export of memory info outside of backend requires documented and extensible data format and I thought it would be a proper way to introduce this format in ONE.

+1

IIUC, @ai-moiseev is trying to do is to create a framework that can visualize the information collected from multiple backends by storing it in a standard format. And that it can be used seamlessly and selectively on the one-cmds workflow. Also, in ONE-vscode, we can make an extension by referring only to this framework.

This method is expected to sufficiently modularize all components, but is there any problem that I am not expecting?

[hyunsik-yoon]

@lemmaa

I write the following,

One way around is

1. for now, how about providing memory info to ONE-vscode? later, if it become certain that the tool can be used by more backends, we can start discussing to introduce such feature into ONE again.

I think it's chicken and egg problem. If we can define a API or format nicely fit into multiple backends, then I guess we could consider to introducing it to ONE. Question is if it is too early or right time to define the format. IMHO the needs of at least two backends should be compared and then the format can be defined.

If 'we can define it now', then @seanshpark 's concern at here may need to be considered.

I also have personal question about the scope of ONE.

Is exposing common features of backends in ONE's scope? 'feature' could be memory consumption, profile result, golden test, etc. Currently, there is one-profile which calls backend profiler. Because we have one-profile, I guess more features could be embraced by ONE such as one-measure-memory or one-value-test, etc.. I wonder whether I think correctly.

[lemmaa]

@hyunsik-yoon ,

If we can define a API or format nicely fit into multiple backends, then I guess we could consider to introducing it to ONE.

Exactly what I mean is this.

Question is if it is too early or right time to define the format. IMHO the needs of at least two backends should be compared and then the format can be defined. so, it seems to be a matter of choice. :)

I thought we could start based on our experience so far. But I also agree with your opinion.

If 'we can define it now', then @seanshpark 's concern at here may need to be considered.

It is not to move the currently developed backend tool itself, but to create a new one here that can be supported including the current backend tool. I don't think there is any problem to work here for the common part like this.

I also have personal question about the scope of ONE.

Is exposing common features of backends in ONE's scope? 'feature' could be memory consumption, profile result, golden test, etc. Currently, there is one-profile which calls backend profiler. Because we have one-profile, I guess more features could be embraced by ONE such as one-measure-memory or one-value-test, etc.. I wonder whether I think correctly.

I think so.

ONE has to show the entire developer workflow end-to-end. Some of these steps may require invocation of the backend tool. Also, I think that ONE-vscode is basically the role of improving the UX of users based on the functions of ONE and the data it generates. In addition, it is important to play a role in connecting with other worlds beyond the scope of the toolchain.

[hyunsik-yoon]

For general relation between ONE, ONE-vscode and backends, I created https://github.com/Samsung/ONE/issues/8743.

[binarman]

@ai-moiseev @SlavikMIPT @BalyshevArtem I think we can use this feature as a visualizer for circle-execution-plan tool. What do you think?