[TVMC] Allow selecting a subset of tasks to be used in `tvmc tune`

[PhilippvK]

This adds a --tasks flag to the tvmc tune command to filter the lists of tasks to be tuned. See examples below.

Motivation

• As auto-tuning can be quite time consuming, it is often desirable to cut down the number of tuned tasks in a session.
• If the tuning session was canceled halfway through, it would be a bad idea to start from scratch. Instead continue with the last untuned task
• Some tasks have more impact on the model performance than others, thus we should be able to train some tasks longer than others

Examples

1. Use --task list to show which tasks are available for tuning

$ tvmc tune toycar.tflite -o out.txt --task list
Available Tasks for tuning:
  0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int...
  1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), 'int16...
  2. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int...
  3. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (128, 128), 'int16...
  4. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16...
  5. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (8, 128), 'int16')...
  6. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16')...
  7. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 8), 'int16'), ('TENSOR', (128, 8), 'int16'), ...
  8. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int...
  9. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 128), 'int16'), ('TENSOR', (640, 128), 'int16...

2. Filter the list of tasks to be tuned:

# Only tune a single task (index 5)
tvmc tune toycar.tflite -o out.txt --tasks 5

# Tunes tasks starting with index 6
tvmc tune toycar.tflite -o out.txt --tasks "6-"

# Tune tasks 1,4,5,6,8,9
tvmc tune toycar.tflite -o out.txt --tasks "1,4-6,8-"

Tests

I added a basic unit test for the filter_tasks utility in tests/python/driver/tvmc/test_autotuner.py.

Open Questions

• ~~While the (truncated) string representations of AutoTVM tasks are quite helpful to pick the correct tasks, using AutoScheduler the tasks can not really be distinguished from each other (only by index). Is there a way to get similar information from AutoScheduler tasks?~~

cc @Mousius @gromero

[PhilippvK]

CC @leandron @giuseros @areusch @tqchen

[areusch]

@PhilippvK i think this one needs a rebase

[PhilippvK]

Short update:

• With my latest changes I can now print useful task names for AutoScheduler as well (using task.desc instead of str(task))
• I added a mention of the --task argument in the TVMC CLI tutorial

[PhilippvK]

@areusch

As I have to retrigger the CI anyway so I want to propose another small addition:

For AutoTVM, in addition to the task description, I could also print len(task.config_space) to give an idea on how large the tuning search space is for a specific task, which can help a lot to estimate the required number of tuning trials.

Example:

Available Tasks for tuning:
  0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640)... (len=640)
  1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), ... (len=35)

Do you think this would be a good addition to tvmc tune --task list?

[PhilippvK]

@areusch Can I have an update on this?

[leandron]

@tvm-bot rerun

[areusch]

sorry @PhilippvK I missed this one. feel free to ping me on discord if that happens again. I retriggered the CI, I'm good for @leandron to merge this if it passes.

Example:

Available Tasks for tuning: 0. Task(func_name=dense_nopack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640)... (len=640)

1. Task(func_name=dense_pack.x86, args=(('TENSOR', (1, 640), 'int16'), ('TENSOR', (128, 640), ... (len=35) Do you think this would be a good addition to tvmc tune --task list?

Sure that seems like a good addition!

[tvm-bot]

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

• Built docs for commit 1e7d1210f498ea266dd9c60b5b60b106c1bd82cc can be found here.

_{Generated by tvm-bot}

[PhilippvK]

Rebased and CI should be fixed now @leandron @areusch.