how to generate accuracy.txt when generated mlperf_log_accuracy.json with int4_offline

[XiaotaoChen]

I'm to reproduce nvidia int4 perf according guide docs

my envs

GeForce RTX 2080
cuda 10.0
trt 6.0.1.5
cudnn 7.6
ubuntu 16.04

performance

i produced the performanceOnly test as belows:

val_dir=/mnt/truenas/upload/xiaotao.chen/Repositories/mlperf_nvidia/open/NVIDIA/imagenet_val_npy/imagenet/ResNet50_int4
val_map_path=/mnt/truenas/upload/xiaotao.chen/Repositories/mlperf_nvidia/open/NVIDIA
mlperf_conf_path=/mnt/truenas/upload/xiaotao.chen/Repositories/mlperf_nvidia/open/NVIDIA/measurements/TitanRTXx4/resnet/Offline/mlperf.conf
user_conf_path=/mnt/truenas/upload/xiaotao.chen/Repositories/mlperf_nvidia/open/NVIDIA/measurements/TitanRTXx4/resnet/Offline/user.conf
batch_size=512
test_model=PerformanceOnly
# test_model=AccuracyOnly
/mnt/truenas/upload/xiaotao.chen/Repositories/mlperf_nvidia/open/NVIDIA/code/resnet/int4/int4_offline \
    -b ${batch_size} \
    -a autoconfig_rtx \
    --test-mode ${test_model} \
    --tensorPath ${val_dir} \
    --mapPath ${val_map_path}/data_maps/imagenet/val_map.txt \
    --lgls_logfile_outdir ./outputs \
    --mlperf_conf_path ${mlperf_conf_path} \
    --user_conf_path ${user_conf_path}

and its' perf seems align with the result as docs tells

rtx 2080, 11GB, cuda core:4352
bs=512 Samples per second: 21719.3

titan rtx, 24GB, cuda core: 4608
bs=1024, Samples per secod: 25178

accuracy

so i change test_mode to AccuracyOnly, there are four output files in outputs directory as belows:

mlperf_log_accuracy.json
mlperf_log_summary.txt
mlperf_log_detail.txt
mlperf_log_trace.json

And the contexts in mlperf_log_accuracy.json as belows. in performanceOnly mode it's null:

[
{ "seq_id" : 0, "qsl_idx" : 27228, "data" : "BA010000" },
{ "seq_id" : 1, "qsl_idx" : 48787, "data" : "BF030000" },
{ "seq_id" : 2, "qsl_idx" : 23135, "data" : "CE020000" },
{ "seq_id" : 3, "qsl_idx" : 701, "data" : "CE020000" },
...

I don't how to generate accuracy.txt in open/NVIDIA/results/TitanRTXx4/resnet/Offline/accuracy/accuracy.txt. Do you have any suggestions? @nvpohanh thanks.

[psyhtest]

@XiaotaoChen You need to use this script. See here how to run it.

You can obtain the labels file (val.txt) with Collective Knowledge as follows:

$ python -m pip install ck --user
$ ck pull repo:ck-env
$ ck install package --tags=image-classification,dataset,imagenet,aux
$ ck locate env --tags=image-classification,dataset,imagenet,aux
/home/dvdt/CK-TOOLS/dataset-imagenet-ilsvrc2012-aux
$ cd `ck locate env --tags=aux`
$ ls -la val.txt

[nvpohanh]

@XiaotaoChen If you look at the Makefile, you will find how we did this:

.PHONY: run_int4_TitanRTXx4_accuracy
run_int4_TitanRTXx4_accuracy: link_dataset_dir
	@mkdir -p $(LOG_DIR)/TitanRTXx4/resnet/Offline/accuracy
	@cd $(INT4_DIR) && ./int4_offline -b 1024 -a autoconfig_rtx --mlperf_conf_path $(PROJECT_ROOT)/measurements/TitanRTXx4/resnet/Offline/mlperf.conf --user_conf_path $(PROJECT_ROOT)/measurements/TitanRTXx4/resnet/Offline/user.conf --test-mode AccuracyOnly --lgls_logfile_outdir $(LOG_DIR)/TitanRTXx4/resnet/Offline/accuracy
	@python3 $(INFERENCE_DIR)/v0.5/classification_and_detection/tools/accuracy-imagenet.py --mlperf-accuracy-file $(LOG_DIR)/TitanRTXx4/resnet/Offline/accuracy/mlperf_log_accuracy.json --imagenet-val-file data_maps/imagenet/val_map.txt --dtype int32 2>&1 | tee $(LOG_DIR)/TitanRTXx4/resnet/Offline/accuracy/accuracy.txt

[XiaotaoChen]

Thanks for your clear guidance, I'll try. @psyhtest @nvpohanh