awesome-ml-demos-with-ios icon indicating copy to clipboard operation
awesome-ml-demos-with-ios copied to clipboard

Published 20 hours ago • verified publisher icon tucan9389

Add new apple's model

Open tucan9389 opened this issue 4 years ago • 9 comments

Models

Vision

Model Name Problem Demo Project Performance Test Demo Gif Added in here
MobileNetV2 Image Classification Done Done Done TODO
Resnet50 Image Classification Done Done TODO TODO
SqueezeNet Image Classification Done Done TODO TODO
MNIST Drawing Classification TODO TODO TODO TODO
YOLOv3-Tiny Object Detection Done Done TODO TODO
YOLOv3 Object Detection) Done Done TODO TODO
DeeplabV3 Image Segmentation Done Done Done TODO
FCRN-DepthPrediction Depth Estimation Done Done Done TODO

Text

Model Name Problem Demo Project Performance Test Demo Gif Added in here
BERT-SQuAD Question Answering TODO TODO TODO TODO

Related Links

tucan9389 avatar Jul 18 '19 00:07 tucan9389

Anyone could contribute! Thanks. 😎

tucan9389 avatar Jul 18 '19 00:07 tucan9389

FCRN-DepthPrediction Demo

https://github.com/tucan9389/DepthPrediction-CoreML

tucan9389 avatar Jul 27 '19 04:07 tucan9389

DeeplabV3 Demo

https://github.com/tucan9389/ObjectSegmentation-CoreML

tucan9389 avatar Jul 27 '19 04:07 tucan9389

MobileNetV2, Resnet50, SqueezeNet Demo

https://github.com/tucan9389/ImageClassification-CoreML/

tucan9389 avatar Jul 27 '19 04:07 tucan9389

YOLOv3, YOLOv3-Tiny Demo

https://github.com/tucan9389/ObjectDetection-CoreML

tucan9389 avatar Jul 27 '19 04:07 tucan9389

Image Classification

Model Size (MB), Minimum iOS Version

Model Size Minimum
iOS Version
MobileNet 17.1 iOS11
MobileNetV2 24.7 iOS11
MobileNetV2FP16 12.4 iOS11.2
MobileNetV2Int8LUT 6.3 iOS12
Resnet50 102.6 iOS11
Resnet50FP16 51.3 iOS11.2
Resnet50Int8LUT 25.7 iOS12
Resnet50Headless 94.4 iOS11
SqueezeNet 5 iOS11
SqueezeNetFP16 2.5 iOS11.2
SqueezeNetInt8LUT 1.3 iOS12

Infernece Time (ms)

Model vs. Device XS XS
Max		 XR X 7+ 7
MobileNet 16 18 19 33 43 35
MobileNetV2 21 18 21 46 64 53
MobileNetV2FP16 20 19 20 48 65 57
MobileNetV2Int8LUT 21 21 20 53 64 53
Resnet50 27 25 26 61 78 63
Resnet50FP16 26 26 27 64 75 74
Resnet50Int8LUT 27 25 26 60 77 75
Resnet50Headless 18 13 18 36 54 53
SqueezeNet 17 17 18 24 35 29
SqueezeNetFP16 17 17 18 24 36 29
SqueezeNetInt8LUT 18 19 18 27 34 30

Total Time (ms)

Model vs. Device XS XS
Max		 XR X 7+ 7
MobileNet 18 20 21 35 46 37
MobileNetV2 23 21 23 48 67 55
MobileNetV2FP16 24 21 23 50 69 60
MobileNetV2Int8LUT 23 23 22 55 67 56
Resnet50 30 28 29 64 82 66
Resnet50FP16 28 28 30 66 78 76
Resnet50Int8LUT 29 28 28 63 80 78
Resnet50Headless 19 13 18 36 54 54
SqueezeNet 18 18 20 25 37 31
SqueezeNetFP16 18 18 19 26 38 31
SqueezeNetInt8LUT 20 20 19 29 37 32

FPS

Model vs. Device XS XS
Max		 XR X 7+ 7
MobileNet 23 23 23 23 20 23
MobileNetV2 23 23 23 20 13 17
MobileNetV2FP16 23 23 23 18 13 15
MobileNetV2Int8LUT 23 23 23 16 13 16
Resnet50 23 23 23 14 11 14
Resnet50FP16 23 23 23 14 11 12
Resnet50Int8LUT 23 23 23 15 11 12
Resnet50Headless 23 23 23 23 16 17
SqueezeNet 23 23 23 23 23 23
SqueezeNetFP16 23 23 23 23 22 23
SqueezeNetInt8LUT 23 23 23 23 23 23

tucan9389 avatar Jul 27 '19 04:07 tucan9389

Object Detection

Model Size (MB), Minimum iOS Version

Model Size Minimum
iOS Version
YOLOv3 248.4 iOS12
YOLOv3FP16 124.2 iOS12
YOLOv3Int8LUT 62.2 iOS12
YOLOv3Tiny 35.5 iOS12
YOLOv3TinyFP16 17.8 iOS12
YOLOv3TinyInt8LUT 8.9 iOS12
MobileNetV2_SSDLite 9.3 iOS12
ObjectDetector 63.7 iOS12

Infernece Time (ms)

Model vs. Device XS XS
Max		 XR X 7+ 7
YOLOv3 108 93 100 356 569 561
YOLOv3FP16 104 89 101 348 572 565
YOLOv3Int8LUT 101 92 100 337 575 572
YOLOv3Tiny 46 41 47 106 165 168
YOLOv3TinyFP16 51 41 44 103 165 167
YOLOv3TinyInt8LUT 45 39 39 106 160 161
MobileNetV2_SSDLite 31 31 31 109 141 134
ObjectDetector 24 26 23 63 86 84

Total Time (ms)

Model vs. Device XS XS
Max		 XR X 7+ 7
YOLOv3 108 93 100 357 569 561
YOLOv3FP16 104 89 101 348 572 565
YOLOv3Int8LUT 102 92 102 338 576 573
YOLOv3Tiny 46 42 48 106 166 169
YOLOv3TinyFP16 51 41 44 104 165 167
YOLOv3TinyInt8LUT 45 39 40 107 160 161
MobileNetV2_SSDLite 32 31 32 109 142 134
ObjectDetector 25 26 23 64 87 85

FPS

Model vs. Device XS XS
Max		 XR X 7+ 7
YOLOv3 8 10 9 2 1 1
YOLOv3FP16 9 10 8 2 1 1
YOLOv3Int8LUT 9 10 9 2 1 1
YOLOv3Tiny 21 22 20 8 5 5
YOLOv3TinyFP16 19 23 21 9 5 5
YOLOv3TinyInt8LUT 21 24 23 8 5 5
MobileNetV2_SSDLite 23 23 23 8 6 6
ObjectDetector 23 23 24 14 10 11

tucan9389 avatar Jul 27 '19 05:07 tucan9389

Table Convert Script

Convert text to markdown table format



dict = {}
device_name = None
device_names = []

f = open("performance-classification.txt", 'r')
lines = f.readlines()
for line in lines:
    componants = line.split()
    # if len(componants) == 0:

    if len(componants) == 0:
        continue

    if len(componants) != 5:
        device_name = " ".join(componants)
        device_names.append(device_name)
        continue
    else:
        model_name = componants[0]
        inference_time = componants[2]
        total_time = componants[3]
        fps = componants[4]

        if model_name not in dict:
            dict[model_name] = {}

        dict[model_name][device_name] = {
            "inference_time": inference_time,
            "total_time": total_time,
            "fps": fps
        }
f.close()




def get_markdown_table(dict, colunm_name = "inference_time"):
    result = ""

    row_text = "| Model vs. Device | "
    row_text = row_text + " | ".join(device_names)
    row_text = row_text + " |"
    result += row_text + "\n"
    # print(row_text)

    row_text = "| ----: | "
    for i in range(len(device_names)):
        row_text += ":----: | "
    result += row_text + "\n"
    # print(row_text)

    for model_name in dict.keys():
        row_text = "| " + model_name + " | "
        for device_name in device_names:
            row_text += dict[model_name][device_name][colunm_name] + " | "
        result += row_text + "\n"

    return result

# inference_time
# total_time
# fps

print()
print()
print()

print("## Object Detection")
print()

print("### Infernece Time (ms)")
print()
print(get_markdown_table(dict=dict, colunm_name="inference_time"))

print("### Total Time (ms)")
print()
print(get_markdown_table(dict=dict, colunm_name="total_time"))

print("### FPS")
print()
print(get_markdown_table(dict=dict, colunm_name="fps"))

Input text file

XS Max
MobileNet           : 18 20 23
MobileNetV2         : 18 21 23
MobileNetV2FP16     : 19 21 23
MobileNetV2Int8LUT  : 21 23 23
Resnet50            : 25 28 23
Resnet50FP16        : 26 28 23
Resnet50Int8LUT     : 25 28 23
Resnet50Headless    : 13 13 23
SqueezeNet          : 17 18 23
SqueezeNetFP16      : 17 18 23
SqueezeNetInt8LUT   : 19 20 23

XR
MobileNet           : 19 21 23
MobileNetV2         : 21 23 23
MobileNetV2FP16     : 20 23 23
MobileNetV2Int8LUT  : 20 22 23
Resnet50            : 26 29 23
Resnet50FP16        : 27 30 23
Resnet50Int8LUT     : 26 28 23
Resnet50Headless    : 18 18 23
SqueezeNet          : 18 20 23
SqueezeNetFP16      : 18 19 23
SqueezeNetInt8LUT   : 18 19 23

7+
MobileNet           : 43 46 20
MobileNetV2         : 64 67 13
MobileNetV2FP16     : 65 69 13
MobileNetV2Int8LUT  : 64 67 13
Resnet50            : 78 82 11
Resnet50FP16        : 75 78 11
Resnet50Int8LUT     : 77 80 11
Resnet50Headless    : 54 54 16
SqueezeNet          : 35 37 23
SqueezeNetFP16      : 36 38 22
SqueezeNetInt8LUT   : 34 37 23

7
MobileNet           : 35 37 23
MobileNetV2         : 53 55 17
MobileNetV2FP16     : 57 60 15
MobileNetV2Int8LUT  : 53 56 16
Resnet50            : 63 66 14
Resnet50FP16        : 74 76 12
Resnet50Int8LUT     : 75 78 12
Resnet50Headless    : 53 54 17
SqueezeNet          : 29 31 23
SqueezeNetFP16      : 29 31 23
SqueezeNetInt8LUT   : 30 32 23

Output text




## Object Detection

### Infernece Time (ms)

| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- | 
| MobileNet | 18 | 19 | 43 | 35 | 
| MobileNetV2 | 18 | 21 | 64 | 53 | 
| MobileNetV2FP16 | 19 | 20 | 65 | 57 | 
| MobileNetV2Int8LUT | 21 | 20 | 64 | 53 | 
| Resnet50 | 25 | 26 | 78 | 63 | 
| Resnet50FP16 | 26 | 27 | 75 | 74 | 
| Resnet50Int8LUT | 25 | 26 | 77 | 75 | 
| Resnet50Headless | 13 | 18 | 54 | 53 | 
| SqueezeNet | 17 | 18 | 35 | 29 | 
| SqueezeNetFP16 | 17 | 18 | 36 | 29 | 
| SqueezeNetInt8LUT | 19 | 18 | 34 | 30 | 

### Total Time (ms)

| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- | 
| MobileNet | 20 | 21 | 46 | 37 | 
| MobileNetV2 | 21 | 23 | 67 | 55 | 
| MobileNetV2FP16 | 21 | 23 | 69 | 60 | 
| MobileNetV2Int8LUT | 23 | 22 | 67 | 56 | 
| Resnet50 | 28 | 29 | 82 | 66 | 
| Resnet50FP16 | 28 | 30 | 78 | 76 | 
| Resnet50Int8LUT | 28 | 28 | 80 | 78 | 
| Resnet50Headless | 13 | 18 | 54 | 54 | 
| SqueezeNet | 18 | 20 | 37 | 31 | 
| SqueezeNetFP16 | 18 | 19 | 38 | 31 | 
| SqueezeNetInt8LUT | 20 | 19 | 37 | 32 | 

### FPS

| Model vs. Device | XS Max | XR | 7+ | 7 |
| ---- | ---- | ---- | ---- | ---- | 
| MobileNet | 23 | 23 | 20 | 23 | 
| MobileNetV2 | 23 | 23 | 13 | 17 | 
| MobileNetV2FP16 | 23 | 23 | 13 | 15 | 
| MobileNetV2Int8LUT | 23 | 23 | 13 | 16 | 
| Resnet50 | 23 | 23 | 11 | 14 | 
| Resnet50FP16 | 23 | 23 | 11 | 12 | 
| Resnet50Int8LUT | 23 | 23 | 11 | 12 | 
| Resnet50Headless | 23 | 23 | 16 | 17 | 
| SqueezeNet | 23 | 23 | 23 | 23 | 
| SqueezeNetFP16 | 23 | 23 | 22 | 23 | 
| SqueezeNetInt8LUT | 23 | 23 | 23 | 23 |

tucan9389 avatar Jul 27 '19 06:07 tucan9389

The results are weird. Maximum FPS is 23.. (Not 30) Need to test again. 😓

tucan9389 avatar Jul 30 '19 09:07 tucan9389