edgetpu-yolo
edgetpu-yolo copied to clipboard
jveitchmichaelis
Unable to interpret prediction values by Ultralytics Yolov8 tflite edgetpu model inference on Coral USB Accelerator
Hi man, I have exported my yolov8 model to edge tpu model using ultralytics. I'm able to perform the inference using USB coral accelerator. But I'm not sure how to handle the predictions. I'm sharing the input and output details of my interpreter.
Model : yolov8_int8_edgetpu.tflite
Input_details: [{'name': 'serving_default_images:0', 'index': 0, 'shape': array([ 1, 1280, 1280, 3], dtype=int32), 'shape_signature': array([ 1, 1280, 1280, 3], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
Output_details: [{'name': 'PartitionedCall:0', 'index': 517, 'shape': array([ 1, 251, 33600], dtype=int32), 'shape_signature': array([ 1, 251, 33600], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
INFERENCE_LOG: python3 detect_image.py -m yolov8_int8_edgetpu.tflite -i image.jpg -o inference/detection.jpg
Note: The first inference is slow because it includes loading the model into Edge TPU memory. ----INFERENCE TIME---- 21368.82 ms
Output Details : [{'name': 'PartitionedCall:0', 'index': 517, 'shape': array([ 1, 251, 33600], dtype=int32), 'shape_signature': array([ 1, 251, 33600], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}] Output Data Shape: (1, 251, 33600) Output Data : [[[6.8984828e-03 8.4312847e-03 1.4987010e-02 ... 9.1422123e-01 9.3176806e-01 9.5353013e-01] [8.2258508e-03 8.4541356e-03 8.1687719e-03 ... 9.7039759e-01 9.6757478e-01 9.6137410e-01] [1.3865395e-02 1.7534753e-02 1.8626634e-02 ... 2.2020416e-01 1.8798628e-01 1.2658052e-01] ... [8.6820194e-07 1.4569782e-06 1.3537380e-06 ... 4.9203813e-05 3.5585672e-05 3.4364115e-05] [1.6130493e-06 1.2267126e-06 2.5768195e-06 ... 4.6514473e-05 4.1418200e-05 3.8001206e-05] [4.9328037e-07 1.4711668e-06 1.1877161e-06 ... 3.2545999e-05 3.1352149e-05 2.2766470e-05]]]
CONFUSION: I'm not sure how to handle the output tensor which has shape [ 1, 251, 33600]. Please guide I'm stuck here.
Have you tried the example code in this repository? Please try the detect.py script here and pass in your model + label file as arguments - you can use -h to see help.
There are several steps you need to perform to convert the output but generally the format is:
You get one prediction vector per anchor point. You need to first rescale your result from the quantized values (int > float) and then perform softmax and non-maximum suppression.
On Fri, 25 Aug 2023 at 14:15, Arslan Mehmood @.***> wrote:
I'm using PyCoral api default procedure.
— Reply to this email directly, view it on GitHub https://github.com/jveitchmichaelis/edgetpu-yolo/issues/11#issuecomment-1693346604, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYDMJ4U5P4IVDMLNX6SRWLXXCQP7ANCNFSM6AAAAAA36PZ52E . You are receiving this because you commented.Message ID: @.***>
Also I havent followed anything in your repo. I have followed the Pycoral basic detection workflow and used the 'detect.py'. One thing I dont get is that the output tensor length is 33600. How will it get mapped to [coordinates x1,x2,x3,x4,probability] (5-values).
Each row in the tensor is an object prediction. The first four valued are xy coordinates representing the box. The remaining values are class probabilities. Almost all of them should be zero and you need to run NMS to filter the ones that work.
4 + 247 = 251 columns
Please try using detect.py - it does all the post processing.
Though yolov8 is anchor free I believe. If you're not sure what the output tensor should be then you should post an issue on ultralytics' repo
Also see https://github.com/ultralytics/ultralytics/issues/751
It's possible the order of the values is changed in v8 but I doubt it.
Also just a note that the detect.py from the coral team assumes you're using an SSD model (eg the examples from Google), so I'm not sure it will work unless the output shapes/interpretations are comparable.
On Fri, 25 Aug 2023 at 14:28, Arslan Mehmood @.***> wrote:
Also I havent followed anything in your repo. I have followed the Pycoral basic detection workflow and used the detect.py. One thing I dont get is that the output tensor length is 33600. How will it get mapped to coordinates x1,x2,x3,x4,probability http://5-values.
— Reply to this email directly, view it on GitHub https://github.com/jveitchmichaelis/edgetpu-yolo/issues/11#issuecomment-1693363414, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYDMJ2LI4ACYI6NHTYLTX3XXCSAXANCNFSM6AAAAAA36PZ52E . You are receiving this because you commented.Message ID: @.***>
understood. A little confusion: if 251 is classes plus bounding box. then 33600 are the no of detections right.? if yes, then what if my image contains no object, why would it still predict such 33600 predictions.
Getting this error when trying this repo detect.py for my model and image.
python3 detect.py --model yolov8_edgetpu.tflite --names label.yaml --image image.jpg
ERROR:
INFO:EdgeTPUModel:Confidence threshold: 0.25
INFO:EdgeTPUModel:IOU threshold: 0.45
INFO:EdgeTPUModel:Loaded 247 classes
INFO:EdgeTPUModel:Successfully loaded /home/arslan/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/yolov8_edgetpu.tflite
Traceback (most recent call last):
File "detect.py", line 51, in
Hmm, I'm not sure why your model is expecting a float input, it should be looking for int8.
On Fri, 25 Aug 2023 at 16:42, Arslan Mehmood @.***> wrote:
Getting this error when trying this repo detect.py for my model and image.
ERROR: INFO:EdgeTPUModel:Confidence threshold: 0.25 INFO:EdgeTPUModel:IOU threshold: 0.45 INFO:EdgeTPUModel:Loaded 247 classes INFO:EdgeTPUModel:Successfully loaded /home/arslan//learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/yolov8_edgetpu.tflite Traceback (most recent call last): File "detect.py", line 51, in model.forward(x) File "/home/arslan/nixense_vixion/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/edgetpumodel.py", line 160, in forward self.interpreter.set_tensor(self.input_details[0]['index'], x) File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 572, in set_tensor self._interpreter.SetTensor(tensor_index, value) ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: inputs_0
— Reply to this email directly, view it on GitHub https://github.com/jveitchmichaelis/edgetpu-yolo/issues/11#issuecomment-1693473092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYDMJ7Z5AEMLO5ATMNZAVLXXC2UFANCNFSM6AAAAAA36PZ52E . You are receiving this because you commented.Message ID: @.***>
Is your model actually quantised?
On Fri, 25 Aug 2023 at 17:02, Josh Veitch-Michaelis < @.***> wrote:
Hmm, I'm not sure why your model is expecting a float input, it should be looking for int8.
On Fri, 25 Aug 2023 at 16:42, Arslan Mehmood @.***> wrote:
Getting this error when trying this repo detect.py for my model and image.
ERROR: INFO:EdgeTPUModel:Confidence threshold: 0.25 INFO:EdgeTPUModel:IOU threshold: 0.45 INFO:EdgeTPUModel:Loaded 247 classes INFO:EdgeTPUModel:Successfully loaded /home/arslan//learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/yolov8_edgetpu.tflite Traceback (most recent call last): File "detect.py", line 51, in model.forward(x) File "/home/arslan/nixense_vixion/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/edgetpumodel.py", line 160, in forward self.interpreter.set_tensor(self.input_details[0]['index'], x) File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 572, in set_tensor self._interpreter.SetTensor(tensor_index, value) ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: inputs_0
— Reply to this email directly, view it on GitHub https://github.com/jveitchmichaelis/edgetpu-yolo/issues/11#issuecomment-1693473092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYDMJ7Z5AEMLO5ATMNZAVLXXC2UFANCNFSM6AAAAAA36PZ52E . You are receiving this because you commented.Message ID: @.***>
Yes , after converting the yolov8 model to tf saved model using ultralytics. I have quantized my tf saved model to INT8 using tf lite converter and a small calibration dataset from my training images.
You also need to use the edgetpu compiler, otherwise I don't see how the model is running on the device?
https://coral.ai/docs/edgetpu/compiler/
But either way, it seems like your model is not quantised because the input layer is asking for a float tensor instead of the int8 one that we're passing in:
ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: inputs_0
sorry for my confusing explainations and all. I'm telling the details from start.
First of all I have exported my yolov8 model to edgetpu model using ultralytics as ultralytics provide this export feature. After successful export, I get all these models as shown in screenshot.
and I'm using my_yolov8_full_integer_quant_edgetpu.tflite for inference by your detect.py script in repo. I'm getting this error.
python3 detect.py --model my_yolov8_full_integer_quant_edgetpu.tflite --names label.yaml --image image.jpg
INFO:EdgeTPUModel:Confidence threshold: 0.25
INFO:EdgeTPUModel:IOU threshold: 0.45
INFO:EdgeTPUModel:Loaded 247 classes
INFO:EdgeTPUModel:Successfully loaded /home/arslan/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/my_yolov8_full_integer_quant_edgetpu.tflite
Traceback (most recent call last):
File "detect.py", line 51, in
Note: The last time the model which was successfully performing inference on edgetpu using pycoral API, I have deleted all previous files, and I'm starting from scratch.
You can try just changing the cast to np.int8 here: https://github.com/jveitchmichaelis/edgetpu-yolo/blob/784d9be1bb13ce4b8b3c1bad729a02a69cca97bb/edgetpumodel.py#L158
from here I have two approaches, either I can use this edgetpu model provided by ultralytics or I can quantize the savedtf model to INT8 only using tflite converter with a calibration dataset. The above error is of 1st approach.
one thing I'm not finding good is that inference is taking a lot of time and my local cpu usage gets high when I start inference which means a part of model in being executed on my local cpu, and this thing cancels the idea of edgetpu for fast inference. :(
man I would be very thankful to you for helping me in this debugging and successful execution of yolov8 on edgeTPU.
Specs of model: Model : my_yolov8_full_integer_quant_edgetpu.tflite
input: [{'name': 'serving_default_images:0', 'index': 0, 'shape': array([ 1, 1280, 1280, 3], dtype=int32), 'shape_signature': array([ 1, 1280, 1280, 3], dtype=int32), 'dtype': <class 'numpy.int8'>, 'quantization': (0.01865844801068306, -14), 'quantization_parameters': {'scales': array([0.01865845], dtype=float32), 'zero_points': array([-14], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
output: [{'name': 'PartitionedCall:0', 'index': 462, 'shape': array([ 1, 251, 33600], dtype=int32), 'shape_signature': array([ 1, 251, 33600], dtype=int32), 'dtype': <class 'numpy.int8'>, 'quantization': (0.003920518793165684, -128), 'quantization_parameters': {'scales': array([0.00392052], dtype=float32), 'zero_points': array([-128], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
The edgetpu compiler will tell you which bits of the model are executed on CPU, but this: [ 1, 1280, 1280, 3] is pretty large and you're probably running out of RAM on the accelerator. Try with much smaller input sizes; in this repo I could only get 224 to compile fully I think.
yes I had the idea that input size is very large and edgetpu ram is only 7-8mb.
inference results: python3 detect.py --model my_yolov8_full_integer_quant_edgetpu.tflite --names label.yaml --image image.jpg INFO:EdgeTPUModel:Confidence threshold: 0.25 INFO:EdgeTPUModel:IOU threshold: 0.45 INFO:EdgeTPUModel:Loaded 247 classes INFO:EdgeTPUModel:Successfully loaded /home/arslan/nixense_vixion/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/my_yolov8_full_integer_quant_edgetpu.tflite INFO:main:Testing on user image: image.jpg INFO:EdgeTPUModel:Attempting to load image.jpg
But I cant find any image with detections in my local repo dir. one thing I can do is print the detections received by post processing function.
You should get a JPEG out, there's a process function which is called during model.predict here:
https://github.com/jveitchmichaelis/edgetpu-yolo/blob/784d9be1bb13ce4b8b3c1bad729a02a69cca97bb/edgetpumodel.py#L131C12-L131C12
Yes I've checked and understood the source code for detection.
in the working dir, there is no .jpeg file. Meanwhile, I've restarted the inference and this time detections will also be printed out in terminal. Because postprocessing and image saving can only occur if detections are not empty
If you put a print statement here: https://github.com/jveitchmichaelis/edgetpu-yolo/blob/784d9be1bb13ce4b8b3c1bad729a02a69cca97bb/edgetpumodel.py#L130
Is it generating an output path?
Though you're right, currently if there aren't any detections then nothing will be saved.
as suspected
python3 detect.py --model my_yolov8_full_integer_quant_edgetpu.tflite --names label.yaml --image image.jpg INFO:EdgeTPUModel:Confidence threshold: 0.25 INFO:EdgeTPUModel:IOU threshold: 0.45 INFO:EdgeTPUModel:Loaded 247 classes INFO:EdgeTPUModel:Successfully loaded /home/arslan/learning/9_edge_tpu/yolo_edge_repo/edgetpu-yolo/my_yolov8_full_integer_quant_edgetpu.tflite INFO:main:Testing on user image: image.jpg INFO:EdgeTPUModel:Attempting to load image.jpg
Detections : []
I suggest stepping through the various models and seeing where you start to lose detections. Start with the full precision one and then check the quantised ones.
alright that's one approach, but fp32 and fp16 models are useless, they won't even execute on edgeTPU, yes to check for detections getting lost, we can follow it
I would suggest ignoring the TPU for now, just check that you actually get sensible predictions on CPU for all models. Presumably the fp32 models actually work? You will get a significant drop in performance moving to quantised (8-bit), especially for small objects,but the EdgeTPU compilation process shouldn't affect the results any further.
You can also significantly lower the confidence threshold and try some of your training images to see if it helps.