Google Edge TPU - Submissions for EXPORT competition

[batrlatom]

Compete and Win

Use this discussion thread for YOLOv5 🚀 EXPORT Competition submissions in the Google Edge TPU category. Good luck!

[bogdannedelcu]

Hi, please check my repo for a detailed tutorial on how to setup all required packages and running YoloV5 inference on Google Coral Dev Mini board. https://github.com/bogdannedelcu/yolov5-export-to-coraldevmini
I included in the repository some images of the output on 224 and 416 pixels and also the conversion report from edgetpu_compiller for both versions.

I see there are some layers in the NN which are not converted due to compatibility and size (416 is worse than 224).

Another issue to be further investigated are the slightly different results on EdgeTPU than in Google Collab on the same INT8 weights, where some detection boxes have zero width.

I hope this will help,
Bogdan

[ksp7518]

convert log1.txt
compile log.txt
@bogdannedelcu There seems to be a problem when I converting to edgeTPU.
The converted model you uploaded is boxed normally.

[jveitchmichaelis]

I assume image size must not be higher than 224, as all Google samples for EdgeTPU are like this

The EgdeTPU supports arbitrarily large inputs, provided they fit into memory. Eg 1024px is fine if the model is small enough.

[bogdannedelcu]

@jveitchmichaelis looking at yolov5s I see it starts with an image size of 640, so all the internal weights are much larger than if it started at 224, say 4x times larger. As I tested previously even for the 224 input resolution as input value, the compiled model does not get fully compiled on the EdgeTPU probably because the TRANSPOSE operation has too many parameters (as seen in @ksp7518 "compile log.txt" file)

My suggestion is to create a "Yolo5xs" version which starts at a lower resolution and maybe fewer inner layers as to fit completely in EdgeTPU. The 224 value I took from the fact that all Google samples for Coral are built using this value, so basically this could be the upper bound resolution value we could fit inside this processor. Just a supposition though... since I understand Google somehow created the EfficientNets-EdgeTPU by brute force search given the parameters of the EdgeTPU.

See here most of the models benchmarked are at 224: https://coral.ai/docs/edgetpu/benchmarks/

Given the Pixel 6 phone includes the EdgeTPU there is a good motivation to have a YoloV5 version which fits inside the EdgeTPU

[AmeerAlahmadi]

I got the same issue, can anyone lead us to the solution?

[bogdannedelcu]

Hello @AmeerAlahmadi, if you start with a model like YOLOv5n6u fine-tuned with default parameters of Ultralytics hub like this one: https://hub.ultralytics.com/models/VNKIIDQ0ngjulGQyQnSg?tab=train

You may export the model directly from the web interface of Ultralytics with a reasonable 256x256 input size which performs all operations in memory. You may see in this image as the entire model is exported as one operation on the EdgeTPU

Please note how it performs in this type of scenario where I finetuned it on VisDrone dataset: https://youtu.be/cdaIjJBYFMo . Bear in mind performance is not great for small objects as image resolution is low. However I think there could be strategies to mitigate this.

@jveitchmichaelis repo is a great place to start because hi is eliminating torch dependency: https://github.com/jveitchmichaelis/edgetpu-yolo . I recently added some specific code to implement the new output format of the last layer of the Yolov5n6u model which changed since 2020 (the 1x14x1360 last tensor). Please see here: https://github.com/bogdannedelcu/edgetpu-yolo in ms.py and edgetpumodel.py.

By the way I intent to write a paper on this, mainly related to drone usage of Google Coral & Yolo5-8, if anybody is interested to join let me know.

[shivamswarnkar]

Update: We had some issues, and we were not able to complete our dynamic image solution for coral devices in time. Therefore we are withdrawing our submission. However, I'll leave this link in case somebody was following our work. We will push the updated code by the next week.

[Not public yet]
https://github.com/BlueMirrors/Yolov5-TPU

---

Hi, we are also developing a submission for this competition and we just release an alpha version of it. We didn't have access to Edge TPUs, so we tested our solution on colab TPU for now using TensorFlow.

Our TPU solution uses the Tensorflow backend right now, but we plan to release tflite_runtime and pycoral in upcoming weeks. We already have the TFLite backend, we just have to modify interpreter creation logic.

We have built our solution in the form of a pip package with various backend support (TensorRT, Torchscript, ONNX, TensorFlow, TFLite) integrated through a common interface named Detector.

Important points

• In addition to the GitHub repo, our solution also has a pip-package
• You can run our solution directly in google-colab (enable TPU) by installing our tool with pip, pretrained weights are automatically downloaded
• All of our backends support fully dynamic inputs
• We have also optimized the inference pipeline, and the same code is compatible with TensorRT as well as TFLite
• You can use the same pip package and code base to run inference on other supported backends, even for CPU and GPU

We are working on

• Batching
• Adding pycoral and tflite_runtime, and benchmarking on relevant devices
• Releasing fully dynamic TF/TFLite exports
• Tracker Integration

Please share your feedback and let us know if you have any suggestions or feature requests.

Repository | pip-pkg | Detector-Info | Tensorflow-TPU | TFLite | FPS-Info (will update it soon) | Precission Loss Info

[Release: Alpha; Still Developing]

Thank you!
Team: @shivamswarnkar @aditya-dl

[ksp7518]

Hello. I want to try this but I'm using 'Coral USB Accelerator' on the PC.
This support to 'TPU' but the 'Coral USB Accelerator' need 'tflite-runtime' and 'edgeTPU'.
Is this possible to running this on my PC?

[shivamswarnkar]

@ksp7518 We will be pushing out a new update in the next few days that should support both tflite-runtime and edgetpu (we are prioritizing tflite runtime). We had some internal issues so our development got delayed so sorry about that. If you want, I can tag you whenever we release the update.

[ksp7518]

@shivamswarnkar I hope to test new version. Thank you for reply

[jveitchmichaelis]

Would you consider solutions that run on a general Coral accelerator? For example I'm currently using an M2 device which is how many people will probably want to deploy in production, or using the USB-C dongle. In general the setup would be identical in terms of installing the pycoral libaries etc - in my case I have it plugged into the expansion slot on a Jetson Nano.

[jveitchmichaelis]

@batrlatom @glenn-jocher please find my submission here: https://github.com/jveitchmichaelis/edgetpu-yolo

Notes:

• Minimal dependency solution - no Torch required
• Tested on a Jetson Nano, but should work on the Coral too (again, minimal deps). It will probably also work on a PC (@ksp7518)
• Docker container provided with a minimal dev environment
• Extensive installation notes for hardware, software and export
• Speed benchmarks for speed are provided for 96/192/224 px inputs; also some notes on possibilities for larger models.
• I've included an option to benchmark on COCO Val2017 - it will output a prediction json similar to val.py. There is also a script to process the json using pycocotools, but I don't consider this a deployment dep. mAP results are in the readme. Not great, but somewhat expected for a post-quantised model running at very low resolution.

[bogdannedelcu]

please give me a link where I can download your converted model and also the options you used to convert so I can investigate further.

…

On Wed, Aug 25, 2021 at 12:43 PM ksp7518 ***@***.***> wrote: @bogdannedelcu <https://github.com/bogdannedelcu> There seems to be a problem when I converting to edgeTPU. The converted model you uploaded is boxed normally. [image: _result] <https://user-images.githubusercontent.com/60207605/130768128-1aeaaa75-cca3-4352-be15-1d6b905490ca.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3428 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAJSJEC6JB4GKBRONUNJ33T6S3NVANCNFSM457JSF2A> .

[ksp7518]

Hello, I'm using edgeTPU USB Accelerator without GPU on i7 cpu.

When I tested yolov5s-int8_edgetpu_416.tflite with video, I saw that the CPU usage rate was 100%.
I also tested yolov5s.pt (without edgeTPU), it was 100% but even slower.
If I not using a GPU, I'm wondering how to lower the CPU usage. Anyone have any advice?

[bogdannedelcu]

All compiled models with image size > 96 pixels have one TRANSPOSE operation running on CPU probably due to size constraints. Therefore all the subsequent operations run on CPU, which is why you have the 100% usage rate. See above the error "Operation is otherwise supported, but not mapped due to some unspecified limitation" (TRANSPOSE and QUANTIZE)"

[ksp7518]

@bogdannedelcu Thank you for reply. I have another question. Is there a performance difference between 'coco128' and 'coco' in the conversion script?
!python3 models/tf.py --weights yolov5m.pt --cfg models/yolov5m.yaml --tfl-int8 --no-tfl-detect --img 320 --tf-raw-resize --source data/coco128.yaml --ncalib 200
my USB accelerator is possible to running --img 320 after comfile with yolov5m.pt
but the detection performance is not good and I have wonder about the option coco128.yaml
and I had test in colab but the storage was full. so, I'm testing in the local.

[jveitchmichaelis]

The source of the calibration images is not super important. I quantised my models by pointing the script at the val2017 folder and using ~1k samples.

How are you exporting the model? With the latest version of the EdgeTPU compiler, I wasn't able to compile beyond 224 px - I also tried using delegate search which didn't help. I didn't try with the m model actually. I guess it's because you're not using the Detection layer. In that case I was also able to export up to around 320 px with yolov5s, however speed was not particularly good.

Can you post the model here?

[ksp7518]

@jveitchmichaelis
Hello, here is my model
How to check detection layer?
yolov5m-int8_edgetpu_320_coco128.zip

[ksp7518]

@bogdannedelcu Hello, I tested the image size with 96. The comfile log was good ,all operation maped to Edgetpu.
but the CPU usage was same 100%.
I wonder what is the problem.
--------------------------------------------------------------------------------------------------------------------------

[jveitchmichaelis]

@ksp7518 you can use Netron:

Look on the right, you can see the tf_detect layer in the model. Also if the detection layer is in the model you should expect to see layers/ops that look like that. Check the int8 non-edgetpu model. I was guessing, as you use --no-tfl-detect. You can also do a forward run of the model and look at the output shape. If your model includes the detection layer then the output should be pretty big (i.e. what you would pass to the NMS function) - it'll have several thousand rows typically.

If you want to check where the CPU usage is coming from, then just profile your test script? Have a look at cProfile, it's easy to use.

https://docs.python.org/3/library/profile.html

[ksp7518]

@jveitchmichaelis thank you for reply
I test about CPU usage. If it is set to 'cpu' when using edgetpu, isn't there a problem?

<detect.py>
img = torch.zeros((1, 3, *imgsz), device=device) # init img

The cpu seems to go to 100% when accessing the 'img' variable in the loop that processing stream image of frame
How to set device value in case of edgetpu?

[jveitchmichaelis]

You can't set the device to be EdgeTPU - it doesn't work with Torch. What the script probably does is load the data, convert it to Torch and then to Numpy for the TFlite interpreter. This is partly why I dropped Torch in my implementation, because it's unnecessary (you might as well go straight to TFLite). However I don't have a pure numpy implementation of the detection layer currently.

Can you paste the output from the profiler?

[ksp7518]

@jveitchmichaelis It's interesting to be able to go straight to TFLite. I want to test
this is cProfile out. (this scrip is detect.py I used)
profile_output_text(sort_tot).txt
profile_output_text(sort_cum).txt

[jveitchmichaelis]

Have a look at my repository for an example - at least see if you can run it with the models I've converted (they're smaller, but should work). If that works, check to see how your CPU usage is doing. I'm curious if you see a similar load.

Note that OpenCV will often use 100% CPU accessing a camera or something similar, though I understand you're processing a video file?

[ksp7518]

@jveitchmichaelis Hello, I trying testing. but I got the error. I leaved error message to your github.
Thank you

[stefani-kovachevska]

Hi participants,

Thank you all for your submissions, we have already reviewed them and are about to announce the winners! Please send me an email at [email protected] including your full name and GitHub username, so I can share more information with you.

Stay tuned - the results are coming soon!

Stefani Kovachevska
Project Manager at Ultralytics

[jveitchmichaelis]

Thanks Stefani, I've messaged you. Look forward to hearing the results!

[stefani-kovachevska]

Thank you all for contributing and for being a valuable member of our open-source community.

The prize in this category of the YOLOv5 Export Competition goes to @jveitchmichaelis. Congratulations!
Check out the official announcement of all the winners here: https://ultralytics.com/article/export-competition-winners

Stay amazing and keep creating! 🚀
Team Ultralytics

[csikosaron98]

Is it working?

[jveitchmichaelis]

@csikosaron98 yes, at least for the config at the time of submission. It's possible things have changed with some of the more recent nano models. Please check out the repository, if you have any questions/issues, please let me know there.

https://github.com/jveitchmichaelis/edgetpu-yolo

[AmeerAlahmadi]

Hello. I'm using 'Coral USB Accelerator' on the PC. I converted 'yolov5s.pt' to edgeTPU by your colab script. It works out but bounding box is wrong. I think this problem is for anchor box. How to fix this?

[bogdannedelcu]

@AmeerAlahmadi see my previous answer above #3428 (reply in thread)

I fine-tuned one pretrained model and the architecture changed a little bit, but you will see in my repo it is working quite well. No need to use the old script I posted in 2020. You may export the trained model from the Ultralytics web interface.

[AmeerAlahmadi]

I'll try this. thank a lot.