integrations/tensorrt/

[giscus[bot]]

integrations/tensorrt/

Discover the power and flexibility of exporting Ultralytics YOLOv8 models to TensorRT format for enhanced performance and efficiency on NVIDIA GPUs.

https://docs.ultralytics.com/integrations/tensorrt/

[Nit-Rathore]

Can you share any guides on installing TensorRT on Ubuntu?

I saw one of the comments suggesting to install from here:-
https://github.com/Linaom1214/TensorRT-For-YOLO-Series

Only issue in ubuntu is I can't install PyCuda for 20.04 and higher versions so, would it run perfect without that?

[pderrenger]

Hey there! 😊 Installing TensorRT on Ubuntu can indeed streamline your YOLO model deployments on NVIDIA GPUs. For PyCUDA on Ubuntu 20.04, you might face some challenges due to compatibility issues. While PyCUDA isn't strictly necessary for TensorRT itself, it's often used in related workflows.

For installing TensorRT, I recommend following the official NVIDIA documentation here: NVIDIA TensorRT Installation Guide. This guide provides detailed steps for various environments, including Ubuntu.

If you're specifically looking for a workaround for PyCUDA on Ubuntu 20.04, you could try installing it via a Conda environment, which can help manage dependencies more smoothly. Here's a quick example:

conda create -n pycuda-env python=3.8
conda activate pycuda-env
conda install -c conda-forge pycuda

Remember, while PyCUDA is useful, you can run TensorRT without it for many scenarios. If you encounter any specific issues or need further assistance, feel free to ask! 🚀

[lamb412]

Hi, I have a problem when i tried to convert pytorch format to tensorrt format:
[05/23/2024-14:22:32] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support
TF32. Disabling TF32.
[05/23/2024-14:22:32] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support
TF32. Disabling TF32.
[05/23/2024-14:22:32] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[05/23/2024-14:22:32] [TRT] [E] 1: Unsupported SM: 0x500
[05/23/2024-14:22:32] [TRT] [E] 1: [caskUtils.cpp::nvinfer1::rt::task::trtSmToCask::193] Error Code 1: Internal Error (Unsupported SM: 0x500)
TensorRT: export failure ❌ 14.7s: enter

Could anyone tell me how to fix this?

[pderrenger]

Hello! It looks like you're encountering an issue with the TensorRT export due to the TF32 setting and an unsupported SM (Streaming Multiprocessor) version on your hardware.

The error message indicates that TF32 is not supported by your GPU and has been disabled automatically. Additionally, the "Unsupported SM: 0x500" error suggests that your GPU's architecture may not be compatible with the TensorRT version you are using.

Here are a few steps you can take to resolve this issue:

1. Check GPU Compatibility: Ensure that your GPU is compatible with the TensorRT version you are using. You might need to update your GPU drivers or use a different version of TensorRT that supports your GPU's architecture.

2. Disable TF32: Although it seems TF32 is automatically disabled, you can explicitly ensure it's turned off in your export settings if the option is available.

3. Update TensorRT: If you're not using the latest version of TensorRT, consider updating it as newer versions might have better support for different GPU architectures.

4. Consult NVIDIA Documentation: NVIDIA's documentation and forums can be a great resource for troubleshooting specific errors related to GPU and TensorRT compatibility.

If you continue to experience issues, providing more details about your setup (like the specific GPU model and the TensorRT version) might help in diagnosing the problem more effectively.

[lamb412]

Thanks you, My GPU is Nvidia Quadro M1200. About TensorRT version, I downloaded 2 vesion, one is TensorRT-8.6.1.6. from https://developer.nvidia.com/nvidia-tensorrt-8x-download. Other I use "pip install tensortrt". I don't know ecxactly which version is needed.

[pderrenger]

Hello! It looks like your Nvidia Quadro M1200 GPU might not be fully compatible with the TensorRT version you're using, especially with the TF32 setting and the specific SM (Streaming Multiprocessor) version required by TensorRT.

Here's a quick suggestion:

• TensorRT Version: Since you have two versions installed, I recommend sticking with the one installed via pip install tensorrt as it's likely more compatible with your Python environment. Ensure it matches the CUDA version installed on your system.

If the issue persists, you might consider using an older version of TensorRT that supports your GPU architecture or updating your GPU to a newer model if you're aiming for optimal performance with TensorRT. NVIDIA's documentation or forums might also have specific guidance or updates for handling such compatibility issues.

[jirivchi1]

Hi!
I am facing this problem:

Export custom model trained from YOLOv8 Model to Tensorrt

! yolo export model="../../weights/M04-best_V2.pt" format=engine half=True device=0

TensorRT: export failure ❌ 6.3s: 'tensorrt_bindings.tensorrt.IBuilderConfig' object has no attribute 'max_workspace_size'

Any idea?

info:
tensorrt version: 10.0.1

[pderrenger]

Hello!

It looks like you're encountering an issue with the max_workspace_size attribute during the TensorRT export. This issue might be due to a mismatch or an update in the TensorRT API with version 10.0.1 that you are using.

Here’s a quick suggestion:

• Ensure that your installation of the TensorRT library and its Python bindings are correctly configured and compatible with the version of the Ultralytics YOLO you are using.

If the issue persists, you might consider downgrading to a previous version of TensorRT where max_workspace_size is supported, or checking for any updates or patches from NVIDIA that address this attribute issue in version 10.0.1.

Let us know how it goes, or if there's anything else we can help you with! 😊

[AgbajeAyomipo]

Good day, I'm working on a Jetson Orin, everything is set up, but i have an issue.
Low memory:
I'm detecting in real time and when i run my script and visualize real time results, i keep getting "low memory,...", what could be the cause of this. I have confirmed that the model works on the GPU and not CPU, i have checked the stats with the jtop command, memory is accumulated when detecting in real time(3.1G/3.3G), I have tried swapping too.
Could it be that the memory i have isn't sufficient for real time detection?

Any help with this?
Thank You.

[pderrenger]

Hello @AgbajeAyomipo,

Thank you for providing the details. The error AttributeError: 'NoneType' object has no attribute 'create_execution_context' typically indicates that the TensorRT engine file was not loaded correctly, possibly due to version mismatches or issues during the export process.

Here are a few steps to troubleshoot and resolve this issue:

1. Verify TensorRT Version:
   Ensure that the TensorRT version used for exporting the model matches the version installed on your Jetson Orin. You can check the TensorRT version on your Jetson device using:

   dpkg -l | grep nvinfer

2. Re-export the Model:
   If there's a version mismatch, re-export the model using the TensorRT version installed on your Jetson Orin. You can do this directly on the Jetson device to ensure compatibility:

   from ultralytics import YOLO
   
   model = YOLO("yolov8n.pt")
   model.export(format="engine", half=True)  # or int8=True for INT8 precision

3. Check for Compatibility:
   Ensure that the TensorRT version is compatible with the JetPack version installed on your Jetson Orin. Sometimes, upgrading or downgrading JetPack can resolve compatibility issues.

4. Memory Management:
   Given the memory constraints, consider using INT8 precision to reduce the model size and memory footprint. Ensure you have a representative dataset for calibration:

   model.export(format="engine", int8=True, data="coco.yaml")

5. Use Smaller Models:
   If memory issues persist, consider using a smaller model variant like yolov8n (nano) or yolov8s (small) to reduce memory usage.

If the issue persists, please provide a minimum reproducible example of your code and confirm that you are using the latest versions of the packages. This will help us diagnose the problem more effectively. You can find guidance on creating a reproducible example here.

Feel free to reach out if you have any more questions or need further assistance! 😊

[AgbajeAyomipo]

i ran this command "model.export(format = "engine", int8 = True)"

i get this:
[06/25/2024-06:42:01] [TRT] [W] Skipping tactic 2 due to insufficient memory on requested size of 250 detected for tactic 0x0000000000000000.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[06/25/2024-06:42:01] [TRT] [W] Tactic Device request: 450MB Available: 93MB. Device memory is insufficient to use tactic.

Since after training the model on Kaggle and exporting it, loading on the jetson returns "loaded on a different version", this is an attempt to export on the jetson as instructed above.

ii libnvinfer-bin 8.5.2-1+cuda11.4 arm64 TensorRT binaries
ii libnvinfer-dev 8.5.2-1+cuda11.4 arm64 TensorRT development libraries and headers
ii libnvinfer-plugin-dev 8.5.2-1+cuda11.4 arm64 TensorRT plugin libraries
ii libnvinfer-plugin8 8.5.2-1+cuda11.4 arm64 TensorRT plugin libraries
ii libnvinfer-samples 8.5.2-1+cuda11.4 all TensorRT samples
ii libnvinfer8 8.5.2-1+cuda11.4 arm64 TensorRT runtime libraries
ii python3-libnvinfer 8.5.2-1+cuda11.4 arm64 Python 3 bindings for TensorRT
ii python3-libnvinfer-dev 8.5.2-1+cuda11.4 arm64 Python 3 development package for TensorRT

Name: tensorrt
Version: 8.5.2.2
Summary: A high performance deep learning inference library
Home-page: https://developer.nvidia.com/tensorrt
Author: NVIDIA Corporation
Author-email:
License: Proprietary
Location: /usr/lib/python3.8/dist-packages
Requires:
Required-by:

[pderrenger]

Hello @AgbajeAyomipo,

Thank you for providing the detailed information. The error message you're encountering suggests that the TensorRT engine is running out of memory during the export process. Here are a few steps to help resolve this issue:

1. Adjust Workspace Size:
   The error indicates that the requested memory exceeds the available memory. You can try reducing the workspace size during the export process. For example:

   from ultralytics import YOLO
   
   model = YOLO("yolov8n.pt")
   model.export(format="engine", int8=True, workspace=2)  # Reduce workspace size to 2 GiB

2. Reduce Batch Size:
   Lowering the batch size can also help manage memory usage. For instance:

   model.export(format="engine", int8=True, batch=4, workspace=2)  # Reduce batch size to 4

3. Use Smaller Image Size:
   Reducing the image size can significantly decrease memory consumption:

   model.export(format="engine", int8=True, imgsz=320, batch=4, workspace=2)  # Reduce image size to 320

4. Ensure Compatibility:
   Make sure that the TensorRT version used for exporting the model matches the version installed on your Jetson Orin. You can check the TensorRT version on your Jetson device using:

   dpkg -l | grep nvinfer

5. Re-export on Jetson Device:
   If the model was initially exported on a different device, try re-exporting it directly on the Jetson Orin to ensure compatibility:

   from ultralytics import YOLO
   
   model = YOLO("yolov8n.pt")
   model.export(format="engine", int8=True, data="coco.yaml")  # Export directly on Jetson Orin

If the issue persists, please provide a minimum reproducible example of your code and confirm that you are using the latest versions of the packages. This will help us diagnose the problem more effectively. You can find guidance on creating a reproducible example here.

Feel free to reach out if you have any more questions or need further assistance! 😊

[AgbajeAyomipo]

Thanks for the feedback, i really do appreciate

However, here's where I'm lost.
When i run pip show tensorrt, the version i get is 8.5.2.2
However with the command "dpkg -l | grep nvinfer", i get:
libnvinfer-bin 8.5.2-1+cuda11.4 arm64 TensorRT binaries
ii libnvinfer-dev 8.5.2-1+cuda11.4 arm64 TensorRT development libraries and headers
ii libnvinfer-plugin-dev 8.5.2-1+cuda11.4 arm64 TensorRT plugin libraries
ii libnvinfer-plugin8 8.5.2-1+cuda11.4 arm64 TensorRT plugin libraries
ii libnvinfer-samples 8.5.2-1+cuda11.4 all TensorRT samples
ii libnvinfer8 8.5.2-1+cuda11.4 arm64 TensorRT runtime libraries
ii python3-libnvinfer 8.5.2-1+cuda11.4 arm64 Python 3 bindings for TensorRT
ii python3-libnvinfer-dev 8.5.2-1+cuda11.4 arm64 Python 3 development package for TensorRT

Which of these is my tensorrt version?

I trained the model on Kaggle and exported directly.

The last question is this, to re-export on the jetson, can i also pass a workspace argument as 2 as shown earlier.

[pderrenger]

Hello @AgbajeAyomipo,

Thank you for your detailed feedback and for providing the version information. It looks like you have TensorRT version 8.5.2.2 installed, which matches the version shown by dpkg -l | grep nvinfer. This confirms that your TensorRT installation is consistent.

Regarding your questions:

1. TensorRT Version:
   The version you see with pip show tensorrt (8.5.2.2) and the one listed by dpkg -l | grep nvinfer (8.5.2-1+cuda11.4) are essentially the same. The slight difference in formatting is due to how the package managers display version information.

2. Re-exporting on Jetson:
   Yes, you can pass the workspace argument when re-exporting the model on the Jetson device. This helps manage memory allocation during the export process. For example:

   from ultralytics import YOLO
   
   model = YOLO("yolov8n.pt")
   model.export(format="engine", int8=True, workspace=2, batch=4, imgsz=320, data="coco.yaml")

   This command reduces the workspace size to 2 GiB, sets the batch size to 4, and reduces the image size to 320, which should help manage memory usage.

If the issue persists, please provide a minimum reproducible example of your code to help us diagnose the problem more effectively. You can find guidance on creating a reproducible example here.

Feel free to reach out if you have any more questions or need further assistance! 😊

[Everlasting-AM]

Hi, I found a error that I didn't read before. It happen when I try export to a YOLOv10 model, (for example, the pretrained model autodownloaded with "yolov10n.pt" param) to TensorRT INT8. I used the code example on this page, so I don't know how could be resolved. The message error:

Ultralytics YOLOv8.2.64 🚀 Python-3.10.12 torch-2.3.1+cu121 CUDA:0 (Tesla T4, 15102MiB)
YOLOv10n summary (fused): 285 layers, 2,762,608 parameters, 0 gradients, 8.6 GFLOPs

PyTorch: starting from 'yolov10n.pt' with input shape (8, 3, 640, 640) BCHW and output shape(s) (8, 300, 6) (5.6 MB)

ONNX: starting export with onnx 1.16.1 opset 17...
ONNX: export success ✅ 4.3s, saved as 'yolov10n.onnx' (8.9 MB)

TensorRT: starting export with TensorRT 10.2.0.post1...
TensorRT: input "images" with shape(-1, 3, -1, -1) DataType.FLOAT
TensorRT: output "output0" with shape(-1, 300, 6) DataType.FLOAT
TensorRT: building INT8 engine as yolov10n.engine
TensorRT: collecting INT8 calibration images from 'data=data.yaml'
Scanning /content/crowdhuman-complete-yolo/Crowdhuman-splited/valid/labels.cache... 5812 images, 0 backgrounds, 0 corrupt: 100%|██████████| 5812/5812 [00:00<?, ?it/s]
TensorRT: export failure ❌ 7.6s: __enter__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-b2497c04331b> in <cell line: 4>()
      2 
      3 model = YOLO("yolov10n.pt")
----> 4 out = model.export(format="engine", imgsz=640, dynamic=True, verbose=False, batch=8, workspace=2, int8=True, data="data.yaml", device=0)

5 frames
/usr/local/lib/python3.10/dist-packages/ultralytics/engine/exporter.py in export_engine(self, prefix)
    801         # Write file
    802         build = builder.build_serialized_network if is_trt10 else builder.build_engine
--> 803         with build(network, config) as engine, open(f, "wb") as t:
    804             # Metadata
    805             meta = json.dumps(self.metadata)

AttributeError: __enter__

The code executed is:

from ultralytics import YOLO

model = YOLO("yolov10n.pt")
out = model.export(format="engine", imgsz=640, dynamic=True, verbose=False, batch=8, workspace=2, int8=True, data="data.yaml")

[pderrenger]

@Everlasting-AM thank you for reporting this issue. It appears to be related to the TensorRT INT8 export process. Please ensure you are using the latest versions of the Ultralytics package and TensorRT. If the issue persists, it might be a compatibility problem with the specific versions of TensorRT and PyTorch. You can find more details and troubleshooting steps in our TensorRT export documentation. If the problem continues, please provide additional context or error logs for further assistance.

[korkland]

what kind of quantization it used in the example? is that just the implicit quantization or more controlled ptq quantization?

[pderrenger]

@korkland the example uses post-training quantization (PTQ) with TensorRT, specifically employing the "ENTROPY_CALIBRATION_2" algorithm for INT8 precision. This method involves calibrating the model using representative input data to determine scale values for each tensor. For more details, you can refer to the TensorRT export section in the Ultralytics documentation.

[Kavi91]

Hello !,

I have converted my trained .pt model to .engine. but when I try inference with my own code, it says it cannot get class names from this ''names = model.model.names''

[pderrenger]

Hello! The issue you're encountering is likely because the TensorRT model does not retain the names attribute. To resolve this, you can manually load the class names from your dataset configuration file or another source. Ensure your code correctly references the class names separately from the model. If the issue persists, please check if it reproduces with the latest version of the Ultralytics package. For more details on TensorRT export, you can refer to the Ultralytics documentation.

[ShreenidhiH]

Hi,
In the same environment I exported the yolo .pt model files to engine format twice. When I do md5sum and check the File Integrity, it is not matching.
So If we export the model twice in the same environment, engine files will be same? or it will have some differences?

[pderrenger]

@ShreenidhiH hi,

When exporting YOLO models to TensorRT format, slight differences in the resulting .engine files can occur due to non-deterministic operations in the optimization process. This means that even in the same environment, the exported files might not be identical. If you need consistent results, consider using fixed seeds and ensuring all other conditions remain constant. For more details, you can refer to our TensorRT Export Guide.

[thangphung215]

any solution to reuse the .cache file when deploy in many places ? (when using TensorRT int8)

[glenn-jocher]

@thangphung215 to reuse the TensorRT INT8 calibration .cache file across multiple deployments, ensure the calibration data and batch sizes are consistent. You can copy the .cache file to each deployment location. For more details, refer to the TensorRT Export for YOLOv8 Models documentation.

[seungdae251]

Hi. Is there a difference between creating engine file with 'yolo export model=mymodel.onnx format=engine' and with 'trtexec --onnx=mymodel.onnx --saveEngine=mymodel.engine'? Are they interchangeable? Also, I'd like to know the version of tensorrt (e.g., 8.6.2) when using 'yolo export' command. Thanks

[glenn-jocher]

@seungdae251 the yolo export command and trtexec both convert ONNX models to TensorRT, but yolo export automates the process within the Ultralytics framework, potentially offering additional optimizations. They are generally interchangeable, but results may vary slightly due to different configurations. The TensorRT version used by yolo export depends on your local installation. For more details, please refer to the Ultralytics TensorRT documentation.

[Nit-Rathore]

So I have setup TensorRT(10.4) on my AWS instance and it works well for all the pretrained models on yolo that I have tried but the moment I do transfer learnning and try to use any of the the other models it is failing and giving me this error:-

Traceback (most recent call last):
  File "Inference_time.py", line 19, in <module>
    results_tensorrt = tensorrt_model.predict(video_path)
  File "/home/cvpr/miniconda3/envs/Tensor2.0/lib/python3.8/site-packages/ultralytics/engine/model.py", line 555, in predict
    return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
  File "/home/cvpr/miniconda3/envs/Tensor2.0/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 168, in __call__
    return list(self.stream_inference(source, model, *args, **kwargs))  # merge list of Result into one
  File "/home/cvpr/miniconda3/envs/Tensor2.0/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    response = gen.send(None)
  File "/home/cvpr/miniconda3/envs/Tensor2.0/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 274, in stream_inference
    s[i] += self.write_results(i, Path(paths[i]), im, s)
  File "/home/cvpr/miniconda3/envs/Tensor2.0/lib/python3.8/site-packages/ultralytics/engine/predictor.py", line 334, in write_results
    string += f"{result.verbose()}{result.speed['inference']:.1f}ms"
  File "/home/cvpr/miniconda3/envs/Tensor2.0/lib/python3.8/site-packages/ultralytics/engine/results.py", line 663, in verbose
    log_string += f"{n} {self.names[int(c)]}{'s' * (n > 1)}, "
KeyError: 1

[glenn-jocher]

@Nit-Rathore it seems like the error might be related to missing class names in your custom model. Ensure that your model's class names are correctly defined and match the dataset used for training. If the issue persists, please verify your dataset configuration and model export settings. For more guidance, you can refer to the TensorRT export documentation.

[Nit-Rathore]

import time
from ultralytics import YOLO

#Load the Yolo model
yolo_model_path = "/home/cvpr/models/tennis_racket_detect.pt"
yolo_model = YOLO(yolo_model_path)

# Load the TensorRT model
tensorrt_engine_path = "/home/cvpr/models/tennis_racket_detect.engine"
tensorrt_model = YOLO(tensorrt_engine_path, task='detect')

print(tensorrt_model.names)
print(yolo_model.names)

(Tensor2.0) cvpr@ip-10-0-12-208:~$ python Inference_time.py 
Loading /home/cvpr/models/tennis_racket_detect.engine for TensorRT inference...
[09/24/2024-06:01:48] [TRT] [I] Loaded engine size: 18 MiB
[09/24/2024-06:01:48] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +144, now: CPU 0, GPU 162 (MiB)
{0: 'tennis_racket'}
{0: 'tennis_racket'}

Hi, so my doubt is if it's working fine on YOLO model and on ONNX as well, on a dataset taken from roboflow, why is it failing on TensorRT only for the custom trained case?

[Nit-Rathore]

So my other question would be say that assuming my labels are incorrect then how is pytorch working correctly if the labels are not matching, since it is rendering results on yolo model.

[Nit-Rathore]

does it have anything to do with the input format that TensorRT expects?

[glenn-jocher]

If PyTorch works but TensorRT doesn't, it might be due to differences in how each handles model configurations or unsupported layers. Ensure your custom model's export settings align with TensorRT requirements, and verify that all layers are supported. For more details, check the TensorRT export documentation.

[tastafur]

Hello,
I have been looking for information to calibrate to INT8 the custom model trained with your specifications and it is not clear to me the model is based on a yolov 8.2 s detection, which was trained the Jetson Orin NX 16GB of 100 TOPS is the one we are using I have been trying to know whether or not I would have to touch the workspace from 4 to 6 or more to see if it would improve the calibration or batch but I have not found information about it, should I raise it or not and why? and another thing I have a devkit AGX orin of 64 gigas, from what I have understood if I calibrate without adjusting the clock frequencies, I emulate the gpu and cpu the power it has and such, from what I have understood, it is not going to calibrate well and so I would have to use the Jetson Orin NX 16GB to calibrate it is correct?

[glenn-jocher]

@tastafur for INT8 calibration, using the Jetson Orin NX 16GB is recommended to ensure accurate results, as calibration is device-specific. Adjusting the workspace from 4 to 6 GiB may help if you encounter memory issues, but it's best to experiment based on your specific setup.

[tastafur]

I did one with 4 and it goes well and if I raise it to 6 and it still holds I understood that it should if it goes up and holds go better, no? I'm going to try to train another one at workspace that if it holds it should work better no?

[glenn-jocher]

If your calibration with a workspace of 4 GiB is stable, increasing it to 6 GiB might enhance optimization, but it's best to test and see. For accurate INT8 calibration, use the same device for both calibration and deployment, so calibrating on the Jetson Orin NX 16GB is advisable.

[tastafur]

Thank you Glen for your help

[glenn-jocher]

Thank you for your interest! For INT8 calibration, using the Jetson Orin NX 16GB is recommended to ensure accurate results, as calibration is device-specific. Adjusting the workspace from 4 to 6 GiB may help if you encounter memory issues, but it's best to experiment based on your specific setup.

[ckavak]

I have followed all your instructions but when exporting to tensorrt still getting error: ONNX: export success ✅ 1.4s, saved as 'yolov8n.onnx' (12.2 MB)
requirements: Ultralytics requirement ['tensorrt>7.0.0,<=10.1.0'] not found, attempting AutoUpdate...
error: subprocess-exited-with-error

I am using Jetson AGX Orin Jetpack 6.1 r36.4 cuda12.6 tensorrt 10.5 versions.

I have already installed python3-libnvinfer and python3-libnvinfer-dev.

I have some doubts whether Ultralytics supports up to date Jetson packages.

[glenn-jocher]

@ckavak it seems like the issue might be related to compatibility with the Jetson packages. Please ensure that your TensorRT version aligns with the supported range specified in the requirements. You can find more detailed guidance on exporting to TensorRT in our documentation: https://docs.ultralytics.com/integrations/tensorrt/.

[ckavak]

there is nothing related with supported range requirement. Please STOP auto replying Glenn and misguide people which does not help. Trusting your comments people delete and re-install packages for nothing !!! I have already solved the problem myself and there is no helpful guidance on your website.

[glenn-jocher]

I'm glad you resolved the issue. If you have further questions or need specific guidance, feel free to ask here. For detailed compatibility and setup instructions, please refer to our documentation.

[Aniket1210]

Hello, I have a problem when i covert my 'best.pt' file into 'best.engine' so running code i occur following error, can you tell me why this happen or what is the solution for this ?

Speed: 5.5ms preprocess, 6.9ms inference, 137.0ms postprocess per image at shape (1, 3, 640, 640)
Loading best_cls_small.engine for TensorRT inference...
[10/24/2024-11:55:11] [TRT] [I] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
[10/24/2024-11:55:11] [TRT] [I] Loaded engine size: 2 MiB
[10/24/2024-11:55:11] [TRT] [W] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[10/24/2024-11:55:11] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 38 (MiB)
[10/24/2024-11:55:11] [TRT] [E] IExecutionContext::setInputShape: Error Code 3: API Usage Error (Parameter check failed, condition: satisfyProfile. Set dimension [1,3,640,640] for tensor images does not satisfy any optimization profiles. Valid range for profile 0: [1,3,32,32]..[4,3,64,64].)
[10/24/2024-11:55:11] [TRT] [E] IExecutionContext::setInputShape: Error Code 3: API Usage Error (Parameter check failed, condition: satisfyProfile. Set dimension [1,3,640,640] for tensor images does not satisfy any optimization profiles. Valid range for profile 0: [1,3,32,32]..[4,3,64,64].)
Traceback (most recent call last):
File "/home/fev/tsr_inference/main.py", line 50, in
output = det.classify(frame,det.model_det)
File "/home/fev/tsr_inference/object_detection.py", line 100, in classify
results = self.model_cls(crop,verbose=False)
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 176, in call
return self.predict(source, stream, **kwargs)
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 554, in predict
return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 168, in call
return list(self.stream_inference(source, model, *args, **kwargs)) # merge list of Result into one
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
response = gen.send(None)
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 261, in stream_inference
self.results = self.postprocess(preds, im, im0s)
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/models/yolo/detect/predict.py", line 25, in postprocess
preds = ops.non_max_suppression(
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/utils/ops.py", line 264, in non_max_suppression
box, cls, mask = x.split((4, nc, nm), 1)
File "/home/fev/tsr/tsrenv/lib/python3.10/site-packages/torch/_tensor.py", line 921, in split
return torch._VF.split_with_sizes(self, split_size, dim)
RuntimeError: split_with_sizes expects split_sizes to sum exactly to 4 (input tensor's size at dimension 1), but got split_sizes=[4, 31, 0]

[glenn-jocher]

@Aniket1210 it seems like the error is due to a mismatch in the input shape and optimization profiles. Ensure that the input dimensions of your model match the optimization profiles specified during the TensorRT export. You can adjust the imgsz and batch parameters during export to align with your model's requirements. For more details, please refer to the TensorRT export guide.

[Aniket1210]

Hello Team, Im faving this error can you please lat me know what is the exam problem here, I'm using
cuda 12.2 ,
Cudnn 8.9.7.29,
torch 2.2.2+cu121 ,
OpenCv 4.8.1,
all version i'm using in my GPU system, But during tensort export i occur following error, please give me solution for this

WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.1.42 🚀 Python-3.10.12 torch-2.2.2+cu121 CUDA:0 (NVIDIA GeForce RTX 4090 Laptop GPU, 16079MiB)
YOLOv8l summary (fused): 268 layers, 43607379 parameters, 0 gradients, 164.8 GFLOPs

PyTorch: starting from 'best_det.pt' with input shape (16, 3, 640, 640) BCHW and output shape(s) (16, 5, 8400) (83.6 MB)

ONNX: starting export with onnx 1.16.2 opset 17...
ONNX: export success ✅ 4.9s, saved as 'best_det.onnx' (166.4 MB)
WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.1.42 🚀 Python-3.10.12 torch-2.2.2+cu121 CUDA:0 (NVIDIA GeForce RTX 4090 Laptop GPU, 16079MiB)
YOLOv8l summary (fused): 268 layers, 43607379 parameters, 0 gradients, 164.8 GFLOPs

PyTorch: starting from 'best_det.pt' with input shape (16, 3, 640, 640) BCHW and output shape(s) (16, 5, 8400) (83.6 MB)

ONNX: starting export with onnx 1.16.2 opset 17...
ONNX: export success ✅ 5.0s, saved as 'best_det.onnx' (166.4 MB)
TensorRT: export failure ❌ 5.0s: partially initialized module 'tensorrt' has no attribute 'version' (most likely due to a circular import)
TensorRT: export failure ❌ 10.5s: partially initialized module 'tensorrt' has no attribute 'version' (most likely due to a circular import)
Traceback (most recent call last):
File "/home/adas/yolov8_inference/tensorrt.py", line 7, in
model.export(format="engine", imgsz=640, dynamic=True, verbose=False, batch=16, int8=True, data="./data.yaml") # creates 'yolov8n.engine'
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 600, in export
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 285, in call
f[1], _ = self.export_engine()
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 138, in outer_func
raise e
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 133, in outer_func
f, model = inner_func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 659, in export_engine
import tensorrt as trt # noqa
File "/home/adas/yolov8_inference/tensorrt.py", line 7, in
model.export(format="engine", imgsz=640, dynamic=True, verbose=False, batch=16, int8=True, data="./data.yaml") # creates 'yolov8n.engine'
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 600, in export
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 285, in call
f[1], _ = self.export_engine()
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 138, in outer_func
raise e
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 133, in outer_func
f, model = inner_func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 665, in export_engine
check_version(trt.version, "7.0.0", hard=True) # require tensorrt>=7.0.0
AttributeError: partially initialized module 'tensorrt' has no attribute 'version' (most likely due to a circular import)

[glenn-jocher]

@Aniket1210 it seems like the error is due to a circular import issue with the TensorRT module. Please ensure that your script does not name any files or variables as 'tensorrt' to avoid conflicts. If the issue persists, consider checking your TensorRT installation and environment setup.

[Aniket1210]

Hello Team, Still I'm getting same error i have installed tensorrt version

WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.1.42 🚀 Python-3.10.12 torch-2.2.2+cu121 CUDA:0 (NVIDIA GeForce RTX 4090 Laptop GPU, 16079MiB)
YOLOv8l summary (fused): 268 layers, 43607379 parameters, 0 gradients, 164.8 GFLOPs

PyTorch: starting from 'best_det.pt' with input shape (16, 3, 640, 640) BCHW and output shape(s) (16, 5, 8400) (83.6 MB)

ONNX: starting export with onnx 1.16.2 opset 17...
ONNX: export success ✅ 5.0s, saved as 'best_det.onnx' (166.4 MB)
TensorRT: export failure ❌ 5.0s: partially initialized module 'tensorrt' has no attribute 'version' (most likely due to a circular import)
TensorRT: export failure ❌ 10.6s: partially initialized module 'tensorrt' has no attribute 'version' (most likely due to a circular import)
Traceback (most recent call last):
File "/home/adas/Traffic_Sign_Detection/data/tensorrt.py", line 7, in
model.export(format="engine", imgsz=640, dynamic=True, verbose=False, batch=16, int8=True, data="./data.yaml") # creates 'yolov8n.engine'
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 600, in export
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 285, in call
f[1], _ = self.export_engine()
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 138, in outer_func
raise e
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 133, in outer_func
f, model = inner_func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 659, in export_engine
import tensorrt as trt # noqa
File "/home/adas/Traffic_Sign_Detection/data/tensorrt.py", line 7, in
model.export(format="engine", imgsz=640, dynamic=True, verbose=False, batch=16, int8=True, data="./data.yaml") # creates 'yolov8n.engine'
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 600, in export
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 285, in call
f[1], _ = self.export_engine()
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 138, in outer_func
raise e
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 133, in outer_func
f, model = inner_func(*args, **kwargs)
File "/home/adas/tsr/tsrenv/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 665, in export_engine
check_version(trt.version, "7.0.0", hard=True) # require tensorrt>=7.0.0
AttributeError: partially initialized module 'tensorrt' has no attribute 'version' (most likely due to a circular import)
(tsrenv) adas@adas:~/Traffic_Sign_Detection/data$ python3
Python 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorrt as trt
WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
^Z^X
[1]+ Stopped python3
(tsrenv) adas@adas:/Traffic_Sign_Detection/data$
(tsrenv) adas@adas:/Traffic_Sign_Detection/data$ python3
Python 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
print(trt.version)
Traceback (most recent call last):
File "", line 1, in
NameError: name 'trt' is not defined
import tensorrt as trt
WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0
Ultralytics YOLOv8.1.42 🚀 Python-3.10.12 torch-2.2.2+cu121 CUDA:0 (NVIDIA GeForce RTX 4090 Laptop GPU, 16079MiB)
(tsrenv) adas@adas:$
(tsrenv) adas@adas:$ python3
Python 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import cv2
import tensorrt as trt
print(trt.version)
10.6.0

[glenn-jocher]

It looks like the issue might be due to a naming conflict with your script or directory named 'tensorrt'. Please rename it to avoid conflicts with the TensorRT module. If the problem persists, ensure your TensorRT installation is correct and compatible with your setup.

[Aniket1210]

Thank you for your suggestion, and it working now, but while running the export code i got following error [

11/11/2024-18:31:54] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[11/11/2024-18:31:56] [TRT] [E] IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node /model.0/conv/Conv + PWN(PWN(/model.0/act/Sigmoid), PWN(/model.0/act/Mul)).)
TensorRT: export failure ❌ 776.2s: enter
Traceback (most recent call last):
File "/home/adas/Traffic_Sign_Detection/data/convert_trt_tsr.py", line 7, in
model.export(format="engine", imgsz=640, dynamic=True, verbose=False, batch=16, int8=True, data="./data.yaml") # creates 'yolov8n.engine'
File "/home/adas/tsr_v9/yolov9_tsr/lib/python3.10/site-packages/ultralytics/engine/model.py", line 602, in export
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)
File "/home/adas/tsr_v9/yolov9_tsr/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/adas/tsr_v9/yolov9_tsr/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 301, in call
f[1], _ = self.export_engine()
File "/home/adas/tsr_v9/yolov9_tsr/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 143, in outer_func
raise e
File "/home/adas/tsr_v9/yolov9_tsr/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 138, in outer_func
f, model = inner_func(*args, **kwargs)
File "/home/adas/tsr_v9/yolov9_tsr/lib/python3.10/site-packages/ultralytics/engine/exporter.py", line 798, in export_engine
with build(network, config) as engine, open(f, "wb") as t:
AttributeError: enter

[glenn-jocher]

It seems like the error might be related to unsupported operations in TensorRT. Please ensure your model architecture is compatible with TensorRT or consider simplifying the model. If the issue persists, reviewing the TensorRT logs for more detailed error information could help identify the specific problem.

[enfff]

When exporting a .engine model with the .export() method, what version of TensorRT does it use? Can I choose a different version, let's say 10.3?

[glenn-jocher]

@enfff the TensorRT version used during export depends on the version installed in your environment. You can specify a different version by installing it via NVIDIA's package manager or using a Docker container with the desired version.

[smrutiranjanmohapatra]

Reading engine from file /content/engine/yolo11x_fp16.engine
Total Inference Time : 17.17
Total Frame processed : 750
Average Inference FPS : 43.69
Total Feature Time : 75.53
Average feature FPS : 9.93
Total Tracking Time : 14.76
Average Tracking FPS : 50.83 .
How to reduce my inference time using tensorrt engine for fast-reid model . Below i have given the code. I need help

def load_and_inference_fastreid(fastreid_batch_images, engine, fastreid_inputs:np.ndarray,fastreid_outputs:np.ndarray,bindings,stream):
# #load the tensorrt engine
# if not os.path.exists(fastreid_engine_path):
# raise FileNotFoundError(f"Engine file {fastreid_engine_path} not found. Please ensure the path is correct.")

# with trt.Runtime(TRT_LOGGER) as runtime, open(fastreid_engine_path,"rb") as f:
#     fastreid_engine_data = f.read()
#     engine = runtime.deserialize_cuda_engine(fastreid_engine_data)

# fastreid_img = cv2.resize(frame, (256,128), interpolation=cv2.INTER_LINEAR)
# fastreid_input_img = preprocess_image(fastreid_img)
# fastreid_batch_images = np.concatenate([fastreid_input_img], axis=0)

# fastreid_inputs,fastreid_outputs,bindings,stream = allocate_buffers(engine,fastreid_output_shape,profile_idx=0)

#Create excution context
context = engine.create_execution_context()


# List to hold extracted features for each image
extracted_features = []

 # Loop over each image in the batch and process it sequentially
for image in fastreid_batch_images:
    # Set the image as input to the model
    fastreid_inputs[0].host = image

    # Set the input shape for the context (batch size of 1)
    context.set_input_shape('input', image.shape)

    # Perform inference on the single image
    features = do_inference(context, engine, bindings, fastreid_inputs, fastreid_outputs, stream)

    # Flatten the features and append to the list
    extracted_features.append(np.array(features[0]).flatten())

# fastreid_inputs[0].host = fastreid_batch_images

# context.set_input_shape('input', fastreid_batch_images.shape)

# #perform feature extraction
# #extracted_features = get_feature_fastreid(engine,context,fastreid_inputs,fastreid_outputs,bindings,stream)
# extracted_features = do_inference(context, engine, bindings, fastreid_inputs, fastreid_outputs, stream)

# #Print type and structure of extracted_features
# #print(f"Extracted features type: {type(extracted_features)}")
# #print(f"Extracted features content: {extracted_features}")

# # Reshape the extracted features to a 2D array (1, 2048)
# extracted_features = np.array(extracted_features[0]).flatten()
# #print(f"Reshaped extracted features shape: {extracted_features.shape}")

return  extracted_features

def _do_inference_base(inputs, outputs, stream, execute_async_func):
# Transfer input data to the GPU.
kind = cudart.cudaMemcpyKind.cudaMemcpyHostToDevice
[cuda_call(cudart.cudaMemcpyAsync(inp.device, inp.host, inp.nbytes, kind, stream)) for inp in inputs]
# Run inference.
execute_async_func()
# Transfer predictions back from the GPU.
kind = cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost
[cuda_call(cudart.cudaMemcpyAsync(out.host, out.device, out.nbytes, kind, stream)) for out in outputs]
# Synchronize the stream
cuda_call(cudart.cudaStreamSynchronize(stream))
# Return only the host outputs.
return [out.host for out in outputs]

def do_inference(context, engine, bindings, inputs, outputs, stream):
def execute_async_func():
context.execute_async_v3(stream_handle=stream)

# Setup context tensor address.
num_io = engine.num_io_tensors
for i in range(num_io):
    context.set_tensor_address(engine.get_tensor_name(i), bindings[i])
return _do_inference_base(inputs, outputs, stream, execute_async_func)

Define target classes

target_classes = ['car', 'bus', 'truck', 'motorcycle']

def main():
# Input
video_file_name = '/content/30.mp4'
video_path = os.path.join(current_directory, 'data', video_file_name)
cap = cv2.VideoCapture(video_path) # Load video

frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

original_size = (frame_width, frame_height)
#original_size = (1200, 800)
fps = int(cap.get(cv2.CAP_PROP_FPS))

# Model and engine paths
model_name = "yolo11x"
precision = "fp16"  # int8 or fp32 or fp16
#quantization_method = ''
onnx_model_path = os.path.join(current_directory, 'onnx', f'{model_name}_{device.type}.onnx')
engine_file_path = os.path.join(current_directory, 'engine', f'{model_name}_{precision}.engine')

fastreid_onnx_path = "/content/fastreid_model.onnx"
fastreid_engine_path = "/content/fastreid_model.trt"
os.makedirs(os.path.dirname(engine_file_path), exist_ok=True)

# Output shapes expected
output_shapes = [(1, 84, 8400)]
output_shape_ndarray = np.array(output_shapes[0], dtype=np.int32)

# fastreid Output shapes expected
#fastreid_output_shapes = (1,256)   #(1,2048)
fastreid_output_shapes = (1, 2048)
fastreid_output_shape_ndarray = np.array(fastreid_output_shapes, dtype=np.int32)
#print(fastreid_output_shape_ndarray)

# Load or build the yolo TensorRT engine and do inference
with get_engine(onnx_model_path, engine_file_path, precision) as engine, \
        engine.create_execution_context() as context:

    # inputs, outputs, bindings, stream = allocate_buffers(engine, output_shapes[0], profile_idx=0)
    inputs, outputs, bindings, stream = allocate_buffers(engine, output_shape_ndarray, profile_idx=0)

    # Video writer to save the output
    fourcc = cv2.VideoWriter_fourcc(*'mp4v')
    out = cv2.VideoWriter('output_video.mp4', fourcc, fps, original_size)

    # Variable to store the start time of processing for each frame
    start_time = time.time()
    total_inference_time = 0
    total_feature_time = 0
    total_tracking_time = 0
    frame_count = 0

     # Load FastReID engine once at the beginning
    fastreid_engine = load_fastreid_engine(fastreid_engine_path)
    fastreid_inputs,fastreid_outputs,fastreid_bindings,fastreid_stream = allocate_buffers(engine,fastreid_output_shape_ndarray,profile_idx=0)

    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break

        original_frame = frame.copy()
        original_frame = cv2.resize(original_frame, original_size)

        #input setup for yolotensorrt
        img = cv2.resize(frame, (640, 640), interpolation=cv2.INTER_LINEAR)
        input_image = preprocess_image(img)  # Preprocess image
        batch_images = np.concatenate([input_image], axis=0)

        inputs[0].host = batch_images
        context.set_input_shape('input', batch_images.shape)

        # #input setup for fastreidtensorrt
        # fastreid_img = cv2.resize(frame, (256,128), interpolation=cv2.INTER_LINEAR)
        # fastreid_input_img = preprocess_image(fastreid_img)
        # fastreid_batch_images = np.concatenate([fastreid_input_img], axis=0)

        # Inference
        start_inference_time = time.time()
        trt_outputs = do_inference(context, engine=engine, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)
        torch.cuda.synchronize()

        inference_time = time.time()-start_inference_time
        total_inference_time += inference_time

        # Reshape and post-process the output
        t_outputs = np.array(trt_outputs).reshape(output_shapes[0])
        results = non_max_suppression(t_outputs,original_size)


        # Filter results to include only target classes
        filtered_results = []
        for img_idx, (boxes, scores, classes) in enumerate(results):
            filtered_boxes = []
            filtered_scores = []
            filtered_classes = []
            for box, score, classe in zip(boxes, scores, classes):
                if coco_labels[classe] in target_classes:
                    filtered_boxes.append(box)
                    filtered_scores.append(score)
                    filtered_classes.append(classe)
            filtered_results.append((filtered_boxes, filtered_scores, filtered_classes))

        # Initialize lists for DeepSORT
        results_deepsort = []
        embeddings = []
        fastreid_batch_images = []
        appearance_features_batch = []

        # Loop over filtered results to extract appearance features and prepare for tracking
        for img_idx, (boxes, scores, classes) in enumerate(filtered_results):
            for box, score, classe in zip(boxes, scores, classes):
                # Get coordinates of the bounding box
                xmin, ymin, xmax, ymax = map(int, box)

                # Ensure coordinates are within image boundaries
                height, width = frame.shape[:2]
                ymin = max(0, ymin)
                ymax = min(height, ymax)
                xmin = max(0, xmin)
                xmax = min(width, xmax)

                # Calculate the center of the bounding box
                center_x = (xmin + xmax) // 2
                center_y = (ymin + ymax) // 2

                # Line crossing condition: Check if the center_y crosses between the start and end y-coordinates of the line
                if startTrackingPoints[0][1] <= center_y <= endTrackingPoints[0][1]:

                     #print(f"ymin,ymax,xmin,xmax :{ymin,ymax,xmin,xmax}")

                     #cropping the image from frame according to bounding box
                     cropped_image = frame[ymin:ymax, xmin:xmax]

                     # Preprocess cropped image for FastReID
                     fastreid_input_img = preprocess_image(cv2.resize(cropped_image, (256, 128), interpolation=cv2.INTER_LINEAR))
                     #  fastreid_batch_images = np.concatenate([fastreid_input_img], axis=0)
                     fastreid_batch_images.append(fastreid_input_img)

                     # Prepare the detection data (bounding box + confidence + class name)
                     results_deepsort.append([[xmin, ymin, xmax - xmin, ymax - ymin], score, coco_labels[classe]])

        # Get the appearance features using FastReID
        start_feature_time = time.time()
        #appearance_features = get_appearance_features(original_frame, [xmin, ymin, xmax, ymax])
        if len(fastreid_batch_images) > 0:
             appearance_features_batch = load_and_inference_fastreid(fastreid_batch_images,fastreid_engine, fastreid_inputs, fastreid_outputs,fastreid_bindings,fastreid_stream)

        feature_time = time.time() - start_feature_time
        total_feature_time += feature_time
        #embeddings.append(appearance_features)
        
        if len(appearance_features_batch) > 0:
            # Process features for each object in the batch
            for idx, feature in enumerate(appearance_features_batch):
                     # Each 'feature' corresponds to an appearance feature for a single object in the batch
                     embeddings.append(feature)

        # Get the appearance features using FastReID
        start_tracking_time = time.time()

        # Perform DeepSORT tracking
        tracked_objects = tracker.update_tracks(results_deepsort, embeds=embeddings, frame=original_frame)
        #tracked_objects = tracker.update_tracks(results_deepsort, frame=original_frame)

        tracking_time = time.time() - start_tracking_time
        total_tracking_time += tracking_time

        # Draw bounding boxes and track IDs
        for track in tracked_objects:
            if not track.is_confirmed():
                continue

            track_id = track.track_id
            bbox = track.to_tlbr()  # Get top-left to bottom-right coordinates of the bounding box

            # Draw the bounding box
            x1, y1, x2, y2 = map(int, bbox)
            cv2.rectangle(original_frame, (x1, y1), (x2, y2), (0, 0, 0), 5)

            # Label the object with its track ID
            cv2.putText(original_frame, f'ID: {track_id}', (x1, y1 - 10),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 0, 255), 3)

        # # Draw bounding boxes on detection the frame
        # for img_idx, (boxes, scores, classes) in enumerate(filtered_results):
        #     labels = [coco_labels[classe] for classe in classes]
        #     output_frame = draw_bounding_boxes(original_frame, boxes, labels, scores)

        # Calculate FPS for the current frame
        end_time = time.time()
        FPS = 1 / (end_time - start_time)
        start_time = end_time

        # Display FPS on the frame
        cv2.putText(original_frame, f"FPS: {FPS:.2f}", (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 0, 255), 3)

        frame_count += 1
        # Write the frame with detections to the output video
        out.write(original_frame)
        #cv2.imshow("Frame", original_frame)
        # Exit the loop if 'q' is pressed
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

    cap.release()
    #out.release()
    cv2.destroyAllWindows()

    #calculate average inference FPS
    avg_inference_fps = frame_count / total_inference_time
    print(f"Total Inference Time : {total_inference_time: .2f}")
    print(f"Total Frame processed : {frame_count}")
    print(f"Average Inference FPS : {avg_inference_fps: .2f}")

    # calculate average feature FPS
    avg_feature_fps = frame_count / total_feature_time
    print(f"Total Feature Time : {total_feature_time: .2f}")
    print(f"Average feature FPS : {avg_feature_fps: .2f}")

    #calculate average tracking FPS
    avg_tracking_fps = frame_count / total_tracking_time
    print(f"Total Tracking Time : {total_tracking_time: .2f}")
    print(f"Average Tracking FPS : {avg_tracking_fps: .2f}")


free_buffers(inputs, outputs, stream)

[glenn-jocher]

@smrutiranjanmohapatra to reduce inference time using TensorRT for your FastReID model, consider optimizing your engine with INT8 precision and ensuring your batch size and workspace settings are appropriately configured. For detailed guidance, please refer to the Ultralytics TensorRT documentation.

[smrutiranjanmohapatra]

i want help with running batch inference for tensorrt in the load_and_inference_fastreid function. i am storing the detected obj in a batch and sending it for feature extraction where i am using for loop for individual obj insted i want to work on batch how to do that

[glenn-jocher]

To run batch inference in your load_and_inference_fastreid function, ensure that your input data is correctly batched and that the TensorRT engine is configured to handle batch processing. Adjust the input shape and bindings to accommodate the batch size, and avoid looping over individual objects.

[wdcs-fenilpatel]

We are using the .engine model with live camera feeds, and providing a batch of images from each camera. However, when one or more cameras fail, .engine refuses to process the batch due to the discrepancy in batch sizes. Is it possible to process dynamic batches to handle such failures?

[glenn-jocher]

@wdcs-fenilpatel to handle dynamic batch sizes with TensorRT models, you can enable dynamic batching by setting the dynamic=True argument during export. This allows the model to process varying batch sizes, accommodating cases where some camera feeds may fail. If you need further guidance, please refer to the TensorRT export documentation for detailed instructions.

[wdcs-fenilpatel]

gettig this error:
Loading /home/aiml/projects/users/krish/spotem/ai_models/v1/A0009/A0009.engine for TensorRT inference...
[11/22/2024-15:08:59] [TRT] [I] Loaded engine size: 8 MiB
[11/22/2024-15:08:59] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +241, now: CPU 0, GPU 247 (MiB)

[11/22/2024-15:09:00] [TRT] [E] IExecutionContext::setInputShape: Error Code 3: API Usage Error (Parameter check failed, condition: engineDims.d[i] == dims.d[i]. Static dimension mismatch while setting input shape for images. Set dimensions are [3,3,640,640]. Expected dimensions are [15,3,640,640].)
[11/22/2024-15:09:00] [TRT] [E] IExecutionContext::executeV2: Error Code 1: Cask (Cask convolution execution)
[11/22/2024-15:09:00] [TRT] [E] [checkMacros.cpp::catchCudaError::212] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
Traceback (most recent call last):
File "/home/aiml/projects/spotem/ai_models/convert_pt_tensorrt.py", line 33, in
results = model(list1)
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 176, in call
return self.predict(source, stream, **kwargs)
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/ultralytics/engine/model.py", line 567, in predict
return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 168, in call
return list(self.stream_inference(source, model, *args, **kwargs)) # merge list of Result into one
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 249, in stream_inference
with profilers[0]:
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/ultralytics/utils/ops.py", line 46, in enter
self.start = self.time()
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/ultralytics/utils/ops.py", line 61, in time
torch.cuda.synchronize(self.device)
File "/home/aiml/projects/spotem/dept_ai_spotem/CameraAnalytics-API/aiml_spenv/lib/python3.10/site-packages/torch/cuda/init.py", line 892, in synchronize
return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[11/22/2024-15:09:01] [TRT] [E] [graphContext.h::~MyelinGraphContext::84] Error Code 1: Myelin ([impl.cpp:unload_cuda:622] Error 700 destroying event '0x5b77b3a1cae0'.)
[11/22/2024-15:09:01] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaStream::55] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:01] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:02] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:02] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:02] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:02] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:02] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:02] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:02] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[11/22/2024-15:09:03] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (an illegal memory access was encountered)

[glenn-jocher]

The error indicates a mismatch in input batch dimensions while setting input shape for your TensorRT engine. Ensure you use a consistent batch size matching the maximum batch size defined during export. If dynamic batch processing is needed, re-export the model using dynamic=True in your export settings to support flexible input sizes.

[wdcs-fenilpatel]

can i set maximum batch size =150 with dynamic=true??

[glenn-jocher]

Yes, you can set a maximum batch size of 150 along with dynamic=True when exporting your model to TensorRT. This ensures flexibility for processing varying input batch sizes, up to the specified maximum. For details on how to configure export options, refer to the TensorRT guide.

[dilshod-obidov]

Hey there,I have error when converting yolov8m model to tensorrt with precision int8 I cannot find where error is happening please can you check it

[11/30/2024-14:35:17] [TRT] [V] --------------- Timing Runner: /model.7/conv/Conv (CudnnConvolution)
[11/30/2024-14:35:17] [TRT] [V] Tactic: 0x0000000000000000 Time: 5.79847
[11/30/2024-14:35:17] [TRT] [V] Tactic: 0x0000000000000001 Time: 1.26923
[11/30/2024-14:35:17] [TRT] [V] Tactic: 0x0000000000000002 Time: 6.37692
[11/30/2024-14:35:17] [TRT] [W] Tactic Device request: 932MB Available: 321MB. Device memory is insufficient to use tactic.

[glenn-jocher]

@dilshod-obidov the error indicates insufficient device memory for the INT8 conversion process. You can try reducing the values of workspace, batch, or imgsz during export to lower the memory requirements. Additionally, ensure the dataset used for INT8 calibration is properly specified with the data argument to optimize calibration. For more details, refer to our TensorRT Export Guide. Let us know if you need further assistance!

[dilshod-obidov]

@glen-jocher Thank you for your reply
I have tried all the ways you said but nothing is worked in Jetson Orin. It always stoped with same error which Segmentation Error(aborted)
can you help to fix it i have to convert it to int8 we converted to fp16