inference Too low (frame) FPS compared to documentation

Search before asking

[X] I have searched the Inference issues and found no similar bug report.

Bug

Set Up

I use a Basler Camera acA1920-40uc.

It provides ~50 fps as cv2.Image via opencv. I use your sdk to post those images.

On the same device: jetson orin nano (no network latency) docker runs the inference-server-jets-5.1.1 image.

For testing I ran the same setup on my notebook(dell precision 5570 - i7-12700H) with the cpu image.

Problem

The inference takes longer than expected ~200ms (self computed ~ 5 fps). This is disappointing for 2 reasons:

unspecialized cpu on my notebook is as fast (~ 5 fps)
your blog-post made me expect at least 10-15 fps if the performance of orin nano is approx. jetson navier nx, however it must be better than it's predecessor

Question

Is there anything I am not considering so I can improve my performance?

Environment

Inference: roboflow/roboflow-inference-server-jetson-5.1.1:latest
OS: JP 5.1.1
Device: NVIDIA Jetson Orin Nano
Python: 3.9
Model Type: Roboflow 3.0 Instance Segmentation (Accurate)

Minimal Reproducible Example

Sorry - I merged 2 files if something seems odd.

import supervision as sv
import cv2
import json
from inference_sdk import InferenceHTTPClient


# Infer via the Roboflow Infer API and return the result
def infer(img: cv2.typing.MatLike) -> cv2.typing.MatLike:
    with open('roboflow_config.json') as f:
        config = json.load(f)

        ROBOFLOW_API_KEY = config["ROBOFLOW_API_KEY"]
        ROBOFLOW_MODEL = config["ROBOFLOW_MODEL"]
        ROBOFLOW_SIZE = config["ROBOFLOW_SIZE"]

        FRAMERATE = config["FRAMERATE"]
        BUFFER = config["BUFFER"]


    # local inference
    client = InferenceHTTPClient(
        api_url="http://localhost:9001",
        api_key=ROBOFLOW_API_KEY,
    )
    results = client.infer(img, model_id=ROBOFLOW_MODEL)

    # remote inference (slower)
    # from inference import get_model
    # model = get_model(ROBOFLOW_MODEL, ROBOFLOW_API_KEY)
    # results = model.infer(img)[0]

    detections = sv.Detections.from_inference(results)
    if len(detections) == 0:
        return img

    box_annotator = sv.BoxAnnotator()
    label_annotator = sv.LabelAnnotator()
    labels = [
        f"{class_name} {confidence:.2f}"
        for class_name, confidence
        in zip(detections['class_name'], detections.confidence)
    ]

    annotated_image = box_annotator.annotate(
        scene=img, detections=detections)
    annotated_image = label_annotator.annotate(
        scene=annotated_image, detections=detections, labels=labels)

    return annotated_image

'''
A simple Program for grabing video from basler camera and converting it to opencv img.
Tested on Basler acA1300-200uc (USB3, linux 64bit , python 3.5)

'''
from pypylon import pylon
import time

# conecting to the first available camera
camera = pylon.InstantCamera(pylon.TlFactory.GetInstance().CreateFirstDevice())

# Grabing Continusely (video) with minimal delay
camera.StartGrabbing(pylon.GrabStrategy_LatestImageOnly)
converter = pylon.ImageFormatConverter()

# converting to opencv bgr format
converter.OutputPixelFormat = pylon.PixelType_BGR8packed
converter.OutputBitAlignment = pylon.OutputBitAlignment_MsbAligned

while camera.IsGrabbing():
    start_time = time.time()  # start time of the loop

    ########################
    # your fancy code here #
    ########################
    grabResult = camera.RetrieveResult(5000, pylon.TimeoutHandling_ThrowException)

    if grabResult.GrabSucceeded():
        # Access the image data
        image = converter.Convert(grabResult)
        img = image.GetArray()

        #  annotation integration
        img = infer(img)

        cv2.namedWindow('title', cv2.WINDOW_NORMAL)
        cv2.imshow('title', img)

        print("FPS: ", round(1.0 / (time.time() - start_time),2))  # FPS = 1 / time to process loop
        if cv2.waitKey(1) == ord('q'):
            break

    grabResult.Release()

# Releasing the resource
camera.StopGrabbing()

cv2.destroyAllWindows()

Additional

No response

Are you willing to submit a PR?

[X] Yes I'd like to help by submitting a PR!