Workflow for performance investigation?

[jricker2]

Not sure on the best place to ask this, just a question -

Right now I'm looking into an issue where a particular ONNX model performs about the same when fp8/int8 quantization is applied vs. the fp16 version. To look into this further, I have been profiling with trtexec and dumping the layer by layer performance and trying to compare with the fp16 version. One issue with this is that since the models are slightly different, the layer names are also slightly different. There are also different layer-fusions between the two versions, making accurately comparing layer performance difficult. The approach that makes sense to me is to look from the ONNX perspective, IE what common ONNX operators have performance discrepancies. I'm surprised there are no tools out right now to help visualize things like this (AFAIK?). My main method has been just making some small python scripts to associate the TRT metadata back to ONNX ops, and print performance for both models. It is pretty inefficient/tedious.

Anyone have any other methods on this kind of investigation? Something in DL designer to really visualize this interactively would be very cool/useful.

[Saad-Mallebhari]

@jricker2
Your instinct to approach this from the ONNX operator perspective is the right one. A few tools that may help:

1. trtexec with --exportLayerInfo and --dumpProfile

trtexec --onnx=model.onnx --int8 \
        --exportLayerInfo=layer_info.json \
        --dumpProfile \
        --profilingVerbosity=detailed

The exported JSON includes ONNX node provenance metadata that can help you map TRT layers back to their original ONNX ops , which should make your Python scripting approach significantly cleaner. --dumpProfile gives you the per-layer timing you're already working with, but combined with --exportLayerInfo the mapping back to ONNX operators becomes more tractable.

2. Polygraphy for accuracy comparison

If you also want to compare numerical accuracy between the FP16 and quantized versions layer by layer:

polygraphy run model.onnx --trt --onnxrt \
    --atol 1e-2 --rtol 1e-2 \
    --verbose

Note: avoid --trt-outputs mark all for performance investigation as it adds output copies that distort profiling results , keep Polygraphy for accuracy debugging rather than timing comparisons.

3. NVIDIA Nsight Systems

For interactive visualization, Nsight Systems gives you a timeline view of kernel execution correlated with layer names , useful for spotting where quantized kernels are slower than expected at the CUDA level.

The ONNX-to-TRT layer mapping gap you're describing is a real tooling pain point , --exportLayerInfo combined with --dumpProfile is probably the most direct path to what you need right now.