PyTorch model running 170x slower on T4 vs A100. What could cause a bottleneck this extreme? [D]

Hey everyone,

Seeing a ~170× slowdown running a point-tracking model on an NVIDIA T4 compared to an A100. On A100 the tracker takes ~0.5 seconds per half-video. On T4 the same call takes ~85 seconds. Video is 47 frames at 256×256, batch 1. I expect a meaningful gap between these cards, but 170× feels too large to explain by generational hardware differences alone.

Setup:

• Precision: pure FP32
• Architecture: builds local 4D correlation volumes (dense matching between frames) followed by transformer layers for temporal context

Already ruled out:

• GPU is at 99% utilization during the call (via nvidia-smi)
• Model is actually on GPU (torch.cuda.is_available() = True, device prints "cuda")
• Enabling torch.backends.cudnn.benchmark = True had no effect
• Same slowdown on two independent T4 machines, so it's not a driver/setup issue

Given the architecture (4D correlations + transformers) and pure FP32 execution, what would cause a T4 to be this much slower than A100? What should I look for or profile first?