Anonymous authors

FAST-TMFactorized Attention over Space and Time for Video Turbulence Mitigation

Video comparisons

Drag the divider or use arrow keys to compare the same frame.

Barcode, real laboratory turbulence. Tiles, row by row: Turbulent input, MambaTM, DATUM, FAST-TM, Ground truth.
Logo, synthetic turbulence. Tiles, row by row: Turbulent input, MambaTM, DATUM, FAST-TM, Ground truth.
Books, real laboratory turbulence. Tiles, row by row: Turbulent input, MambaTM, DATUM, FAST-TM, Ground truth.
Sign, synthetic turbulence. Tiles, row by row: Turbulent input, MambaTM, DATUM, FAST-TM, Ground truth.
Building with text, real long-range footage. Tiles, row by row: Turbulent input, MambaTM, DATUM, FAST-TM.
Toys, real laboratory turbulence. Tiles, row by row: Turbulent input, MambaTM, DATUM, FAST-TM, Ground truth.
Bridge truss, synthetic turbulence. Tiles, row by row: Turbulent input, MambaTM, DATUM, FAST-TM, Ground truth.

Clips

About these comparisons

Selected for visible turbulence reduction, not representative of all test cases. Each model restores overlapping twelve-frame windows at native resolution; overlapping outputs are averaged. No number on this page comes from these videos.


Abstract

Atmospheric turbulence blurs and warps long-range video. Camera systems that restore such video on an embedded GPU must do so at the frame rate of the camera. We deploy MambaTM, a recent multi-frame turbulence mitigation network, on an NVIDIA Jetson AGX Orin with TensorRT, where it restores a clip of twelve 1024 × 1024 frames at 1.6 frames per second (FPS) in single precision. Our deployment study finds two bottlenecks. First, the state space model of MambaTM processes each clip sequentially and takes most of the inference time. Second, when we replace the state space model, the convolutional encoders and decoders, which start and end at full input resolution, take more than half of the inference time.

We present FAST-TM (Factorized Attention over Space and Time for Turbulence Mitigation), which replaces the state space model with attention (within frames, across frames and over channels) and a local 3D convolution, all native TensorRT operators, and runs the encoder and decoder below input resolution. On the same Jetson and input, FAST-TM runs at 47.1 FPS in mixed precision with 25× less energy per frame and 9× less engine memory than our FP32 engine of MambaTM, and a pipeline that decodes, restores and encodes 720p video sustains 39.3 FPS. With twelve-frame inputs, FAST-TM is 0.75 dB above MambaTM in PSNR on the static scenes of the ATSyn benchmark and at most 0.5 dB below it on the dynamic scenes.


Speed on Jetson

TensorRT engine-only throughput in MAXN power mode, twelve-frame clips.

30 FPS camera rate

  • MambaTM FP32, our engineAGX Orin, 1024 × 10241.6 FPS
  • FAST-TM-0 FP32AGX Orin, 1024 × 10246.3 FPS
  • FAST-TM-0 mixedAGX Orin, 1024 × 102413.4 FPS
  • FAST-TM FP32AGX Orin, 1024 × 102422.8 FPS 14.2× faster
  • FAST-TM mixedAGX Orin, 1024 × 102447.1 FPS 29.4× faster
  • FAST-TM-0 mixedAGX Orin, 1280 × 72015.2 FPS
  • FAST-TM mixedAGX Orin, 1280 × 72053.4 FPS
  • FAST-TM-0 mixedOrin NX, 1024 × 10245.2 FPS
  • FAST-TM mixedOrin NX, 1024 × 102416.1 FPS
  • FAST-TM-0 mixedOrin NX, 1280 × 7206.0 FPS
  • FAST-TM mixedOrin NX, 1280 × 72020.5 FPS
MambaTM: our TensorRT engine of the released model, with a custom FP32 scan plugin, measured only on AGX Orin at 1024 × 1024. Speed-ups depend on this implementation; a faster plugin would reduce them. Mixed precision uses FP16 except GroupNorm and channel L2 normalization in FP32. FAST-TM-0 is our first design without scans. DATUM and TMT were not ported to Jetson.

25.2× lower energy per frame

Mixed vs FP32 · 12.0× at equal FP32

  • MambaTM FP3226.5 J/frame
  • FAST-TM FP322.21 J/frame
  • FAST-TM mixed1.05 J/frame

9.2× lower execution-context memory

Mixed vs FP32 · 3.2× at equal FP32

  • MambaTM FP3210.8 GB
  • FAST-TM FP323.33 GB
  • FAST-TM mixed1.18 GB
Jetson AGX Orin, 1024 × 1024 input. MambaTM: our FP32 TensorRT engine. Energy includes idle power.

Speed against quality

Figure 1 from the paper. Static restoration quality against engine time on Jetson AGX Orin: FAST-TM takes 526 milliseconds in FP32 and 255 milliseconds in mixed precision per twelve-frame clip; our MambaTM FP32 engine takes 7486 milliseconds. The dashed line marks a camera rate of thirty frames per second.
Engine time: AGX Orin, 1024 × 1024 input. Quality: PyTorch, native 512 × 512 frames. The twelve-frame quality protocol favours FAST-TM. Filled/hollow markers: FP32/mixed, sharing the PyTorch PSNR. MambaTM uses our FP32 scan plugin and has the highest PSNR on dynamic scenes.

Quality on ATSyn

Same twelve-frame input for all methods. Static: native 512 × 512 frames. Dynamic: whole frames mirror-padded to 1024 × 1024, scored only on the native region.

Results table
Method Static PSNR (dB) Static SSIM Dynamic PSNR (dB) Parameters (M) Orin FPS, FP32
TMT 23.777 0.7114 27.245 22.95 not ported
DATUM 24.340 0.7395 27.842 5.78 not ported
MambaTM 24.149 0.7399 28.175 7.239 1.6
FAST-TM 24.901 0.7559 27.822 5.452 22.8

Quality on ATSyn and engine speed on Jetson AGX Orin at 1024 × 1024, twelve-frame inputs. Best quality and speed in bold.

MambaTM speed: our TensorRT engine, with its scan in a custom FP32 plugin.

Evaluated together on identical noisy inputs: the first twelve static frames and centre twelve dynamic frames. Gaussian-noise variance is up to 4 × 10⁻⁴ and 10⁻⁴, respectively. TMT uses official overlapping 288 × 288 tiles. These conditions differ from the baselines’ published protocols.

FAST-TM is trained on twelve-frame clips, distilled from MambaTM, and fine-tuned at the static test resolution. The baselines use their released checkpoints, trained on longer clips. This protocol favours FAST-TM, so the static gain of 0.752 dB over MambaTM does not isolate the architecture. The turbulent input scores 21.192 dB on the static set.

From scans to parallel mixers

Our TensorRT implementation of MambaTM scans the clip sequentially in a custom FP32 plugin. FAST-TM replaces recurrent mixing with spatial attention, temporal attention and local 3D convolution.

FAST-TM architecture. (a) An encoder takes the turbulent clip down to one eighth of its resolution; three mixing groups, each with a spatial, a temporal and a local block, mix the twelve frames; a latent phase distortion branch guides the third group; a decoder with shared weights restores the clip. (b) Inside a group: the spatial block uses spatial-reduction attention, the temporal block self-attention over time, and the local block a depthwise 3D convolution, each followed by a feed-forward or channel-attention step.
Architecture of FAST-TM (Figure 2 of the paper).

Within each group: spatial → temporal → local (3D convolution, then channel attention).

MambaTM · our TensorRT scanone forward scan
FAST-TMpositions processed in parallel
Space-first scan → spatial attention
Each frame uses queries at every feature position and keys and values reduced by a learned strided convolution. Shading shows reduction regions: two by two here, eight by eight in the model. Attention weights distinguish regions, not individual distant positions.
Time-first scan → temporal attention
Each position attends across the twelve frames independently. The weights depend on the position; they do not track content moving between positions.
Local Hilbert scan → local 3D convolution
A depthwise 3D convolution mixes a three-by-three-by-three neighbourhood, with zero padding at the borders. It preserves local mixing, not the Hilbert scan’s long-range path.
Channel attention (added)
Each head estimates channel weights from the whole frame, then mixes value channels at each position. It does not transfer values between distant positions. The projected values include local spatial filtering, omitted here.
Nominal mixer support on toy feature grids. Diagrams show unguided operator cores; normalization, projections and residual paths are omitted. The third group uses LPD features for spatial and temporal attention keys and values. Matrix cells mark allowed connections, not learned weights. Animations show computation structure, not elapsed time or GPU scheduling. Each MambaTM order also runs in reverse.

Where the time went

  • MambaTM with scans7486 ms
  • FAST-TM-0 mixers replace the scans; groups six to two; 3.9× faster1905 ms
  • FAST-TM encoder and decoder below input resolution; groups two to three; 3.6× faster526 ms

FAST-TM-0 time by kernel type (AGX Orin, FP16 profile)

  • Full-resolution convolutions, 16 channels: 36 %
  • Other convolutions: 21 %
  • Concatenations and format copies: 21 %
  • Separate GELU and GroupNorm: 14 %
  • Attention: 5 %
  • Other: 3 %
AGX Orin, 1024 × 1024: engine times in FP32; kernel profile in FP16. MambaTM uses our custom FP32 scan plugin. The designs also change the mixing-group count (six, two, three), so these steps do not isolate individual changes.

Limitations

Short windows, by design

Longer windows increase buffering and computation; temporal attention maps grow quadratically. At 30 FPS, estimated buffering plus engine inference delays the first frame by about 0.6 s (AGX Orin, 720p, mixed precision), excluding video I/O.

Window size and attention-map entries at 1280 × 720

  • 12 frames11.1 million pixels144 entries
  • 24 frames22.1 million pixels576 entries
  • 36 frames33.2 million pixels1296 entries

Pixels per windowAttention-map entries per position and head

Dynamic scenes

Under the twelve-frame protocol, FAST-TM is 0.47 dB below MambaTM on cropped clips and 0.35 dB below it on whole frames.

Real footage

The real long-range clip has no ground truth. The laboratory and synthetic clips include a reference. These selected examples support visual comparison, not an aggregate evaluation.

Evaluation scope

Each model comes from one training run. We do not quantify consistency across clip boundaries or isolate the contribution of the LPD guidance.

Deployment scope

The Orin NX video pipeline reaches 18.9 FPS, below camera rate. Agreement between mixed-precision engines and PyTorch is checked on one clip, not the test splits.

BibTeX

@unpublished{fasttm,
  title = {FAST-TM: Factorized Attention over Space and Time for Video Turbulence Mitigation},
  author = {Anonymous},
  note = {Manuscript}
}