Less is More: Token-Efficient Video-QA via Adaptive Frame-Pruning and Semantic Graph Integration

TL;DR. Even state-of-the-art keyframe selectors return prompts full of near-duplicate frames — a video-specific redundancy we call “visual echoes”. Our training-free, plug-and-play refinement layer prunes these echoes with Adaptive Frame-Pruning (AFP) and compensates with a lightweight textual semantic graph, cutting input tokens by up to 82.2% while improving the accuracy of the upstream selector.

82.2%
fewer total input tokens
Training-free
plug-and-play & model-agnostic
Higher accuracy
with far fewer frames

Abstract

The practical application of Multimodal Large Language Models (MLLMs) to Video Question Answering (Video-QA) is severely hindered by the high token cost of processing numerous video frames. While keyframe selection is the dominant strategy for mitigating this, we identify a critical flaw: even state-of-the-art selectors produce prompts suffering from significant temporal redundancy, a challenge unique to video that we term “visual echoes”. This issue leads to context dilution and can paradoxically degrade performance. To address this dual challenge, we propose a novel refinement framework that synergistically combines Adaptive Frame-Pruning (AFP) with a lightweight text-based semantic graph. AFP intelligently prunes “visual echoes” by adaptively clustering frames, while the semantic graph provides crucial, low-cost semantic compensation. Conducting extensive experiments on the LongVideoBench and Video-MME benchmarks against multiple state-of-the-art selectors, our approach demonstrates a drastic reduction in total input tokens by up to 82.2%. Crucially, by creating a concise, high-quality prompt, our framework not only enhances efficiency but also demonstrates a remarkable ability to robustify and improve the accuracy of upstream selectors, achieving results that are highly competitive with, and often superior to, baselines that use vastly more frames.

Method

Our framework is a universal refinement module that sits after any upstream keyframe selector (e.g., AKS, TStar, VSLS). It refines the selector’s output through two complementary stages:

The overall pipeline of AFP: an upstream selector provides initial keyframes; AFP performs fused feature extraction (CLIP + ResNet), adaptive hierarchical clustering (KDE-based threshold), and representative-frame selection; a concurrent LLM builds a textual semantic graph; the pruned frames plus the graph form a compact prompt for the downstream VLM. — **Figure 1. The overall pipeline.** An upstream selector provides initial keyframes. Our **AFP** module then performs (1) fused feature extraction (CLIP + ResNet) and adaptive clustering to produce representative keyframes, and (2) concurrent semantic-graph generation via a lightweight LLM call. Both are combined into an optimized prompt (**4 keyframes + graph ≈ 390 tokens**) for the downstream VLM.

Stage 1 · Pruning

Adaptive Frame-Pruning (AFP)

Adaptively clusters frames and keeps one representative per cluster, collapsing temporally redundant “visual echoes” into a compact, non-redundant set.

Stage 2 · Compensation

Textual Semantic Graph

Injects a lightweight, text-attributed semantic graph directly into the prompt — no GNN required — restoring relational cues at minimal token cost.

Inside AFP — three steps:

Fused Feature Extraction. Each frame is encoded with a ResNet-50 backbone (appearance) and a CLIP ViT-B/32 encoder (semantics), projected into a shared 512-d space and blended:
$f_{\text{fused}} = (1-\alpha)\, f_{\text{ResNet}} + \alpha\, f_{\text{CLIP}}$
Adaptive Hierarchical Clustering. Agglomerative clustering under a combined distance fusing feature similarity and temporal proximity, with a dynamic distance threshold (not a fixed cluster count) so pruning adapts to each video’s redundancy:
$D(i,j) = \beta \cdot \text{cos\_dist}(f_i, f_j) + (1-\beta)\cdot \text{temporal\_dist}(t_i, t_j),\quad \beta = 0.9$
Representative Selection. One representative frame is kept per cluster, yielding a compact visual set (e.g., 32 frames → 4 frames).

Why “less is more”. By removing visual echoes, AFP directly counters context dilution: the MLLM is no longer distracted by many near-identical frames, so the semantic graph’s compensation and the remaining representative frames carry more signal per token — reducing tokens by up to 82.2% while robustifying and improving upstream selectors.

Key Contributions

🔎

Defining “visual echoes”

We formally define and demonstrate the prevalence of “visual echoes” in video, attributing the “less is more” paradox in Video-QA to this redundancy and to context dilution.

🧩

A universal refinement module

A two-pronged framework combining AFP with a lightweight text-based semantic graph — a plug-and-play layer for any upstream keyframe selector.

📊

Extensive validation

Across LongVideoBench and Video-MME, over multiple backbones and selectors, we drastically cut token cost while frequently improving accuracy.

⚡

Training-free & efficient

Off-the-shelf encoders, no fine-tuning, a single non-iterative refinement pass — negligible overhead, drastically lower visual token load.

Evaluated across backbones including GPT-4o and Qwen2.5-VL-7B, on top of upstream selectors such as AKS, TStar, and VSLS. See the paper for full comparisons, ablations, and efficiency analysis.

Citation

@inproceedings{wang2026less,
  title={Less is more: Token-efficient video-qa via adaptive frame-pruning and semantic graph integration},
  author={Wang, Shaoguang and Guo, Weiyu and Chen, Ziyang and Xu, Yijie and Hu, Xuming and Xiong, Hui},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9856--9866},
  year={2026}
}