Explore the project ↓
Fusion, guided by a protocol.
Specify what to preserve. Learn how to instruct.
Fuse complementary evidence with frozen generative models.
Environment state cards, grounding principles and a style profile.
Generate, evaluate and revise offline with multimodal evidence.
Keep the learned protocol fixed. Adapt the prompt to each image pair.
Read the abstract
Infrared and visible image fusion integrates complementary information from both modalities to provide a more comprehensive representation of the scene. We propose ProtoFusion, a protocol-guided generative framework for infrared and visible image fusion. Instead of relying on fixed fusion operators, hand-crafted objectives, or manually specified one-shot prompts, ProtoFusion formulates fusion as protocol-constrained generation. It represents fusion intent with a structured protocol, consisting of environment state cards, grounding principles, and a style profile, which explicitly specifies which evidence should be preserved or suppressed and which properties the fused result should satisfy.
To optimize this protocol, we introduce protocol-guided prompt refinement, a closed-loop learning framework that iteratively performs generation, evaluation, and revision under multimodal evidence. Once learned, the optimized protocol is fixed at inference time and directly guides fusion generation. To address residual source inconsistencies, we further incorporate structured belief construction, which organizes modality reliability, scene evidence, and cross-modal conflicts into a structured representation for conservative correction after generation. Extensive experiments on standard benchmarks show that ProtoFusion improves quantitative performance, achieves favorable results in aesthetic evaluation, and produces visually coherent fusion outputs.
Framework at a glance
Across challenging conditions.
Preserve thermal targets and visible structure—from low light and glare to daylight and overexposure.
10 scenes · night to daylight
Select a scene belowView the complete comparison figures
What does infrared contribute?
Keep the visible image fixed. Compare no infrared, unrelated infrared and the original paired infrared.
Eight examples under the same optimized protocol and generation settings. Unrelated infrared can introduce unsupported targets and structures; paired infrared supplies scene-specific evidence. All highlights are retained from the original figure.
View the full input ablation
Refine the instruction.
Inspect the generation.
PGPR closes the loop between generation, multimodal evaluation and structured protocol revision.
Three displayed candidates per example, with the corresponding prompt excerpts from Figure 7. Candidate indices denote display order; the figure does not identify the accepted rounds or the final retained output.
How much refinement?
FMB · UniPercept
All 280 FMB test pairs; base PGPR without SBC. T = 6 leads in IQA and Overall; T = 9 gives marginal gains in IAA and ISTA. The highest value for the selected metric is highlighted.
What belongs in the protocol?
Component ablation on all 280 FMB test pairs. Full PGPR uses T = 6 without SBC; the initial and optimized protocols are different experimental settings.
Why a structured protocol?
Open-ended VLM prompt optimization starts from the human prompt. PGPR instead refines explicit protocol slots through source-grounded diagnosis and validated edits.
FMB · UniPercept
VLM-optimized prompting improves IAA but lowers IQA and ISTA relative to the human prompt. PGPR improves all three; optional SBC further improves the reported scores.
Full prompt optimization and SBC results
A second look
at the evidence.
SBC organizes source reliability and cross-modal conflicts to make a conservative, scoped correction after generation.
Compare base PGPR with PGPR + SBC in four examples. SBC performs at most one additional generation; when no valid correction is available, the prior output is retained.
Fewer errors. Residual errors remain.
Comprehensive human review of all 280 FMB test pairs. Only definite errors enter the numerator. Categories can overlap; “Any error” counts each affected image once. These measurements describe the reviewed test set, not a guarantee of source fidelity.
View the complete SBC examples and error table
One protocol. Two stages.
Learn the protocol offline, then keep it fixed while composing a new prompt for each infrared–visible pair.
Inside the protocol
Three complementary slots make the fusion intent explicit. The excerpts below come from the initial protocol provided in the manuscript appendix.
What can change during refinement?
PGPR can add an environment item, revise a grounding principle or add a grounding principle. The style profile stays fixed. Invalid edits leave the incumbent unchanged; worse or fatal candidates are rejected.
The retained critic score is non-decreasing by selection. This is a property of the selection rule and does not imply monotonically improving true image quality. The final optimized protocol is not included in the current manuscript.
Original method diagrams
Protocol analysis: factors and control allocation
A shared protocol.
Three benchmarks.
A single protocol, refined on FMB training data, is evaluated on the full test sets of FMB, MSRS and RoadScene.
UniPercept · FMB · Overall
Higher is better ↑ProtoFusion uses PGPR + SBC on all three benchmarks. UniPercept reports image aesthetics (IAA), image quality (IQA), structure and texture (ISTA) and its aggregate Overall score. All methods and values are retained below.
Full UniPercept results
Full image-quality results · nine metrics
Directions are shown in each column. Bold marks the best result; an underline marks second best. Values retain the manuscript’s four-decimal precision.
Full source-referenced results · CE, Q_AB/F and VIF
Thirteen methods on FMB, MSRS and RoadScene. Lower CE and higher Q_AB/F and VIF are better. Bold marks the best result; an underline marks second best.
Downstream semantic segmentation
A frozen CLIPer evaluates the fused images on the FMB test set. The manuscript reports eight classes, with mean IoU and mean accuracy summarized below.
Road · Building · Sky · Car
Truck · Bus · Vegetation · Pole
Full segmentation results · eight reported classes
One protocol.
Different generators.
Four qualitative examples reuse the same fixed protocol with Qwen-Image-Edit and GPT Image 2.
Original image proportions are preserved. These examples illustrate qualitative transfer; they are not a matched quantitative comparison of the generators.
View the full cross-generator figure
Runtime configurations
~188sQwen-Image-Edit configuration
~9sQwen-Image-2.1-Turbo configuration
Reported end-to-end base-inference times under different runtime configurations. Offline protocol learning and optional SBC are excluded. The manuscript does not provide a matched quality evaluation for these two configurations.