EVA

Editing for Versatile Alignment
against Jailbreaks

IEEE TPAMI 2026

Yi Wang 1 Hongye Qiu 1 Yue Xu 1 Sibei Yang 2 Zhan Qin 3 Minlie Huang 4 Wenjie Wang 1†

1ShanghaiTech University, ASPIRE Lab

2 Sun Yat-sen University 3 Zhejiang University 4 Tsinghua University

†Corresponding author.

Paper Code IEEE TPAMI Hugging Face

Impact

Safety alignment through model editing.

Jailbreaks can exploit both textual and visual inputs. EVA treats safety alignment as a targeted model editing task: it identifies harmful key representations, optimizes safe target values, and updates selected MLP weights. The framework applies to both LLMs and VLMs, with regularization to limit changes on benign inputs.

Experiments show reduced jailbreak success with largely retained utility across text and multimodal benchmarks.

Before and after EVA

InternVL3.5-8B: safety and general utility before and after EVA.
Metric Before After EVA
MM-SafetyBench · ASR (%) ↓
SD
14 3
TYPO
48 6
SD+TYPO
48 3
General utility · Score ↑
MM-Vet-v2 69.1 69.3
MMMU 57.6 57.6
MMStar 68.3 67.5

Tables 3 & 5 · SD: harmful images generated by Stable Diffusion; TYPO: harmful text in images (OCR attacks); SD+TYPO: combines both.

Overview

Where does EVA fit?

EVA applies safety edits during downstream adaptation, after foundational alignment. The updated weights are then deployed in place of the original weights.

EVA in the Model Lifecycle

EVA in the model lifecycle: a model released after foundational alignment is vulnerable to a jailbreak, and localized safety editing changes its harmful response into refusal during downstream adaptation.

Why Use Targeted Model Editing After Release?

A released model can encounter jailbreaks that its initial safety alignment did not anticipate. Broad safety fine-tuning can alter useful capabilities, while external defenses add components to the inference pipeline. EVA instead uses localized parameter edits to modify the model’s response to harmful concepts without adding inference-time defense components.

Methodology

How Does EVA Edit a Model?

EVA editing pipeline: extract harmful text tokens, generate query variations, select related image tokens through attention, aggregate representative keys, optimize safe values, and update MLP down-projection weights.

From Harmful Keys to Safe Values

EVA identifies harmful textual and visual key representations, optimizes target values that favor safe refusals, and writes the new mappings into selected MLP down-projection weights.

Together, these steps incorporate targeted safety corrections directly into the model weights. Once editing is complete, the updated model can be deployed without an additional inference-time defense module.

Results

How Does EVA Balance Safety and Utility?

Compare model performance

Visual jailbreak attacks · ASR ↓

Table 3
Visual jailbreak attacks · ASR ↓. Table 3.
Method HarmBench AdvBench JailbreakBench MaliciousInstruct
FigStepADV-16HadesFigStepADV-16HadesFigStepADV-16HadesFigStepADV-16Hades
Vanilla 32% 15% 19% 26% 18% 12% 4% 8% 56% 7% 17% 47%
JailGuard 16% 8% 13% 15% 7% 2% 2% 7% 26% 3% 10% 29%
AdaShield-A 10% 9% 1% 9% 0% 4% 0% 0% 31% 4% 0% 12%
VLGuard 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0%
EVA (text) 3% 0% 1% 1% 0% 2% 0% 0% 2% 1% 0% 5%
EVA (Ours) 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%

Multimodal safety benchmarks · ASR ↓

Table 3
Multimodal safety benchmarks · ASR ↓. Table 3.
Method MM-SafetyBench MultiTrust
SDTYPOSD + TYPOTypographicMultimodalCross-modal
Vanilla 11% 40% 51% 3% 35% 10%
JailGuard 9% 30% 37% 2% 25% 8%
AdaShield-A 5% 27% 27% 0% 6% 5%
VLGuard 2% 11% 11% 1% 8% 10%
EVA (text) 4% 19% 22% 0% 14% 0%
EVA (Ours) 1% 0% 0% 0% 0% 0%

Attack success rate (ASR, %), lower is better. ADV-16 images are optimized against LLaVA-1.5-7B and transferred to the other VLMs. EVA (text) uses only textual keys for editing.

Across five LLMs and three VLMs, EVA substantially reduces jailbreak ASR under diverse textual and visual attacks while retaining most general task utility.

How Does Visual Token Selection Affect Robustness?

We compare edits built from scene content, OCR text, and composite inputs. Select a variant to inspect its permitted visual token region and measured attack success.

Which image regions matter?

Editing image
Figure 8 editing image: EVA · composite editing

Highlighted: permitted visual token selection region.

EVA · composite editing

Use composite editing images and allow visual token selection across both scene and OCR regions.

Original model Selected variant ASR (%) ↓
SD 14% → 3%
TYPO 48% → 6%
SD+TYPO 48% → 3%
Compare all editing variants Table 8
Visual editing variants on MM-SafetyBench. Table 8.
MethodSD ↓TYPO ↓SD+TYPO ↓
Vanilla 14% 48% 48%
EVA 3% 6% 3%
EVA (Image-Focused) 6% 17% 7%
EVA (Image-Only) 3% 36% 16%
EVA (TYPO-Focused) 10% 4% 27%
EVA (TYPO-Only) 13% 2% 39%

MM-SafetyBench ASR (%), lower is better. Focused variants use composite images with restricted token selection; Only variants use images containing either scene content or rendered text.

How Many Visual Tokens Should Be Selected?

Selecting one visual token reduces ASR from 48% to 3% while keeping MM-Vet-v2 at 69.3, compared with 69.1 for the original model. Selecting more tokens lowers ASR further but reduces utility in this ablation.

InternVL3.5-8B

Visual token selection
MM-SafetyBench ASR ↓

3%

Original 48%

MM-Vet-v2 score ↑

69.3

Original 69.1

Critical layers Figures 3 & 4

Layer sweeps identify layer 8 for Qwen2.5-VL-7B and layer 9 for InternVL3.5-8B as providing the best safety utility trade-offs. Both match the critical layers of their LLM backbones.

Qwen2.5-VL layer sweep comparing attack success and MM-Vet-v2 utility, with layer 8 marked.
InternVL3.5 layer sweep comparing attack success and MM-Vet-v2 utility, with layer 9 marked.
Do edits transfer to other harmful behaviors? Figures 5 & 6

We edit one HarmBench behavior category at a time and evaluate all six categories. On Llama2-7B-chat under GCG and LLaVA-1.5-7B under Hades, the heatmaps show reduced attack success on categories not used for editing.

Cross-behavior ASR heatmap on Llama2-7B-chat under GCG. Rows denote evaluation categories and columns denote editing categories.
Llama2-7B-chat · GCG
Cross-behavior ASR heatmap on LLaVA-1.5-7B under Hades. Rows denote evaluation categories and columns denote editing categories.
LLaVA-1.5-7B · Hades

Columns denote editing categories, and rows denote evaluated categories. Off-diagonal cells show transfer to other categories; original ASR values appear in parentheses. ASR is shown on a 0–1 scale (lower is better).

What do the key representations show? Figure 7

We visualize harmful key representations at the selected editing layers using PCA. Textual keys from Llama2 and visual keys from Qwen2.5-VL show substantial overlap across behavior categories and datasets, suggesting shared representation patterns among harmful inputs.

PCA projections of Llama2 harmful text token keys, colored by behavior category.
Llama2 · text token keys across behaviors
PCA projections of Llama2 harmful text token keys, colored by dataset.
Llama2 · text token keys across datasets
PCA projections of Qwen2.5-VL harmful image token keys, colored by behavior category.
Qwen2.5-VL · image token keys across behaviors
PCA projections of Qwen2.5-VL harmful image token keys, colored by dataset.
Qwen2.5-VL · image token keys across datasets

Editing Cost and Deployment

Training time ↓

0.8 h

EVA requires 0.8 hours of training on InternVL3.5-8B, compared with 2.0 hours for VLGuard and 1.2 hours for AdaShield-A.

Single NVIDIA A40 GPU · Averaged over five runs.

JailGuard —
AdaShield-A 1.2 h
VLGuard 2.0 h
EVA 0.8 h

Training time from Table 9. "—" indicates no training is required. Lower is better.

Citation

If you find EVA useful for your research, please consider citing our paper:

@ARTICLE{wang2026eva,
  author={Wang, Yi and Qiu, Hongye and Xu, Yue and Yang, Sibei and Qin, Zhan and Huang, Minlie and Wang, Wenjie},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  title={EVA: Editing for Versatile Alignment Against Jailbreaks},
  year={2026},
  volume={48},
  number={10},
  pages={11776-11791},
  keywords={Automatic speech recognition;Modeling;Safety;Visualization;Large language models;Conferences;Optimization;Educational institutions;Light emitting diodes;Tuning;Safety alignment;jailbreak attacks;model editing;large language models;vision language models},
  doi={10.1109/TPAMI.2026.3694189}}