CertViT

Certifying pre-trained vision transformers by Lipschitz bounding — and measuring where certification stops working.

Certifying a Model You Did Not Train

Certified robustness normally assumes you own the training run: you regularize the Lipschitz constant from scratch and pay for it in accuracy. That assumption does not hold for the models people actually deploy, which arrive pre-trained.

CertViT takes ImageNet-pretrained ViT, DeiT, and Swin transformers, from 5M to 300M parameters, and constrains their Lipschitz bounds after the fact using a Douglas-Rachford proximal projection. The projection alternates between a proximal step that shrinks per-layer Lipschitz bounds and a step that keeps the network close to its pre-trained weights, so the model is certified without training from scratch.

The Negative Result

Certified accuracy is non-zero only for the tiny variants — ViT-T/16 and DeiT-T. It is 0% for ViT-S, ViT-B, ViT-L, DeiT-S, DeiT-B, and every Swin variant.

The reason is structural, not a tuning failure: the certificate rests on a global Lipschitz bound obtained by composing per-layer bounds, and that composition loosens multiplicatively with depth. Past a certain depth and parameter count the bound is so loose that the certified radius collapses to zero, whatever the projection does to the weights.

What It Implies

The limit is the bound, not the optimizer. Because the looseness compounds with depth, a better projection or a longer schedule does not recover a non-vacuous certificate at these sizes — the global Lipschitz bound is simply too pessimistic once the network is deep enough. Deployed models are typically orders of magnitude larger than the variants this method certifies non-vacuously.

Published at the AdvML Frontiers Workshop, ICML 2023, with Kavya Gupta.