Semgrep tested GLM 5.2, an open-weight model from Zhipu AI, on their internal benchmark for detecting IDOR vulnerabilities (Insecure Direct Object Reference, access control issues). The model achieved an F1 score of 39%, beating Claude Code (32%) and even Claude Opus 4.8, at a cost of roughly $0.17 per vulnerability found. The test used just a direct prompt, without the infrastructure (harness) that normally accompanies these systems in production.

Why it matters: this is the first time an open-weight model has beaten a proprietary frontier model on a vertical security task under controlled conditions. If you work with AI for code analysis, security review automation, or vulnerability triage, it means open-weight options are becoming competitive on specialized tasks too, not just generic benchmarks. You can run them in-house, control them, and pay less — without sacrificing much accuracy compared to closed models.

Semgrep’s multimodal pipeline (combining the model with a custom-built harness) still reaches 53–61% F1, so the surrounding infrastructure still matters. But the gap is closing, and for anyone building internal automated security tools, having a performant model you can self-host changes the game.

Where to look: Semgrep’s post explains how they structured the benchmark and what their multimodal pipeline does. If you want to replicate the test or learn how to build a similar harness for your security tasks, that’s your primary source.