We Pointed Helix Cyber at OWASP Juice Shop
Oct 29, 2026
A real Helix Cyber engagement, end to end: a gray-box web pentest against OWASP Juice Shop, run by open-weight models on self-hosted inference. Then the same job on two models, GLM-5.3-Flash and DeepSeek-V4.1-Flash: 67 and 58 independently verified findings, and each one caught serious bugs the other missed.
We ordered the same gray-box pentest twice from Helix Cyber, once per open-weight model, against OWASP Juice Shop. GLM-5.3-Flash confirmed 67 findings and DeepSeek-V4.1-Flash confirmed 58. Each model caught serious bugs the other missed, including an anonymous feed of every password hash that only one of them found.
Here is how the engagement ran, what it found, and how the two models differed.
Juice Shop is built to be broken, so it is a fair test of whether a tool finds things. It is a poor stand-in for a real production app. Read the numbers as a coverage check and a model comparison, not a league table.
An order, not a script
We ordered a web engagement with the gray box package through the customer portal, with two ordinary customer accounts: the fastest way to find broken access control is to log in as one user and read the other's data. The backend refuses to start anything until a customer admin signs the rules of engagement, which are bound to the exact host and test window.
How the test runs
- 1Recon, twiceUnauthenticated, then as each account. Here: 60 API entries, 44 app routes and a 53-row role matrix.
- 2Parallel lanesConfig, authentication, session, authorization, business logic, injection and XSS, SSRF, client-side: one agent and one sandbox each, inside a one-origin scope and a request budget.
- 3Independent verificationThe agent that files a candidate never confirms it. A separate verifier replays it from scratch with fresh sessions and argues the case against it.
- 4Escalation chainsVerified findings are joined into attack paths, so two Lows that combine into account takeover are reported as the chain.
The first report
The first GLM-5.3-Flash run filed 44 candidates, and 33 made the report (2 Critical, 9 High, 14 Medium, 8 Low or Info), each with replay evidence. The Criticals were forged alg=none admin tokens, and a credential table any customer can read. The second one did not survive a closer look; see Same bug, different severity.
The report also named what it could not reach: the login rate-limiter's keying, the socket.io protocol past the handshake, and the external OAuth flow. Coverage gaps a scanner never reports are the part an auditor actually reads.
Two models, one job
The whole roster runs on an open-weight model on our own inference; nothing goes to a frontier API. So we ran the engagement on both GLM-5.3-Flash and DeepSeek-V4.1-Flash.
DeepSeek's first pass looked weak: 9 findings in 20 minutes. That was our fault, not the model's. Only ten sandboxes could run at once, so most lanes never started. We fixed it (runs now queue at a higher cap) and reran both models on equal footing: same target, same accounts, same instructions, same comprehensive scope.
- GLM-5.3-Flash2 · 9 · 14 · 6 · 233
- DeepSeek-V4.1-Flash · our sandbox limit, not the model1 · 2 · 3 · 3 · 09
- GLM-5.3-Flash3 · 13 · 24 · 18 · 967
- DeepSeek-V4.1-Flash4 · 12 · 19 · 21 · 258
Numbers read Critical · High · Medium · Low · Info, then the total. Bars share one scale.
On the fixed infrastructure, DeepSeek went from 9 findings to 58, with more Criticals than GLM (4 vs 3). GLM confirmed 67, mostly by going wider on Medium and Info. Each took a working day of wall clock, with interruptions: the shared target went down mid-run once, and sandboxes were reaped once.
The totals hide the interesting part. Matching the two reports by hand, roughly 40 issues appear in both. About 10 appear only in DeepSeek's report and about 25 only in GLM's, so neither is a superset of the other:
The signature issues both reruns caught, then the most severe ones only one model reported. Severity is each model's own rating.
| Issue | DeepSeek-V4.1 | GLM-5.3 |
|---|---|---|
| SQL injection in login → admin session | Critical | Critical |
| Unsigned (alg:none) JWTs accepted | Critical | Critical |
| SQL injection in product search | High | Critical |
| Credential table readable by any signed-in customer | Medium | Critical |
| Sign up as admin (mass assignment) | Critical | Medium |
| Anonymous feed of password hashes + 2FA seeds | Critical | not found |
| Pre-2FA token creates administrators | High | not found |
| Erasure request filed against another user | High | not found |
| KeePass vault in the public /ftp folder | High | Medium |
| Another user's address taken over | not found | High |
| Password change trusts the hash inside the JWT | not found | High |
| AI chat trusts client-supplied identity | not found | Medium |
Both caught the signature issues: login SQL injection, forged JWTs, the exposed file server. Beyond that they diverged. DeepSeek dug into authentication and found an anonymous endpoint leaking every password hash and 2FA seed, plus a pre-2FA token that can create administrators. GLM spread across more surfaces: address takeover, a password change that trusts the JWT, and the AI chat endpoint.
Same bug, different severity
The two models disagreed on the severity of five bugs both found. We checked each one against the evidence in both reports and against Juice Shop's source. Our rule: rate what was shown on this target, and name what would raise it.
| Issue | GLM | DeepSeek | Our call | Why |
|---|---|---|---|---|
| SQL injection in product search | Critical | High | Critical | Unauthenticated, and GLM pulled the whole user table in one request. DeepSeek's own impact text says “any table”. Its rating undersold its evidence. |
| Sign up as admin (mass assignment) | Medium | Critical | Mediumhere | Juice Shop's server never checks the admin role; it only hides the admin page in the browser. DeepSeek claimed full admin control but showed nothing a customer can't already do. GLM's tests hit routes that refuse every role, real admins included, so neither model proved the impact. Critical in any app that enforces admin on the server. |
| Credential table open to any customer | Critical | Medium | Medium | The source masks both the password and the 2FA seed in this response. GLM rated what might be in the table, not what came back. |
| KeePass vault in public /ftp | Medium | High | High | Neither cracked it. DeepSeek checked the key derivation (one round, not 100,000+), which makes offline guessing cheap. GLM stopped after 1,287 guesses. |
| Paid tier without payment | Medium | High | Mediumhere | A free upgrade with no money on this demo shop. High on a product that charges. |
- SQL injection in product searchGLM CriticalDeepSeek HighOurs Critical
Unauthenticated, and GLM pulled the whole user table in one request. DeepSeek's own impact text says “any table”. Its rating undersold its evidence.
- Sign up as admin (mass assignment)GLM MediumDeepSeek CriticalOurs Mediumhere
Juice Shop's server never checks the admin role; it only hides the admin page in the browser. DeepSeek claimed full admin control but showed nothing a customer can't already do. GLM's tests hit routes that refuse every role, real admins included, so neither model proved the impact. Critical in any app that enforces admin on the server.
- Credential table open to any customerGLM CriticalDeepSeek MediumOurs Medium
The source masks both the password and the 2FA seed in this response. GLM rated what might be in the table, not what came back.
- KeePass vault in public /ftpGLM MediumDeepSeek HighOurs High
Neither cracked it. DeepSeek checked the key derivation (one round, not 100,000+), which makes offline guessing cheap. GLM stopped after 1,287 guesses.
- Paid tier without paymentGLM MediumDeepSeek HighOurs Mediumhere
A free upgrade with no money on this demo shop. High on a product that charges.
GLM's rating held on three of the five and DeepSeek's on two, and neither model was simply harsher. The disagreements came from evidence: a test that could not tell the roles apart, a key-derivation check one model ran and the other skipped, an impact claim nobody demonstrated. That is why every finding carries its severity rationale, and why a human reviews it before it ships.
- GLM-5.3-Flash67 of 89 confirmed · 16 rejected · 6 to a human
- DeepSeek-V4.1-Flash58 of 60 confirmed · 2 rejected
The verifier earned its keep. It rejected 16 of GLM's claims, four of them re-filed from an earlier run that no longer reproduced, and sent six to a human. DeepSeek filed fewer claims, and nearly all of them held up.
Our takeaway: both models are capable of a real assessment, and neither is a superset of the other. Running both and merging behind one independent verifier finds more than either alone.
What this shows, and what it doesn't
It shows that open-weight models on self-hosted inference can run a multi-agent web assessment end to end, verify every finding, and write it up with coverage gaps included. The code, traffic and findings never left infrastructure we control.
It does not show that we beat anyone. Juice Shop is designed to be found. What survives contact with a real production target is the discipline: a second agent reproduces every finding, the report states what was not tested, and a human merges the fixes.
Want the same pointed at your own application? That is Helix Cyber, from web app and API assessments on up.
Appendix: the engagement, in portal terms
- Type, package: web, gray box.
- Scope: one signed host (OWASP Juice Shop v20.3.0), two customer accounts; the container host was out of scope.
- Authorization: rules of engagement signed before provisioning; detect-only; independent verification required for every finding.
- Delivery: one Helix project per engagement, agents in sandboxes, findings synced to the portal as tickets, report rendered from the structured records.