Episode 0066: Equally Unproven, Unequally Inspectable: Jev versus Needle 3

Why it matters. Two companies shipped a calibrated typed-decision product within seventy-two hours of each other, and the obvious episode about them is not available: there is no shared benchmark between TypeSafe's Jev and Cactus Compute's Needle 3, and their ground truths are not even the same kind of object, one being agreement with two competitors' frontier models and the other exact match against human-authored gold labels. So we run the comparison on the openness axis instead. Both products are essentially unproven in the same specific way, because neither company has published a single calibration measurement for the property both of them are selling. But only one of them can be checked at all, and this episode is what happens when you check it: Needle's own BFCL v4 tab reverses its "beats models ten times its size" headline, its advertised size range is arithmetically impossible against its own published parameter count and quantisation floor, and its open issue tracker contains bugs nobody has answered. Every error in this episode was found on the open side, and none of it because Cactus is worse, but because Cactus is checkable. If you grade companies by how many mistakes the public caught, you will reliably conclude that the one which showed you nothing is the better one.

---

What the episode covers

• The file. Needle 3's calling convention, its published geometry, and the ladder that makes every depth from two to twenty layers a deployable model sliced from one set of weights.
• Two scoreboards that do not meet. Why any side-by-side accuracy number between Jev and Needle would be manufactured, and the one honest quantitative comparison that is available.
• The number that argues with the headline. Needle's BFCL v4 result, TypeSafe's Invoice Processing result, and the pattern both share.
• Two calibration companies, zero calibration measurements.
• The number in the name. A four-way measurement of the shipped binary against an advertised range.
• What open is doing here — genuinely Apache-2.0 weights, and the three asterisks that do not cancel it.

Verification note

Three load-bearing claims from the research dossier were independently re-verified from primary sources before this script was written. All three passed.

• File size. needle3.cact = 35,335,380 bytes, confirmed via the Hugging Face tree API size field, the git-LFS pointer record, the x-linked-size header on the blob response, and content-length on the CDN redirect target. At 121,021,910 parameters that is 2.336 bits per parameter. A 29 MB decimal file would require 1.917 and a 29 MiB file 2.010, both below the format's documented 2.125-bit floor.
• The benchmark decode. The dossier's figures came from reading point geometry out of a client-rendered React chart. They were re-derived independently from assets/benchmarks.svg in the Apache-2.0 model repository, a static chart whose every value is an explicit text label. 57 of 57 per-suite cells match exactly, all 18 aggregate means recompute to the stated decimal, and four figures match the plain text of the Show HN post. The fine-tuning figure was re-verified the same way, all twenty values exact.
• The paper. arXiv:2607.18363 re-fetched from the arXiv API: title, submission date (20 July 2026), eight-author list and abstract figures all confirmed.

Two caveats carried on air: no published BFCL leaderboard entry for Needle exists, so the 50.2 is Cactus's own run of a public harness rather than an independently scored result; and only the shipped twenty-layer file was measured, so the 8 MB bottom of the advertised range is unmeasured and is not claimed to be wrong.

Sources

Needle 3 / Cactus Compute - Launch page — https://cactuscompute.com/needle - Model weights and card, Apache-2.0 — https://huggingface.co/Cactus-Compute/needle3 - Source repository — https://github.com/cactus-compute/needle - Issue #118, "Where are the libneedle sources?" — https://github.com/cactus-compute/needle/issues/118 - Issue #121, frozen acceptance suites — https://github.com/cactus-compute/needle/issues/121 - Show HN: Cactus Needle 3 — https://news.ycombinator.com/item?id=49748553 - The .cact format — https://cactuscompute.com/blog/cact-format - Intelligence ladders — https://cactuscompute.com/blog/intelligence-ladders - Leveraging Needle's confidence — https://cactuscompute.com/blog/needle-confidence - Designing tools for Needle — https://cactuscompute.com/blog/designing-tools-for-needle - Fine-tuning Needle — https://cactuscompute.com/blog/finetuning-needle

The paper - A Controlled Study of Attention-Only Transformers, Ndubuaku, Mosoyan, Mroz, Cylich, Kumar, Sandhu, Shemet & Lee — https://arxiv.org/abs/2607.18363

Jev / TypeSafe — covered in full in episode 0063, Jev and the System One Model: A New Calling Convention.

Benchmarks referenced - Berkeley Function Calling Leaderboard — https://gorilla.cs.berkeley.edu/leaderboard.html

Corrections and standards

Numbers attributed to a vendor are labelled as such on air. Hacker News and Reddit quotations are attributed as a commenter's words, never as established fact. Reception figures — Show HN scores, Hugging Face download counts, Reddit vote totals — are deliberately not quoted, because Needle 3 was hours old at research time and any such figure would be stale by air date. An ethos pass was run against c4573.org and its recent posts before this script was finalised.

Notes for DTF:FTL episode 0066. Script and research: data/episodes/0066/.