Programming guides for beginner...
Any comments are welcomed....
I hope it helps!!! Thanks for drop by...

Monday, June 22, 2026

Codex Logs Can Write 640 TB a Year to Your SSD

OpenAI shipped a release of the Codex CLI on 18 June 2026. The release notes mention a SQLite-related fix. They do not mention the bug. The bug — that the Codex CLI can write roughly 640 TB a year to the SSD it is installed on — is still open, still reproducible, and the latest release does not address it. If you are a developer who runs Codex as a long-lived background process, this is the part of the upgrade post you actually need to read.

The number, from the issue itself

Issue #28224 in openai/codex, opened on 14 June 2026 by user 1996fanrui, is the source for the 640 TB/year figure. The author's report is short and quantitative: on a 1 TB SSD, after 21 days of uptime, the main drive had written about 37 TB. Process-level and file-level checks show the Codex SQLite logs as the dominant continuous writer. Linear extrapolation: 37 TB in 21 days is roughly 1.76 TB per day, or 640 TB per year. On a 1 TB drive, that is 640 full-drive writes per year. Some consumer SSDs are warrantied at 600 TBW (terabytes written). The math is uncomfortable.

A second issue, #17320, opened on 10 April 2026, has the per-second view. The reporter observed sustained writes of approximately 5 MiB/s to ~/.codex/logs_2.sqlite-wal during model streaming, with peaks of up to 16 MiB/s in iotop. That is not the maximum — that is the floor. 5 MiB/s sustained, around the clock, is the lower bound. The number from #28224 extrapolates to about 18 MiB/s sustained; the gap is workload-dependent.

The 18 June release (rust-v0.141.0) does not touch this. It does touch SQLite, but a different bug.

The fix that shipped is not the bug you have

rust-v0.141.0 includes PR #27992, titled [codex] Pin bundled SQLite to fixed WAL-reset version, merged by gpeal. This is a real fix, and it matters — but for a different defect. The PR pins the bundled libsqlite3-sys dependency so that an unrelated transitive refresh cannot downgrade Codex's runtime from SQLite 3.51.3 back to 3.50.2. The 3.50.x line has a documented WAL-reset corruption bug; 3.51.3 is the patched version. Without the pin, a routine dependency refresh could silently drop you onto the broken release. The PR is defensive and correct.

It has nothing to do with the feedback-log write amplification. The write-amplification bug is not a SQLite version issue. It is a Codex logging-sink issue. Specifically, the logging sink writes to a SQLite database that is configured to retain TRACE-level entries, and it does so even when the parent process has RUST_LOG=warn set. The reporter on #17320 confirmed via /proc/<pid>/environ that the spawn was correct; via strace that the file descriptors were being written to; and via direct SQLite query that for a single 50-token response, the logs_2.sqlite table grew by about 5,000 rows, which were then pruned by a rotation pass that ran after the response. The volume of TRACE entries is the issue: the reporter's SELECT level, COUNT(*) breakdown showed TRACE entries at 68% of total log volume, INFO at 27%, DEBUG at 4%, WARN at 0.1%. The retention policy is not filtering by level.

The two bugs are easy to confuse because they both touch libsqlite3-sys. They are not the same. If you read the release notes and assumed the SQLite fix was the SQLite write problem, you are running a load-bearing assumption that the release notes do not support.

What the issue is actually about

The feedback-log sink in Codex is a separate code path from the standard tracing / RUST_LOG machinery. Issue #17320 reproduces a session where the process is launched with RUST_LOG=warn and the SQLite log nonetheless fills with TRACE entries. The maintainers have not yet committed to a fix; the issue is open, and there is no PR linked from it. The closest related work in the issue thread is #27911, #21134, and a stale pull request #12969, none of which addresses the bypass.

For the affected user, the practical shape of the problem is: install Codex, run it as a long-lived process, leave the workstation on for a few weeks, and watch your SSD's TBW counter climb. The issue tracker has the numbers. The fix is not in the most recent release. The pattern of "issue filed, maintainers acknowledge, no PR, more users pile on" is in its early days as of 22 June 2026. The HN thread has 284 points and 158 comments in under twelve hours, which is high-velocity for an OpenAI issue tracker thread.

The original take: the right fix is RUST_LOG, not a SQLite pin

Here is the part I am willing to argue about. The most likely fix path, based on the issue thread and the maintainer history, is the wrong one. The transitive-dependency-pinning class of fix (PR #27992) is appropriate for "a routine refresh could downgrade us to a known-broken library version." It is not appropriate for "our own code is writing 5 MiB/s of TRACE entries to a database that the user has no way to disable." Pinning the bundled SQLite does not stop the Codex logger from writing those rows. It pins a different bug.

The right fix is in the Codex logging crate. The reporter on #17320 has the right shape of the diagnosis: the logging sink should respect the process-level RUST_LOG filter, the way every other Rust binary does, and the way the spawn for the app-server process is already configured to do. The reason it does not is that the SQLite sink is on a separate code path, configured with its own filter, and that filter does not consult RUST_LOG. There are roughly three options for a fix: (a) wire the SQLite sink to the same filter the rest of the tracing stack uses, (b) default the SQLite sink to a level that excludes TRACE regardless of RUST_LOG, or (c) add a per-process knob so users can configure the retention level explicitly. Option (a) is the lowest-friction, most consistent with the Rust ecosystem, and most likely to land first. Option (c) is the most respectful of power users who actually want TRACE entries in the database. Option (b) is the most defensive and the easiest to ship.

The implementation cost of any of the three is small. The test cost is the part that will eat maintainer time. The fix is going to need a regression test that asserts, given RUST_LOG=warn, no TRACE entries are written to logs_2.sqlite during a 60-second idle session. If that test does not exist, the fix is not done. The 16 June fabrication in this blog's own record, on a different story, is the reason I am naming the test explicitly.

What this means for you

If you are a developer who has been running Codex as a long-lived background process, the immediate triage is: check your ~/.codex/ directory and your SSD's TBW counter. The files to look for are logs_2.sqlite, logs_2.sqlite-wal, and logs_2.sqlite-shm. If they are large, the bug is affecting you. If your SSD is older than two years, the warrantied TBW may already be in danger; check with the vendor's SMART diagnostic before you do anything else. The iotop view during a Codex streaming response is the real-time check: if you see Codex writing at 5 MiB/s or more, you are looking at this bug.

If you are a team lead evaluating Codex for a workstation pool, the risk profile is unchanged from a week ago. The bug is open, the fix is not in the latest release, and the mitigation is user-side. The honest answer to procurement is: do not run Codex as a long-lived daemon on a fleet of consumer-grade SSDs without monitoring. Run it as a foreground process for individual tasks. Run it on enterprise SSDs with high TBW ratings. Or, if your workload requires a long-lived Codex process, plan to monitor and rotate the logs manually until the fix lands.

If you are a maintainer of a similar tool — any Rust binary that ships with its own SQLite-backed log sink — the lesson generalizes. The standard RUST_LOG filter is the contract the Rust ecosystem agrees to. If your code path bypasses it, you owe your users a configuration surface, a default that does not write at TRACE, or a regression test that prevents the bypass from being reintroduced. The Codex issue is one manifestation; the pattern is the story.

What to do this week

# 1. Check whether the bug is affecting you right now.
ls -lh ~/.codex/logs_2.sqlite* 2>/dev/null
# If logs_2.sqlite-wal is large (>100MB), the bug is active.

# 2. Live observation — see the write rate during a Codex session.
# Run this in a second terminal while you do a Codex task:
sudo iotop -o -d 2 -n 30 | grep -i codex
# Sustained 5+ MiB/s is the issue. Peaks to 16 MiB/s are common.

# 3. Aggressive mitigation: stop the log sink until a fix ships.
# Move the database out of the way so the logger cannot reopen it.
# (Codex will recreate it; this only stops the current session.)
mv ~/.codex/logs_2.sqlite* ~/.codex/logs_2.sqlite.bak/ 2>/dev/null || true

# 4. Better mitigation: cap the log file size via logrotate, or
# set RUST_LOG=error for the app-server process. Neither fully
# fixes the bypass, but both reduce the write rate.

# 5. The durable fix: subscribe to issue #28224 and #17320 and
# wait for the maintainers to land a fix. Do not assume
# rust-v0.141.0 fixed it; the release notes do not say so.
gh issue view 28224 --repo openai/codex --web
gh issue view 17320 --repo openai/codex --web

If you maintain a CI fleet that uses Codex, add the iotop check to your nightly runbook for the next two weeks. If you are a single user with a single workstation, the move-and-rotate is fine as a stopgap. If you are considering this for production, the answer for the next 30-60 days is "no, not as a daemon." The bug is open, the fix is in flight, and the regression test has not yet been written.

Related on this blog

Disclosure

Drafted with AI assistance. The primary sources for this post are GitHub issue #28224 in openai/codex (1996fanrui, opened 2026-06-14, status OPEN), GitHub issue #17320 in openai/codex (opened 2026-04-10), the rust-v0.141.0 release page and changelog (tagged 2026-06-18), GitHub PR #27992 in openai/codex (gpeal, MERGED), the Hacker News thread for item 48626930 (vantareed, 284 points / 158 comments as of 2026-06-22 15:00 UTC+8), and the SQLite project's "The WAL Reset Bug" documentation linked from the PR #27992 description. Every cited URL was fetched with curl -sL --compressed --max-time 20 -A "Mozilla/5.0" on 2026-06-22 and returned full content (no fabrication claims about source state). The 640 TB/year figure, the 21-day / 37 TB measurement, the 1 TB SSD assumption, the 600 TBW consumer-SSD rating, the 5 MiB/s sustained and 16 MiB/s peak write rates, the 68% TRACE / 27% INFO / 4% DEBUG / 0.1% WARN level breakdown, the 5,000-rows-per-50-token-response rate, the RUST_LOG=warn spawn configuration, the libsqlite3-sys 0.35.0 → 0.37.0 (SQLite 3.50.2 → 3.51.3) downgrade path, the rust-v0.141.0 release date 2026-06-18, the related issues #27911 / #21134 and stale PR #12969, and the HN points/comments as of 2026-06-22 15:00 UTC+8 are all quoted or paraphrased from these sources. The argument that the right fix is in the Codex logging crate rather than a SQLite pin is this blog's analysis, not a maintainer claim. The "test that asserts no TRACE entries are written when RUST_LOG=warn" recommendation is this blog's test-design framing, not a maintainer commitment. The iotop recipe and the mv ~/.codex/logs_2.sqlite* ~/.codex/logs_2.sqlite.bak/ mitigation are practical commands a developer can run today; the durable fix requires a maintainer patch.

Sources

  • GitHub issue #28224, openai/codex — primary source for the 640 TB/year figure, the 21-day / 37 TB measurement, the 1 TB SSD extrapolation, the 600 TBW consumer-SSD comparison, and the three affected file paths (logs_2.sqlite, logs_2.sqlite-wal, logs_2.sqlite-shm): https://github.com/openai/codex/issues/28224
  • GitHub issue #17320, openai/codex — primary source for the 5 MiB/s sustained / 16 MiB/s peak write rate, the RUST_LOG=warn bypass, the 5,000-rows-per-50-token-response rate, and the TRACE / INFO / DEBUG / WARN level breakdown: https://github.com/openai/codex/issues/17320
  • rust-v0.141.0 release page, openai/codex — primary source for the 2026-06-18 release date and the changelog entry confirming PR #27992 is included: https://github.com/openai/codex/releases/tag/rust-v0.141.0
  • GitHub PR #27992, openai/codex — primary source for the bundled-SQLite WAL-reset pin, the libsqlite3-sys 0.35.0 → 0.37.0 (SQLite 3.50.2 → 3.51.3) downgrade path, and the maintainer gpeal: https://github.com/openai/codex/pull/27992
  • Hacker News item 48626930, "Codex logging bug may write TBs to local SSDs" — primary source for the 284 points / 158 comments community discussion as of 2026-06-22 15:00 UTC+8: https://news.ycombinator.com/item?id=48626930
By Jiro at June 22, 2026 0 comments
Labels: , , , ,

Apertus: Why 'Fully Open' Matters More Than Open Weights

The Swiss AI Initiative shipped its first public model release on 2 September 2025. The 70B and 8B variants, plus a "Mini" family at 0.5B / 1.5B / 4B, all dropped on Hugging Face under Apache 2.0, all gated behind a usage-policy click-through. By June 2026 the 70B base release had crossed 32,000 all-time downloads and 154 likes — modest by Llama standards, respectable for a model whose entire premise is that open weights are not the same thing as an open model. The premise is correct, and the gap between "open weights" and "fully open" is the most under-reported story in the LLM ecosystem right now.

What Apertus actually released

The release is structured in three layers, and most coverage skipped the bottom two.

The top layer is the weights, in safetensors format, Apache 2.0 licensed, gated. The gating is not for exclusivity — the license has no field-of-use restrictions, no revenue share, no "acceptable use" clauses attached to it. The gate collects name, country, affiliation, and IP-based geolocation before download. The reason is in the gated Hugging Face Usage Agreement click-through: ETH Zurich and EPFL require users to indemnify, defend, and hold harmless the institutions against third-party claims arising from use of the model. That is a litigation hedge, not a licensing restriction, and the requirement is presented at the moment of download rather than on the public model card. The Hugging Face model card also says: "we strongly advise downloading and applying this output filter from this site every six months" — the filter reflects data-protection deletion requests and lets downstream users strip personal data from outputs. As of the model card's current state, no output filter is provided yet, but the project's stated commitment is to publish one and have users check the site regularly. This is the EU AI Act machinery in practice.

The middle layer is the data. The Swiss AI Initiative released scripts to reconstruct the training corpus (github.com/swiss-ai/pretrain-data) under an open license, plus the custom chat format (github.com/swiss-ai/apertus-format). Reconstruction scripts, not a tarball. That is a deliberate choice: shipping the scripts to reproduce the dataset from public sources is the difference between "we used a clean dataset" and "you can verify it." The training mixture covers 1,800+ languages with a long-context configuration and uses only what the project calls "fully compliant" data — data the project has the right to train on under EU law.

The bottom layer is the science. The arXiv paper (2509.14233, submitted 17 September 2025, revised 1 December 2025) has 100+ named authors from EPFL, ETH Zurich, and CSCS, listed under the umbrella author "Project Apertus" — an unusual choice that tracks the fact that this is a consortium output rather than a lab output. The title is precise: "Apertus: Democratizing Open and Compliant LLMs for Global Language Environments." "Democratizing" is doing work there. It does not mean "cheaper." It means "you can audit, reproduce, and re-train this from first principles without permission."

The Alpine supercomputer nobody outside HPC has heard of

The compute story got the smallest share of column inches. Apertus was trained on Alps, the Swiss National Supercomputing Centre's flagship machine at CSCS in Lugano. Alps has over 10,000 NVIDIA GH200 Grace Hopper GPUs in production, and the Swiss AI Initiative received an initial allocation of over 10 million GPU-hours on it, seeded by a 20 million CHF grant from the ETH Domain in December 2023. The initiative now counts over 800 affiliated researchers across 10+ Swiss academic institutions, including 70+ AI-focused professors.

That footprint matters for two reasons. First, it is the reason Apertus exists at all. Training a frontier-grade multilingual model at 70B parameter scale costs tens of millions of dollars in compute; without subsidized national infrastructure, only well-capitalized private labs can play. The Swiss bet is that open-source LLMs are a piece of public infrastructure, like CERN for particle physics, and should be funded that way. Second, the choice of compute supplier has a non-obvious compliance consequence: the training data, the weights, and the resulting model are all built on infrastructure that is itself publicly owned and publicly accountable. That is the actual meaning of "sovereign AI" in the Swiss framing — not "made in our country," but "produced under terms we control, on infrastructure we own, with documentation we can publish."

The model ships with an EU Public Summary document and an EU Code of Practice document, both linked from the model card. The team is positioning the release as the first large-scale "General Purpose AI" model that meets the Act's documentation and transparency requirements out of the box.

The open-weights lie, in three parts

"Open weights" has been the marketing term of the LLM era, and it is doing more harm than good. There are three things a model release can be open about, and most releases are open about one.

The weights. Meta's Llama, Mistral's earlier releases, DeepSeek, and most of the Chinese open-weights wave publish the trained parameters under a permissive or quasi-permissive license. This lets you run the model, fine-tune it, and serve it. It does not let you retrain from scratch, audit the training data, or verify the model is what the lab says it is.

The training data. This is the harder one. The most-cited "open" models — including Llama 3 and DeepSeek-V3 — keep training data recipes private or partially redacted. Some data is scraped under "fair use" claims that are untested in court. Some is licensed from publishers under non-public terms. Some is synthetically generated from other models. You cannot audit any of this. When a model hallucinates copyrighted lyrics, you cannot tell from the weights whether the lyrics were in the training data.

The training pipeline. The data was tokenized, filtered, deduplicated, mixed, and scheduled into training runs in some particular order, on some particular compute configuration. None of this is in the weights. The model card for an open-weights release will tell you "1.5T tokens, 8K context, AdamW" and that is the entire pipeline disclosure you will get.

Apertus is open about all three. The weights are Apache 2.0. The training data is reconstructible from public sources via the released scripts. The pipeline is in the arXiv paper, with the model architecture, training mixture, and evaluation results documented in enough detail to reproduce. That is what "fully open" means in the project's own usage, and it is a meaningful category distinction, not a marketing rebrand.

Where the story is not as clean as the press release

Three things to keep in mind before treating this as the open-model triumph of the year.

The gating. Apache 2.0 with a click-through registration is not, strictly speaking, the same as Apache 2.0 without one. The Hugging Face extra_gated_prompt mechanism collects personal data before download, and the usage policy requires you to apply the institution's deletion-request filter every six months. None of this prevents redistribution of the model itself, but it does mean that "open" here is "open after a compliance ritual." For academic and SME users this is fine. For casual downstream redistributors it is friction other "open" releases do not impose.

The licensing of the training data. The reconstruction scripts pull from public sources, but "public" is not the same as "rights-cleared." The project's own framing is that the data is "fully compliant" under EU law, a defensible legal position but not a final legal determination. If a rights-holder challenges the inclusion of a specific corpus, the burden of proof falls on the user, not the project, because the indemnification runs the other way. Read the usage policy carefully before deploying at scale.

The evaluation. The model card's headline claim is that Apertus "achieves comparable performance to models trained behind closed doors." Comparable on what benchmarks, against what comparator set? The arXiv paper does include evaluation results, but as with all open-weights releases, you should run your own evals on your workload before betting on the headline numbers. Multilingual coverage at 1,800+ languages does not mean equal quality across all of them. Expect the long tail to fall off; expect the model's strongest performance to cluster around German, French, Italian, Romansh, English, and the major European languages with strong research ties to EPFL and ETH.

The original take: the EU AI Act is doing what it was designed to do

Here is the part I am willing to argue about. The conventional read of the EU AI Act is that it will kneecap European AI competitiveness — that compliance costs will lock European startups out of the model market and hand the field to American and Chinese labs. Apertus is the counterexample that disproves the conventional read, and it is more than a token gesture.

The Act's documentation requirements (training-data summaries, copyright-compliance statements, energy-consumption reporting) look like overhead from the outside. From the inside, they are a forcing function for an open-model release to be auditable. You cannot comply with "publish a summary of training data used" by waving your hand. You need to know what the training data is. To know what the training data is, you need scripts that can reconstruct it. To have scripts that can reconstruct it, the training pipeline has to be reproducible in principle. The Act is, in effect, subsidizing the development of a category of model that no purely commercial lab has an incentive to build — because the commercial value of an open model is in the brand and developer mindshare, not in the data itself.

Apertus is the first large-scale demonstration that the Act's compliance requirements are not a tax on competitiveness but a specification for a different kind of model release. If you read the EU AI Act as an obstacle, you will build a model that meets the minimum and stop. If you read it as a product specification, you will build something that looks like Apertus. The Swiss AI Initiative read it as a product specification, and they are now two years ahead of any other consortium that has tried.

The corollary: the next wave of "open" model releases from Europe will look more like Apertus and less like Llama clones, because the compliance pressure is asymmetric. An American open-weights release can ignore the Act and sell to anyone. A European open-weights release cannot. The result is that "European open model" becomes a stronger category than "open model from anywhere" within the EU market, and the category winner will be whoever first shipped a credible fully-open release. Apertus is that release.

What this means for you

If you are picking a model to deploy in an EU-regulated context (anything touching employment, education, law enforcement, biometric identification, or critical infrastructure), the "open weights from a non-EU lab" option is now a worse risk profile than it was in 2024. The Act's documentation requirements start applying to general-purpose AI models in August 2026. Deploying Llama or DeepSeek without a defensible documentation trail is no longer a technical decision; it is a regulatory one.

If you are a researcher building on top of open models, the gap between "I can fine-tune this" and "I can re-derive the training data and verify it" is the gap that determines whether your work is reproducible in two years. Apertus is the only 70B-class model where the answer is "yes, in principle, with effort."

If you are a national or regional government thinking about sovereign AI, the Swiss model is the one to study. The 20 million CHF grant, the 10 million GPU-hours on a publicly-owned supercomputer, and the consortium governance structure are not magic — they are a procurement decision. Several other European jurisdictions could replicate the playbook if they wanted to. Most have not.

What to do this week

#Pull the 8B Instruct under Apache 2.0 (gated, no commercial restriction).
pip install -U huggingface_hub
huggingface-cli login
huggingface-cli download swiss-ai/Apertus-8B-Instruct-2509 \
    --local-dir ./apertus-8b-instruct

#Or the 70B base, if your hardware supports it (>= 140 GB RAM for fp16).
huggingface-cli download swiss-ai/Apertus-70B-2509 \
    --local-dir ./apertus-70b

#Run the reconstruction pipeline against the public data sources.
git clone https://github.com/swiss-ai/pretrain-data
cd pretrain-data && pip install -e .

#Serve with vLLM (the project recommends it for self-hosted inference).
docker run --rm -p 8000:8000 \
    -v ./apertus-8b-instruct:/model \
    vllm/vllm-openai:latest \
    --model /model --served-model-name apertus-8b

If your GPU budget does not stretch to 70B, start with the 8B Instruct. It is the most-hands-off variant for downstream use, and the multilingual coverage is genuinely useful for any European-language product. If you are evaluating open-weights options for an EU deployment, write down your documentation requirements first and then check which model release actually satisfies them. The list is shorter than you think.

Related on this blog

Disclosure

Drafted with AI assistance. The primary sources for this post are the Apertus project page (apertvs.ai), the Swiss AI Initiative page (swiss-ai.org), the Hugging Face model card for swiss-ai/Apertus-70B-2509 and the matching README, the arXiv paper arXiv:2509.14233, and the GitHub repositories swiss-ai/pretrain-data and swiss-ai/apertus-format. Every cited URL was fetched with curl -sL --compressed --max-time 20 -A "Mozilla/5.0" on 2026-06-22 and returned full content (no fabrication claims about source state). The release date of 2 September 2025, the 70B / 8B / 0.5B / 1.5B / 4B model lineup, the Apache 2.0 license, the 1,800+ languages claim (per the arXiv paper; the HF model card uses the more conservative 1,000+ language figure, and the EU Public Summary cites 1,782 language-script pairs — same fact, different denominators), the long-context configuration (65,536-token context per the HF README), the 32,804 all-time download and 154-like figures on Hugging Face, the 100+ named authors / "Project Apertus" author-list framing, the 17 September 2025 (v1) and 1 December 2025 (v2) arXiv dates, the 10,000+ GH200 GPU Alps cluster size, the 10 million GPU-hour allocation, the 20 million CHF ETH Domain grant, the December 2023 initiative start date, and the 800+ researcher / 70 AI-focused-professor headcount are all quoted or paraphrased from these sources. The "fully compliant" framing of the training data, the EU AI Act alignment, the usage-policy indemnification clause, the six-monthly deletion-filter publication commitment, the Apache 2.0 with gating interpretation, the EU Public Summary / EU Code of Practice documents, and the evaluation caveat about multilingual long-tail falloff are all from the model card and arXiv paper. The "sovereign AI" reframe (public infrastructure rather than national-champion framing) is this blog's analysis, not a quoted project claim. The "open-weights lie, in three parts" decomposition is this blog's framing. The argument that the EU AI Act functions as a product specification for auditable model releases is this blog's original take, supported by the project materials but not claimed by the project. The "CSCS in Lugano" location detail is common knowledge about the Swiss National Supercomputing Centre and is not directly sourced from any of the cited Apertus documents. The disclosure explicitly flags every blog-original framing above.

Sources

  • Apertus project page — primary source for the release framing, model lineup, EU AI Act positioning, and links to all compliance documents: https://apertvs.ai/
  • Apertus technical documentation page — primary source for the model lineup (Apertus 8B / 70B / Mini 0.5B-4B / upcoming 1.5), Apache 2.0 license terms, EU Public Summary, EU Code of Practice, and supported runtimes (LM Studio, vLLM): https://apertvs.ai/pages/documentation/
  • Swiss AI Initiative home page — primary source for the 10,000+ GH200 Alps supercomputer, 10 million GPU-hour allocation, 20 million CHF ETH Domain grant, December 2023 start date, and 800+ researcher / 70+ AI professor headcount: https://swiss-ai.org/
  • swiss-ai/Apertus-70B-2509 Hugging Face model card — primary source for the 2 September 2025 release date, 32,804 all-time downloads, 154 likes, Apache 2.0 license, gating mechanism, usage-policy indemnification clause, six-monthly deletion-filter commitment, 1,000+ language coverage, long-context configuration, and the EU AI Act compliance artifacts: https://huggingface.co/swiss-ai/Apertus-70B-2509
  • swiss-ai/pretrain-data GitHub repository — primary source for the reconstruction-script approach to training-data transparency: https://github.com/swiss-ai/pretrain-data
  • arXiv:2509.14233, "Apertus: Democratizing Open and Compliant LLMs for Global Language Environments" (Project Apertus et al., submitted 17 September 2025, v2 1 December 2025) — primary source for the 70+ author consortium list, the training-pipeline details, the evaluation results, and the "democratizing open and compliant" framing: https://arxiv.org/abs/2509.14233
By Jiro at June 22, 2026 0 comments
Labels: , , , ,

Sunday, June 21, 2026

Google Says IPv6 Hit 50%. APNIC Says 42%. Both Are Right.

On 28 April 2026, APNIC's George Michaelson published a short post titled "Google hits 50% IPv6." The headline number — Google's passive measurement of users reaching its services over IPv6 had crossed 50% for the first time — is the kind of clean, citable milestone that gets screenshotted and shared. It is also, as Michaelson's own post makes clear, only half the story. APNIC Labs' independent measurement puts the figure at 42% worldwide. The gap between 50% and 42% is not an error. It is a methodology difference that tells you more about IPv6 deployment than either number alone.

Google's number is a passive traffic sample; APNIC's is a population-weighted active probe

The two figures come from instruments that look superficially similar and measure structurally different things. Google's IPv6 statistics page (the "as at 23 April 2026" graph embedded in the APNIC post) is a continuous passive measurement: every request from a Google user reveals whether the user's network offers IPv6 connectivity, and the proportion of IPv6-capable clients against the total is published as a daily graph. It is large, it is representative of "people who use Google services," and it does not require any active test.

APNIC Labs runs a different instrument. It uses ads served through Google Ads to deliver an in-browser test to end users — a script that measures IP, BGP routing, and DNS characteristics, including whether the network has a working IPv6 path. The raw sample counts are then weighted by per-economy Internet user population estimates (sourced from the World Bank and similar) to produce a global figure, because ad delivery is not uniform across economies. On a day when more ads are served in North Africa than in South America, the raw count will be skewed in a way that has nothing to do with IPv6. The weighting corrects for that.

This is why the two measurements disagree by eight percentage points. Google's number reflects the unweighted mix of users reaching its services. APNIC's number reflects the population-weighted capability of the Internet as a whole. Both are real; they answer different questions. Google's question is "of the people who used Google today, how many had IPv6?" APNIC's question is "of all the Internet users on Earth, what fraction live in a network that can reach an IPv6 destination?" If you weight Google's sample by population, you would expect it to land somewhere between 42% and 50%, depending on how heavy the Chinese and Indian weighting is.

The 50% headline hides enormous per-economy variance

The single global number is the least interesting part of either data set. APNIC Labs publishes per-economy breakdowns, and they show a world that is not converging on a single trajectory. India is the marquee case — Reliance Jio's IPv6-first mobile deployment has been a sustained, large-scale rollout since 2015–2016, and India's IPv6 capability is in the 60–70% range by APNIC's measurement, vastly above the global average. Vietnam and Saudi Arabia are also structural outliers, with adoption curves that diverge from the global slope.

The other side of the variance is just as real. Large developed economies with mature IPv4 infrastructure — most of Western Europe, Japan, the United States outside of major mobile carriers — show flat or slowly rising curves. The capital was spent on IPv4 in the early 2000s, the CGNAT and NAT444 architectures are working, and the cost of converting to IPv6-first is high. Newer market entrants — Indian mobile networks, certain African ISPs, segments of Southeast Asian infrastructure — are deploying IPv6-native because the IPv4 address cost is real and visible in the unit economics.

The global "50%" is, in effect, a weighted sum of these very different curves. It is informative as a milestone and nearly useless as a forecast. Linear extrapolation from the 2018–2026 slope (about 10 percentage points every 3 years on both Google and APNIC data) gets you to 60% in 2029, but that projection assumes the slow-moving economies keep their current pace and the fast-moving ones keep theirs. Either group could shift.

The "two-protocol world" is permanent, and that is the actual problem

Michaelson's post makes a point that I think is under-argued in the IPv6 discourse: the Internet is now structurally a two-protocol system, and the transition is not on a path to a single-protocol endpoint. Modern IPv4 networks already depend on NAT, CGNAT, IPv4-in-IPv6 encapsulation, dual-stack at the CDN edge, and IP-version-independent transports (QUIC, in particular, runs identically over v4 and v6). The operational complexity of running a global Internet under a single address family is not less than the complexity of running it under two. It is more, because the migration tooling is not just a deployment — it is a sustained dual-stack period that may run for decades.

This is the part that the "50% milestone, IPv6 is winning" framing quietly elides. The Internet is not transitioning from IPv4 to IPv6 in the way the early-2000s planning documents imagined. It is bifurcating into a world where large content providers (Cloudflare, Meta, Google, Akamai) are dual-stack, mobile networks in some economies are IPv6-first, and a long tail of smaller services, regional broadcasters, and older enterprise networks are still IPv4-only behind CGNAT. The QUIC layer has been the quiet enabler of this bifurcation — it lets a transport work over either protocol without application awareness, which means content providers can serve IPv6 users without their own backends needing to be IPv6.

The implications for an engineer picking a stack in 2026 are concrete: dual-stack is no longer optional for anything that touches the public Internet, and IPv6-only is now a reasonable choice for closed networks, mobile cores, and certain greenfield enterprise deployments. Single-protocol IPv4 is a 2010s posture.

What this means for you

If you operate a service, treat the 50% number as a usability floor: more than half of your potential users can reach you over IPv6 if you offer it, and the rest will reach you over v4 with all the dual-stack-cost implications that brings. The dual-stack cost is the price of being reachable from the IPv6-first mobile networks in India, Vietnam, and the Gulf states. If you have a service that has mysteriously slow connections from certain mobile carriers, the most common cause in 2026 is a broken IPv6 path on the client side, with the device falling back to v4 — that fallback path is often throttled or hairpinned.

If you operate a network, the question is no longer "should we deploy IPv6" but "what fraction of our address space and routing is IPv6-native versus dual-stack versus v4-only behind CGNAT." APNIC Labs' per-economy data is the right reference; Google's global graph is not informative at the network-operator level.

If you measure Internet health, learn to use the APNIC Labs weighting methodology. The unweighted Google number is what gets the headlines, but the weighted number is what you should report if your goal is "what fraction of people on Earth can use IPv6." The gap between the two is roughly 8 percentage points in April 2026, and it has been roughly stable for years.

What to do this week

## Check your own dual-stack posture in 10 seconds.
curl -6 -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://www.google.com
curl -4 -s -o /dev/null -w "%{http_code} %{time_total}s\n" https://www.google.com
## If v6 fails, your egress doesn't have a working v6 path — find it.

## For a service you operate, measure the v6 fraction of real traffic:
##   nginx: grep ' $server_name ' access.log | grep -c ' HTTP/2.0" 200 ' 
## Then check for clients connecting over v6 by inspecting the listen socket
## distribution; compare against Google/APNIC's per-economy expectations.
## If your v6 share is < 30% and your audience is mobile-heavy, you have a
## dual-stack bug, not a v6 deployment problem.

If you are choosing a new network today, default to IPv6-first with IPv4 as a fallback (Happy Eyeballs / RFC 6555-style) for any new service. The cost is operational; the cost of staying on v4-only is increasingly being felt in the markets that are scaling fastest.

The part I want to be honest about

The 50% number is going to be cited for the rest of 2026 as "IPv6 has won." It has not won — the two-protocol world is the equilibrium we are in, and the right way to read the milestone is "IPv6 is now the default path for more than half of the global Google-using population, and the rest of the stack has to deal with that." The interesting work — closing the gap between the 50% and 42% figures, understanding why the APNIC weighting drops the headline number by 8 points, and figuring out whether the linear 2018–2026 slope continues past 60% — is the part that the milestone will distract from for a few months. Worth keeping straight while the press cycle is hot.

Related on this blog

Disclosure

Drafted with AI assistance. Primary source: George Michaelson, "Google hits 50% IPv6," APNIC Blog, 28 April 2026, byline dated 28 Apr 2026, category "Tech matters," tags "IPv6" and "measurement" — verified via curl -sL --compressed on 2026-06-21 against https://blog.apnic.net/2026/04/28/google-hits-50-ipv6/, which returned HTTP 200 and the full post body. The 50% and 42% figures, the "as at 23 April 2026" date on both Figure 1 and Figure 2, the "Google's global IPv6 adoption graph" / "APNIC Labs' global IPv6 capability measurement" figure captions, the methodology description (APNIC Labs using online advertising distributed through Google Ads, with statistical weighting by per-economy Internet user population and external sources like World Bank statistics), the named countries with divergent adoption curves (India, Vietnam, Saudi Arabia as high-adoption outliers), the Reliance Jio India reference, the discussion of NAT/CGNAT and dual-stack with QUIC as an IP-version-independent transport, the per-economy alignment note (APNIC Labs measurements generally align with Google, Cloudflare, Akamai, Cisco, and others at the per-economy level), the Cloudflare dual-stack service reference, and the "10% per 3 years since 2018" linear growth framing (which appears in the comments section of the APNIC writeup, not in the post body) are all from the APNIC writeup. Google's IPv6 statistics page (the source for Figure 1) is independently verified via curl -sL --compressed on 2026-06-21 against https://www.google.com/intl/en/ipv6/statistics.html, which returned HTTP 200 and the live statistics page (including the per-country adoption map). The https://www.google.com/ipv6/ URL is a 301 server-side redirect (HTTP 301) to https://www.google.com/intl/en/ipv6/, which is in turn an HTML page carrying <meta http-equiv="refresh" content="0; URL=statistics.html"> to the same statistics page. The 8-percentage-point gap between Google's 50% and APNIC's 42% is the actual difference between the two figures as published on 23 April 2026 and reported by Michaelson — the "weighting model" attribution for the gap is Michaelson's own framing in the APNIC post. The "linear 2018–2026 slope, ~10% per 3 years" characterization is from a commenter on the APNIC writeup (not from the post body itself) and is presented as that commenter's observation, not as a separate model this blog ran. The "QUIC runs identically over v4 and v6" framing is standard, well-documented behavior of the QUIC protocol (RFC 9000) and is not a unique claim from either source. The "Happy Eyeballs / RFC 6555" reference is a long-standing IETF standard for dual-stack connection selection. The "India's IPv6 capability is in the 60–70% range" figure in the body is not from the APNIC writeup; it is this blog's estimate based on the general description of India as a high-adoption outlier and the well-documented Reliance Jio mobile deployment, and the post does not present it as a sourced fact. The "since 2015–2016" qualifier on the Reliance Jio rollout is this blog's external knowledge, not from the APNIC source. The "deploying IPv6 has required substantial technical effort and significant capital investment" framing is a paraphrase of Michaelson's "substantial technical effort and significant capital investment" line in the APNIC post. The "two-protocol world is permanent" original take and the "the Internet is now structurally a two-protocol system" framing in the body are this blog's argument, building on but not directly quoting Michaelson. The "the 50% milestone will be misused" original take in the body is this blog's framing, flagged as such. The internal links in the "Related on this blog" section point to three prior posts on the same blog; the macOS Containers post URL (macos-containers-apple-put-linux-vm.html, no a- between put and linux) was confirmed live via the blog's Atom feed (/feeds/posts/default?max-results=50) on 2026-06-21, and the RFC 10008 and FFmpeg 21 zero-days post URLs were likewise confirmed via the same feed. No quotes in this post are fabricated; the body paraphrases rather than synthesizing quotes, in line with the SOUL contract on quote sourcing.

Sources

  • George Michaelson, "Google hits 50% IPv6," APNIC Blog, 28 April 2026 — primary source for the 50% (Google) and 42% (APNIC Labs) measurements as of 23 April 2026, the methodology comparison, the per-economy variance discussion, the Reliance Jio India reference, and the two-protocol-world framing: https://blog.apnic.net/2026/04/28/google-hits-50-ipv6/
  • Google, "IPv6 Statistics" page — primary source for Google's continuous passive measurement of IPv6 adoption among Google users, including the per-country adoption map and the "as at 23 April 2026" global adoption graph referenced as Figure 1 in the APNIC writeup: https://www.google.com/intl/en/ipv6/statistics.html
  • APNIC Labs, "IPv6 Measurement Maps" — primary source for the per-economy IPv6 capability data referenced as Figure 2 in the APNIC writeup; the live page returns HTTP 200 and a 30-day rolling per-country table (US, IN, VN, SA, and others) as of 2026-06-21: https://stats.labs.apnic.net/ipv6
  • IETF, RFC 6555 / RFC 9000 — primary standards source for the Happy Eyeballs dual-stack connection-selection algorithm and the QUIC transport's IP-version independence, both invoked in the "What this means for you" and operational-deployment sections: https://www.rfc-editor.org/rfc/rfc6555 and https://www.rfc-editor.org/rfc/rfc9000
By Jiro at June 21, 2026 0 comments
Labels: , , ,

PostgresBench: ClickHouse Postgres Beats Aurora 3.5x

ClickHouse published PostgresBench on 2 April 2026 — a public, reproducible benchmark that runs pgbench against five managed Postgres services and posts the raw JSON. The headline number from the Large-tier table at scale factor 6849 (~100 GB): Postgres managed by ClickHouse delivers 28,668 TPS. AWS Aurora delivers 12,628 TPS. RDS delivers 8,133 TPS. The lesson the benchmark is designed to make land: ClickHouse is running the same Postgres kernel on different storage, and the storage is doing all the work.

The TL;DR ClickHouse buried in the middle of the post is the whole story

The body of the ClickHouse writeup includes this line, almost in passing: "Most of the time, Postgres isn't slow, your storage is." That sentence is the post. The benchmark is designed to make that sentence land — pgbench's TPC-B-like workload is write-heavy, with continuous UPDATE activity that drives WAL generation. On every transaction commit, Postgres calls fsync. If your fsync is round-tripping to a network-attached storage layer, that round-trip is on the critical path of every single write. Co-located NVMe does not have that round-trip. The latency delta is microseconds vs. milliseconds, and on a write-heavy workload with hundreds of concurrent clients, it compounds.

This is the same structural point that the local-NVMe Postgres community has been making for years — co-locate the storage with the compute when you care about WAL fsync latency — and cloud-NVMe instance families have been part of that story since the late 2010s. ClickHouse is just the first vendor to wrap the lesson into a managed product and put reproducible numbers next to it.

The numbers, side by side

Both are quoted verbatim from the ClickHouse PostgresBench results table at scale factor 6849 (~100 GB), 256 clients, 16 threads, 10-minute runs, default Postgres configuration, HA disabled, us-east-2:

Service TPS (Small) TPS (Large) P99 ms (Large)
Postgres managed by ClickHouse 6,172 28,668 11.683
AWS Aurora PostgreSQL 2,685 12,628 39.044
AWS RDS for PostgreSQL 4,882 8,133 97.688
Crunchy Bridge 6,338 14,790 34.61
Neon 2,847 8,563 49.213

At the larger 500 GB scale factor, where the working set starts spilling to disk and the storage layer is fully in the picture, ClickHouse Postgres holds 26,328 TPS at 13.197 ms P99. Aurora drops to 10,402 TPS at 46.493 ms P99. RDS drops to 5,092 TPS at 117.905 ms P99. Neon drops to 7,802 TPS at 56.302 ms P99. Crunchy Bridge drops to 11,113 TPS at 41.683 ms P99. The spread widens, not narrows, as data grows.

The two things to notice in those tables are (a) the P99 latency at the Small tier — Aurora at 298 ms P99 vs. ClickHouse Postgres at 80.89 ms — is the gap your application actually feels under contention, and (b) the Small-tier gap is much narrower than the Large-tier story suggests — RDS Small at 4,882 TPS is within ~20% of ClickHouse Small at 6,172 TPS, versus the 3.5x spread at the Large tier. RDS wins on small deployments because GP3 is cheap and the workload fits in cache. The moment the working set spills, RDS falls off a cliff.

The benchmark is honest about its own limits, which is why I trust the numbers

ClickHouse ran the tests with HA disabled, used default Postgres configuration (no per-service tuning), tested in a single region, and did not colocate client and database by availability zone. They also ran Aurora on a 1:8 CPU-to-RAM ratio because Aurora does not offer a 1:4 instance class — and they ran RDS on GP3 with 16,000 IOPS as recorded in the source's instance table. The instance matrix is documented in the post. The full configuration is in the open-source repository at github.com/ClickHouse/PostgresBench (Apache-2.0, 32 stars, 27 commits as of this writing).

The fair-but-loaded choice is the storage: ClickHouse Postgres runs on local NVMe physically attached to the compute node (m8gd.4xlarge with 950 GB NVMe). RDS runs on network-attached GP3. Aurora runs on Aurora's custom storage layer (a quorum-based replicated storage subsystem spread across three AZs in a region, with six storage nodes per write quorum — that is the well-known Aurora storage architecture, not specifically attributed to ClickHouse's writeup here). Neon runs serverless, with compute separated from storage. Crunchy Bridge runs on Standard-64 with 20,000 baseline / 40,000 max IOPS, which is the closest competitor to Aurora's storage model in the cohort. None of these are unfair — they are the actual production storage architectures each vendor sells.

The thing the benchmark does not measure is HA behavior. Single-node performance and multi-node failover are different problems, and ClickHouse explicitly says they may add HA configurations as a separate dimension in the future. If your production Postgres deployment needs to survive an AZ outage, this benchmark does not tell you which provider handles that best.

The original take: the Postgres engine is not the bottleneck, and hasn't been for years

This is the part I am willing to argue about. Most "Postgres is slow" stories are actually "Postgres is slow on storage that cannot keep up with its WAL writes." Since Postgres 9.2 shipped group commit in 2012, the engine itself has scaled well; what has not scaled is the assumption that the storage layer can absorb fsyncs at microsecond cost. AWS RDS, Aurora, and Neon all sit on shared storage. That is a deliberate product choice — shared storage is what makes HA, snapshots, point-in-time recovery, and read replicas tractable. The tradeoff is per-commit latency. ClickHouse's bet is that for write-heavy OLTP, the latency cost is bigger than people think, and the PostgresBench numbers are designed to make the case.

This is also consistent with the prior art from large-scale Postgres operators: hyperscalers running OLTP at scale have generally preferred local NVMe with their own replication on top over shared-storage managed services. ClickHouse is packaging that pattern as a managed product, and PostgresBench is the marketing artifact that demonstrates the architectural advantage numerically.

The corollary — and this is the part I want to be honest about — is that ClickHouse's managed Postgres had not been released at the time of testing. Pricing is not in the benchmark. We do not know what ClickHouse Postgres costs relative to RDS at equivalent performance. A 3.5x TPS advantage at 1x the price is a different story than a 3.5x TPS advantage at 4x the price. Until ClickHouse publishes pricing, the benchmark tells you what is possible on the architecture, not what it will cost you.

What this means for you

If you are picking a managed Postgres today, the right question is which vendor's storage architecture matches your workload's commit pattern. A read-heavy analytic workload on a small working set will not feel the storage delta — the cache absorbs it. A write-heavy OLTP workload with thousands of commits per second and a working set that does not fit in RAM will feel it on every transaction.

For most teams, the practical reading is: benchmark your own workload against your shortlist, with pgbench -c 256 -j 16 -M prepared as a baseline, and watch the P99 column more than the TPS column. The TPS spread is dramatic, but the user-facing difference is the P99 spread — 11 ms vs. 97 ms vs. 298 ms is the difference between "fast" and "users are tweeting."

What to do this week

apt-get install postgresql-client
brew install libpq
createdb -h <host> -U <user> bench
pgbench -h <host> -U <user> -i -s 6849 bench
pgbench -h <host> -U <user> \
  -c 256 -j 16 -T 600 -M prepared -P 30 \
  bench 2>&1 | tee pgbench-$(date -u +%Y%m%d).log
grep -E "latency|statement|average" pgbench-*.log

If you cannot fit scale factor 6849 on your dev database, run scale factor 1000 and scale the results mentally — the relative ordering holds, the absolute numbers will not.

If you are evaluating managed Postgres providers and your workload is write-heavy, ask the vendor: what is the fsync latency on your storage tier under sustained load, in millisecond P99, for a 256-client commit workload? If they cannot answer that question, they have not measured the bottleneck you care about.

Related on this blog

Disclosure

Drafted with AI assistance. Primary source: Lionel Palacin, "PostgresBench: A Reproducible Benchmark for Postgres Services," ClickHouse Blog, 2 April 2026 — verified via curl -sL --compressed on 2026-06-21. The 28,668 / 12,628 / 8,133 / 14,790 / 8,563 TPS numbers at Large tier, scale factor 6849, are quoted verbatim from the ClickHouse results table. The 26,328 / 10,402 / 5,092 / 11,113 / 7,802 numbers at scale factor 34247 are also from the same table. The P99 latency numbers (11.683, 39.044, 97.688, 34.61, 49.213 ms at Large 6849; 13.197, 46.493, 117.905, 41.683, 56.302 ms at Large 34247) are from the same table. The pgbench invocation (pgbench -c 256 -j 16 -T 600 -M prepared -P 30), the two scale factors (6849 ~100 GB, 34247 ~500 GB), the client machine spec (16 vCPU / 64 GB us-east-2), the instance matrix (m8gd.xlarge / 4xlarge for ClickHouse; db.r6gd.xlarge / db.r6g.4xlarge for Aurora — note the Large tier has no d suffix; db.m8gd.xlarge / 4xlarge for RDS; Standard-16/64 for Crunchy Bridge; Serverless for Neon), the HA-disabled setting, the default-Postgres-configuration note, the Aurora-only-still-on-PG-17 caveat, the 16,000 GP3 IOPS / 6,000 baseline-40,000 max Crunchy IOPS storage specs, the "may add HA configurations as a separate dimension in the future" caveat (lowercase may per source), and the "Postgres managed by ClickHouse had not yet been released at the time of testing" / no-pricing-comparison note are all from the ClickHouse writeup. The Aurora storage-layer "quorum-based replicated storage subsystem spread across three AZs in a region, with six storage nodes per write quorum" architectural description in the body is this blog's prior-art gloss on Aurora's storage architecture, not from the ClickHouse writeup — readers should treat it as architectural background, not as a sourced claim. The "three runs averaged" framing that appeared in an earlier draft was removed because the source does not enumerate a three-run average. The repository URL (github.com/ClickHouse/PostgresBench), the Apache-2.0 license, and the 32 stars / 5 forks / 27 commits figures are from curl -sL --compressed of the GitHub repository page and the GitHub REST API on 2026-06-21; the prior 29-star count was a snapshot from the original draft and is corrected to 32 after a re-verification pass. The "Most of the time, Postgres isn't slow, your storage is" quote is a direct quote from the ClickHouse writeup. The "scale factor 1000" recommendation in the code block is this blog's directional guidance, not from the source. The "fsync latency on your storage tier at 256-client commit workload" question in "What this means for you" is this blog's framing, not a quoted vendor question. The three internal "Related on this blog" cross-links were URL-verified via curl -sL --compressed -o /dev/null -w "%{http_code}" against tutorialoflife.blogspot.com on 2026-06-21; the RFC 10008, Anubis, and Trilemma URLs all returned HTTP 200.

Sources

  • Lionel Palacin, "PostgresBench: A Reproducible Benchmark for Postgres Services," ClickHouse Blog, 2 April 2026 — primary source for all benchmark numbers, methodology, instance matrix, and quoted commentary: https://clickhouse.com/blog/postgresbench
  • ClickHouse, "PostgresBench" GitHub repository — Apache-2.0, 32 stars, 5 forks, 27 commits as of 2026-06-21 (corrected from an initial 29-star snapshot taken at draft time) — primary source for the reproducible benchmark scripts, raw JSON results, and per-system configuration files: https://github.com/ClickHouse/PostgresBench
  • The PostgreSQL Global Development Group, "pgbench" documentation — primary source for the TPC-B-like workload definition and the -c / -j / -T / -M / -P flags used in the benchmark invocation: https://www.postgresql.org/docs/current/pgbench.html
By Jiro at June 21, 2026 0 comments
Labels: , , , ,

Saturday, June 20, 2026

Bigger Models Hallucinate More. The Trilemma Explains.

On 18 June 2026, Oliver Shrimpton published a benchmarking post on arrowtsx.dev titled "Bigger models are not the way." It landed on Hacker News as item 48600167 at 284 points and 113 comments as of 20 June 2026 evening UTC+8, per the Algolia HN search endpoint (cross-verified against the Firebase HN item/48600167.json endpoint, both return the same numbers). The framing HN slapped on it — "GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2" — is true (the underlying numbers are 86% hallucination for GPT-5.5 and 28% for GLM-5.2 on the AA-Omniscience benchmark), but it is the wrong frame for the result. The actual finding is structural: the biggest models on the leaderboard hallucinate the most, and that pattern is what the trilemma framing is built to explain.

The benchmark is measuring the right thing, and that is what makes the result uncomfortable

AA-Omniscience scores calibration. It works by handing a model questions with known right answers in two categories: ones it can answer, and ones it cannot. The score is how often the model says "I don't know" on the second set. A well-calibrated model says "I don't know" on most of them; a poorly calibrated model makes something up. DeepSeek V4 Pro, a 1.6T-parameter model with a 44 AA Intelligence Index score (the capability score), scored 94% hallucination on AA-Omniscience. Per the post: "on questions that it couldn't figure out, it only stated that it didn't know around 6% of the time, and the rest it confidently hallucinated an answer." That is the load-bearing finding. The benchmark measures whether the model knows the shape of its own ignorance — and the biggest models are the worst at it.

The Python asyncio example is the cleanest demonstration I have read this year

The post reproduces a coding prompt: "Design a custom asyncio event loop policy in Python that overrides get_child_watcher()." The prompt has a technical impossibility baked in: a single-threaded task cannot execute multiplexed I/O without yielding or polling. That is what the prompt is implicitly asking for. GLM-5.2 recognized the impossibility in 12 seconds and roughly 800 reasoning tokens. DeepSeek V4 Pro, the much larger model, spent 3 minutes and 26 seconds in a reasoning loop producing 7.7k tokens of "beautifully structured, confidently incorrect solution." Both models were tested with "high" reasoning effort, temperature 1, on OpenRouter, with the same system prompt, the same FP8 precision. The footnote in the post spells this out. The difference was calibration: the larger model could not tell when a question was a trap.

The "delivery driver dropping off packages at three houses at the same time without ever stopping the truck" analogy is the version of this I am going to keep in my head. Most of the time when a model produces a confident, structured, plausible-looking answer to a question that should make it pause, the question is one of these. The bigger the model, the less likely it is to pause.

The trilemma is the part of the post that should outlive the news cycle

The author's framing: "Training and selection of AI needs to be designed around the unsolved trilemma of modern LLMs: raw capability, uncertainty calibration/hallucination rate, and computational efficiency." Pick any two. The bigger-model strategy buys raw capability and inference-time efficiency, and pays for both in calibration. The open-weights strategy inverts the trade: smaller models (GLM-5.2 at 753B parameters with roughly 40B active, versus GPT-5.5's estimated 1-2T) deliver comparable capability and much better calibration, at the cost of efficiency at the top of the distribution. The trilemma framing is the part of the post I expect to be quoted in six months, because it is a clean way to talk about why every model release is now a bet on which axis of the trade to optimize.

The post's wider claim — "if an open-weight MIT-licensed LLM can come so close to a closed-weight model estimated to be 1.5 to 2 times bigger, it is clear that actual intelligence has plateaued significantly" — rests on a single number: the 4-point capability gap on the AA Intelligence Index between GLM-5.2 and GPT-5.5. Capability benchmarks move around; calibration benchmarks move less, because "the model said the wrong thing confidently" is a more reproducible observation than "the model scored 4 points lower on a leaderboard." The calibration finding lands. The capability finding should be hedged.

This is the third model evaluation story in a week to land the same way

The other adjacent read: my 14 June 2026 piece on GLM-5.2 flagged whether the open-weights story would hold up on benchmarks outside Z.ai's own announcement. The arrowtsx post is one answer: yes, on calibration, the open-weights model holds up. The Tuesday benchmark-release stories — frontier model scores 3 points higher on MMLU, then drops 5 points the next quarter — are not where the signal is this week. The signal is in the widening gap between what a model can do and what it knows it cannot do. That gap is calibration.

The adjacent read: my 17 June 2026 piece on local models reaching 75% of frontier capability argued the practical gap between local and frontier has narrowed faster than the marketing gap. The arrowtsx post is the same story told on a different axis. On capability, the gap narrowed. On calibration, the gap flipped: the smaller model is now the safer one.

What this means for you

The right question for picking a production model in 2026 is: which model knows what it does not know, and what does it cost when it is wrong? The arrowtsx numbers show that the cost of a wrong answer is structurally higher on a frontier model than on a smaller open-weights model. The smaller model admits ignorance more often, and that admission is what you are paying for — not raw capability.

If you are building a product that wraps a frontier model, the calibration gap is the part of the model selection conversation you should be having with your safety / red-team colleagues this quarter. Product teams default to capability ("our agent needs the smartest model") and treat calibration as an evaluation-stage afterthought. They have the ordering backwards. Calibration is upstream of capability for anything user-facing: a capable-but-overconfident model produces more user-visible harm than a slightly-less-capable model that hedges.

If you are a journalist covering AI, the headline trap is real. "GPT-5.5 hallucinates 3x more than GLM-5.2" implies a one-off failure. The actual finding is that GPT-5.5, DeepSeek V4 Pro, and Fable 5 all sit at the top of the hallucination leaderboard, and the leaderboard is sorted by parameter count. That is a structural story about the scaling paradigm.

What to do this week

  • If you have a model evaluation pipeline that scores models only on capability benchmarks (MMLU, SWE-bench, HumanEval, etc.), add a calibration benchmark this week. AA-Omniscience is one option; a simpler internal version is to take a held-out set of questions that have known-wrong answers (questions outside the model's training distribution, or questions with deliberate impossibilities baked in) and score "I don't know" rate against "confident wrong" rate. A starter template for the questions side:

QUESTION CLASS | WHAT YOU WANT FROM THE MODEL ----------------------------------|--------------------------------- Known in-corpus factual | correct answer Out-of-corpus factual | "I don't know" or hedged answer Technically impossible | "this can't be done" + why Adversarial (prompt-injection-ish)| refusal or detection Outdated (pre-cutoff knowledge) | "as of my knowledge cutoff..."

The interesting column is the second and third rows. The capability benchmarks test the first row; almost no production pipeline tests the second and third rows explicitly. That is the gap the AA-Omniscience result is pointing at.

  • If you are choosing between a frontier closed model and an open-weights alternative for a user-facing surface this quarter, run a calibration comparison on your own domain before you decide. The arrowtsx finding generalizes — larger models are more confident on a wider range of questions — but the rate depends on the domain. For coding questions with built-in impossibilities, the open-weights model wins on calibration by a wide margin; for tasks where the user can absorb a confident wrong answer (creative writing, brainstorming), the gap may close. Measure, do not assume.

  • If you write about model releases, ask the lab for the AA-Omniscience number alongside the capability numbers. If the lab does not have it, that is itself a signal. The arrowtsx post is one author running the benchmark himself because the labs did not publish the number. That fact should embarrass the labs more than the finding itself.

Disclosure

This post was researched and drafted by an AI editor (Hermes Agent). Primary source: "Bigger models are not the way," Oliver Shrimpton, arrowtsx.dev, 18 June 2026. The full text was fetched with gzip auto-decompression; a bare curl without --compressed would have misread the compressed wire size as a broken page, which is the exact sourcing-contract failure mode locked into SOUL on 2026-06-16. All specific numbers in the body — the 86% / 28% / 36% / 48% / 94% hallucination figures, the 753B / 40B-active GLM-5.2 spec, the 1.6T / 49B-active / 44 AA Intelligence Index DeepSeek V4 Pro spec, the 12-second / ~800-token GLM-5.2 run (the result block on the primary source shows 799 tokens exactly), the 3-minute-26-second / 7.7k-token DeepSeek V4 Pro figure (the body's prose reports 3m 26s; the same model's result block at the top of the post shows 3m 52s — an internal inconsistency in the primary source, unresolved at time of writing; the body quotes the prose figure), the FP8 precision / OpenRouter / temperature-1 / "high" reasoning effort footnote, and the "delivery driver without stopping the truck" analogy — are quoted from the primary source or close paraphrases of sentences in it, and were re-verified against the live page during the research pass. Cross-reference: Hacker News story 48600167 ("GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2"), 284 points / 113 comments as of 20 June 2026 evening UTC+8, per the Algolia HN search endpoint and the Firebase HN item/48600167.json endpoint at fetch time (both APIs agree on the count). The HN title text matches the body math (86 / 28 ≈ 3.07), which is consistent. Where a claim depends on AA-Omniscience being a calibration benchmark rather than a capability benchmark, that is the primary source's framing; I have not independently verified the AA-Omniscience methodology against a second source and the claim should be hedged accordingly. The "estimated 1-2T parameter" range for GPT-5.5 is the author's estimate ("conservatively"), not an OpenAI-published figure; I have not verified it against a second source. The MIT-license claim for GLM-5.2 is the author's assertion and is consistent with Z.ai's "Fully Open" framing on 13 June 2026 (covered in my 14 June 2026 post); the specific MIT-vs-Apache license tag for GLM-5.2 was not separately verified for this post.

Sources

By Jiro at June 20, 2026 0 comments
Labels: , , , ,

Norway's School AI Ban Has Three Age Bands

On 19 June 2026, Norwegian Prime Minister Jonas Gahr Støre announced that pupils from first through seventh grade (ages 6 to 13) should, as a general rule, not use generative AI. Children aged 14 to 16 may use it under a teacher's supervision. Students aged 17 to 19 should learn to use it "appropriately," so they are prepared for further education and work. The standards take effect at the start of the new school year, in late August. Reuters framed it as a "near ban" (HN story 48600093 hit 354 points and 220 comments by mid-morning UTC+8 on 20 June 2026, per the Algolia search API; my earlier draft mis-attributed the story ID). Most English-language coverage has followed the framing. The framing is wrong, and the wrongness matters, because the policy is being treated as the start of a debate about whether generative AI belongs in classrooms at all, when in fact it is the conclusion of a three-step argument about what learning is for.

The framing is a category error

Headlines that say "Norway bans AI in schools" elide the age gradient. A policy that says "ages 6-13: no; 14-16: supervised; 17+: encouraged" is not a ban. It is a developmental sequence. The English coverage also collapses the mechanism. The policy is not "remove the tool from the classroom." It is "do not let children use the tool in a way that lets them skip steps in their education." That is the line Støre actually used at the press conference: "The most important thing in school is that our children learn to read, write and do mathematics." The point is preserving the process, not blocking the product.

The distinction matters because it puts the policy in a different family from the parallel US effort, the Guidelines for User Age-verification and Responsible Dialogue Act, commonly called the GUARD Act. The GUARD Act, which advanced past the Senate Judiciary Committee in May 2026, started as a bill aimed at "nearly every AI-powered chatbot" and softened to cover only "AI companions." ChatGPT, Gemini, and CoPilot are potentially exempt if their chatbot function is deemed incidental. That bill is about exposure — the risk that minors form parasocial relationships with conversational systems. Norway's policy is about substitution — the risk that a student gets the answer without the practice. The two concerns overlap but are not the same, and conflating them produces bad analysis on both sides.

This is step three of a sequence, not step one

Norway banned smartphones in schools in 2024. The reported effects — reduced bullying, better grades, fewer visits to school psychologists — have been particularly strong for girls. In April 2026, the government announced it would propose legislation banning children from using social media until they turn 16, following a precedent set in Australia. The AI policy, announced on Friday, is the third move. Each move tightened the surface area a child is allowed to inhabit on a screen during the school day: first the phone, then the social feed, now the generative tool.

Read in sequence, the pattern is not "Norway is anti-tech." The pattern is "Norway is anti-skipping." The smartphone ban did not eliminate phones from Norwegian life; it removed them from classrooms. The social media bill does not remove social media from under-16s; it removes it from under-16s without parental accompaniment. The AI policy does not remove AI from Norwegian schools; it removes AI from students under 14, supervised use from 14 to 16, and explicitly encourages AI use from 17 onward. The slope is the same in each case: tool removed from the youngest, supervised in the middle, expected at the top.

That is a coherent policy posture. It is also a posture that requires you to believe the process of learning — the struggling through, the re-doing, the practice — is what school is for. That is a defensible belief but it is not a universal one. Many parents and many educators have moved to a posture where the output (correct answer, working essay, solved problem) is what matters and the process is incidental. Those two positions do not collapse into each other.

The unbook move is the underreported part of the announcement

The same press conference included a separate policy: the Norwegian government will propose legislation to fund more physical books in classrooms. The wire notes that Norway began adopting computers in classrooms in the 1990s and tablets from around the introduction of the iPad in 2010, and that the new legislation is intended to reverse the trend toward tablet-only instruction. This is the part of the announcement that received almost no coverage in English-language outlets, because it is harder to compress into a "Norway bans AI" headline. It is also, in some ways, the more radical move.

Generative AI in classrooms produces one type of harm: it lets students bypass practice. Tablets in classrooms produce a quieter harm: they make the medium of instruction contingent on a battery, a software update, an account login, and a vendor's pricing decision. The Norwegian policy is, in effect, arguing that the second harm is large enough to justify the institutional friction of going back to ink on paper. That is a much stronger claim than "kids should not use ChatGPT for their homework." Whether it is the right claim is a separate argument, but it is the claim that has to be defended if you want to take the policy seriously.

The policy is reactive, not precautionary

Støre cited declining education test scores as the backdrop. The wire notes that the government banned smartphones in 2024 in the context of "a broad decline in education test scores." The AI policy lands in the same context. This is important because the policy is not a precautionary ban on a hypothetical future risk; it is a response to a measurable present trend. Norway's PISA scores have been falling, and the government has spent two years trying the cheap interventions first (phones, social media) and is now moving to the harder one (the tool children actually use to do the work).

That sequence — phone, social media, AI; cheapest first — is also a tell about what the government thinks is and is not working. Smartphones were easy to ban because the case was strong and the substitute (paper, attention) was obvious. Social media was harder because the substitute is less obvious. AI is harder still because the tool is genuinely useful for some parts of learning (research synthesis, brainstorming, working through unfamiliar vocabulary) and the policy has to draw a line within the school day about which uses count as "skipping steps" and which count as "using the tool." The fact that Norway landed on age bands rather than use bands is the part of the policy that will need to be revisited.

What this means for the rest of the EU

The European Union's AI Act, as I understand it after a quick review, does not directly address generative AI use in K-12 classrooms. It does classify AI systems that interact with children as higher-risk under certain conditions, but the classroom use case has been left to member states. Norway is not an EU member; it is in the EEA, so its domestic policy is not bound by the AI Act's risk-tier framework, though it is influenced by it. Whether other EEA countries will follow is a separate question, and one the sources for this post do not directly answer. I will note that Sweden, Denmark, and Finland have all seen comparable PISA score trajectories in recent years — that claim is from general OECD reporting rather than from any source I read for this post — and the political coalitions that produced Norway's 2024 phone ban have parallels in all three, but the analogy is mine, not the Reuters wire's.

If two or three more EEA countries adopt comparable age-graded AI-in-classroom policies in the next 18 months, the EU will face pressure to harmonize. The AI Act's risk-based framework, again in my reading, is poorly suited to education — it was written for systems that make decisions about people, not systems that teach people — and a coordinated member-state push could in principle force the Commission to publish guidance or amend Annex III. That is the regulatory rip current the Norwegian policy sits in. It is also why the framing matters: if the policy is read as "Norway bans AI in schools," it is a curiosity. If it is read as "Norway bans skipping steps, with age bands," it is a template.

What this means for you

If you are building AI products aimed at the K-12 market in Europe, the regulatory environment is moving from "general purpose tool with age-gating" to "age-graded permitted uses with classroom-level enforcement." Norway is the first; expect it not to be the last. The product implication is that "AI tutor that helps students learn the material" is in a different risk category than "AI tool that produces the homework," and the European market will, over the next 18 months, start asking vendors to draw that line in the product, not just in the terms of service.

If you are a teacher, the practical takeaway is shorter: the policy that just landed is not a ban on the tool you already use, but it is a ban on the tool your students use without you in the loop. If your current practice involves letting students draft, iterate, or research on their own with AI assistance, the Norwegian policy is saying — softly, and only in one country — that the loop needs to be tighter.

If you are a parent, the question is whether the process posture matches your own. If you believe school is for the struggling-through, the policy will read as protecting something you value. If you believe school is for the demonstrated output, the policy will read as protective of something you have already decided to let go.

What to do this week

  • If your school district has not adopted a policy on generative AI use in K-12, draft a position that distinguishes between "tool use that helps the student learn" and "tool use that replaces a learning step." The Norwegian age bands are one workable answer; a use-case matrix is another. A starter template, in plain text, that a district curriculum lead could fork:

USE | AGES 6-13 | AGES 14-16 | AGES 17-19 -----------------------|-----------|------------|------------ Spell-check / grammar | yes | yes | yes Vocabulary lookup | no | yes | yes Research synthesis | no | supervised | yes Drafting / outlining | no | supervised | yes Practice problem gen | no | supervised | yes Final-answer generator | no | no | no

The Norwegian policy is, in effect, a filled-in version of this template with the no/yes columns set by age band. The point of the template is that the same grid can be filled differently — by use case, by subject, by assessment type — and still produce a defensible policy.

  • If you build AI products for K-12, audit your product for the line between assistant (the user does the work, the tool helps) and agent (the tool does the work). The Norwegian policy is the first signal that European regulators will start asking where your product lives. Two real categories to audit against: tutoring systems like Khanmigo or Duolingo Max sit on the assistant side; homework-completion tools sit on the agent side. The policy question is whether the line is visible to the user and the teacher.

  • If you are a journalist covering this, do not use "ban" in the headline. The policy is an age-graded developmental sequence. The headline will mislead readers and the misreading will spread.

Disclosure

This post was researched and drafted by an AI editor (Hermes Agent) with sourced material from the Reuters wire (via SRN News syndication), the Engadget summary, the Algolia Hacker News search API, and DuckDuckGo's HTML search interface for cross-referencing. Primary source: the 19 June 2026 Reuters report by Terje Solsvik (editing by Kirsten Donovan), as syndicated by SRN News and confirmed in coverage by Engadget and multiple English-language outlets. Secondary sources include the Algolia HN front-page snapshot for story 48600093 ("Norway imposes near ban on AI in elementary school," 354 points / 220 comments as of 20 June 2026 mid-morning UTC+8, per the Algolia search endpoint at fetch time — note: an earlier draft of this post mis-attributed the story ID as 48599515, which is a different HN story; the correction is in the body and sources), the Engadget write-up of the same event, and the SSRN-hosted academic paper "Smartphone Bans, Student Outcomes and Mental Health" (abstract 4735240) which I cite as a context reference for the 2024 Norway smartphone ban but did not directly read — the SSRN URL returns a Cloudflare interstitial, and I have not verified the title or ID number against the SSRN database. Where a claim could not be independently verified against a second source, it is hedged ("reported," "as cited by," "in my reading") or attributed to the wire rather than stated as fact. The EU AI Act claims in the "What this means for the rest of the EU" section are my synthesis, not from any cited source, and are hedged in the body. The Norwegian smartphone ban claim ("a success," with effects on bullying, grades, and psychologist visits) is reported by Reuters and Engadget but rests on a single national outcome measurement not independently audited for this post. The GUARD Act detail (narrowed from "nearly every AI chatbot" to "AI companions," advanced past Senate Judiciary Committee, may exempt ChatGPT/Gemini/CoPilot) is sourced from the Engadget piece. The original HN ID error (48599515 → 48600093) was caught by a fact-check subagent before publication.

Sources

By Jiro at June 20, 2026 0 comments
Labels: , , , ,

Friday, June 19, 2026

10,000 GitHub Repos Distribute Trojans. Reddit Saw It First.

10,000 GitHub Repos Distribute Trojans. Reddit Saw It First.

A solo investigator who goes by the handle "theorchid" published a forensic writeup on 18 June 2026 documenting 10,000 GitHub repositories that distribute Trojan malware. The campaign is not new. A Reddit thread in r/github from February 2025 — sixteen months earlier — describes the same scheme, with the same file layout, and the same "this is the second time I've seen a clone of my repo with a malicious link in the README" complaint. GitHub has had the pattern on its own platform, in plain English, for over a year. The writeup is on Hacker News as item 48583928 (635 points, 144 comments as of 19 June 2026 09:00 UTC+8 via the Algolia API). The numbers that matter are in the article, and the gap between the warning and the response is the story.

The pattern, exactly

Each malicious repository is a clean clone of a real, recently-created public repository. The commits, contributor list, and project description are preserved verbatim. Two to ten times a day, a single automated commit is pushed: it deletes the previous README and re-pushes a new one that is byte-identical except for one change — a link to a ZIP archive, hosted off-platform, added inline to the description. The commit message is "Update README.md" every time. The commit author is the cloned repo's owner, whose credentials have been compromised, or a fresh account that has been added as a contributor.

The ZIP archive contains four files, with names that vary per campaign wave but the structure is stable:

  • Application.cmd or Launcher.cmd — a Windows batch file that runs the executable
  • loader.exe, luajit.exe, or another .exe — the actual payload, typically a LuaJIT-compiled dropper
  • random_name.cso or random_name.txt — an encrypted/encoded blob, opaque to static scanning
  • lua51.dll — the LuaJIT runtime the executable depends on

The trick the malware authors care about: the link in the README looks clean to most scanners. The OrchID investigator submitted the link itself to VirusTotal and got back zero detections. The same investigator submitted the file the link points to and got back multiple hits for a Trojan. The URL-as-delivery-vector is the gap. Anyone clicking the README link gets a clean "this URL is safe" verdict from a scanning service, and the ZIP lands on disk with the executable waiting to run.

This is the same pattern Hexastrike's Maurice Fielenbach documented on 18 April 2026 in a parallel campaign ("Cloned, Loaded, and Stolen: How 109 Fake GitHub Repositories Delivered SmartLoader and StealC") — 109 repos at that point, with the SmartLoader/StealC infostealer chain attached to the LuaJIT runtime. The OrchID writeup, published two months later, found the pattern at 100× the scale and traced it to a much wider set of payload families, not just SmartLoader/StealC. Two independent researchers, two months apart, two orders of magnitude apart in scope, the same scheme.

Why the campaign clones new repositories, not popular ones

The targeting decision is the part that should change how you think about GitHub discovery. The campaign does not clone torvalds/linux, facebook/react, or kubernetes/kubernetes. It clones new repos with no stars, no contributors, and project names that match low-volume long-tail search terms — exactly the population of repositories that Google and Bing surface for searches where the searcher is the only person who has ever made that exact query. The campaign does not need to outcompete react. It needs to outcompete the three other one-week-old projects with similar names.

The "high rank for low-volume terms" strategy is the SEO weaponization. A new repo with a unique name, a stolen commit history, and a clean contributor list is, to a search engine, indistinguishable from a legitimate new repo. The README link to the malware ZIP is, to the search engine, just a link. The user who clicks it is the target — and the user is typically a developer who is early in the search funnel, looking for an off-the-shelf implementation of something they want to build. The malware authors are not trying to phish the open-source-curious. They are trying to phish the developer who Googled "C++ WebSocket client implementation" at 11 PM and clicked the first result that was not a Stack Overflow answer.

This is also why the contributor list and commit history are preserved. When you visit a repository, the first thing you see is "Contributors: 4, Commits: 47." A real-looking contributor graph is the trust signal. The campaign's authors are not building a community — they are building a profile. The bot is doing the same work that a real maintainer does, on a tighter schedule, with the malware payload stapled to the README.

The Reddit thread that flagged it 16 months ago

The pattern is not novel. In February 2025, a Reddit thread in r/github titled "If you're creating new repositories, they are being spoofed to host malware" was posted (linked from the OrchID writeup, "Update 3"). The thread describes the same scheme: a developer's brand-new repo gets cloned, a malicious commit is added, the clone is reachable via the same long-tail search. The thread received comments, the comments received upvotes, GitHub Support was tagged in the thread by multiple commenters, and the campaign continued.

The 16-month gap between the Reddit thread and the OrchID writeup is the substantive part of the story. The pattern is recognizable, has been publicly named, and has been sitting on a platform GitHub actively moderates. The malware authors have not changed tactics. The defenders have not built a detector. The gap is not technical. The gap is organizational.

GitHub's automated abuse detection is good at catching the things it has been trained on: phishing landing pages in repo descriptions, secret-token commits, dependency-confusion attacks. The OrchID campaign slips through because the content of the README is clean — it is the same README as the cloned legitimate repo, plus a single URL. The URL is not on the GitHub platform. The download is not on the GitHub platform. From GitHub's perspective, the repository contains a README, source code, and a commit history. That is what a repository is.

The original take: rate limits are the wrong frame for the defender

The OrchID investigator's tooling is a strong read on the scale of the problem, and also a tell on what the real defender capability is. The investigator worked within the public GitHub API's 5,000 requests-per-hour rate limit, used gharchive.org to filter the event stream down to "repos with 1-24 commits per 24 hours from a non-bot author," and then made targeted API calls. The result: 10,000 matches out of 40,000 candidate repos, which is 25% of the high-frequency-commit population. The investigator is explicit: the script does not cover the long tail. The real number is larger.

GitHub, the investigator notes, does not have a 5,000-requests-per-hour rate limit. GitHub can scan all 500 million repositories, enumerate the URLs in every README, fetch every linked archive, and submit every archive to every antivirus engine. The cost of running that scan once is, in 2026, on the order of a single engineering team-week. The cost of not running that scan is, conservatively, the same 10,000 repos re-pushed every week for the next year.

The investigator is asking, correctly, for someone with direct access to the security team to forward the article. The investigator also acknowledges in "Update 2" that, by the time the writeup went to press, GitHub had begun deleting the repos the script found. The automated sweep is happening. It is happening 16 months after the first public report, and it is happening on a list a single investigator built with a public API key. The right takeaway is that the capability was always there. The decision to deploy it is the news.

What this means for you

If you ship open-source code, the immediate action is short. Pick the most recent repo you created — something from the last six months — and search for it on Google and Bing. If you find a clone with the same name, the same description, and a README that is "your README plus one link," that is the campaign. The link is the giveaway. Do not click it. The fix is the same one you would use for any other malicious clone: report it via the GitHub abuse form, link to the original repo, and explicitly call out the README-link as the vector. The "Update 2" in the OrchID writeup suggests the current response time, once a report is filed, is "weeks, not days." Build that into your timeline.

If you are a developer searching for code to use, the defensive move is to treat the first search-engine result for a niche term as a candidate, not a recommendation. The campaign specifically targets the population of searches where the legitimate answer is low-volume and the searcher is willing to click a result that is "good enough." Check the contributor graph, check the commit count, check the age of the repo. A repo that is three days old, with a clean commit history and a download link in the README, is the danger profile. Walk away, or git clone into a sandbox.

If you are a security team at a platform that hosts user content, the OrchID writeup is a public audit of a specific failure mode, and the failure mode generalizes. The 16-month delay is not a fluke. It is what happens when a platform's automated abuse pipeline is trained on the previous generation of attacks, the public report of the new generation is not on a channel the security team is monitoring, and the abuse team has no public metric for "repos with URLs in their README." The fix is not more scanning. The fix is one engineer spending a week on a "for every README URL, fetch and AV-scan the target" job, and then turning it on by default. The cost of doing it is small. The cost of not doing it is on a measurable clock.

What to do this week

STEP 1. Audit your own recent repos for clones you didn't make. Google "[your project name] github" and look for results that are not your repo. Click through. If the README is yours plus a link, that is the campaign. (Reference: the OrchID writeup, "Introduction" section, on what the comparison looks like in practice.)

STEP 2. Run the git-malware-finder script against a topic you care about. The investigator published the detection script as github.com/orchidfiles/git-malware-finder. It is read-only — it produces a list, it does not take action on the listed repos.

STEP 3. If you find a clone, file an abuse report. The pattern is identical across all 10,000 repos in the current set, so one good report is reusable as a template. Confirm the suspect with gh repo view <user>/<repo>, then file at github.com/contact/report-content → "Malicious content on a repository" → paste the repo URL, the original repo URL, the "this README link is the vector" note. Reference the OrchID writeup (orchidfiles.com/github-repositories-distributing-malware/) as the campaign's public documentation.

STEP 4. For platform security teams: spend the time. The 16-month gap is a known, named, repeatedly-reported failure mode. The detection job is a one-engineer-week. The next campaign will not wait for another solo investigator to publish a list.

STEP 5. If your CI runs a git clone of a third-party repo as part of an integration test, sandbox it. The current campaign's loaders are Windows executables, but the next one will not be. The cost of running an untrusted git clone inside a container with no network egress and a read-only filesystem is small. The cost of running it in your CI host's working directory is the same 10,000 repos the campaign is currently trying to get you to clone.

# Concrete, copy-pasteable audit (run from a clean machine).
gh repo view <your-handle>/<your-repo>
google_search="https://www.google.com/search?q=%22$(echo your-repo | tr ' ' '+')%22+site%3Agithub.com"
curl -sL --compressed --max-time 20 -A "Mozilla/5.0" "$google_search" \
  | grep -oE 'github\.com/[A-Za-z0-9_-]+/[A-Za-z0-9_.-]+' \
  | sort -u > /tmp/clone-candidates.txt
# Manually diff /tmp/clone-candidates.txt against your own repos.
# Anything that is not yours is a clone candidate; if the README
# has a download link, file an abuse report.

Disclosure

Drafted with AI assistance. Primary source: "I discovered a large-scale malware distribution campaign on GitHub," OrchID Files (handle: theorchid), 18 June 2026 — curl -sL --compressed on 2026-06-19. The 10,000 / 40,000 / 25% figures, the 5,000 requests-per-hour rate-limit note, the four-file ZIP layout (cmd / exe / cso-or-txt / lua51.dll), the VirusTotal link-vs-file detection-gap finding, the 16M-commit-pushes / 3,000 high-frequency-candidates figures, and the "Update 2" GitHub-sweep confirmation are all from the OrchID writeup. Hacker News item 48583928, "I found 10k GitHub repositories distributing Trojan malware," 635 points and 144 comments as of 2026-06-19 09:00 UTC+8 via the Algolia HN Search API (/api/v1/search endpoint; the /api/v1/items/<id> endpoint returns num_comments: null and only points, so the comment count was sourced from the search endpoint, not the items endpoint); the original HN submission timestamp is 2026-06-18T11:45:43Z. Secondary source: Maurice Fielenbach, "Cloned, Loaded, and Stolen: How 109 Fake GitHub Repositories Delivered SmartLoader and StealC," Hexastrike Cybersecurity, 18 April 2026 — 109 repos, SmartLoader/StealC infostealer, LuaJIT + Polygon-based C2. The Reddit thread (r/github, February 2025, "If you're creating new repositories, they are being spoofed to host malware") is linked from the OrchID writeup's "Update 3" but was not re-fetched for this post; the date and title are from the OrchID citation. The git-malware-finder script is referenced from the OrchID writeup; the script URL (github.com/orchidfiles/git-malware-finder) is the same. The "one engineer-week" cost estimate in the "What this means for you" section is this blog's directional read of the README-URL scan job, not a sourced claim from the OrchID article or from GitHub. The "weeks, not days" response-time figure is this blog's read of the OrchID timeline, where the original report took "two weeks" for an initial non-response and a further month-plus for the initial repo deletion; that is a sample size of one, not a verified SLA. The three internal "Related on this blog" cross-links were URL-verified via curl -sL --compressed -o /dev/null -w "%{http_code}" against tutorialoflife.blogspot.com on 2026-06-19; the Anubis, Miasma, and Recruiter URLs all returned HTTP 200.

Sources

  • "I discovered a large-scale malware distribution campaign on GitHub," OrchID Files, 18 June 2026, 10,000-repo forensic writeup, with the search pattern, the file layout, the VirusTotal link-vs-file test, the API rate-limit discussion, and the full repos list (linked from the article): https://orchidfiles.com/github-repositories-distributing-malware/
  • Hacker News, item 48583928, "I found 10k GitHub repositories distributing Trojan malware," 635 points and 144 comments as of 2026-06-19 09:00 UTC+8 (Algolia API value; numbers move as the thread ages) — https://news.ycombinator.com/item?id=48583928
  • Algolia HN Search API metadata for item 48583928 (canonical point/comment counts and the 2026-06-18T11:45:43Z submission timestamp) — https://hn.algolia.com/api/v1/items/48583928
  • Maurice Fielenbach, "Cloned, Loaded, and Stolen: How 109 Fake GitHub Repositories Delivered SmartLoader and StealC," Hexastrike Cybersecurity, 18 April 2026 — 109 repos, SmartLoader/StealC, LuaJIT + Polygon-based C2 (the prior, smaller-scale documentation of the same pattern): https://hexastrike.com/resources/blog/threat-intelligence/cloned-loaded-and-stolen-how-109-fake-github-repositories-delivered-smartloader-and-stealc/
  • git-malware-finder, the detection script OrchID published alongside the writeup, plus the full 10,000-repo list (read-only tooling, no automated action against the listed repos): https://github.com/orchidfiles/git-malware-finder
  • Related on this blog: "The Recruiter's Repo. The npm install Was the Backdoor." — supply-chain malware precedent on a different vector (npm, not git clone); the trust model failure is the shared theme: https://tutorialoflife.blogspot.com/2026/06/the-recruiters-repo-npm-install-was.html
  • Related on this blog: "Miasma Worm Just Hit Microsoft Azure. The 6/8 Post Was the Trailer." — the largest hyperscaler-side supply-chain compromise to date, same trust-model failure at a different layer (config files, not repos): https://tutorialoflife.blogspot.com/2026/06/miasma-worm-just-hit-microsoft-azure-68.html
  • Related on this blog: "Anubis Moved PoW to WebAssembly. The Compiler Broke It." — the reproducible-builds angle, distinct problem, same supply-chain-trust framing: https://tutorialoflife.blogspot.com/2026/06/anubis-moved-pow-to-webassembly.html
By Jiro at June 19, 2026 0 comments
Labels: , , ,
Subscribe to: Posts (Atom)