HackerRank's ATS Is Open Source. The Luck Is the Feature.

On the morning HackerRank published their open-source applicant tracking system, a developer named Dan Kinsky opened a terminal, pointed his own resume at it a hundred times, and watched the same document score anywhere from 66 to 99 out of 100. The repo is real, the runs are reproducible, and the bottom line is the design choice everyone in hiring tooling has been quietly making for three years.

The tool in question is interviewstreet/hiring-agent: a Python pipeline that parses a PDF resume, calls a local LLM (default: gemma3:4b) six times to pull structured fields out of work history, education, skills, projects, and awards, optionally enriches the result with GitHub repository scans, and then asks the model to grade the whole bundle out of 100. Up to 20 bonus points get stacked on top for startup experience, a portfolio site, or a technical blog. MIT-licensed, 3,592 stars on GitHub at time of writing, 253 open issues — most of which are the same complaint from different people. HackerRank didn't appear out of nowhere either: the repo dates to July 2025, but the link only went viral after a LinkedIn and r/leetcode pass that started roughly two months later, which matches Kinsky's correction footnote on the post (one LinkedIn post linked; one Reddit thread linked, both in his footnote 1). Anyone who has been watching the AI-in-hiring discourse knows the pattern by now: an LLM is wired into a pipeline that touches millions of decisions, the LLM's behavior changes under load, and nobody on the buying side inspects which version of stochastic they actually deployed.

Kinsky's experiment is the part that should change how the industry talks about the space. With the tool set to its default temperature — 0.1, a setting most people would call "effectively deterministic" — the same resume gets graded on the same rubric and the same rubric returns a 33-point spread on 100 trials. Toggling DEVELOPMENT_MODE off, hard-coding the inputs, and changing nothing except deleting a print() statement would already shift the score by 16 points; looping the model produces the full range. Re-running with Gemini instead of gemma3:4b tightens the distribution — but to a 48-64 band, which still has a 16-point spread and would still fail any cutoff in that range on roughly 28% of submissions (Kinsky's number for a 60-cutoff, not a separate reproduction). The non-determinism is a sampling problem, and the sampling never goes away.

The numbers that matter

Most resume-screeners, including this one, grade on a 100-point rubric anchored to a handful of weighted categories. Hiring-agent's breakdown is unusually explicit about what it's optimizing for: 35 points for open source contributions, 30 for personal projects, 25 for work experience, 10 for technical skills, plus up to 20 in bonus. Read it once and you see what the tool is for: a fairly specific kind of engineer with a specific kind of artifact trail. Candidates whose work happens inside a corporation and stays there — the majority of working engineers, by every measure — start the test at a structural disadvantage that has nothing to do with their quality.

That structural tilt is what makes the non-determinism land so hard. Kinsky ran the tool against the "technical skills" category and watched it score 8 out of 10 in 98 of 100 trials — almost a hard rule, because "did this candidate list React" is the kind of check that any extraction model can do reliably. The "work experience" category came back 25/25 in every run, including against a stripped-down resume listing only one internship — the rubric is two lines long, contains no anchor examples, and the LLM has nothing to vary on, so it just agrees with itself. Categories with something to judge are exactly the categories the tool can't judge consistently. Projects swings wildly. Open source, with the rubric actually reading like a rubric, swings less than it used to but still swings. Kinsky's resume got marked as one that its projects "lack architectural complexity" or, with comparable frequency, projects that "demonstrate real-world deployment" — two opposite readings from the same input, sampled roughly evenly across runs, and the only meaningful distinction between those phrasings is the random seed the sampler hit.

Temperature 0 is a story the model tells you

The HN thread on Kinsky's post spent the first hundred comments litigating the same argument, and it happens to be the part of the story that most confidently deserves a closer reading. In theory, "temperature 0" produces deterministic outputs from a sampling model. In theory-theory — which is the theory library developers actually mean when they quote it — temperature 0 doesn't really exist as a fixed point. The softmax becomes a spike function in the limit, but a discrete tokenizer with a finite vocabulary doesn't carry a true Dirac; it carries a Dirac comb, which collapses to the single highest-logit token only when there's a unique highest-logit token at every position. Floating-point quirks normally paper over that, but the assumption that no two logits will ever tie is exactly the kind of assumption you don't want underwriting a hiring decision.

The deeper issue is that the model is asked to do two jobs with one set of weights: parse a document into structured fields (the part LLMs are good at), and score a candidate against a rubric (the part LLMs are uniquely bad at, because rubric scoring is a discriminative task and chat models are trained to be generative). The tool's own prompt for experience is two lines long, per Kinsky's quoted rubric — read the Production section in the repo: instructions about analyzing work and volunteer sections for real-world or internship experience, plus a special-consideration line that awards extra for founder or early-stage engineer roles. No anchors. No examples. No definition of "real-world." The model is being asked to invent a calibration it was never trained on, and the result is whatever happens to come out of the sampler. That's why an intern and a principal engineer both get 25/25: the prompt can't tell them apart, and neither can the model.

The reproducibility budget is the only metric that matters

Most AI-in-hiring coverage focuses on bias — and deservedly so; the Brookings April 2025 study on gender, race, and intersectional bias in LLM-driven resume retrieval put real numbers behind the failure mode. But reproducibility is the failure mode people who aren't in the literature are about to discover, and it doesn't need a bias-detection study to demonstrate — it just needs Kinsky's terminal loop. A tool whose identical inputs produce non-identical outputs is a tool whose identical candidates produce non-identical outcomes. At any fixed cutoff, the failure rate of "this qualified candidate didn't make it past the screen" is structurally non-zero, and the candidates that fall on the wrong side of the cutoff are random with respect to merit. That's the function the tool is performing. Calling it a "filter" understates it; calling it a "luck filter" catches it.

There are two things worth keeping separate, even though they often get tangled together. The first is LLM bias — outputs that differ systematically across groups, the bias problem the literature has spent two years measuring. The second is LLM noise — outputs that differ across identical inputs, the reproducibility problem Kinsky is documenting. The first matters because fairness is a legal category and a moral category. The second matters because anything with this much noise is unfit for the actual decision even if you fix the bias. A noise-free version of a biased tool is still biased. A noise-heavy version of a fair tool is unfit to use.

Open source changed the optics but not the math

The interesting decision HackerRank made was opening the source. A closed-source LLM screener with 33-point variance would be the kind of "actuarial non-decision" enterprise software tends to hide; an open-source one is a reproducible experiment. Kinsky's loop is the unit-test the entire industry should have been writing since AI resume screeners started shipping in 2022. Anyone can replicate it — and many will, because the cost of doing so is a laptop, a pip install, and an hour. What they will find is what Kinsky found: the tool's accuracy, as a filter, is the same as flipping a weighted coin. Whatever signal the company thought they were buying is in the noise floor.

That distinction matters even more at the buyer side. A screening tool produces a ranking function whose top-K is unstable across runs — meaning its top-K is arbitrary. Companies buying these tools should be asking, before they wire one into Workday, Greenhouse, or Lever, what the tool's reproducibility budget is for the population they're screening. If your top-of-funnel conversion is 10% and your screener has a 30% pass rate at the cutoff, the screen is responsible for roughly half of your funnel noise. Halving the variance by switching to a smaller, deterministic model and tighter prompts would do more for hire quality than any number of model upgrades. Anyone who's been on the receiving end of an unexplained rejection knows this already.

What to do this week

If you're a job seeker:

• Assume a non-trivial share of the screen is a coin flip. Use that as license to apply to roles your gut says you're a fit for, even when your heuristic says you're not.
• The resume rubric HackerRank-style tools optimistically measure is heavy on open source and personal projects. If you have those, surface them more prominently — GitHub README polish, a one-paragraph portfolio, a working demo URL. The tool is explicitly grading on artifacts that look like artifacts.
• If you have none of those, your path through this filter is rougher regardless of quality. Lean on referrals and on company-specific application tracks that bypass the automated screen.

If you're an engineer with a say in how your company screens:

• Run Kinsky's loop on your own tool with your own population. The "100 runs against the same resume" test is the smallest possible reproducible experiment and you should have its output before you trust it.
• Treat any LLM-based screener that returns a single candidate score as inadmissible. Demand either a structured decomposition (the model returns per-rubric scores so you can audit which parts are stable) or a calibration band (each score comes with a standard deviation across N runs).
• If the screener doesn't expose its rubric, what you have is a vibe check with extra steps. The vibe check is the part you don't want.

If you're running the screener yourself:

• Lower the temperature only after you have measured the temperature=1 distribution — the noise floor has to be known to be lowered.
• Replace single-call score generation with multi-sample consensus, or with discriminative models trained on labeled paired comparisons (the actual right tool for the job).
• The single most valuable line in the open-source repo is the temperature: 0.1 default. Change it to 0, document the new spread, and ship the difference.

The feature, renamed

The industry-wide reflex when a reproducibility paper appears is to call the problem "non-determinism" and promise a fix in the next model. Non-determinism is the property, not a bug to patch — and it's a direct consequence of how these models generate text. A model that returns 100/100 with seed 0 and 73/100 with seed 1 is doing exactly what it was trained to do; the prompt engineer has not yet built a system that constrains the sampler. The fix is to stop pretending the model is a sensor when it's a sampler, and to put determinism back into the pipeline by routing it through a part of the system that actually has it. Structured extraction can be done deterministically. Rubric scoring, with the right anchors, can be done deterministically. The middle distance — "judge me on my projects, please" — is where the sampler takes over, and the sampler is supposed to take over there. The honest answer is to admit that's a part of the decision a human has to make.

Kinsky's post is honest about that in a way the industry usually isn't. He isn't angry at HackerRank. He's angry at himself for thinking the tool was testing something it wasn't. Plenty of other readers will be angry at HackerRank; they're right to be, but only about the secondary thing. The primary thing is that the entire category of tool is built on a category error, and the open-source release is the moment that became undeniable. Once you see the same resume swing from 66 to 99 on a hundred deterministic-looking runs, every score that came out of every other LLM screener starts to look like the same number — just with a different seed you can't reproduce.

Disclosure

Drafted with AI assistance. Primary source: Dan Kinsky's 28 Jun 2026 post at danunparsed.com/p/hackerrank-open-source-ats, fetched and cached locally on 29 Jun 2026. GitHub repo interviewstreet/hiring-agent confirmed live via the GitHub REST API on the same date. Brookings 25 Apr 2025 piece on bias is cited only for the bias vs. noise distinction in the body, not for any specific finding. Per-claim attribution and live numbers are in the Sources section below.

Sources

• HackerRank's open-source ATS — Dan Kinsky, "HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74/100. No — 88/100. Actually 83/100.", danunparsed.com/p/hackerrank-open-source-ats, 28 Jun 2026. Primary source for all experimental claims in the body (66–99 spread, 65% cutoff failure rate, 48–64 Gemini band, 98/100 technical-skills consistency, 25/25 experience rubric outcome). Fetched 29 Jun 2026.
• The GitHub repo itself — github.com/interviewstreet/hiring-agent, MIT-licensed Python project, 3,592 stars / 745 forks / 253 open issues at time of writing. Repo created 2025-07-29; first viral LinkedIn/Reddit pass ~Oct 2025 per Kinsky's footnote. Confirmed via GitHub REST API on 29 Jun 2026.
• The HN discussion — Hacker News item 48713832. 730 points / 309 comments at time of writing; thread moving. Used for the temperature-zero analysis and the broader engineering reaction.
• Brookings 25 Apr 2025 on bias in LLM-based resume screening — Kyra Wilson and Aylin Caliskan, "Gender, race, and intersectional bias in AI resume screening via language model retrieval," brookings.edu/articles/gender-race-and-intersectional-bias-in-ai-resume-screening-via-language-model-retrieval/. Used only for the bias vs. noise distinction; no specific findings paraphrased.
• The Reddit r/leetcode pass — referenced in Kinsky's correction footnote (footnote 1) as one of the two original viral-sharing surfaces, 28 Jun 2026. Linked but not directly fetched (Reddit returned a block page to my fetch attempt).

Anubis Moved PoW to WebAssembly. The Compiler Broke It.

Xe Iaso's "I hate compilers" hit the front page of Hacker News on 18 June 2026 with 111 points, and the title undersells what is actually a reproducible-build horror story dressed up as a WASM-to-JavaScript engineering writeup. Anubis — the proof-of-work reverse proxy that this blog covered recently as the de facto answer to the LLM-scraper DDoS problem — is moving its challenge logic from SHA-256 to WebAssembly so administrators can swap in custom PoW schemes. The goal is clean: define the check logic once, run the same bytes on both client and server. The reality is that getting the same bytes out of clang twice in a row is the actual hard part.

The lesson generalizes well beyond Anubis — to anyone shipping compiled artifacts (WASM modules, native binaries, LLVM bitcode, kernel modules) from CI and expecting the bytes to be stable.

Angle 1: Why your WebAssembly binary has a different hash on every rebuild

The first demonstration in Xe's post is the reproducible-builds thesis in twenty lines of C++. The example defines __DATE__ and __TIME__ as compiler builtins that stamp the build timestamp into the output, then compiles the same hello.cpp twice in a row. The two outputs differ in the embedded timestamp. Identical source, different bytes — on every run, for a reason no one designing a "reproducible build" would have invented.

Compiler nondeterminism shows up in three places that the Anubis writeup hits in order: embedded timestamps via __DATE__ / __TIME__ (trivial); tooling the compiler shells out to, like Clang silently invoking wasm-opt from $PATH (surprising); and address-sensitive codegen, where pointer values leak into the order of try_table blocks in Clang's exception-handling path (genuinely hard). Xe observed the last one as a 29-byte drift between consecutive builds of the same wasm2js on the same machine with the same flags. Structurally meaningless, byte-for-byte meaningful.

@pertymcpert identified the mechanism in the HN comments: Clang iterating over a DenseMap (a hash-map with non-deterministic iteration order) on some code path when generating try_table blocks; the fix is to swap for a MapVector (preserves insertion order, with some runtime/memory cost). One-line fix in Clang. Until it ships, every WASM binary built from C++ with exception handling will drift on every build.

Angle 2: The tooling supply chain is the actual attack surface

The most operationally alarming finding is the chain clang → wasm-opt → binaryen → wasi-sdk → Clang's bundledwasm2js`. Every one has its own version, schedule, and vendoring story. Thewasm-optXe had on a DGX Spark ARM machine was 108. The version on his x86 workstation, from Homebrew, was 130. The version Clang reaches for depends on$PATH. When the installedwasm-optis too old to understand the WebAssembly Exceptions extension thatwasi-sdk` emits by default, the build fails silently — looks like a Clang bug, is a binaryen version mismatch.

The lesson: the compiler's "implicit dependencies" are not in your lockfile. Nix picks this up — @crvdgc pointed out in the comments that Nix sets the build time to epoch to make hash calculation stable — but most CI pipelines do not. Pinning clang alone is insufficient; pin every binary the compiler can shell out to.

For Anubis — where the WASM binary is the trust anchor for the entire proof-of-work challenge — the compiler's nondeterminism lands as a security boundary. Reproducible builds are the property that lets an independent party re-build your binary, compare hashes, and be confident they got what you shipped. Without it, the "is this WASM actually from the Anubis project?" question becomes unanswerable.

Angle 3: The fallback chain is more honest than most production stacks

The original WASM-based PoW challenge had one failure mode: a client with WebAssembly disabled (privacy settings, browser policy, an old embedded device, Tor Browser) cannot solve the challenge and gets locked out. Xe did not want to exclude those users, so:

1. Primary: WASM check, runs on both client and server, fast.
2. Fallback when WASM is disabled: wasm2js recompiles the same WASM module into JavaScript at build time. Slower, but it runs on any browser.
3. Why both artifacts stay byte-equal: the WASM and the JS both encode the same source, so the PoW logic is identical. The browser picks one.

The original-recipe implementation uses wasm2js from the Linux distribution's package manager. That's where the reproducibility problem comes in: Debian's version is too old, Homebrew's produces different output, and the version Clang produces depends on $PATH. Xe's fix is to bundle a copy of wasm2js compiled to WASM with wasi-sdk, and ship it inside the Anubis repo. Single-architecture, single-toolchain, byte-stable (modulo the Clang bugs above).

A generic "WASM is the answer" stack would ship the WASM-only path and add a "supported browsers" list. Xe's stack is "if you can't run WASM, run our slower JS port, and we keep both artifacts under the same reproducibility guarantee." The fallback is part of the product, not a TODO.

Angle 4: This is the second anti-AI-bot arms escalation that depends on toolchain trust

The first escalation was the original Anubis PoW: a SHA-256 challenge that proves the client spent CPU. It works because SHA-256 is in WebCrypto on every browser and the CPU cost is honest. The second escalation moves the challenge itself into a WASM module, giving the server operator control over the PoW scheme — memory-hard, GPU-unfriendly, custom preimage format, all without coordinating with the Anubis core team.

The new attack surface is the WASM module itself. With SHA-256, the trust chain was Anubis project → npm package → your server → browser. With WASM, it is Anubis project → WASM binary built by someone → mirrored to a CDN → loaded by the browser. The honest defense is reproducible builds. Xe's whole post is an open admission that the reproducible-builds half of that defense is missing for the toolchain he is using, plus a working note on the patches he applied to make it so.

Angle 5: The HN thread shows the canonical mistakes

Three top comments identify the three common wrong responses to "this build is non-deterministic":

• @charcircuit: byte-identical output is an arbitrary restriction, equivalent programs are equivalent regardless of the build hash, the right defense is signature verification. Cryptographically correct in the narrow sense. Wrong for Xe's use case: Anubis is community-run and the trust model is anyone can rebuild and verify, not trust the single signing key holder.
• @dyauspitr: LLMs should be trained on and directly output binary. The "skip the compiler" position. The determinism problem goes away when the model is the compiler — except it does not, it just moves.
• @ComputerGuru pushed back on the title as clickbait, noting that compilers literally made the project possible. The right read. Xe hates compilers the way a structural engineer hates gravity: gravity is a real force, and you design around it anyway.

All three replies are partially correct in isolation. None engages with the actual problem: "I need this WASM binary reproducible so downstream operators can verify it."

The original take: the compiler is the supply chain

The honest read of "I hate compilers" is that the modern compiled-artifact supply chain has the same trust properties as a software dependency graph, and most projects are not treating it that way. You pin npm versions. You audit container base images. You run cargo audit or npm audit. You do not, as a rule, audit your clang's implicit wasm-opt dependency.

The reproducible-builds community has been saying this for fifteen years. Debian's reproducible-builds project has been patching individual nondeterminism sources across the archive. Nix, Guix, and Bazel-with-remote-execution each take a swing at the hermetic-build problem. None of them is the default.

Xe's post is, in this reading, a public service announcement that the Anubis team is one of the few projects in the WASM ecosystem taking the question seriously. They ship their own vendored wasm2js, accept the 29-byte Clang-exception-handling drift as a known-unfixed upstream bug, and document the patch trail. That is not "I hate compilers." That is "I have read the source code of my compiler and I am not happy about what I found, but here is the patch."

What this means for you

If you ship a WASM module, native binary, or any compiled artifact that downstream parties verify, ask this week:

1. Two consecutive builds on the same machine — same bytes? Run three times, sha256sum the outputs.
2. Two different machines, both pinned — same bytes? Pin clang, pin wasm-opt, pin everything clang can shell out to. strace -f -e execve the build, read what it invokes.
3. If a downstream operator runs your build today, do they get the same bytes you got last month? If the answer is no, your signing story is the only thing standing between "trust us" and "trust us, plus our key." Decide before the audit asks.

If you are using Anubis (or any tool that ships a WASM PoW check), ask your vendor whether the WASM module you load is reproducible from a clean checkout. If they cannot answer, the "is this WASM actually from the project?" question is one CDN compromise from being unanswerable.

What to do this week

Pick a compiled artifact you ship and run this three times — same source, fresh build each time, hash the output:

make clean && make my-wasm-module
sha256sum my-wasm-module
make clean && make my-wasm-module
sha256sum my-wasm-module
make clean && make my-wasm-module
sha256sum my-wasm-module

If the three hashes disagree, the artifact is non-reproducible. The usual culprits, in order of frequency: embedded timestamps (__DATE__, __TIME__, build epoch); source paths in debug info (-ffile-prefix-map helps); compiler-shelled-out-to tooling (strace your build); address-sensitive codegen (MapVector vs DenseMap, etc.).

For Nix users the fix is partially built in:

nix-build -A my-wasm-module
nix-build -A my-wasm-module  # second build, same hash?

If the two builds disagree and you are not on Nix, the path forward is either Nix (heavy lift, real fix) or a hand-pinned toolchain inside a container with the tool versions frozen in the Dockerfile (lighter lift, recurring maintenance). Xe chose the second path for Anubis. Most projects do not choose either, and ship non-reproducible binaries anyway.

Disclosure

Drafted with AI assistance. Primary source (Xe Iaso's "I hate compilers") and the HN thread (item 48581070) were both retrieved via direct HTTP fetches on 2026-06-18 around 13:30 UTC. All quoted comments are paraphrased, not blockquoted; the compiler-nondeterminism claims (__DATE__ / __TIME__, Clang's silent wasm-opt shell-out, DenseMap vs MapVector for try_table ordering, the 29-byte drift) are sourced from Xe's writeup, with the MapVector mechanism confirmed in the comment by @pertymcpert. The 111-point HN figure is from the Algolia API at the fetch timestamp (live-page counter was 113 at the same moment; the API value is the canonical figure for citation). Xe Iaso is the author of Anubis; weight that into any verification claims about the toolchain.

The compiler is the supply chain. You are not auditing it.

Sources

• Xe Iaso, "I hate compilers" — the primary writeup, with the full reproducible-builds walkthrough (published 2026-06-18, 1665 words): https://xeiaso.net/notes/2026/anubis-wasm-vendor-binary/
• HN discussion, item 48581070, "I hate compilers" (111 points per Algolia API as of 2026-06-18 13:30 UTC fetch; live-page counter was 113 at the same moment): https://news.ycombinator.com/item?id=48581070
• Anubis project, the proof-of-work proxy whose WASM-port this post is about: https://github.com/TecharoHQ/anubis
• Binaryen / wasm2js, the WebAssembly-to-JavaScript transpiler Xe is vendoring for the deterministic-builds fix: https://github.com/WebAssembly/binaryen
• wasi-sdk, the WASI-flavored Clang toolchain Xe used to compile wasm2js to WASM: https://github.com/WebAssembly/wasi-sdk
• Related on this blog: "An AI Agent Burned $6,531 on AWS to Scan a Hobby Network Nobody Asked It To" — covers Anubis as the standard answer to LLM-scraper DDoS: https://tutorialoflife.blogspot.com/2026/06/an-ai-agent-burned-6531-on-aws-to-scan.html
• Related on this blog: "Linear Is Fast Because the Browser Is the Database" — different problem, same supply-chain-trust theme: https://tutorialoflife.blogspot.com/2026/06/linear-is-fast-because-browser-is.html