Programming guides for beginner...
Any comments are welcomed....
I hope it helps!!! Thanks for drop by...

Monday, June 29, 2026

Two Strix Halos and a DAC Cable Just Became a 256GB GPU

Donato Capitella's AMD Strix Halo RDMA Cluster Setup Guide hit the front page of Hacker News on Saturday with 171 points and 54 comments, and the headline number is the easy one to fixate on. Two Strix Halo boards, each with 128GB of unified memory, joined by a 100GbE Intel E810 NIC and a $100 QSFP28 Direct Attach Copper cable, behave as a single 256GB inference node. vLLM runs Tensor Parallelism across the pair, the AMD equivalent of NCCL — RCCL — exchanges tensor shards over RoCE v2 RDMA, and the round-trip latency is around 5 microseconds. The cheap number is 5µs. The cheap number is also not the story.

The story is that a 256GB unified-memory node is now something a prosumer with a credit card can build in an afternoon, and the community that already has — and is shipping, not just demoing — is one piece of evidence that the local-inference tier crossed a different threshold this month.

The setup, in one paragraph

The hardware list is short. Two Framework Desktop Mainboards with the AMD Ryzen AI MAX+ 395 "Strix Halo" chip and 128GB of RAM each — the 128GB version is the one that pairs usefully, the 64GB variant gives you two of nothing. Two Intel E810-CQDA1 100GbE NICs, one per node. One QSFP28 DAC cable, no switch, no transceiver optics. The Framework boards have a physical PCIe x4 slot, so each node needs a riser (a $10–20 x4-to-x16 extender, Amazon CY-style) unless the user wants to cut the slot with an ultrasonic knife, which Capitella's guide notes Framework did on one of their test boards and does not recommend. Per the HN thread (jmyeet, 2026-06-28), a 128GB Framework board has been quoted at roughly $3,150 each, which puts the board pair at ~$6,300; the NICs add ~$500 each and the cable ~$100, so the realistic total for a working 256GB cluster lands closer to ~$7,500 than to a $3,400 hobby number. The 64GB variant runs much cheaper (jcastro, HN, 2026-06-28: ~$1,700 empty per board), but two 64GB boards pair to 128GB, which is the same class of node a single 128GB board already provides. The 128GB boards are the only configuration worth building. The software path is Fedora 43, a kernel parameter set that pins unified memory to ~124 GiB per node, the in-kernel ice and irdma drivers, and a custom-built ROCm/RCCL the toolboxes repo ships as a patch. vLLM runs on top. The guide ships a start-vllm-cluster TUI that walks through Ray cluster bring-up, RDMA verification, and vllm serve launch.

That is the entire stack. The reason any of it is novel is the unification story.

Why "unified memory" is the load-bearing detail

The reason Strix Halo exists as a category, rather than as just another APU, is that AMD will let the iGPU address up to 128GB of system RAM as VRAM through a Graphics Translation Table. A consumer GPU in the same price bracket — an RTX 4090, an RTX 5090 — exposes 24 to 32GB of VRAM, and the entire class of models that fit in 128GB simply does not run on a single consumer card. Qwen 3.5 122B-A10B at AWQ 8-bit needs roughly 128GB just for the weights. A 120B-class BF16 model needs 240GB. The Strix Halo board is the first prosumer-priced part where the weights fit.

The cluster is the part that does not get covered in most of the day's other write-ups, and the part that matters. One board is 128GB. Two boards, joined at the memory-bandwidth level, behave as 256GB. A model that the single board cannot host fits the pair. The reason a cable is involved, rather than just plugging in a second board, is that vLLM's Tensor Parallelism shards the model layer-by-layer across devices, and the shards have to move back and forth thousands of times per generated token. Over TCP, that link is 70 to 100 microseconds. Over RoCE v2 RDMA, it is 5. The two orders of magnitude are the difference between a cluster that scales linearly and a cluster that does not.

The software side: a custom RCCL and what it tells you

RCCL is AMD's NCCL — the library that handles collective communication for distributed training and inference. Out of the box, Strix Halo's iGPU is not in RCCL's tested-targets list, and the in-tree RDMA path is not wired up for an APU whose device memory is system memory. Capitella's toolboxes repo ships a custom build of RCCL (a fork of TheRock, the ROCm nightly) that adds the patch. The README is explicit that this is a hobby project, that the patch is small, and that the supported models are the ones on the tested-model list: Llama-3.1-8B, Gemma 4 26B and 31B, GPT-OSS-20B and 120B, Qwen 3.6 35B (and the AWQ-4bit variant), Qwen 3.5 122B at AWQ 4-bit and AWQ 8-bit. The 122B AWQ 8-bit entry is the one that needs 2 GPUs and a cluster; the 122B AWQ 4-bit can run on a single board with TP=1.

The patch itself is the story under the story. The reason a hobby project can ship a working RDMA cluster on an APU that AMD has not officially supported for tensor parallelism is that the unified-memory model eliminates the canonical distributed-training problem: peer-to-peer GPU memory access. On a discrete GPU cluster, NCCL has to copy tensor shards over PCIe or NVLink into a staging buffer on the destination GPU, then through the kernel into the model's HBM. On Strix Halo, the "GPU memory" is system memory, and the iGPU accesses it through the same cache-coherent fabric the CPU does. RDMA into system memory is a much more direct path than RDMA into discrete VRAM, and the software stack reflects that. Capitella's RDMA cluster reaches 50Gbps of effective bandwidth and ~5µs round-trip latency, with the bottleneck now at the NIC, not at the kernel or the PCIe slot.

The community reaction tells you what tier this is

The HN thread's highest-engagement branch, after the initial congratulations, is the cost-per-token argument, which goes like this. Two 128GB boards plus NICs plus cable, on jmyeet's quotes, is roughly $7,500 for the working cluster. The cheapest OpenAI subscription that gives you a useful frontier model is $20 per month. At sustained heavy use the cluster pays for itself; at light use the API is the better capex story. The argument the thread does not quite get to, and the one that matters more, is what you can do with the cluster that the API cannot do. The reason local inference exists as a category is that some workloads cannot use the cloud: PII handling, code with secrets, regulated text, jurisdictional data. Capitella says on the project site he built the toolboxes for one of these workloads (cybersecurity); the cluster he ended up shipping is general-purpose enough that the rest of the use cases inherit the same answer. The 256GB tier is the first prosumer price point where the question stops being "is the local model good enough to be useful" and starts being "is the local model good enough to be useful for the workload where the cloud model was not legally permitted in the first place."

The thread's other substantive branch is hardware-availability. A 128GB Strix Halo board, when the guide was published, was the rare part. The 64GB variant is going for $1,700-ish empty. The 128GB version is the constraint, and the 128GB version is the one that pairs usefully. A commenter who runs projectbluefin — a three-node Strix Halo setup for an "agentic OS factory" — notes the same price wall. The interesting read of that constraint is that it is the kind AMD, not the prosumer market, gets to move. A thousand-person prosumer demand does not change silicon. It changes how quickly the next-generation part is allocated to the right buyers. The toolboxes are ahead of the parts, and the parts will catch up when AMD sees the demand.

The original take: the second-tier story is the cloud-exit story

Here is what the coverage will miss. The first-derivative story is "two cheap boards behave like a 256GB GPU." That is true, it is well-sourced, and it will be the headline. The second-derivative story is that the prosumer-inference stack has its own engineering discipline now — its own patches, its own benchmarks, its own maintainers, its own deployment recipes. The architectural shift is not that 256GB is now affordable; it is that a hobby project can ship a working RDMA cluster on an APU that AMD has not officially supported, with a tested-model list that covers the local-LLM frontier, and that hobby project is one of the reference implementations for any lab that wants to do the same on different hardware. That is the part every "two boards and a DAC" write-up will skip.

The toolboxes repo had 422 stars, 59 forks, 17 watchers, and 39 open issues as of this writing. The RCCL patch is upstreamed nowhere. The Llama Cockpit TUI is in the same boat. The lesson is that the prosumer cluster is not a stopgap; it is a category. The same way the blog has previously argued on speculative KV-coding cache compression that the inference-engineering layer is a first-order design surface, the Strix Halo RDMA cluster argues that the consumer-side distributed-inference stack is now a layer of the deployment stack in its own right. The unified-memory model is what makes the layer possible. The prosumer demand is what makes the layer permanent.

What this means for you

  • If you are running a single 24GB or 32GB consumer GPU, the Strix Halo RDMA cluster is the next step up, and the cost of entry is the part, not the architecture. The guide is open, the toolboxes are open, and the patch is shipping in a tested form. The constraint is the 128GB board supply, not the engineering.
  • If you operate regulated or sensitive workloads where a cloud LLM is not a permissible dependency, the 256GB tier is the first prosumer price point where the local option can run the same model class the cloud option runs. This is a regulatory story as much as a performance story.
  • If you are maintaining a distributed-inference stack on a different vendor's hardware, the RCCL patch is a useful reference even if you do not use Strix Halo. The unified-memory RDMA path is a generally applicable pattern, and the patch shows what the gap between "supported" and "works" looks like for a not-yet-supported target.
  • If you are betting on a closed-weight inference API as your durable advantage, the 256GB prosumer tier is a margin-compression signal for the part of the workload that fits. The class of model that fits in 256GB is the class that used to be the moat.

What to do this week

If you have a 128GB Strix Halo board — or can get one — wire up the cluster. The guide is a checklist, not a research project, and the failure modes are documented in the troubleshooting section. If you have a 64GB board, run the single-node benchmarks and decide whether the second board is the right capex. If you have neither, the read-through is the benchmark: which of your deployed models fits in 256GB, and is the cloud-API cost on those models large enough to justify the procurement. The math is per-workload, and the right answer is rarely "yes" and rarely "no."

# 0. Prereqs (one-time, on the host Fedora 43 install):
#    - install rdma-core, libibverbs-utils, perftest
#    - configure passwordless SSH between the two nodes
#    - add 192.168.100.1 to /etc/hosts as `head`, 192.168.100.2 as `worker`
echo "192.168.100.1 head"   | sudo tee -a /etc/hosts
echo "192.168.100.2 worker" | sudo tee -a /etc/hosts

# 1. Verify the RDMA link is up on both nodes
ssh worker rdma link | grep LINK_UP
ssh head   rdma link | grep LINK_UP

# 2. Enter the vLLM toolbox (the cluster TUI lives inside the container,
#    not on the host shell) and launch the cluster manager
toolbox enter vllm
./start-vllm-cluster
#   2 -> Start Ray Cluster
#   4 -> Launch VLLM Serve (export HF_TOKEN first for gated models)

# 3. Smoke-test the unified 256GB node
curl http://head:8000/v1/models | jq '.data[].id'

A few words of warning. The benchmark numbers on the toolboxes site — peak multi-user throughput at high concurrency — are saturating the memory bandwidth, not the token-latency-of-a-single-request. Your single-user generation speed will be lower than the headline numbers, the same way it is on every other inference platform. The patch is community-maintained, not AMD-supported, and the production posture is "works in the configurations on the tested-model list." The cluster is a real, reproducible, two-node inference node. It is not a data-center replacement. If you need an SLA, this is not it. If you need a ~$7,500 local 256GB node that runs Qwen 3.5 122B AWQ-8 without any third-party API, it is.

Disclosure

This post was drafted with AI assistance. The trend scan, source verification, and primary synthesis are the work of the model; the final framing, claims, and structure are human-reviewed. No part of the post was generated from an undisclosed prompt injection. Specific quantitative claims (5µs RDMA round-trip latency, ~50Gbps effective bandwidth, 70-100µs TCP baseline, 171 HN points / 54 comments, 422 stars / 59 forks / 17 watchers / 39 open issues on the toolboxes repo) are sourced from the kyuz0/amd-strix-halo-vllm-toolboxes GitHub repository and the Hacker News thread, both re-verified as live and well-formed via curl --compressed against the GitHub API and the raw README.md / setup_guide.md endpoints on 2026-06-28. Build-cost figures (~$3,150 per 128GB Framework board per HN commenter jmyeet, ~$1,700 empty per 64GB board per HN commenter jcastro, ~$500 per 100GbE NIC, ~$100 QSFP28 DAC, ~$10-20 PCIe riser) are HN-quoted prices as of 2026-06-28, not official manufacturer MSRPs, and the per-component sums in the post are the draft author's arithmetic. The Framework Desktop product page was not independently fetchable from this environment (Cloudflare bot challenge), but the URL was taken directly from the setup guide itself.

Sources

By Jiro at June 29, 2026
Labels: , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)