Building a Fully Local 120B / 72B / 31B AI Cluster with Three 128GB Strix Halo Mini PCs

Loading one large model and organizing several models into everyday infrastructure are different tasks.

My goal was the latter: a local research environment that keeps multiple models available, assigns different workloads to different endpoints, and can support future Dify and Sanapedia research workflows without requiring an external model API for inference.

The result is a three-node system built from GMKtec NucBox EVO-X2 mini PCs. Each has an AMD Ryzen AI MAX+ 395, Radeon 8060S integrated graphics, 128GB unified memory, and a 96GB BIOS UMA allocation. The captured runtime inventory contains ten llama-server processes: one on the first node, one on the second, and eight on the third.¹²

1. The Design Question Was Not Only “Can It Run?”#

A single-model experiment asks whether a particular model fits and responds. This architecture asks how several capabilities can remain available together.

A research pipeline may need deep synthesis, visual interpretation, embeddings, reranking, query classification, structured extraction, and text transformation. My design does not assume that the largest available model should perform every step.

Instead, the largest text and visual models receive dedicated nodes. Smaller support models share a third node. This is a service-allocation choice, not a measured claim that every smaller model is faster or more accurate for its assigned task.

2. Design Goals#

The first goal is local inference: source documents do not have to be submitted to an external AI service for the model-processing stage.

The second is persistent availability: the core model set stays resident while the cluster is online, avoiding repeated model-loading steps inside a workflow.

The third is functional separation: model selection follows the task rather than a single default endpoint.

The fourth is long-document capacity: the system should accommodate books, reports, and research collections where the evidence can be much larger than a short chat exchange.

The fifth is extensibility: the same service layer should support command-line testing now and more complete Dify research workflows later.

“Local” is a design boundary, not an air-gap certification. External source collection and optional integrations remain separate questions.

3. Hardware and Software Baseline#

These values come from the system captures and the operator-confirmed BIOS setting.¹

The software is not identical across all three nodes:

All three report 7.0.0-31-generic, but the captured kernel build strings differ between the Ubuntu 24.04 and 26.04 installations. The shared release label is not evidence of identical kernel packages.¹

Unit convention: hardware and BIOS labels retain 128GB and 96GB; byte-derived binary quantities use GiB. In this allocation, Linux reports about 30 GiB of conventional system RAM. That is distinct from the 96 GiB GPU-visible region.

4. Overall Architecture#

                         Control-PC
                 Windows 11 / Dify / WSL2
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
       LLM-Server01   LLM-Server02   LLM-Server03
             |              |              |
       gpt-oss-120B    Qwen2.5-VL-72B       +-- Gemma-4-31B-it
                                           +-- Embeddings x2
                                           +-- Rerankers x2
                                           +-- Router
                                           +-- Extractor
                                           +-- Coder

The Windows control environment and its Dify/WSL2 role are described in the construction notes. The Linux process and socket inventory was collected directly.³²

This is distributed services, not distributed inference for one model. The machines do not present a pooled 384GB context or weight-memory space. A request goes to a service on a particular node.

5. Why Three Nodes Instead of One?#

Memory isolation keeps the 120B text model and 72B VLM from competing for the same allocation. Bandwidth separation puts their work on different APUs rather than one shared memory subsystem.

Failure separation also matters during experimentation. A model change on one machine need not require a change on the others. That is not automatic failover: workflows that depend on the unavailable role may still fail.

The third reason is functional specialization. The utility node hosts several narrower roles instead of reserving its entire memory budget for one model. Whether a specific routing or extraction choice produces the best quality will be evaluated with real workloads; it is not established by process counts.

6. Model and Service Allocation#

The model assignments and configured arguments are visible in the runtime capture; effective slot values come from the matching startup records. The gpt-oss-120B production setting is documented separately from the sampled 256k experimental process.²⁴⁵

“Role” means the intended workflow responsibility. It does not imply that all ten roles have passed an end-to-end Dify acceptance test.

7. LLM-Server01: The 120B Long-Context Node#

The first node hosts gpt-oss-120B MXFP4. Its two GGUF shards total 63,387,345,440 bytes, or approximately 59.03 GiB. This is on-disk file size, not runtime VRAM usage.⁶

The production launch pattern is:

"$HOME/llama.cpp-vulkan/build/bin/llama-server" \
  -m "$HOME/models/gpt-oss-120b-MXFP4-00001-of-00002.gguf" \
  -ngl 999 \
  --host 0.0.0.0 \
  --port 8080 \
  -c 131072 \
  -b 2048 \
  -ub 256 \
  --parallel 1 \
  --flash-attn on

The model path is normalized for publication; arguments preserve the production-script record. The two shards must be available together.⁵

Security note: 0.0.0.0 binds to all IPv4 interfaces on the host. It does not mean “LAN only.” Restrict access through network controls and appropriate authentication. Do not expose an unauthenticated inference endpoint directly to the public Internet.

A separate experiment requested 262,144 tokens. The runtime reported n_ctx_train = 131072 and capped the effective slot at 131,072. The standalone GGUF dump did not succeed on this node, so that model-context figure is explicitly runtime-recognized, not independently extracted metadata.⁶⁴

8. LLM-Server02: The 72B Multimodal Node#

The second node hosts Qwen2.5-VL-72B-Instruct Q4_K_M. Its model GGUF is 47,736,872,640 bytes (44.46 GiB), and the associated F16 projector is 1,410,222,816 bytes (1.31 GiB). The original lowercase paths are symbolic links; these sizes describe their resolved targets, not the links themselves.⁶

The model metadata identifies qwen2vl, with 80 blocks, an embedding length of 8,192, and a declared context of 128,000. The service requests 131,072, while its startup log reports an effective slot of 128,000.⁶⁴

The intended role includes scans, diagrams, maps, and illustrated sources. Loading the model and projector establishes deployment state; it does not yet establish accuracy across those document categories.

9. LLM-Server03: Eight Services on One Node#

The utility node hosts Gemma-4-31B-it UD-Q4_K_XL, an 18,822,970,304-byte model GGUF (17.53 GiB), with a 1,200,726,496-byte BF16 projector (1.12 GiB). Its metadata declares 262,144 tokens and the startup log reports the same slot value.⁶⁴

Seven smaller services run alongside it. The process table and listening sockets confirm the eight-process, eight-port topology.²

This establishes concurrent residence, not sustained concurrent throughput. The services still share the APU, memory bandwidth, operating system, and host-level failure boundary.

10. The Software Stack#

Research workflow / Dify
          |
          v
Capability-specific local HTTP APIs
          |
          v
llama-server -> llama.cpp -> Vulkan -> Mesa RADV -> amdgpu
          |
          v
AMD Strix Halo unified-memory hardware

The API layer separates workflow intent from GGUF paths and launch mechanics. It does not make every capability interchangeable: chat, embeddings, and reranking have different contracts. The official server documentation distinguishes OpenAI-compatible chat/embedding routes from its reranking interface.⁷

Part 4 explains that boundary and the evidence available for Dify registration.

11. Persistent Deployment#

Each node uses a llama-cluster.service unit to invoke its start and stop scripts. The captured units contain Type=forking, Restart=on-failure, RestartSec=10, LimitMEMLOCK=infinity, and LimitNOFILE=65535.²

On Server03, one unit launches multiple model processes. This is not the same as eight independently supervised systemd units. The captured state establishes that the unit and processes were running, not that each child has independently verified automatic recovery.

The design objective is zero model-load cold start while the core services remain loaded. It does not imply zero request latency or guaranteed immediate service when an endpoint is busy.

12. Memory Usage in Practice#

The construction notes contain this representative monitor series, attributed there to radeontop / amdgpu_top observation:³

The MB labels are retained as recorded. These are representative monitor snapshots, not a fixed memory requirement or a byte-exact substitute for DRM totals. The tool/version and capture conditions are not fully attached to every value in this earlier series.

A later DRM capture gives a separate series:

Every DRM total is exactly 96 GiB. The two series are not simultaneous, and their counters and denominators are not identical. The available record does not establish that timing alone explains every difference.¹

Server02 has the highest occupancy in both series. Apparent free memory is not a guarantee of sufficient headroom under every workload. The memory-management details belong in Part 2.

13. A Recorded 129,722-Token Prefill Test#

The construction notes record a long input described as roughly three books:³

105.46 tokens/s is the recorded prefill rate, not generation speed. Dividing the displayed input count by 1,230 seconds gives approximately 105.47 tokens/s; the small final-digit difference is retained as a reporting-precision issue rather than silently changing the original result.

This is one historical observation. The notes report no observed leak, but the series does not present a controlled leak test. A paired raw timing trace, full request, and exact test-build attribution are not included in the published baseline, so this is not presented as a reproducible performance comparison.

14. Configured Context Is Not Always Effective Context#

The experimental gpt-oss-120B process requested 262,144 and exposed 131,072. Qwen2.5-VL-72B requested 131,072 and exposed 128,000. Jina Reranker v2 requested 8,192 and exposed 1,024.⁴

Those are observations from the captured builds, not a universal formula for every llama.cpp version or model. Part 3 separates artifact metadata, operator requests, effective slots, and actual request evidence.

15. Batch Parameters Can Also Change at Startup#

Server03's embedding and reranking logs report that the requested n_batch = 2048 was adjusted to n_ubatch = 256.⁴

The deployment command describes intent. The startup log describes interpretation. Both belong in a reproducibility record.

16. Problems Encountered#

The construction notes describe earlier ROCm allocation failures and Vulkan device-loss incidents. The later kernel capture directly records paired amdgpu_vm_validate() and command-submission memory errors on Server02.³²

The exact workload associated with that kernel burst is unresolved. It is not presented as proof that the current VLM process caused it. Part 2 preserves the symptoms, reported changes, and limits of the causal evidence.

17. Evidence and Interpretation#

The reports distinguish four things: directly captured state, behavior recorded during construction, architectural interpretation, and unresolved hypotheses.

A logged memory error is evidence of a reported failure. A lower micro-batch followed by an apparently successful run is a useful construction observation. Neither automatically proves a universal mechanism or guarantees multi-hour stability.

18. Intended Research Work#

The platform is intended for long-form analysis, historical sources, multimodal documents, retrieval, structured extraction, and development support. The complete Sanapedia/Dify pipeline is still a planned integration stage.

The service inventory makes that work possible to attempt. Workload-specific quality, latency, and recovery behavior remain to be measured.

19. Limits of This Baseline#

This first phase does not establish controlled generation speed, wall power, thermal characteristics, multi-hour stability, repeated-run variance, or end-to-end quality. It is also not a controlled comparison between the operating-system versions.

The inference nodes are captured in detail. The full control-plane software inventory, provider/plugin versions, and functional request traces for every Dify model are not part of this collection.

20. Security and Public Examples#

Public text uses LLM-Server01, LLM-Server02, LLM-Server03, Control-PC, $HOME, and <user>. Local paths are normalized without changing model filenames. Private hostnames, user identities, IP/MAC addresses, credentials, and unredacted raw logs are omitted.

Normalization protects documentation hygiene; it does not itself secure the live network. The access-control discussion is in Part 4.

21. From Running a Model to Operating Infrastructure#

The design result is a division of responsibilities: a large text node, a large visual node, and a multi-service utility node. The captured records establish that this deployment exists; they do not complete every operational test that an organization might require before adopting it.

That is why the baseline is worth preserving before the next phase. The architecture can now be tested as a system rather than as a collection of isolated model launches.

Continue with Part 2: Memory, Vulkan, and AMD Strix Halo.

Evidence and Reading Notes#