--- title: Building a Fully Local 120B / 72B / 31B AI Cluster with Three 128GB Strix Halo Mini PCs subtitle: Architecture, model allocation, memory usage, and a recorded 129,722-token prefill test description: Architecture, model roles, software versions, and memory observations from three 128GB Strix Halo nodes hosting ten local AI services. lang: en author: Deshimaru Sakaguchi series: Local AI Infrastructure Lab series_part: 1 baseline_date: '2026-09-23' baseline_timezone: Asia/Tokyo last_updated: '2026-09-23' revision: '1.0' canonical: https://deshimarusakaguchi.com/lab/local-ai-cluster/architecture/ permalink: /lab/local-ai-cluster/architecture/ tags: - local-ai - strix-halo - llama-cpp - research-infrastructure --- # Building a Fully Local 120B / 72B / 31B AI Cluster with Three 128GB Strix Halo Mini PCs [Lab Hub](https://deshimarusakaguchi.com/lab/local-ai-cluster/) · **Part 1** · [Part 2](https://deshimarusakaguchi.com/lab/local-ai-cluster/memory-vulkan-strix-halo/) · [Part 3](https://deshimarusakaguchi.com/lab/local-ai-cluster/context-runtime/) · [Part 4](https://deshimarusakaguchi.com/lab/local-ai-cluster/persistent-services/) **Baseline captured: September 23, 2026 (JST)** **Author: Deshimaru Sakaguchi** *Architecture, model allocation, memory usage, and a recorded 129,722-token prefill test* Loading one large model and organizing several models into everyday infrastructure are different tasks. My goal was the latter: a local research environment that keeps multiple models available, assigns different workloads to different endpoints, and can support future Dify and Sanapedia research workflows without requiring an external model API for inference. The result is a three-node system built from GMKtec NucBox EVO-X2 mini PCs. Each has an AMD Ryzen AI MAX+ 395, Radeon 8060S integrated graphics, 128GB unified memory, and a 96GB BIOS UMA allocation. The captured runtime inventory contains ten `llama-server` processes: one on the first node, one on the second, and eight on the third.[^system][^runtime] ## 1. The Design Question Was Not Only “Can It Run?” A single-model experiment asks whether a particular model fits and responds. This architecture asks how several capabilities can remain available together. A research pipeline may need deep synthesis, visual interpretation, embeddings, reranking, query classification, structured extraction, and text transformation. My design does not assume that the largest available model should perform every step. Instead, the largest text and visual models receive dedicated nodes. Smaller support models share a third node. This is a service-allocation choice, not a measured claim that every smaller model is faster or more accurate for its assigned task. ## 2. Design Goals The first goal is **local inference**: source documents do not have to be submitted to an external AI service for the model-processing stage. The second is **persistent availability**: the core model set stays resident while the cluster is online, avoiding repeated model-loading steps inside a workflow. The third is **functional separation**: model selection follows the task rather than a single default endpoint. The fourth is **long-document capacity**: the system should accommodate books, reports, and research collections where the evidence can be much larger than a short chat exchange. The fifth is **extensibility**: the same service layer should support command-line testing now and more complete Dify research workflows later. “Local” is a design boundary, not an air-gap certification. External source collection and optional integrations remain separate questions. ## 3. Hardware and Software Baseline | Item | Configuration per compute node | |---|---| | System | GMKtec NucBox EVO-X2 | | APU | AMD Ryzen AI MAX+ 395 | | CPU | 16 cores / 32 threads | | Integrated GPU | Radeon 8060S | | Physical memory | 128GB unified memory | | BIOS setting | 96GB BIOS UMA allocation | | DRM-reported visible VRAM | 103,079,215,104 bytes = 96 GiB | | Inference backend | `llama.cpp` with Vulkan / Mesa RADV | These values come from the system captures and the operator-confirmed BIOS setting.[^system] The software is not identical across all three nodes: | Node | OS | Kernel release label | BIOS | Mesa RADV | llama.cpp build / commit | |---|---|---|---|---|---| | LLM-Server01 | Ubuntu 24.04.3 LTS | 7.0.0-31-generic | EVO-X2 1.09 | 25.2.8 | 10858 / `64e9bceb2` | | LLM-Server02 | Ubuntu 26.04.1 LTS | 7.0.0-31-generic | EVO-X2 1.11 | 26.0.8 | 11047 / `b23701f77` | | LLM-Server03 | Ubuntu 26.04.1 LTS | 7.0.0-31-generic | EVO-X2 1.11 | 26.0.8 | 11061 / `b23efaa2e` | All three report `7.0.0-31-generic`, but the captured kernel build strings differ between the Ubuntu 24.04 and 26.04 installations. The shared release label is not evidence of identical kernel packages.[^system] **Unit convention:** hardware and BIOS labels retain `128GB` and `96GB`; byte-derived binary quantities use `GiB`. In this allocation, Linux reports about 30 GiB of conventional system RAM. That is distinct from the 96 GiB GPU-visible region. ## 4. Overall Architecture ```text Control-PC Windows 11 / Dify / WSL2 | +--------------+--------------+ | | | v v v LLM-Server01 LLM-Server02 LLM-Server03 | | | gpt-oss-120B Qwen2.5-VL-72B +-- Gemma-4-31B-it +-- Embeddings x2 +-- Rerankers x2 +-- Router +-- Extractor +-- Coder ``` The Windows control environment and its Dify/WSL2 role are described in the construction notes. The Linux process and socket inventory was collected directly.[^notes][^runtime] This is **distributed services**, not distributed inference for one model. The machines do not present a pooled 384GB context or weight-memory space. A request goes to a service on a particular node. ## 5. Why Three Nodes Instead of One? **Memory isolation** keeps the 120B text model and 72B VLM from competing for the same allocation. **Bandwidth separation** puts their work on different APUs rather than one shared memory subsystem. **Failure separation** also matters during experimentation. A model change on one machine need not require a change on the others. That is not automatic failover: workflows that depend on the unavailable role may still fail. The third reason is **functional specialization**. The utility node hosts several narrower roles instead of reserving its entire memory budget for one model. Whether a specific routing or extraction choice produces the best quality will be evaluated with real workloads; it is not established by process counts. ## 6. Model and Service Allocation | Node | Port | Model / quantization | Configured context | Startup-reported slot | Intended role | |---|---:|---|---:|---:|---| | LLM-Server01 | 8080 | gpt-oss-120B / MXFP4 | 131,072 production | 131,072 in the separate 256k experiment | Deep text analysis | | LLM-Server02 | 8080 | Qwen2.5-VL-72B-Instruct / Q4_K_M + F16 projector | 131,072 | 128,000 | Large multimodal analysis | | LLM-Server03 | 8080 | Gemma-4-31B-it / UD-Q4_K_XL + BF16 projector | 262,144 | 262,144 | General VLM / long-context work | | LLM-Server03 | 8081 | BGE-M3 / Q8_0 | 8,192 | 8,192 | Embedding | | LLM-Server03 | 8082 | BGE Reranker v2 M3 / Q8_0 | 8,192 | 8,192 | Reranking | | LLM-Server03 | 8083 | GTE-Qwen2-1.5B-instruct / Q8_0 | 8,192 | 8,192 | Secondary embedding | | LLM-Server03 | 8084 | Jina Reranker v2 / Q8_0 | 8,192 | 1,024 | Secondary reranking | | LLM-Server03 | 8085 | Qwen2.5-3B-Instruct / Q8_0 | 8,192 | 8,192 | Routing | | LLM-Server03 | 8086 | Qwen2.5-7B-Instruct / Q8_0 | 16,384 | 16,384 | Structured extraction | | LLM-Server03 | 8087 | Qwen2.5-Coder-7B / Q8_0 | 16,384 | 16,384 | Coding / preprocessing | The model assignments and configured arguments are visible in the runtime capture; effective slot values come from the matching startup records. The gpt-oss-120B production setting is documented separately from the sampled 256k experimental process.[^runtime][^startup][^production] “Role” means the intended workflow responsibility. It does not imply that all ten roles have passed an end-to-end Dify acceptance test. ## 7. LLM-Server01: The 120B Long-Context Node The first node hosts **gpt-oss-120B MXFP4**. Its two GGUF shards total **63,387,345,440 bytes**, or approximately **59.03 GiB**. This is on-disk file size, not runtime VRAM usage.[^artifacts] The production launch pattern is: ```bash "$HOME/llama.cpp-vulkan/build/bin/llama-server" \ -m "$HOME/models/gpt-oss-120b-MXFP4-00001-of-00002.gguf" \ -ngl 999 \ --host 0.0.0.0 \ --port 8080 \ -c 131072 \ -b 2048 \ -ub 256 \ --parallel 1 \ --flash-attn on ``` The model path is normalized for publication; arguments preserve the production-script record. The two shards must be available together.[^production] > **Security note:** `0.0.0.0` binds to all IPv4 interfaces on the host. It does not mean “LAN only.” Restrict access through network controls and appropriate authentication. Do not expose an unauthenticated inference endpoint directly to the public Internet. A separate experiment requested 262,144 tokens. The runtime reported `n_ctx_train = 131072` and capped the effective slot at 131,072. The standalone GGUF dump did not succeed on this node, so that model-context figure is explicitly **runtime-recognized**, not independently extracted metadata.[^artifacts][^startup] ## 8. LLM-Server02: The 72B Multimodal Node The second node hosts **Qwen2.5-VL-72B-Instruct Q4_K_M**. Its model GGUF is **47,736,872,640 bytes** (44.46 GiB), and the associated F16 projector is **1,410,222,816 bytes** (1.31 GiB). The original lowercase paths are symbolic links; these sizes describe their resolved targets, not the links themselves.[^artifacts] The model metadata identifies `qwen2vl`, with 80 blocks, an embedding length of 8,192, and a declared context of 128,000. The service requests 131,072, while its startup log reports an effective slot of 128,000.[^artifacts][^startup] The intended role includes scans, diagrams, maps, and illustrated sources. Loading the model and projector establishes deployment state; it does not yet establish accuracy across those document categories. ## 9. LLM-Server03: Eight Services on One Node The utility node hosts **Gemma-4-31B-it UD-Q4_K_XL**, an **18,822,970,304-byte** model GGUF (17.53 GiB), with a **1,200,726,496-byte** BF16 projector (1.12 GiB). Its metadata declares 262,144 tokens and the startup log reports the same slot value.[^artifacts][^startup] Seven smaller services run alongside it. The process table and listening sockets confirm the eight-process, eight-port topology.[^runtime] This establishes concurrent residence, not sustained concurrent throughput. The services still share the APU, memory bandwidth, operating system, and host-level failure boundary. ## 10. The Software Stack ```text Research workflow / Dify | v Capability-specific local HTTP APIs | v llama-server -> llama.cpp -> Vulkan -> Mesa RADV -> amdgpu | v AMD Strix Halo unified-memory hardware ``` The API layer separates workflow intent from GGUF paths and launch mechanics. It does not make every capability interchangeable: chat, embeddings, and reranking have different contracts. The official server documentation distinguishes OpenAI-compatible chat/embedding routes from its reranking interface.[^api] [Part 4](https://deshimarusakaguchi.com/lab/local-ai-cluster/persistent-services/) explains that boundary and the evidence available for Dify registration. ## 11. Persistent Deployment Each node uses a `llama-cluster.service` unit to invoke its start and stop scripts. The captured units contain `Type=forking`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, and `LimitNOFILE=65535`.[^runtime] On Server03, one unit launches multiple model processes. This is not the same as eight independently supervised systemd units. The captured state establishes that the unit and processes were running, not that each child has independently verified automatic recovery. The design objective is **zero model-load cold start** while the core services remain loaded. It does not imply zero request latency or guaranteed immediate service when an endpoint is busy. ## 12. Memory Usage in Practice The construction notes contain this representative monitor series, attributed there to `radeontop` / `amdgpu_top` observation:[^notes] | Node | Workload label in the notes | Monitor value as recorded | Approximate ratio | |---|---|---:|---:| | LLM-Server01 | gpt-oss-120B | 69,550 MB / 98,148 MB | 70.9% | | LLM-Server02 | Qwen2.5-VL-72B | 87,790 MB / 98,148 MB | 89.4% | | LLM-Server03 | Gemma-4-31B-it + seven support services | 61,941 MB / 98,148 MB | 63.1% | The `MB` labels are retained as recorded. These are representative monitor snapshots, not a fixed memory requirement or a byte-exact substitute for DRM totals. The tool/version and capture conditions are not fully attached to every value in this earlier series. A later DRM capture gives a separate series: | Node | `mem_info_vram_used` | Binary conversion | Total reported by DRM | |---|---:|---:|---:| | LLM-Server01 | 72,928,403,456 bytes | 67.92 GiB | 96 GiB | | LLM-Server02 | 92,056,068,096 bytes | 85.73 GiB | 96 GiB | | LLM-Server03 | 66,990,362,624 bytes | 62.39 GiB | 96 GiB | Every DRM total is exactly 96 GiB. The two series are not simultaneous, and their counters and denominators are not identical. The available record does not establish that timing alone explains every difference.[^system] Server02 has the highest occupancy in both series. Apparent free memory is not a guarantee of sufficient headroom under every workload. The memory-management details belong in [Part 2](https://deshimarusakaguchi.com/lab/local-ai-cluster/memory-vulkan-strix-halo/). ## 13. A Recorded 129,722-Token Prefill Test The construction notes record a long input described as roughly three books:[^notes] | Metric | Recorded result | |---|---:| | Input | 129,722 tokens | | Prefill time | 20 min 30 sec | | Reported prompt-processing throughput | 105.46 tokens/s | | Monitor fields during processing | Graphics Pipe and Shader Interpolator reported at 100% | | Outcome in the notes | Prefill completed without a reported Vulkan crash | **105.46 tokens/s is the recorded prefill rate, not generation speed.** Dividing the displayed input count by 1,230 seconds gives approximately 105.47 tokens/s; the small final-digit difference is retained as a reporting-precision issue rather than silently changing the original result. This is one historical observation. The notes report no observed leak, but the series does not present a controlled leak test. A paired raw timing trace, full request, and exact test-build attribution are not included in the published baseline, so this is not presented as a reproducible performance comparison. ## 14. Configured Context Is Not Always Effective Context The experimental gpt-oss-120B process requested 262,144 and exposed 131,072. Qwen2.5-VL-72B requested 131,072 and exposed 128,000. Jina Reranker v2 requested 8,192 and exposed 1,024.[^startup] Those are observations from the captured builds, not a universal formula for every `llama.cpp` version or model. [Part 3](https://deshimarusakaguchi.com/lab/local-ai-cluster/context-runtime/) separates artifact metadata, operator requests, effective slots, and actual request evidence. ## 15. Batch Parameters Can Also Change at Startup Server03's embedding and reranking logs report that the requested `n_batch = 2048` was adjusted to `n_ubatch = 256`.[^startup] The deployment command describes intent. The startup log describes interpretation. Both belong in a reproducibility record. ## 16. Problems Encountered The construction notes describe earlier ROCm allocation failures and Vulkan device-loss incidents. The later kernel capture directly records paired `amdgpu_vm_validate()` and command-submission memory errors on Server02.[^notes][^runtime] The exact workload associated with that kernel burst is unresolved. It is not presented as proof that the current VLM process caused it. Part 2 preserves the symptoms, reported changes, and limits of the causal evidence. ## 17. Evidence and Interpretation The reports distinguish four things: directly captured state, behavior recorded during construction, architectural interpretation, and unresolved hypotheses. A logged memory error is evidence of a reported failure. A lower micro-batch followed by an apparently successful run is a useful construction observation. Neither automatically proves a universal mechanism or guarantees multi-hour stability. ## 18. Intended Research Work The platform is intended for long-form analysis, historical sources, multimodal documents, retrieval, structured extraction, and development support. The complete Sanapedia/Dify pipeline is still a planned integration stage. The service inventory makes that work possible to attempt. Workload-specific quality, latency, and recovery behavior remain to be measured. ## 19. Limits of This Baseline This first phase does not establish controlled generation speed, wall power, thermal characteristics, multi-hour stability, repeated-run variance, or end-to-end quality. It is also not a controlled comparison between the operating-system versions. The inference nodes are captured in detail. The full control-plane software inventory, provider/plugin versions, and functional request traces for every Dify model are not part of this collection. ## 20. Security and Public Examples Public text uses `LLM-Server01`, `LLM-Server02`, `LLM-Server03`, `Control-PC`, `$HOME`, and ``. Local paths are normalized without changing model filenames. Private hostnames, user identities, IP/MAC addresses, credentials, and unredacted raw logs are omitted. Normalization protects documentation hygiene; it does not itself secure the live network. The access-control discussion is in Part 4. ## 21. From Running a Model to Operating Infrastructure The design result is a division of responsibilities: a large text node, a large visual node, and a multi-service utility node. The captured records establish that this deployment exists; they do not complete every operational test that an organization might require before adopting it. That is why the baseline is worth preserving before the next phase. The architecture can now be tested as a system rather than as a collection of isolated model launches. Continue with [Part 2: Memory, Vulkan, and AMD Strix Halo](https://deshimarusakaguchi.com/lab/local-ai-cluster/memory-vulkan-strix-halo/). ## Evidence and Reading Notes [^system]: Author-retained system baseline, September 23, 2026 (JST): CPU, firmware, OS/kernel strings, Mesa/Vulkan versions, and byte-valued DRM counters. BIOS allocation was also confirmed by the operator. [^runtime]: Author-retained runtime baseline, September 23, 2026 (JST): process arguments, systemd unit definitions/properties, listening ports, and Server02 kernel-error context. [^artifacts]: Author-retained model/GGUF baseline, September 23, 2026 (JST): resolved targets, exact sizes, split files, model/projector metadata, and the failed gpt-oss-120B Python dump. [^startup]: Author-retained startup/context capture, September 23, 2026 (JST): current model-load messages, slot context, batch correction, and the separate gpt-oss-120B experiment. Older Server01 RAG logs are excluded from the current service count. [^production]: Operator clarification and production-script excerpt: gpt-oss-120B uses `-c 131072`; the sampled `-c 262144` process was a retained experiment. [^notes]: Author's construction notes: control-plane description, intended roles, earlier monitor readings, prefill record, and historical troubleshooting. These are distinguished from newly captured raw runtime evidence. [^api]: External reference, checked during editorial preparation: [official llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md). Current documentation explains interfaces; the captured local builds remain the source for observed behavior. --- [↑ Back to the Lab Hub](https://deshimarusakaguchi.com/lab/local-ai-cluster/) [Next → Part 2: Memory, Vulkan, and AMD Strix Halo](https://deshimarusakaguchi.com/lab/local-ai-cluster/memory-vulkan-strix-halo/)