--- title: Local AI Infrastructure Lab subtitle: Three 128GB Strix Halo mini PCs and ten persistent local AI services description: 'A dated field record of a three-node Strix Halo local AI cluster: architecture, memory, context, persistent services, and future workload studies.' lang: en author: Deshimaru Sakaguchi series: Local AI Infrastructure Lab series_part: 0 baseline_date: '2026-09-23' baseline_timezone: Asia/Tokyo last_updated: '2026-09-23' revision: '1.0' canonical: https://deshimarusakaguchi.com/lab/local-ai-cluster/ permalink: /lab/local-ai-cluster/ tags: - local-ai - strix-halo - llama-cpp - research-infrastructure --- # Local AI Infrastructure Lab **Lab Hub** · [Part 1](https://deshimarusakaguchi.com/lab/local-ai-cluster/architecture/) · [Part 2](https://deshimarusakaguchi.com/lab/local-ai-cluster/memory-vulkan-strix-halo/) · [Part 3](https://deshimarusakaguchi.com/lab/local-ai-cluster/context-runtime/) · [Part 4](https://deshimarusakaguchi.com/lab/local-ai-cluster/persistent-services/) **Baseline captured: September 23, 2026 (JST)** **Author: Deshimaru Sakaguchi** ## A local AI research cluster built around three 128GB AMD Strix Halo mini PCs This project is a persistent local AI infrastructure for long-document analysis, multimodal research, retrieval-augmented generation (RAG), structured extraction, and supporting development work. Its design keeps model inference on the local network rather than requiring an external model API. The system uses three GMKtec NucBox EVO-X2 compute nodes, each with an AMD Ryzen AI MAX+ 395, Radeon 8060S integrated graphics, and 128GB unified memory. The BIOS UMA allocation is set to 96GB on every node. The captured process and socket records show ten `llama-server` processes across the three machines, including eight on the utility node.[^baseline] This page is the permanent entrance to the architecture, deployment observations, troubleshooting records, and later operational studies. It is an index, not a substitute for the detailed reports. ## Baseline at a Glance | Item | September 2026 baseline | |---|---| | Compute nodes | 3 × GMKtec NucBox EVO-X2 | | APU / integrated GPU | AMD Ryzen AI MAX+ 395 / Radeon 8060S | | Physical memory | 128GB unified memory per compute node | | BIOS setting | 96GB BIOS UMA allocation per node | | DRM-reported GPU-visible VRAM | 103,079,215,104 bytes, exactly 96 GiB per node | | Largest text model | gpt-oss-120B, MXFP4 | | Large vision-language model | Qwen2.5-VL-72B-Instruct, Q4_K_M | | General multimodal model | Gemma-4-31B-it, UD-Q4_K_XL | | Service topology | 10 model processes: 1 + 1 + 8 | | Inference stack | `llama.cpp` → Vulkan → Mesa RADV → `amdgpu` | | Control environment | Windows 11 / PowerShell / WSL2, with Dify as the orchestration layer | | Main purpose | Local research and knowledge-processing infrastructure | Hardware and runtime entries are grounded in the captured system records. The control environment and intended workflow roles are described in the construction notes; the complete Sanapedia workflow is a later phase.[^baseline][^notes] ## What This Is This is not a single-model benchmark machine. The objective was not simply to find the largest model that could be loaded. It was to keep different capabilities available for different jobs: long-form reasoning, visual analysis, embeddings, reranking, routing, structured extraction, and coding or preprocessing. The nodes do **not** combine their memory to run one distributed model. They host separate services. Each model executes on its assigned node, and a workflow selects the relevant endpoint. The design question is therefore not only **“Can this model run?”** It is also **“Can these models operate together as persistent infrastructure?”** ## Architecture at a Glance ```text Control-PC Windows 11 / Dify / WSL2 | +--------------+--------------+ | | | v v v LLM-Server01 LLM-Server02 LLM-Server03 128GB memory 128GB memory 128GB memory 96GB BIOS UMA 96GB BIOS UMA 96GB BIOS UMA | | | gpt-oss-120B Qwen2.5-VL-72B +-- Gemma-4-31B-it +-- Embedding x2 +-- Reranking x2 +-- Router +-- Structured extractor +-- Coder / preprocessor ``` ### LLM-Server01: Long-Context Text Analysis The first node hosts **gpt-oss-120B MXFP4**. Its intended role is deep analysis and synthesis across large source collections. The production startup script requests 131,072 tokens. A separate 262,144-token configuration was used experimentally and is not the production setting.[^production] ### LLM-Server02: Large Multimodal Analysis The second node hosts **Qwen2.5-VL-72B-Instruct Q4_K_M** with an F16 vision projector. Its intended role includes scans, maps, diagrams, and illustrated research material. Although the command requests 131,072 tokens, the captured startup log reports an effective slot of 128,000.[^startup] ### LLM-Server03: A Multi-Service Utility Node The third node hosts **Gemma-4-31B-it** with a BF16 projector, plus two embedding models, two rerankers, a Qwen2.5-3B router, a Qwen2.5-7B extractor, and a Qwen2.5-Coder-7B service. The runtime records show eight processes and listening ports `8080` through `8087`.[^baseline] Simultaneous residence is the observation here. It is not a benchmark of eight simultaneous inference workloads. ## Read the Technical Series | Report | What it covers | |---|---| | [Part 1: Architecture and System Design](https://deshimarusakaguchi.com/lab/local-ai-cluster/architecture/) | Node roles, software versions, model allocation, representative memory snapshots, and a recorded 129,722-token prefill test | | [Part 2: Memory, Vulkan, and AMD Strix Halo](https://deshimarusakaguchi.com/lab/local-ai-cluster/memory-vulkan-strix-halo/) | UMA, TTM/GTT observations, earlier ROCm experiments, Vulkan deployment, memory failures, and micro-batch tuning | | [Part 3: Model Context vs. Runtime Context](https://deshimarusakaguchi.com/lab/local-ai-cluster/context-runtime/) | Model metadata, requested context, runtime-recognized limits, effective slot context, and captured request rejections | | [Part 4: Persistent AI Services and Operations](https://deshimarusakaguchi.com/lab/local-ai-cluster/persistent-services/) | Startup scripts, the actual systemd supervision boundary, API contracts, Dify integration, remote management, and service-health checks | ## Selected Baseline Observations The most useful findings are not all performance numbers. The startup logs show that `llama.cpp` capped the experimental gpt-oss-120B slot from 262,144 to 131,072, the Qwen2.5-VL-72B slot from 131,072 to 128,000, and the Jina Reranker v2 slot from 8,192 to 1,024. They also record automatic batch adjustment for the utility node's embedding and reranking services.[^startup] The construction notes record a **129,722-token prefill workload**, a processing time of **20 min 30 sec**, and a reported average of **105.46 tokens/s**. This is a single recorded prompt-processing observation, not text-generation speed or a repeated-run benchmark.[^notes] The memory records include both earlier monitor snapshots and later byte-valued DRM readings. The detailed articles keep those measurement series separate rather than combining them into one fixed consumption figure. ## What the Evidence Establishes The captured baseline establishes hardware and software identities, GPU-visible memory totals, running model processes, listening sockets, systemd settings, model-file sizes, available GGUF metadata, and startup interpretation. The gpt-oss-120B model-context value comes from the runtime's `n_ctx_train` report. A standalone GGUF dump for that model was not obtained during this collection because the Python environment lacked NumPy. That limitation is retained, not filled in by assumption.[^artifacts][^startup] A listening port is not a complete functional test. Nor does a successful startup establish long-duration stability, automatic recovery of every child process, or research-output quality. The series distinguishes direct capture, construction notes, design intent, and unresolved questions. ## What Has Not Yet Been Measured The next phase will address controlled generation throughput, wall power, temperature, sustained-load behavior, repeated-run variance, and real research-workload telemetry. End-to-end latency, failure/retry rates, and model utilization will be measured when the Sanapedia and Dify research workflow is operating. No values for those areas are estimated here, and no publication date is promised for the next phase. ## Scope, Security, and Updates “Local” describes the inference design and intended data-handling boundary. It is not a claim that the machines are air-gapped or that every optional tool, plugin, or research source is offline. Public examples normalize hostnames, usernames, paths, and network identifiers. Raw private logs are not bundled with these articles. The detailed network and credential cautions are in [Part 4](https://deshimarusakaguchi.com/lab/local-ai-cluster/persistent-services/). Later revisions will preserve the September 23, 2026 baseline rather than silently assigning old measurements to new software. This is a field record of one evolving system, not a hardware certification or a universal recipe. **Phase 1 documents what the infrastructure is. Phase 2 will measure what it does.** ## Evidence Notes [^baseline]: Author-retained system and runtime captures collected September 23, 2026 (JST): hardware/OS/firmware identification, DRM memory values, process commands, systemd properties, and listening sockets. BIOS UMA settings were also confirmed by the operator. These private captures are summarized here using normalized identifiers. [^notes]: Author's construction notes: control-PC layout, intended service roles, earlier memory snapshots, and the 129,722-token prefill record. Historical observations are not presented as newly repeated measurements or as a complete set of raw benchmark traces. [^production]: Operator clarification and startup-script excerpt supplied during the September 23, 2026 collection: `-c 131072` is the gpt-oss-120B production setting; the sampled process still contained the earlier `-c 262144` experiment. [^startup]: Author-retained startup/context capture, September 23, 2026 (JST): gpt-oss-120B, Qwen2.5-VL-72B, and Server03 service logs. Older RAG log files found on Server01 are not counted as current Server01 services. [^artifacts]: Author-retained model/GGUF capture, September 23, 2026 (JST): resolved file sizes, split GGUF inventory, model/projector metadata, and the NumPy import failure on Server01. --- [Start with Part 1: Building a Fully Local 120B / 72B / 31B AI Cluster with Three 128GB Strix Halo Mini PCs](https://deshimarusakaguchi.com/lab/local-ai-cluster/architecture/)