Local AI Infrastructure Lab

A local AI research cluster built around three 128GB AMD Strix Halo mini PCs#

This project is a persistent local AI infrastructure for long-document analysis, multimodal research, retrieval-augmented generation (RAG), structured extraction, and supporting development work. Its design keeps model inference on the local network rather than requiring an external model API.

The system uses three GMKtec NucBox EVO-X2 compute nodes, each with an AMD Ryzen AI MAX+ 395, Radeon 8060S integrated graphics, and 128GB unified memory. The BIOS UMA allocation is set to 96GB on every node. The captured process and socket records show ten llama-server processes across the three machines, including eight on the utility node.¹

This page is the permanent entrance to the architecture, deployment observations, troubleshooting records, and later operational studies. It is an index, not a substitute for the detailed reports.

Baseline at a Glance#

Hardware and runtime entries are grounded in the captured system records. The control environment and intended workflow roles are described in the construction notes; the complete Sanapedia workflow is a later phase.¹²

What This Is#

This is not a single-model benchmark machine.

The objective was not simply to find the largest model that could be loaded. It was to keep different capabilities available for different jobs: long-form reasoning, visual analysis, embeddings, reranking, routing, structured extraction, and coding or preprocessing.

The nodes do not combine their memory to run one distributed model. They host separate services. Each model executes on its assigned node, and a workflow selects the relevant endpoint.

The design question is therefore not only “Can this model run?” It is also “Can these models operate together as persistent infrastructure?”

Architecture at a Glance#

                         Control-PC
                 Windows 11 / Dify / WSL2
                            |
             +--------------+--------------+
             |              |              |
             v              v              v
       LLM-Server01   LLM-Server02   LLM-Server03
       128GB memory   128GB memory   128GB memory
       96GB BIOS UMA  96GB BIOS UMA  96GB BIOS UMA
             |              |              |
       gpt-oss-120B    Qwen2.5-VL-72B       +-- Gemma-4-31B-it
                                           +-- Embedding x2
                                           +-- Reranking x2
                                           +-- Router
                                           +-- Structured extractor
                                           +-- Coder / preprocessor

LLM-Server01: Long-Context Text Analysis#

The first node hosts gpt-oss-120B MXFP4. Its intended role is deep analysis and synthesis across large source collections. The production startup script requests 131,072 tokens. A separate 262,144-token configuration was used experimentally and is not the production setting.³

LLM-Server02: Large Multimodal Analysis#

The second node hosts Qwen2.5-VL-72B-Instruct Q4_K_M with an F16 vision projector. Its intended role includes scans, maps, diagrams, and illustrated research material. Although the command requests 131,072 tokens, the captured startup log reports an effective slot of 128,000.⁴

LLM-Server03: A Multi-Service Utility Node#

The third node hosts Gemma-4-31B-it with a BF16 projector, plus two embedding models, two rerankers, a Qwen2.5-3B router, a Qwen2.5-7B extractor, and a Qwen2.5-Coder-7B service. The runtime records show eight processes and listening ports 8080 through 8087.¹

Simultaneous residence is the observation here. It is not a benchmark of eight simultaneous inference workloads.

Read the Technical Series#

Selected Baseline Observations#

The most useful findings are not all performance numbers.

The startup logs show that llama.cpp capped the experimental gpt-oss-120B slot from 262,144 to 131,072, the Qwen2.5-VL-72B slot from 131,072 to 128,000, and the Jina Reranker v2 slot from 8,192 to 1,024. They also record automatic batch adjustment for the utility node's embedding and reranking services.⁴

The construction notes record a 129,722-token prefill workload, a processing time of 20 min 30 sec, and a reported average of 105.46 tokens/s. This is a single recorded prompt-processing observation, not text-generation speed or a repeated-run benchmark.²

The memory records include both earlier monitor snapshots and later byte-valued DRM readings. The detailed articles keep those measurement series separate rather than combining them into one fixed consumption figure.

What the Evidence Establishes#

The captured baseline establishes hardware and software identities, GPU-visible memory totals, running model processes, listening sockets, systemd settings, model-file sizes, available GGUF metadata, and startup interpretation.

The gpt-oss-120B model-context value comes from the runtime's n_ctx_train report. A standalone GGUF dump for that model was not obtained during this collection because the Python environment lacked NumPy. That limitation is retained, not filled in by assumption.⁵⁴

A listening port is not a complete functional test. Nor does a successful startup establish long-duration stability, automatic recovery of every child process, or research-output quality. The series distinguishes direct capture, construction notes, design intent, and unresolved questions.

What Has Not Yet Been Measured#

The next phase will address controlled generation throughput, wall power, temperature, sustained-load behavior, repeated-run variance, and real research-workload telemetry. End-to-end latency, failure/retry rates, and model utilization will be measured when the Sanapedia and Dify research workflow is operating.

No values for those areas are estimated here, and no publication date is promised for the next phase.

Scope, Security, and Updates#

“Local” describes the inference design and intended data-handling boundary. It is not a claim that the machines are air-gapped or that every optional tool, plugin, or research source is offline.

Public examples normalize hostnames, usernames, paths, and network identifiers. Raw private logs are not bundled with these articles. The detailed network and credential cautions are in Part 4.

Later revisions will preserve the September 23, 2026 baseline rather than silently assigning old measurements to new software. This is a field record of one evolving system, not a hardware certification or a universal recipe.

Phase 1 documents what the infrastructure is. Phase 2 will measure what it does.

Evidence Notes#