--- title: Memory, Vulkan, and AMD Strix Halo subtitle: Operating 96GB BIOS UMA for 120B and 72B local models with llama.cpp description: 'A 96GB UMA field record: Vulkan/RADV deployment, TTM/GTT differences, memory snapshots, historical ROCm failures, and micro-batch observations.' lang: en author: Deshimaru Sakaguchi series: Local AI Infrastructure Lab series_part: 2 baseline_date: '2026-09-23' baseline_timezone: Asia/Tokyo last_updated: '2026-09-23' revision: '1.0' canonical: https://deshimarusakaguchi.com/lab/local-ai-cluster/memory-vulkan-strix-halo/ permalink: /lab/local-ai-cluster/memory-vulkan-strix-halo/ tags: - local-ai - strix-halo - llama-cpp - research-infrastructure --- # Memory, Vulkan, and AMD Strix Halo [Lab Hub](https://deshimarusakaguchi.com/lab/local-ai-cluster/) · [Part 1](https://deshimarusakaguchi.com/lab/local-ai-cluster/architecture/) · **Part 2** · [Part 3](https://deshimarusakaguchi.com/lab/local-ai-cluster/context-runtime/) · [Part 4](https://deshimarusakaguchi.com/lab/local-ai-cluster/persistent-services/) **Baseline captured: September 23, 2026 (JST)** **Author: Deshimaru Sakaguchi** *Operating 96GB BIOS UMA for 120B and 72B local models with llama.cpp* A machine with 128GB unified memory appears to offer a simple route to large local models: reserve memory for the GPU, load a quantized model, and run inference. The construction record for this cluster is more complicated. It includes earlier ROCm allocation failures, a move to Vulkan through Mesa RADV, reported improvements after micro-batch changes, and later captures of kernel-level memory errors. This article preserves those observations without treating them as a controlled comparison or a universal configuration recipe. The September 2026 baseline is an observed running deployment. Multi-hour stability and repeated-run failure rates belong to the next phase. ## 1. Memory Is More Than the Weight File On each node, the operator selected a **96GB BIOS UMA allocation** from **128GB unified memory**. DRM reports a **96 GiB GPU-visible VRAM** region, while Linux reports roughly 30 GiB of conventional system RAM.[^system] The layers are easier to understand as a map than as one number: ```text 128GB physical unified memory | v 96GB BIOS UMA setting | +-- Linux-visible system RAM +-- GPU-visible VRAM | v Driver / memory-management interfaces | v Model weights + context state + compute / vision buffers ``` The model file's on-disk size is not a complete runtime-memory budget. The measurements below describe different interfaces and different capture moments. They should not be added together as if they were independent physical memory banks. ## 2. The 96 GiB DRM Baseline All three nodes report the same total through the DRM memory interface:[^system] ```text mem_info_vram_total = 103079215104 bytes mem_info_vis_vram_total = 103079215104 bytes ``` That byte count is exactly **96 GiB**. Representative `amdgpu` boot messages are: ```text VRAM: 98304M Detected VRAM RAM=98304M, BAR=98304M 98304M of VRAM memory ready ``` The raw `M` spelling is preserved inside the log excerpt. In explanatory prose, byte-derived binary values use `GiB`. The BIOS label remains `96GB` because that is the operator-reported setting. These are GPU-visible allocations in an integrated-memory system, not a claim that separate 96 GiB discrete VRAM chips exist. ## 3. Same UMA Setting, Different GTT and TTM Values The captured values are: | Metric | LLM-Server01 | LLM-Server02 | LLM-Server03 | |---|---:|---:|---:| | BIOS UMA setting | 96GB | 96GB | 96GB | | DRM visible VRAM | 96 GiB | 96 GiB | 96 GiB | | `mem_info_gtt_total`, bytes | 104,857,600,000 | 16,363,974,656 | 16,361,816,064 | | Derived GTT, GiB | 97.66 | 15.24 | 15.24 | | TTM `pages_limit` | 25,600,000 | 3,995,111 | 3,994,584 | | TTM `page_pool_size` | 25,600,000 | 3,995,111 | 3,994,584 | | TTM `dma32_pages_limit` | 524,288 | 524,288 | 524,288 | All columns come from the same collection of system-baseline records.[^system] Server01 reports: ```text 100000M of GTT memory ready mem_info_gtt_total = 104857600000 bytes pages_limit = 25600000 page_pool_size = 25600000 ``` Server03 reports: ```text 15603M of GTT memory ready mem_info_gtt_total = 16361816064 bytes pages_limit = 3994584 page_pool_size = 3994584 ``` The GTT figures are approximately **97.66 GiB**, **15.24 GiB**, and **15.24 GiB**. Earlier command output rounded these to `98G` and `16G`; the table retains the byte counts to avoid treating that display rounding as exact capacity. **Observed:** identical BIOS UMA settings coexist with different GTT/TTM reports. **Not established:** why those values differ. The machines differ in firmware, distribution, package builds, and potentially configuration history. The same short kernel release label does not establish identical kernel packages. No controlled A/B comparison isolated these variables. The record does not justify attributing the difference specifically to Mesa, Ubuntu, or a single kernel setting. It also does not establish that the reported GTT total is an additional physically reserved pool on top of VRAM and system RAM. ## 4. Earlier ROCm Experiments The construction notes record an earlier large-model offload failure with messages including:[^notes] ```text ggml_backend_cuda_buffer_type_alloc_buffer: allocating 29632.42 MiB on device 0: cudaMalloc failed: out of memory llama_model_load_from_file_impl: using device ROCm0 (Radeon 8060S Graphics) - 29980 MiB free ``` The `cuda` spelling is preserved from the recorded message even though the experiment was described as using ROCm/HIP. It is not relabeled as evidence of NVIDIA hardware. The defensible conclusion is narrow: **the recorded ROCm configuration failed during a large allocation**. The complete earlier ROCm version, build configuration, and workload trace are not attached to this publication baseline. This does not establish a permanent “30GB ROCm limit,” a universal TTM one-third allocation rule, or a current comparison between ROCm and Vulkan. Those would require separate evidence. ## 5. Why the Deployment Moved to Vulkan The construction notes describe moving the large-model workloads to `llama.cpp` built with `GGML_VULKAN=ON`. The later system records identify Mesa RADV, and the runtime captures show the large models loaded through the resulting deployment.[^notes][^system][^startup] The official repository is now published under `ggml-org/llama.cpp`. The following preserves the documented build pattern, with a recorded revision selected explicitly rather than implying that the latest branch reproduces the baseline.[^build] ```bash # Historical build pattern, not an instruction to upgrade a running node. sudo apt update sudo apt install -y \ build-essential cmake vulkan-tools libvulkan-dev \ glslc glslang-tools libshaderc-dev spirv-headers \ mesa-vulkan-drivers # Run in a separate workspace rather than over an existing installation. git clone https://github.com/ggml-org/llama.cpp.git llama.cpp-vulkan cd llama.cpp-vulkan # Recorded Server01 source revision. Use the target node's recorded revision. git checkout 64e9bceb2c3a856efed96feda784a50947049feb cmake -B build -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release cmake --build build -j"$(nproc)" ``` Package repositories can change. These commands alone do not reconstruct the historical compiler, distribution packages, firmware, or driver. They are the build approach used in the notes, not a newly tested installation script or a full environment lockfile. ## 6. Current Vulkan and Runtime Identity | Node | OS | Kernel release label | BIOS | Mesa RADV | llama.cpp build / commit | |---|---|---|---|---|---| | LLM-Server01 | Ubuntu 24.04.3 LTS | 7.0.0-31-generic | EVO-X2 1.09 | 25.2.8 | 10858 / `64e9bceb2` | | LLM-Server02 | Ubuntu 26.04.1 LTS | 7.0.0-31-generic | EVO-X2 1.11 | 26.0.8 | 11047 / `b23701f77` | | LLM-Server03 | Ubuntu 26.04.1 LTS | 7.0.0-31-generic | EVO-X2 1.11 | 26.0.8 | 11061 / `b23efaa2e` | The version records also distinguish the Vulkan instance from the device API:[^system] | Node | Vulkan instance version | Radeon device API version | Compiler reported by llama.cpp | |---|---|---|---| | LLM-Server01 | 1.3.275 | 1.4.318 | GNU 13.3.0 | | LLM-Server02 | 1.4.341 | 1.4.335 | GNU 15.2.0 | | LLM-Server03 | 1.4.341 | 1.4.335 | GNU 15.2.0 | The exact retained identities are: ```text LLM-Server01 Kernel package build: #31~24.04.1-Ubuntu Mesa package: 25.2.8-0ubuntu0.24.04.2 llama.cpp: 0.4.0-dev, build 10858 Commit: 64e9bceb2c3a856efed96feda784a50947049feb LLM-Server02 Kernel package build: #31-Ubuntu Mesa package: 26.0.8-1ubuntu0.3 llama.cpp: 0.4.1-dev, build 11047 Commit: b23701f77d47dad9de834d59ebfcbe25c9e8b46f LLM-Server03 Kernel package build: #31-Ubuntu Mesa package: 26.0.8-1ubuntu0.3 llama.cpp: 0.4.1-dev, build 11061 Commit: b23efaa2ef147f547ee75cbf0c621d61904de80e ``` These are the identifiers reported locally. Referencing current upstream documentation does not replace the captured version history. ## 7. What `LimitMEMLOCK=infinity` Establishes All three captured service definitions include: ```ini LimitMEMLOCK=infinity LimitNOFILE=65535 ``` The associated systemd properties report those values as active.[^runtime] `LimitMEMLOCK` controls the process resource limit for locked memory. It is permission to lock memory, not evidence that a process actually locked its model, removed every driver allocation limit, or gained additional physical RAM.[^limits] The record contains no isolated comparison with and without this setting. Its justified status is **part of the captured service configuration**, not **a proven prerequisite for all Vulkan inference**. The launch commands also do not, by themselves, establish which memory-locking operations occurred internally. ## 8. Keep the Two Memory Measurement Series Separate The construction notes contain representative monitoring values attributed to `radeontop` / `amdgpu_top` observation:[^notes] | Node | Workload label in the notes | Monitor value as recorded | Approximate ratio | |---|---|---:|---:| | LLM-Server01 | gpt-oss-120B | 69,550 MB / 98,148 MB | 70.9% | | LLM-Server02 | Qwen2.5-VL-72B | 87,790 MB / 98,148 MB | 89.4% | | LLM-Server03 | Gemma-4-31B-it + seven support services | 61,941 MB / 98,148 MB | 63.1% | These retain the recorded `MB` unit label. The source does not attach a complete tool/version/time record to every snapshot, so the numbers are not silently converted into a different counter's unit system. The later direct DRM readings are: | Node | `mem_info_vram_used` | Binary conversion | Total reported by DRM | |---|---:|---:|---:| | LLM-Server01 | 72,928,403,456 bytes | 67.92 GiB | 96 GiB | | LLM-Server02 | 92,056,068,096 bytes | 85.73 GiB | 96 GiB | | LLM-Server03 | 66,990,362,624 bytes | 62.39 GiB | 96 GiB | Their denominator is exactly **103,079,215,104 bytes = 96 GiB**.[^system] The earlier monitor denominator of `98,148 MB` and the DRM total are not presented as identical measurements. Counter definitions, reservations, display units, and capture timing may matter; the record does not isolate their contributions. Both series place Server02 closest to its reported ceiling. They do not establish a safe maximum occupancy or the peak allocation required by a future image or long-context request. ## 9. Model Residence and Submission Errors Are Separate Observations The captured Server02 runtime shows a loaded Qwen2.5-VL-72B service. Its kernel history also contains this repeated pair:[^runtime][^startup] ```text amdgpu_vm_validate() failed. Not enough memory for command submission! ``` The first shown pair occurs around kernel monotonic time `41896.499` seconds, followed by a burst of similar messages. Those timestamps identify the event within the kernel log; they do not identify the application request that triggered it. **What is established:** the driver reported the errors, and the later collection also found a running VLM process. **What is not established:** whether the same VLM process was running during that burst, which prompt or image was active, whether another GPU client contributed, or whether GTT/VRAM occupancy was the direct cause. Loading a model and successfully executing a workload are different tests. This record illustrates why both should be captured, but it is not a completed causal reconstruction of that particular incident. ## 10. Earlier Vulkan Device-Loss Reports The construction notes preserve earlier messages including:[^notes] ```text radv/amdgpu: The CS has been cancelled because the context is lost. This context is innocent. ggml_vulkan: device lost on Vulkan0 srv update_slots: decode() failed: vk::Queue::submit: ErrorDeviceLost ``` The notes associate those incidents with aggressive long-prompt or multimodal testing and report improvement after reducing micro-batch size. That is useful historical information. It does not prove a GPU timeout as the sole cause, demonstrate elimination of every device-loss condition, or establish a long-run failure rate. A controlled reproduction with paired settings and workload traces is outside this baseline. ## 11. The Operational Batch Pattern The large-model launch commands use: ```text -b 2048 -ub 256 ``` In the documented interface, `-b` is the logical maximum batch size and `-ub` the physical maximum micro-batch size.[^batch] The gpt-oss-120B production pattern is: ```text -c 131072 -b 2048 -ub 256 --parallel 1 --flash-attn on ``` The first value is from the production script. The earlier sampled 256k process is retained as a separate experiment, not blended into the production record.[^production] The construction history explains why a smaller micro-batch was selected. It does not establish that `256` is universally optimal, or that the same value should be copied without testing another model or build. ## 12. Embedding and Reranking Settings Are Adjusted On Server03, the captured startup logs for the embedding and reranking services include:[^startup] ```text embeddings enabled with n_batch (2048) > n_ubatch (256) setting n_batch = n_ubatch = 256 to avoid assertion failure ``` The effective batch setting is therefore different from the literal command line for those services. Older RAG logs also existed on Server01, with different micro-batch and slot settings. The current process inventory shows only the 120B service there. Those historical files are not evidence of additional currently running services and are not merged with Server03's active configuration. The practical rule is: **record the process arguments and the matching startup interpretation, not just a script or any file that happens to be in the log directory.** ## 13. Different Messages Describe Different Layers | Observation | Evidence category | What it does not establish | |---|---|---| | Earlier ROCm allocation failure | Construction-note excerpt | A permanent backend capacity ceiling | | Vulkan `ErrorDeviceLost` | Construction-note excerpt | One universal driver-failure mechanism | | `amdgpu_vm_validate()` / submission error | Direct kernel capture | The responsible request or process | | Context cap | Direct startup capture | A theoretical limit under every extension method | | Batch correction | Direct startup capture | A universal optimum for all embedding models | A warning or automatic correction is not necessarily a failure. Layer and severity should be identified before attempting a diagnosis. ## 14. Conditions of the Running Baseline The running deployment combines 96GB BIOS UMA, Vulkan/RADV, separated large-model nodes, the recorded service limits, 256-token micro-batches, distinct per-model logs, and unallocated memory at the captured moments. Some are design choices. Some are settings confirmed by capture. Micro-batch improvement is reported in the construction history. Their individual contribution to stability was not isolated. Calling these **the conditions of the observed baseline** is more accurate than calling each one a proven fix. Long-duration stability remains explicitly unmeasured. ## 15. What I Would Not Generalize I would not attribute the GTT difference to the distribution name alone. I would not describe an older ROCm result as its current behavior. I would not treat `DeviceLost` as a complete diagnosis or a successful load as a stability guarantee. I would also not infer that an OS memory counter is a simple measure of how much model weight can be added. It belongs to a particular interface, allocation policy, and capture state. ## 16. Reproducibility Record A useful memory-related comparison needs hardware and BIOS identity, UMA setting, full OS/kernel package identity, Mesa and Vulkan details, exact runtime build, model and projector files, requested and effective context, logical and physical batch sizes, parallel slots, service limits, and workload details. Where an error is central to a claim, the record should preserve the relevant application and kernel logs with correlated timestamps. Peak usage under the workload is more informative than an unrelated idle snapshot. The current series contains substantial portions of that record. It does not pretend the missing historical test context has been recovered. ## 17. Current Interpretation The captures establish a deployment of large quantized models on this 96GB UMA configuration. The construction history explains the route taken to reach it. The central lesson is not simply that “96GB is enough.” It is that capacity, the driver/runtime stack, workload requirements, and operational policy have to be considered together. ## 18. Open Questions The reason for the GTT/TTM differences remains unresolved. So does the exact workload behind the Server02 error burst. A new ROCm comparison would require a separately recorded experiment, and a safe sustained occupancy threshold for the VLM node would require longer workload tests. These are future questions, not prerequisites for publishing an accurately bounded construction baseline. ## 19. Next: Context Is a Separate Question A model can be loaded while its service exposes less context than the command requested. The next report follows that distinction from model metadata through startup logs to actual rejected requests. Continue with [Part 3: Model Context vs. Runtime Context](https://deshimarusakaguchi.com/lab/local-ai-cluster/context-runtime/). ## Evidence and Reading Notes [^system]: Author-retained system-baseline capture, September 23, 2026 (JST): hardware, BIOS, OS/kernel identity, Vulkan instance/device reports, Mesa packages, llama.cpp versions/commits, TTM parameters, and DRM memory values. Operator confirmation supplies the BIOS setting. [^runtime]: Author-retained runtime capture, September 23, 2026 (JST): systemd configuration/properties, process list, and Server02 kernel-error context. The responsible workload for the kernel burst was not identified. [^startup]: Author-retained startup/context capture, September 23, 2026 (JST): model loading, effective contexts, and batch adjustment. Current and older retained logs are treated separately. [^production]: Operator-supplied production-script clarification: `-c 131072` is the intended gpt-oss-120B production value; `-c 262144` belonged to a retained experiment. [^notes]: Author's construction notes: historical ROCm/Vulkan error excerpts, reported micro-batch improvement, and representative monitor snapshots. These do not constitute a controlled backend comparison. [^build]: External reference: [official llama.cpp repository](https://github.com/ggml-org/llama.cpp) and [build documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md), checked during editorial preparation. The recorded local commits, not the current default branch, identify the baseline. [^limits]: External clarification: [systemd.exec documentation source](https://raw.githubusercontent.com/systemd/systemd/main/man/systemd.exec.xml), resource-limit definitions. This explains the setting; it is not evidence that memory locking occurred in these processes. [^batch]: External terminology reference: [official llama.cpp server parameter documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md). Local startup logs remain authoritative for observed batch adjustment. --- [← Part 1: Building a Fully Local 120B / 72B / 31B AI Cluster with Three 128GB Strix Halo Mini PCs](https://deshimarusakaguchi.com/lab/local-ai-cluster/architecture/) [↑ Back to the Lab Hub](https://deshimarusakaguchi.com/lab/local-ai-cluster/) [Next → Part 3: Model Context vs. Runtime Context](https://deshimarusakaguchi.com/lab/local-ai-cluster/context-runtime/)