Persistent AI Services and Operations

Running a large local model and operating a collection of models are different problems. A benchmark can load a model, run one prompt, record a result, and exit. A research environment needs identifiable capabilities, predictable addresses, startup control, logs, and a way to determine whether each endpoint can do its intended work.

This cluster organizes ten model processes across three inference nodes. The captured inventory contains one process on each large-model node and eight on the utility node. The construction notes describe a Windows/Dify control environment above them.¹²

The transition is from running models to operating services. This article explains the current arrangement while distinguishing captured deployment state from future automation and untested recovery behavior.

1. Why Persistent Services Matter#

A research task may classify an input, create an embedding, retrieve evidence, rerank candidates, invoke a language or visual model, and extract structured results.

Loading a different model before every stage would make model lifecycle part of the workflow. My design moves that concern to deployment: keep the core model set resident and call the required capability.

This is an architectural preference for predictable availability, not a measured claim that it is always more economical than dynamic loading. Idle power and utilization have not yet been quantified.

2. The Ten-Service Topology#

                          Control-PC
                    Windows 11 / Dify / WSL2
                              |
                Capability-specific local HTTP APIs
                              |
          +-------------------+-------------------+
          |                   |                   |
          v                   v                   v
    LLM-Server01        LLM-Server02        LLM-Server03
       :8080               :8080              :8080-8087
          |                   |                   |
    gpt-oss-120B        Qwen2.5-VL-72B             +-- Gemma-4-31B-it
                                                  +-- Embeddings x2
                                                  +-- Rerankers x2
                                                  +-- Router
                                                  +-- Extractor
                                                  +-- Coder

The nodes do not jointly execute one model. They expose separate roles. The inference topology is captured directly; the control-plane design comes from the environment notes.¹²

3. One Process per Role, Not One systemd Unit per Role#

Each role has a separate llama-server process, model selection, port, arguments, and log. This makes attribution easier when inspecting a model load or runtime message.

However, service has two meanings here. An application service is a model endpoint. A systemd service is a host-supervision unit.

The captured deployment uses one llama-cluster.service unit per node. On Server03, that unit invokes a script which launches eight model processes. They are not eight independently supervised systemd services.¹

That boundary matters for restart behavior, readiness, and failure isolation.

4. Port Mapping as Service Identity#

The process arguments and listening sockets establish this map.¹

The roles describe intended workflow responsibilities. A dedicated router endpoint exists, but the captured process list is not a test of routing quality or proof that the complete research workflow is already automated.

5. What systemd Currently Manages#

A normalized version of the captured unit is:

[Unit]
Description=Llama.cpp Multi-Model Cluster
After=network.target

[Service]
Type=forking
User=<user>
WorkingDirectory=/home/<user>

LimitMEMLOCK=infinity
LimitNOFILE=65535

ExecStart=/home/<user>/scripts/start_all_services.sh
ExecStop=/home/<user>/scripts/stop_all_services.sh

Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

<user> is a publication placeholder and must be replaced before use. This is the observed arrangement, not a new hardened deployment template. The captured properties include ActiveState=active, SubState=running, and the displayed restart/resource-limit settings.¹

Supervision limitation: a multi-process Type=forking unit without an explicit PIDFile does not establish independent failure detection and restart for every child. Upstream systemd documentation specifically cautions about main-process identification in such arrangements.³

The baseline does not include a fault-injection recovery test. Separate units or an explicit supervisor would be a possible future revision, not a change silently applied to this record.

6. Startup Scripts as Deployment Maps#

The production 120B launch follows this pattern:

#!/bin/bash

LLAMA_SERVER="$HOME/llama.cpp-vulkan/build/bin/llama-server"
MODEL_DIR="$HOME/models"
LOG_DIR="$HOME/logs/llama"

mkdir -p "$LOG_DIR"

"$LLAMA_SERVER" \
  -m "$MODEL_DIR/gpt-oss-120b-MXFP4-00001-of-00002.gguf" \
  -ngl 999 \
  --host 0.0.0.0 \
  --port 8080 \
  -c 131072 \
  -b 2048 \
  -ub 256 \
  --parallel 1 \
  --flash-attn on \
  > "$LOG_DIR/llm-8080.log" 2>&1 &

Paths are normalized and shell quoting is made explicit. The production arguments preserve the operator's script; the earlier -c 262144 process was a separate experiment.⁴

Security note: 0.0.0.0 binds all host IPv4 interfaces, not just a trusted LAN. This example does not establish authentication or firewall isolation. Do not publish an unauthenticated inference endpoint to the Internet.

The script backgrounds the process. Its exit does not certify that model loading succeeded. The > redirection also replaces the previous log on the next start, so evidence needed for a baseline must be archived before restart. No log-rotation or archival automation is claimed here.

7. Zero Model-Load Cold Start#

In this series, zero model-load cold start means that a request normally does not initiate a fresh model load while the core service is already resident.

It does not mean zero time to first token. Requests can still wait for an occupied slot, perform long prefill, initialize workload-specific state, or encounter errors.

The captured processes use --parallel 1. Persistent availability should therefore not be confused with unlimited concurrent request capacity.¹

The trade-off is reserved memory and idle power. Quantitative comparison with dynamic loading belongs to Phase 2.

8. API Compatibility Is Capability-Specific#

A common LLM API base can be represented as:

http://LLM-Server01:8080/v1

But the entire ten-service platform is not one interchangeable API. Upstream llama-server documentation distinguishes OpenAI-compatible chat and embedding routes from a separate reranking interface.⁵

A /v1 prefix alone does not prove that an arbitrary client supports every endpoint behind it.

The orchestration layer needs to know capability and API contract. It need not know the private GGUF path, memory layout, or model-launch mechanics.

9. Dify Model Registration and Its Evidence Boundary#

The construction notes list model registrations for local LLM/VLM, embedding, and reranking services. Those registrations are described there under an OpenAI-compatible provider, with model names, base URLs, vision settings, and contexts.²

For publication, the accurate architectural description is: Dify registers local endpoints as models with capability-appropriate provider support. A router or extractor is a workflow role, not necessarily a separate native Dify model type. Dify's provider documentation distinguishes model names/types and their credentials.⁶

The collection does not include exact Dify/provider-plugin versions or successful request traces for every registration. In particular, the generic OpenAI-compatible label should not be read as proof that the installed plugin handles the reranking contract.

The runtime cap also matters. A registration that still says 131,072 for the Qwen2.5-VL service must not be treated as evidence of a 131,072-token usable slot when its server reports 128,000.

10. Role-Based Routing#

The intended routing pattern is:

Research request
      |
      v
Task classification
      |
      +-- Retrieval needed? -> Embed -> Retrieve -> Rerank
      |                                          |
      +------------------------------------------+
      |
      v
Select text or visual analysis endpoint
      |
      v
Optional structured extraction / post-processing

This is a design pattern for the future workflow, not an exported and validated Dify graph. Not every request requires every stage.

11. Why Assign a 3B Router?#

The router's intended task is narrow classification: decide whether a request belongs to text analysis, visual analysis, retrieval, extraction, or coding support.

My design reserves the large-model endpoints for work where deeper synthesis is expected to matter. The smaller router is a candidate implementation for a smaller job, not a proven latency or accuracy winner in this baseline.

A routing evaluation must later test incorrect classifications and their consequences, not only whether the endpoint responds.

12. Structured Extraction Is a Separate Responsibility#

The Qwen2.5-7B-Instruct endpoint is assigned to transform analysis into machine-readable records, for example:

{
  "title": "...",
  "entities": [],
  "dates": [],
  "claims": [],
  "sources": []
}

This is an illustrative output shape, not a captured successful response. Naming a model “extractor” does not guarantee schema compliance. The workflow needs validation and error handling, with any structured-output mechanism tested against the selected runtime/provider.

Reasoning and serialization can be separated without pretending that the second stage is automatically correct.

13. Coding and Preprocessing#

Qwen2.5-Coder-7B is assigned to code assistance, normalization, text transformations, and development support.

The role provides generated suggestions. It does not imply that generated code is automatically executed on the host. Any execution stage requires its own permissions, review, isolation, and validation policy.

This separation keeps inference capability distinct from authority to change files or run system commands.

14. Embedding and Reranking Remain Separate Stages#

The retrieval design is:

Query -> Embedding -> Candidate retrieval -> Reranking -> Selected evidence

The two embedding services and two rerankers provide room for comparison and future specialization. Their simultaneous deployment is confirmed; a superiority ranking or automatic fallback policy is not.¹

They also have different effective context and batching behavior. The Jina service's captured slot is 1,024, and the embedding/reranking startup logs adjust batch settings to 256. Full-sized payload acceptance must be tested separately rather than inferred from the listening port.⁷

15. Control Plane and Inference Plane#

The environment notes describe Windows 11, PowerShell, WSL2, Docker-based services, and Dify on the control side. The three Linux nodes form the inference side.²

This is a useful responsibility split: the control plane coordinates requests and lifecycle operations; the inference plane hosts model endpoints.

The Linux stack was captured in detail. Equivalent versioned control-plane evidence has not been collected in this baseline, so this article is not a fully pinned Dify installation guide.

16. Remote Lifecycle Management#

The construction notes describe Wake-on-LAN for startup, SSH for inspection, and remote shutdown. Public examples omit real MAC/IP addresses and authentication details.²

A status-only PowerShell example is:

# Illustrative read-only inspection; replace <user> and configure name resolution.
$servers = @("LLM-Server01", "LLM-Server02", "LLM-Server03")
foreach ($server in $servers) {
    Write-Host "=== $server ==="
    ssh "<user>@$server" "free -h && ss -lnt"
    if ($LASTEXITCODE -ne 0) {
        Write-Warning "Inspection failed for $server."
    }
}

Power-on is not model readiness. Before shutdown, active research jobs should be stopped or drained. Any passwordless administrative permission should be limited to the required command rather than broad unrestricted sudo access.

The supplied evidence does not establish automated job draining, power-on health gating, or a tested cluster-wide recovery sequence.

17. Operational State Checks#

The captured investigation used complementary read-only views:

systemctl status llama-cluster.service --no-pager
ps -eo pid,args | grep '[l]lama-server'
ss -lntp

The first checks unit state, the second process presence and arguments, and the third socket state. Startup logs add model-load and configuration evidence.

A representative request is still needed to establish functional behavior. A future health check should distinguish service readiness from a completed task of the correct type.

18. What Server03 Actually Demonstrates#

The runtime capture shows eight model processes and listeners on every port from 8080 to 8087. Its startup logs identify loaded models and initialized slots.¹⁷

That is stronger evidence than a deployment script alone. It is still narrower than proving eight simultaneous successful inference jobs, automatic recovery of any failed child, or correct completion of a whole research workflow.

19. Logging as Part of Reproducibility#

The startup scripts assign separate log paths:

llm-8080.log
rag-8081.log
rag-8082.log
rag-8083.log
rag-8084.log
router-8085.log
extractor-8086.log
coder-8087.log

Those logs made the context caps and batch corrections in Parts 2 and 3 visible.⁷

Log existence is not sufficient provenance. Old RAG logs remained on Server01 despite its current single-model process list. A preserved record should identify the node, capture time, process/run, and associated configuration before a historical file is cited as current behavior.

20. Service Health Is Layered#

Unit active
    -> Expected process exists
    -> Expected port listens
    -> Model load completes
    -> API responds
    -> Representative request succeeds
    -> Output meets the task's acceptance criteria

The baseline is strongest on the first four layers, with additional request evidence for the gpt-oss context experiment. It does not contain a uniform functional/quality test suite across all ten endpoints.

This distinction matters more than a single green “active” indicator.

21. Failure Isolation Has Limits#

Separate machines provide useful host boundaries. Separate processes make logs and launches easier to attribute.

Within Server03, however, the services still share hardware, driver state, memory bandwidth, an operating system, and one systemd unit. A GPU failure or unit-level stop can affect several endpoints. A non-main child failure may not trigger the expected unit restart.³

The architecture offers separation, not a demonstrated high-availability system. There is no captured automatic failover or independently measured per-service recovery guarantee.

22. The Security Boundary#

The endpoints are intended for local use, but intention is not enforcement. The shown --host 0.0.0.0 binding is not a LAN restriction.

Access control, authentication, and any external tool connections must be considered independently. A placeholder API-key string entered in a client is not proof of server-side authentication. The excerpts do not establish the full firewall, router, or credential configuration, and those private details are not published here.

Public text uses LLM-Server01, LLM-Server02, LLM-Server03, Control-PC, $HOME, and <user>. Raw logs, private paths, IP/MAC addresses, keys, tokens, and VPN/router credentials are excluded.

This is documentation normalization, not a substitute for a security assessment of the live deployment.

23. Stable Roles, Replaceable Models#

The role “deep analysis” can outlive the particular model currently implementing it. The same is true of routing, extraction, multimodal analysis, or reranking.

A stable endpoint and role reduce workflow coupling. They do not make a replacement automatically compatible. Model identifiers, templates, tool handling, context, output formatting, vision payloads, and resource requirements still require validation.

This is especially important for embeddings: the captured BGE-M3 artifact reports 1,024 dimensions, while GTE-Qwen2 reports 1,536. A role or URL remaining unchanged does not preserve vector compatibility or remove the need to review indexing and re-embedding.⁸

24. Infrastructure as Roles#

The role table above is intended to be more durable than the current model list. Model changes can be documented as implementation changes behind those responsibilities.

Acceptance criteria should also attach to the role. A router needs a classification test; an extractor needs a schema test; an embedding/retrieval path needs a corpus-level test; a visual model needs representative visual sources.

Those are proposed evaluation boundaries, not completed measurements.

25. The Research Workflow Pattern#

The complete Sanapedia pipeline is not yet implemented. Its planned shape is:

Research request
      |
      v
Dify coordination and task classification
      |
      +-- Optional retrieval / reranking
      |
      v
Selected LLM or VLM
      |
      +-- Optional structured extraction
      +-- Optional reviewed post-processing
      |
      v
Research result with evidence and validation

This is deliberately not presented as an existing exported workflow. The current infrastructure supplies candidate services for building it.

26. What Is Not Claimed as Automated#

The baseline does not establish a complete Sanapedia pipeline, centralized metrics, workload-aware routing, automatic failover, health-based model selection, model-utilization telemetry, or power/thermal-aware scheduling.

It also does not establish independent child-process supervision or automatic archival of each overwritten model log. Those are operational development areas, not hidden guarantees of the current scripts.

27. Why Not Dynamic Loading in This Phase?#

The design prioritizes resident core models, predictable endpoints, and fewer load/unload decisions in workflow logic.

Dynamic loading could be evaluated later. The present choice is not a claim that it has lower total cost, higher efficiency, or better latency under every utilization pattern.

The relevant comparison needs measured idle cost and actual workload demand.

28. Operational Trade-Offs#

The intended benefits are fewer model-loading transitions, clear service identities, and simpler role-based calls. The costs include reserved memory, idle power, shared-host contention, and the need for lifecycle and health management.

The process/socket captures show the chosen topology. Quantitative benefits and costs remain Phase 2 work.

29. Reproducibility Checklist#

For each endpoint, preserve its role, node/port, exact model/projector, runtime build, configured and effective context, batch settings, supervision unit, log identity, API contract, provider/plugin version, and functional acceptance result.

For host operations, preserve the distinction between an existing setting and a tested recovery procedure. Do not infer reliability from a restart directive without observing the relevant failure path.

30. What the First Four Reports Establish#

Part 1 documents machines and role allocation. Part 2 separates memory interfaces, construction history, and observed failures. Part 3 traces context from artifact or runtime declaration to effective slot and available request evidence. Part 4 describes the service and control boundaries.

Together, they form a dated engineering record of one local research environment. They are not a certification that every application-level, security, or reliability test has already passed.

31. Phase 2: Sanapedia and Real Workloads#

The next phase will evaluate the system while it performs research work: model invocation counts, request sizes, routing decisions, retrieval frequency, end-to-end latency, failure/retry rates, throughput, utilization, power, temperature, sustained behavior, and repeated-run variance.

Those measurements will follow the actual workflow rather than being invented to complete this first publication.

Phase 1 documents what the infrastructure is. Phase 2 will measure what it does.

Evidence and Reading Notes#