Model Context vs. Runtime Context

A launch option such as -c 131072 looks as though it answers a simple question: how much context does this service provide?

In this cluster, the answer required more than the command line. Model metadata, the runtime's recognized context, the requested allocation, startup correction, and actual request handling were not always identical.

The September 2026 captures include matched configurations, capped configurations, and services deliberately configured below the model-declared value. The clearest example requested 262,144 tokens for gpt-oss-120B; the server exposed 131,072 and subsequently rejected larger requests.¹

The value passed to -c records a request. The deployment record must also show what the running server exposed.

1. Why -c Is Easy to Misread#

A startup command is visible and easy to copy. That can make it seem like the final authority.

The observed path is more detailed:

Model artifact / metadata
           |
           v
Runtime-recognized model context
           |
           +-- Operator request: -c N
           |
           v
Server initialization and validation
           |
           v
Effective slot context
           |
           v
Actual request: accepted or rejected

This diagram describes the captured deployment. It is not a universal algorithm for every version of llama.cpp, every context-extension mechanism, or every parallel-slot configuration.

2. Four Different Meanings of Context#

GGUF-declared context is the architecture-specific value extracted from the artifact, such as qwen2vl.context_length, gemma4.context_length, qwen2.context_length, or bert.context_length.

Runtime-recognized model context is the value the loaded runtime reports as n_ctx_train. The name refers to training context in the implementation, but the log is not an independent audit of the model's training history or long-context quality.

Requested runtime context is the operator's -c value. It is one part of the service configuration.

Effective slot context is the startup-reported n_ctx_slot for this server deployment. It describes the initialized slot, not the maximum guaranteed size of an unformatted source document.

The active service records use --parallel 1. Earlier Server01 RAG logs with multiple slots are historical and are not used to infer the current single-slot behavior.²¹

3. The Main Comparison#

The model-side metadata comes from the artifact capture except for gpt-oss-120B. Slot values come from the startup logs; configured values are corroborated by the runtime inventory.³²¹

The gpt-oss-120B production script requests 131,072, not 262,144. The first row intentionally describes the retained experiment, not the intended production command.⁴

4. Case Study: gpt-oss-120B and the 256k Experiment#

The experimental process requested:

-c 262144

The captured startup messages include:¹

n_ctx_seq (262144) > n_ctx_train (131072) -- possible training context overflow
the slot context (262144) exceeds the training context of the model (131072) - capping
initializing, n_slots = 1, n_ctx_slot = 131072, kv_unified = 'false'

Only logging prefixes are omitted from these excerpts.

The result was a 262,144-token request and a 131,072-token slot. A standalone GGUF inspection had failed because NumPy was unavailable on Server01. Consequently, 131,072 is documented here as a runtime-recognized model value, not a separately extracted GGUF value.³

5. Real Requests Confirmed Enforcement#

The same retained log contains rejected requests:¹

request (223232 tokens) exceeds the available context size (131072 tokens), try increasing it
request (134331 tokens) exceeds the available context size (131072 tokens), try increasing it

The 223,232-token error appears twice in the record. These messages show that the exposed limit was enforced for those requests.

The generic “try increasing it” wording does not override the preceding capping evidence. Increasing the CLI request alone had not produced a larger slot in this experiment.

6. The 129,722-Token Workload Is Different Evidence#

The construction notes separately record successful prefill of 129,722 tokens. That is close to 131,072, but it is not part of a fully preserved, same-session boundary sweep.⁵

The subtraction 131,072 - 129,722 = 1,350 describes the numerical gap. It does not establish the remaining output budget in a particular chat request, because the complete request formatting and accounting are not included here.

The completed prefill is not a measurement of answer quality across the full document, generation throughput, or the maximum raw document payload.

7. Case Study: Qwen2.5-VL-72B#

The resolved model GGUF declares:³

qwen2vl.context_length = 128000

The process requests -c 131072. Its startup record reports:²¹

n_ctx_seq (131072) > n_ctx_train (128000) -- possible training context overflow
the slot context (131072) exceeds the training context of the model (128000) - capping
initializing, n_slots = 1, n_ctx_slot = 128000, kv_unified = 'false'

The resulting evidence chain is:

GGUF-declared: 128,000
Requested:    131,072
Slot:         128,000

This is a startup observation. A separate near-limit visual-input acceptance test is not included in the baseline.

8. “128k” Is Not an Exact Integer#

In this collection, the relevant values include both 128,000 and 131,072. Their difference is 3,072 tokens.

For reproducibility, the shorthand “128k” cannot replace the actual integer. The same applies to “256k” when the precise value is 262,144.

This series uses informal size labels in introductory prose, but exact integers in configuration and measurement tables.

9. Case Study: Gemma-4-31B-it#

The GGUF declares gemma4.context_length = 262144, the service requests -c 262144, and the startup log reports n_ctx_slot = 262144.³²¹

This is the matched case:

Declared = Requested = Startup-reported slot = 262,144

It establishes that the service initialized that slot. It does not establish a completed 262,144-token request, reliable recall across the entire window, or a quality comparison with the other models.

10. Case Study: Jina Reranker v2#

The deployed Jina GGUF contains:³

bert.context_length = 1024

Its configured -c 8192 is reduced at startup:¹

n_ctx_seq (8192) > n_ctx_train (1024) -- possible training context overflow
the slot context (8192) exceeds the training context of the model (1024) - capping
initializing, n_slots = 1, n_ctx_slot = 1024, kv_unified = 'false'

The service therefore exposes a startup-reported slot of 1,024 in this deployment. This is a claim about this GGUF and runtime combination, not every implementation carrying the Jina model name.

For a reranking request, the encoded query/document pair and its special tokens matter. A 1,024-token slot is not automatically a 1,024-token document allowance, and the record does not establish that one pair can consume the whole slot through the selected endpoint.

11. Case Study: A Deliberately Smaller Coder Service#

The Qwen2.5-Coder-7B GGUF declares qwen2.context_length = 131072. Its deployment requests 16,384 and initializes a 16,384-token slot.³²¹

This is not a failed attempt to expose 131,072. It is a smaller service allocation selected for coding and preprocessing work.

The useful distinction is:

Artifact declaration: 131,072
Service allocation:    16,384
Observed slot:         16,384

The declaration is not itself a quality guarantee, and the smaller allocation is not evidence of a limitation in the model file.

12. Utility-Node Allocations#

These entries are supported by the artifact, process, and startup records.³²¹

The role-based intent is to preserve a useful eight-service node, not to maximize the allocation of every model. The resulting memory saving was not isolated model by model in this baseline.

13. Declaration, Allocation, and Capability Are Not Synonyms#

A model's metadata is evidence of what the artifact declares. An operator's allocation is evidence of deployment policy. Neither alone proves usable quality across that range.

The same caution applies to the phrase “model supports 131k.” For this series, the precise statement is “the deployed artifact declares 131,072” unless an actual capability test is also presented.

That wording keeps a metadata inventory from becoming an unintended performance claim.

14. A Three-Layer Deployment Model#

ARTIFACT / RUNTIME DECLARATION
What the deployed file and runtime report
                 |
                 v
SERVICE POLICY
What the operator requests and the server exposes
                 |
                 v
REQUEST BUDGET
What the fully assembled request consumes

A workflow should use the final service budget while retaining the evidence behind it. Dify does not need to parse the GGUF on every request, but its configured limits should not be copied blindly from a filename or a nominal model-family label.

15. Context Allocation and Memory#

For generation services, larger context allocations can increase KV-cache requirements. The exact behavior depends on architecture, cache implementation, data types, and runtime configuration; it is not the same for all embedding and reranking architectures.⁶

A slot cap also does not, by itself, prove how much memory was returned after initialization. In the gpt-oss experiment, the log establishes the available slot, not a controlled comparison of 128k and 256k allocation footprints.

This article therefore treats smaller contexts as an allocation policy, while leaving quantitative memory savings to a separate test.

16. Read the Matching Startup Log#

The useful values in this collection are n_ctx_train, n_ctx_seq, and n_ctx_slot.

A file named rag-8084.log is not necessarily the current deployment record merely because it exists. The collection found old RAG files on Server01 even though its current process list contained only gpt-oss-120B.²¹

Correlating node, process, launch arguments, log history, and timestamp prevents historical settings from being mistaken for current behavior.

Configuration documents intent. A matched startup record documents runtime interpretation.

17. The Actual GGUF Is Part of the Record#

Relevant fields include general.architecture, general.name, general.file_type, general.quantization_version, and the architecture-specific context field.

For visual models, the associated projector file also matters. The captured Qwen2.5-VL and Gemma deployments include separate projector metadata.³

A generic filename such as mmproj-BF16.gguf is insufficient provenance by itself. The resolved file, extracted metadata, and eventual repository/revision or file hash should accompany a stronger reproducibility record. This baseline does not invent missing download provenance.

18. Public Context Notation#

The preferred public format is explicit:

Qwen2.5-VL-72B-Instruct
GGUF-declared context: 128,000
Requested runtime context: 131,072
Effective slot observed at startup: 128,000

For gpt-oss-120B, the first label changes:

gpt-oss-120B, separate experiment
Runtime-recognized model context (n_ctx_train): 131,072
Requested runtime context: 262,144
Effective slot observed at startup: 131,072

The production request remains 131,072. Evidence labels are as important as the numbers.

19. What n_ctx_train Does Not Prove#

The runtime's recognized value is not a mathematical proof that no context extension can ever work. Nor is it a quality guarantee up to the reported integer.

The captured result is narrower: these services initialized the recorded slots, and the gpt-oss experiment rejected the captured oversized requests.

Different builds or model-specific extension methods require separate tests. No universal min(requested, n_ctx_train) rule is asserted for every deployment.

20. Context Extension Is Experimental Here#

The earlier construction notes mention positional-scaling options as possible experiments. No successful extended-context configuration, quality evaluation, or long-run result was established for this baseline.⁵

The series therefore does not publish an untested RoPE command as a recommended production recipe. Technical acceptance of a longer sequence and useful reasoning over that sequence would remain different tests.

21. A Slot Is Not Raw Document Capacity#

Generation requests may include system instructions, chat templates, role markers, history, tool definitions, retrieved material, and output still to be generated. Multimodal requests also depend on how the deployed model accounts for non-text input.⁶

The budget should be applied to the assembled request, not to raw character count or a document's page count.

For embedding and reranking, batch/micro-batch constraints and endpoint-specific handling add a separate concern. Server03's logs reduced n_batch and n_ubatch to 256 for those services. A reported 8,192-token slot therefore does not establish acceptance of an arbitrary 8,192-token single input; that needs a functional test.¹

22. An Illustrative Workflow Budget#

For a generation service with an effective context of 131,072, an example allocation is:

Effective context                  131,072
System / template allowance          3,000
Output reserve                       4,000
Operational safety margin            5,000
------------------------------------------
Remaining source budget            119,072

This is arithmetic for illustration, not a measurement from the cluster. The real request must be counted using the deployed tokenizer and formatting. The example does not prescribe those allowances for every model or endpoint.

23. Why Retrieval Still Has a Role#

The architecture includes retrieval because a corpus can be larger than any one context window, and because different services deliberately receive different budgets.

The intended pipeline selects candidate evidence, reranks it, and supplies a bounded selection to a generation model. This is a design rationale, not a measured claim that one retrieval pair is universally superior or that RAG always improves quality.

The upcoming Sanapedia workflow is where those choices can be evaluated against actual research tasks.

24. Three Observed Configuration Classes#

Matched: Gemma-4-31B-it declares, requests, and initializes 262,144.

Capped in this build: Qwen2.5-VL-72B requests 131,072 but initializes 128,000; the gpt-oss experiment and Jina deployment provide other examples.

Intentionally constrained: the coder declares 131,072 while the service requests and initializes 16,384.

These describe deployment states, not quality rankings.³¹

25. Reproducibility Checklist#

A context test should retain the exact model artifact, extracted context field, runtime build/commit, full launch arguments, parallel-slot settings, startup interpretation, tokenizer/template, input accounting, output allowance, and acceptance or rejection result.

The current collection is strongest on artifact/runtime/slot comparisons. The gpt-oss experiment also has captured rejections. Near-limit acceptance and quality are not equally demonstrated for every model.

26. What the Captures Establish#

They establish the capped slots for gpt-oss-120B, Qwen2.5-VL-72B, and Jina; the matched Gemma slot; and smaller utility-model allocations. They also establish the two distinct oversized request counts rejected by the gpt-oss server.¹

They do not establish full-window quality, a context-extension success, all endpoint payload limits, or an exact controlled acceptance boundary around the historical 129,722-token prefill.

27. The Operational Rule#

Never document context from the CLI flag alone. Preserve the artifact declaration where available, the runtime-recognized value, the requested allocation, the initialized slot, and real request evidence.

When a layer is missing, label it as missing rather than reconstructing it from the model name.

28. Context as a Resource-Allocation Decision#

This multi-service node is designed around useful role allocations. The question is not only how much context a model declares, but how much a particular service should receive for its job.

That separates deployment policy from model specification and creates a clearer basis for later memory, latency, and quality experiments.

29. Next: Persistent Services and Operations#

Parts 1 through 3 establish the topology, memory observations, and context interpretation. The final article asks how those endpoints are managed, addressed, observed, and eventually composed into a research workflow.

Continue with Part 4: Persistent AI Services and Operations.

Evidence and Reading Notes#