Architecture primer · 01 September 2026 snapshot

On-prem clinical LLMs: size the service, not the model.

A reported 128 GB / $5,000 / 15–20 ICU beds system can be plausible as a compact unified-memory appliance. It is not, by itself, evidence of clinical capacity. The decisive variables are the exact model and quantization, p95 context and output length, KV-cache headroom, request mix, retrieval quality, validation, and safe failure behavior.

Current as of 2026-09-01Claims, prices, availability decay quicklyNot legal, regulatory, or clinical advice

The 128 GB reality check

Loading a model is not the same as serving a workload.

A 128 GB box may fit quantized weights. Runtime reserve, KV cache, OS memory, batches, and safety margin determine whether it can respond reliably under clinical use.

Weightsmeasured model file

KV cachep95 tokens × sequences

Runtimekernels · graphs · reserve

Marginfragmentation · safety

Capacity statement to challenge: “It fits in 128 GB.” Question to ask: “At our p95 prompt/output lengths, how many simultaneous sequences fit with measured latency and headroom?”

Start smaller, compare honestly

Begin with a 20B–32B dense or 20B–120B MoE candidate, then compare it against a larger quality reference on a frozen local clinical suite.

Context is a budget

Long prompts cost prefill time; their KV state persists. Bed count is not concurrency. Measure tokens and arrival patterns.

The model is one layer

A credible assistant also needs controlled RAG, structured validation, access control, auditability, and clinician review.

Control of weights is not clinical readiness.

“Open-weight” is the operationally useful term: a hospital can run parameters in its own environment. It does not establish open-source provenance, clinical validation, unrestricted use, support, or security.

Term	What is available	Deployment implication
Open source	Source code under an OSI-style license; ideally reproducible training code and data details for LLMs.	Software can be inspected and modified, but weights and training data may still be unavailable.
Open weights	Downloadable trained parameters; corpus, full training code, and reproducibility may be absent.	Enables on-prem inference or fine-tuning. License, use policy, redistribution, provenance, and support obligations remain.
Source-available / custom license	Weights and some code public under bespoke terms or an acceptable-use policy.	Legal review matters, especially for hospital use, redistribution, and geographic restrictions.
Closed model / API	Neither weights nor most internals are available.	The provider operates it; data residency and BAA or contract terms become central.

License review target: counsel should approve the exact checkpoint license and revision hash, not merely a family name. OpenAI gpt-oss is open-weight under Apache 2.0 with a usage policy; Qwen3’s cited 235B checkpoint is Apache 2.0; Llama 4 uses Meta’s custom community license and has an EU restriction in its multimodal license grant; Gemma has its own terms and prohibited-use policy.

A practical current shortlist

Family / example	Architecture and size	Why it matters	128 GB-class fit
gpt-oss-20b	MoE; 21B total / 3.6B active per token; 128K context	Tool use, structured outputs, reasoning workflow experiments; native MXFP4 weights stated at about 16 GB.	Comfortable; KV and concurrency headroom.
gpt-oss-120b	MoE; 117B total / 5.1B active; 128K context	Higher-capability reasoning and tool use; official MXFP4 model stated at about 80 GB.	Possible; remaining memory must cover everything else.
Qwen3 dense	0.6B–32B dense range	Thinking/non-thinking modes, multilingual, agent/tool workflows, local deployment.	14B–32B are strong operational baselines.
Qwen3-30B-A3B / 235B-A22B	MoE; largest: 235B total / 22B active; 94 layers, 128 experts / 8 active, GQA; 32K native / 131K YaRN	Strong agent and reasoning family.	235B requires aggressive quantization with little practical headroom.
Gemma 3 4B / 12B / 27B	Dense multimodal; 128K context	Compact text/image, document understanding, summarization, multilingual use.	12B / 27B practical; validate imaging separately.
Llama 3.3 70B	Dense; 128K context	Mature ecosystem; useful quality reference.	BF16 cannot fit one 128 GB box; 4-bit leaves much less headroom.
Llama 4 Scout	MoE; 109B total / 17B active; multimodal; 10M advertised context	Capable multimodal model with very long advertised context.	Loading is not serving; long context is KV-limited. Review license and eligibility.
DeepSeek-V3	MoE; 671B total / 37B active; 128K context	Frontier open-weight reference; MLA/MoE and multi-token prediction.	Not a realistic single 128 GB target.

Model-selection posture

Production candidate: Qwen3 14B/32B, Gemma 3 12B/27B, or gpt-oss-20b.
Quality reference: gpt-oss-120b or a 70B-class model where hardware allows.
Compare both on the same frozen clinical evaluation suite.

Decision rule

If the smaller model has comparable clinician-review acceptance and materially better latency and concurrency, it is normally the better system. Public benchmarks do not substitute for ICU notes, local abbreviations, language mix, EHR exports, and known failure cases.

Memory has two jobs: weights and live conversations.

Parameter count drives weight storage, but it is not all of runtime memory. In MoE models, active parameters influence compute; all experts’ weights usually still need to be resident.

weight memory ≈ P × b / 8 × (1 + overhead)

P = parameters; b = bits per stored weight. Overhead includes scales, zero points, metadata, unquantized tensors, runtime layout, and fragmentation.

Model	BF16 / FP16	8-bit	4-bit
8B dense	16 GB	8 GB	4 GB
20B dense	40 GB	20 GB	10 GB
32B dense	64 GB	32 GB	16 GB
70B dense	140 GB	70 GB	35 GB
117B MoE	234 GB	117 GB	58.5 GB
235B MoE	470 GB	235 GB	117.5 GB

△

These are lower bounds, not provisioning figures. Add runtime work buffers, kernels, KV cache, framework reservations, OS memory, and fragmentation. Use measured model-file size plus loaded-server measurement. Highest precision that meets the latency/cost envelope is a starting point; task-specific validation must prove the quantized model preserves accuracy, calibration, JSON reliability, multilingual behavior, and rare-term handling.

The request has two phases

Context phase

Prefill

Processes the prompt and creates KV cache. Long documents increase time to first token; attention work grows sharply with prompt length. Queueing, retrieval, and prompt construction also affect TTFT.

Generation phase

Decode

Generates one or a few tokens from cached history. Often memory-bandwidth limited; determines inter-token latency and streaming smoothness.

Latency vocabulary

TTFT: request receipt to first streamed token.
ITL: time between generated tokens; how responsive streaming feels.
Throughput: aggregate tokens/sec; it can rise while individual TTFT worsens.
Concurrency: admitted requests, not hospital beds.

SLOs to require

Set p95 TTFT, p95 total response time, p95 inter-token latency, availability, error rate, and queue duration. Test those against a realistic mixed clinical trace, not a favorable benchmark prompt.

KV cache is the hidden capacity constraint

Cached history, not a footnote.

Prior attention keys and values keep the model from recomputing the whole history for every next token. Context × simultaneous sequences is a durable memory claim.

allocated blocksactive requestfree / shareable capacity

PagedAttention allocates KV memory in blocks/pages on demand, reducing fragmentation and allowing block sharing. The original vLLM paper reported 2–4× throughput versus evaluated baselines at comparable latency; that is a historical paper result, not a guarantee. Continuous (in-flight) batching lets new requests join while older ones decode; prefix caching reuses identical prefixes and speeds prefill, not novel generation; chunked prefill prevents huge prompts from monopolizing decode traffic; speculative decoding is worthwhile only when acceptance rates and kernels prove favorable.

KV bytes ≈ 2 × L × T × Hkv × Dh × Bkv

For standard Transformer GQA: L = layers; T = total cached tokens over concurrent sequences; Hkv = KV heads; Dh = head dimension; Bkv = bytes per KV value (2 for BF16/FP16, 1 for FP8). The 2 covers keys and values. MLA, sliding-window, and hybrid attention require a model-specific formula.

↗

Worked example: an 80-layer GQA model with 8 KV heads, 128-dimensional heads, BF16/FP16 KV, and 8,192 tokens: 2 × 80 × 8,192 × 8 × 128 × 2 = 2.68 GB per active sequence. Eight simultaneous 8K-token conversations consume about 21.5 GB before allocator/runtime overhead. Doubling context or concurrency roughly doubles KV memory.

Sizing worksheet

Input	Fill in	Why it matters
Model and exact revision		Config, license, tokenizer, regression baseline
Weight format and measured file size		Weight residency
Accelerator / usable memory		Not nominal system RAM alone
Framework memory reservation		Runtime, graphs, kernels, fragmentation
Max prompt / output tokens		Prefill/decode time and total KV
Mean / p95 input and output tokens		Real capacity, not theoretical maximum
Concurrent active sequences		KV-cache demand
Required p95 TTFT / ITL		Scheduler configuration
Shared-prefix hit rate		Value of prefix caching
Retrieval chunk count and token budget		Keeps RAG context bounded
Failure behavior		Timeout, fallback, queue, clinician notification

usable accelerator memory > weights + KV_p95 + runtime reserve + safety margin

Provision with a measured peak under a realistic trace, not this formula alone.

Serve with a stack you can measure and operate.

For a low-concurrency hospital unit, one well-sized GPU or coherent-memory appliance is simpler than multi-GPU tensor parallelism. Add replicas for resilience before complexity unless a model simply cannot fit.

Stack	Best fit	Watch-outs
vLLM	General-purpose high-throughput serving; OpenAI-compatible API; PagedAttention, continuous batching, prefix cache, speculative decoding, distributed execution.	Fast-moving releases: pin versions and benchmark exact model / quantization.
SGLang	Complex generation programs, structured agents, prefix-heavy workloads; RadixAttention KV reuse.	Validate checkpoint support and operational maturity.
TensorRT-LLM + Triton	NVIDIA-specific performance engineering where optimization effort is justified.	Engine builds, CUDA coupling, deployment complexity; strongest when hardware is fixed.
llama.cpp	Small/simple local service; CPU, Apple Silicon, AMD/HIP, Vulkan, hybrid CPU+GPU, GGUF.	Usually not first choice for high-concurrency GPU serving; excellent for prototypes and UMA devices.
Transformers	Reference implementation, testing, correctness baseline.	Usually not operational high-throughput server.
TGI / LMDeploy / Ollama / LM Studio	Useful alternatives by hardware and integration needs.	Test observability, concurrency control, auth, and exact model support—not convenience alone.

Use two stacks in evaluation

Keep one reference stack (for example, Transformers) for repeatable regression tests and one serving stack (vLLM or TensorRT-LLM) for load tests.

Parallelism tradeoffs

Tensor parallelism splits layer matrices but adds frequent inter-GPU communication. Pipeline parallelism splits layers but causes bubbles and latency. Data parallelism is full replicas for throughput/availability. Expert parallelism distributes MoE experts with routing and network complexity.

Hardware capacity is not bandwidth or resilience.

VRAM is dedicated accelerator memory. System RAM overflow is generally slow. Unified memory shares a physical pool between CPU and GPU: it can hold larger models locally, but its capacity is shared and bandwidth can be far below datacenter HBM.

Hardware	Memory	Bandwidth / power	Architectural meaning
NVIDIA DGX Spark / GB10	128 GB LPDDR5x unified	273 GB/s; GB10 SoC TDP 140 W; 240 W external PSU	Plausible reading of a reported 128 GB appliance. Do not equate its bandwidth with HBM GPUs.
RTX 5090	32 GB GDDR7	575 W board power	Powerful consumer card; 32 GB constrains capacity and long-context concurrency. Multiple cards add chassis/power/topology complexity.
RTX PRO 6000 Blackwell	96 GB GDDR7 ECC	1,792 GB/s; up to 600 W	Strong single-GPU workstation option, closer to a production single-GPU footprint.
NVIDIA L40S	48 GB GDDR6 ECC	864 GB/s; 350 W	Mature PCIe option; two cards add memory but are not seamless without supported TP.
NVIDIA H200	141 GB HBM3e	4.8 TB/s; up to 700 W	One accelerator with enough high-bandwidth memory for large work; costly server procurement.
AMD MI300X	192 GB HBM3	5.3 TB/s; up to 750 W	High-capacity datacenter alternative with ECC/RAS; validate ROCm compatibility.
Ryzen AI Max+ 395	Up to 128 GB unified LPDDR5x	256 GB/s (AMD developer platform); 120 W	Compact lower-power alternative; up to 96 GB graphics memory. Better for experiments/smaller workloads than assumed datacenter serving.
Apple Silicon Mac Studio	Up to 512 GB unified	Up to 1.2 TB/s for M5 Ultra	Large local-memory alternative with Metal/llama.cpp-style stacks; different ecosystem from CUDA servers.

The $5,000 claim: plausible for DGX Spark-class hardware. NVIDIA’s developer forum said Founders Edition MSRP moved from $3,999 to $4,699 in February 2026 due to memory supply constraints. Street price, tax, storage, service, support, and availability vary. Official GB10 documentation calls it 128 GB unified system memory, not discrete HBM VRAM; “models up to 200B” is a capacity claim, not a latency/concurrency claim.

One box or replicas?

A single box offers locality and simplicity but is a single point of failure. For 15–20 beds, two independent smaller replicas may give a better resilience story than one giant multi-GPU host. Decide whether the workload requires throughput, availability, or a larger model.

Facilities & operations

Rack/deskside location, temperature, airflow, acoustic level, power circuit, UPS.
ECC, GPU error monitoring, page retirement, driver/firmware maintenance.
Encrypted NVMe, secure boot, TPM, hardening, network segmentation.
Artifact checksums, immutable release registry, GPU/KV/queue/TTFT/ITL/OOM metrics.
Cold start, restart runbook, failover, and named patch/support ownership.

The LLM sits inside a controlled clinical system.

RAG gives the model evidence; it does not make it truthful. Constrained output can guarantee syntax; it cannot make a claim clinically true. Design every layer so uncertainty can surface safely.

InputsEHR, notes, policy corpusAccess control and minimum-necessary data selection.

EvidenceRetrieval pipelineEmbed query → retrieve approved chunks → rerank/filter → attach provenance.

GenerationLLM orchestrationTask prompt + patient context → bounded generation → constrained schema or tool call.

Proof / safetyDeterministic validationJSON/schema, citations, clinical rules/ranges, abstain/escalate policy.

AccountabilityClinician review + audit + feedbackHuman action remains visible; evaluation and feedback close the loop.

RAG rules

Use versioned, approved corpus: local policy, pathways, formulary, curated guidelines—not uncontrolled web content.
Keep coherent chunks, document/version/section metadata, and a token cap.
Use a separate embedding model for recall and reranker for precision.
Show source title, version/date, section, retrieval score, and exact supporting text/link.
Evaluate recall@k, evidence relevance, staleness, wrong-patient/document failures; defend against retrieval prompt injection.

Structured extraction & tools

Define types, required fields, code sets, dates, units, and unknown/not-documented states.
Request JSON/tool call; constrain decoding where supported.
Validate deterministically; reject, repair, or route uncertainty to review.
Never silently coerce an invalid clinical fact into a valid-looking record.

Tool layer: narrow allow-list, patient/context binding, RBAC, parameter validation, least privilege, idempotency/confirmation for writes, timeout/retry, audit, and a strict line between recommendation and order.

Minimum safety posture

Human review

No autonomous diagnoses, orders, medication changes, or high-risk recommendations.

Grounding

Show sources and uncertainty; make “insufficient evidence” a valid answer.

Evaluation

De-identified historical cases, clinician adjudication, subgroups/languages, hard cases, abstention.

Change control

Freeze versions and revalidate model, prompt, index, embeddings, reranker, driver, and serving stack.

Start in shadow mode alongside normal workflow; audit disagreements and unsafe suggestions. Audit only what is necessary for safety: authorized identity, permitted patient/context IDs, checkpoint hash, prompt template, retrieval sources/versions, tool requests/results, response, validator result, clinician action, and timestamps. Avoid chain-of-thought logging by default: it can expand sensitive-data exposure without clinical value.

Pass the gates in order.

A fast model is not a deployment. Do not use price, parameter count, or tokens/sec as a shortcut around clinical quality and operational ownership.

Gate	Pass condition
Intended use	Clear non-autonomous scope, owner, user, and clinician accountability.
Data / security	On-prem data flow, access control, segmentation, logging, retention, and incident response approved.
License	Exact weights/revision and dependencies approved for hospital use.
Clinical quality	Independent task-specific evaluation meets predeclared acceptance thresholds.
Safety	Grounding, abstention, validation, human review, escalation pass adversarial tests.
Performance	p95 TTFT/ITL, queue time, concurrency, and failure behavior meet workflow SLOs.
Reliability	Maintenance, rollback, monitoring, capacity, and failover are owned and rehearsed.
Economics	Includes hardware, support, power/cooling, storage, staff, validation, and refresh—not GPU price alone.

Benchmark discipline: vendor numbers use favorable prompts, batches, precisions, hardware, and output lengths. Tokens/sec is incomplete without prompt length, output length, concurrency, quantization, TTFT, ITL, and percentile latency. Use dated reseller quotes with service/support terms.

Questions that turn a sales call into an architecture review.

Ask for traces and measurements on the exact release—not an assurance that it works in principle.

Workload, model, and memory

01What exact tasks are served: summarization, extraction, Q&A, draft note, triage, coding, or recommendations?
02What are p50/p95 prompt and output tokens for each task?
03What are peak active requests, arrival rates, and duty cycle—not just beds?
04What p95 TTFT, p95 ITL, total-time, and availability SLOs are required?
05What happens when overloaded, uncertain, or unavailable?
06Which exact checkpoint, revision hash, chat template, tokenizer, precision, and quantization are in use?
07How many total versus active MoE parameters does it have?
08What model-specific evaluation used representative local clinical material?
09What acceptance threshold, adjudication process, and failure taxonomy apply?
10How does the quantized build differ from BF16 on extraction, JSON validity, rare terms, and hallucinations?
11Does it emit internal reasoning; what is retained, displayed, or redacted?
12Show weights, KV cache, runtime, OS, and safety margin as a memory budget.
13At p95 lengths, how many sequences fit before eviction, preemption, or OOM?
14What p50/p95 TTFT and ITL occur under the expected mixed workload?
15Is prefix caching enabled, and what is its measured hit rate?
16What are max model length, max sequences, max batched tokens, and output-token limits?
17What happens to 8K, 32K, and 128K input?

Serving architecture

18Which stack/version is used and why: vLLM, SGLang, TensorRT-LLM, llama.cpp, or another?
19Is it single-GPU, tensor parallel, pipeline parallel, or independent replicas?
20What is the GPU/CPU interconnect topology and observed collective-communication bottleneck?
21Are prefill and decode colocated? If not, what are the network and failover designs?
22How long do cold start and model reload take?

RAG, tools, safety, and operations

23Which documents are retrievable, who approves them, and how are version and retention handled?
24What embedding/reranking models and retrieval metrics have passed?
25How is provenance shown to the clinician?
26Are outputs schema-constrained and independently validated?
27Which tools may the model call; are they read-only, allow-listed, patient-bound, authenticated, and audited?
28How are prompt injection and untrusted retrieved content handled?
29What is the abstention policy, and how is it measured?
30Which logs are retained, where, for how long, and who can access them?
31What monitoring covers unsafe output, retrieval drift, latency, OOM, faults, and feedback?
32What triggers rollback, and can a prior model/index/prompt return in minutes?
33Which updates require revalidation: weights, quantization, prompts, corpus, embeddings, server, driver, firmware?
34What security protects ePHI at rest, in use, and in transit?
35Who owns patching, escalation, incident response, and clinical-governance approval?

A focused 60-minute learning path.

Use this sequence before a procurement or architecture review; it makes the memory and safety questions concrete before comparing vendor claims.

Time	Focus	Outcome
0–10 min	Executive takeaways and open weights	Separate local control from “open source” and clinical validation.
10–20 min	Prefill, decode, KV cache, TTFT, ITL	Understand why context/concurrency—not beds—determine capacity.
20–30 min	Memory formulas and worksheet	Challenge “it fits in 128 GB” claims.
30–40 min	gpt-oss-20b, gpt-oss-120b, Qwen3 32B, Gemma 3 27B, 70B reference	Form a shortlist without treating leaderboards as clinical proof.
40–50 min	vLLM / SGLang / TensorRT-LLM roles and hardware	Ask why a stack and topology were selected.
50–60 min	Engineer questions and clinical controls	Turn the next meeting into a structured architecture review.

A compact working glossary.

Active parameters: MoE parameters used for a token; total weights still matter for memory.
Batch: Requests/tokens processed together to improve accelerator utilization.
Context window: Maximum prompt plus generated tokens available to the model.
Decode: Next-token generation phase.
Embedding: Vector representation used to retrieve semantically related text.
GQA / MQA: Grouped/multi-query attention; fewer KV heads and a smaller KV cache.
KV cache: Stored attention keys/values for prior tokens.
MoE: Mixture of experts; selected expert networks run for each token.
Paged attention: Block-based KV management that reduces fragmentation and enables sharing.
Prefill: Processing prompt tokens and constructing KV cache.
Quantization: Lower-bit representation of weights/KV state to reduce memory and often improve speed.
RAG: Retrieval-augmented generation: controlled evidence before generation.
Reranker: Second model that reorders retrieved candidates for precision.
Structured output: Machine-readable output constrained to a schema.
TTFT / ITL: Time to first token / inter-token latency.

Primary sources and technical references.

Model availability, licensing, benchmarks, hardware pricing, and serving support change quickly. Re-check these before procurement or production design.

OpenAI gpt-oss model cardmodel architecture / footprint Introducing gpt-ossarchitecture / footprint OpenAI open-weight licensing FAQlicensing Qwen3-235B-A22B model cardmodel architecture / license Gemma overview and termsopen-model policy Gemma 3 model cardmodel details Llama 4 model cardmodel details Llama 4 Community Licenselicense terms Llama 4 use policypermitted use DeepSeek-V3 model card / technical reportmodel details PagedAttention papervLLM system research vLLM documentationserving features TensorRT-LLM scheduling docsin-flight batching NVIDIA TensorRT-LLM documentationserving / quantization llama.cpp repositoryGGUF / backend support DGX Spark hardware guideGB10 memory / power RTX 5090 specificationsconsumer GPU RTX PRO 6000 specificationsworkstation GPU NVIDIA L40S specificationsdatacenter GPU NVIDIA H200 specificationsHBM GPU AMD MI300X specificationsaccelerator AMD Ryzen AI Max+ 395unified-memory platform Apple M5 Ultra announcementMac Studio bandwidth NVIDIA DGX Spark price announcementdated MSRP note EmbeddingGemma model cardretrieval model HHS HIPAA Security RuleePHI safeguards NIST AI 600-1GenAI risk management ONC decision support requirementstransparency / source attributes FDA digital health guidance indexevolving guidance landscape