AI inference optimization begins with a practical question: what makes this particular workload slow or expensive? A model generating a short answer for one user has different constraints from a service processing long documents for hundreds of concurrent users. Reducing model size can help, while leaving the actual bottleneck almost untouched.
Three techniques receive much of the attention. Quantization reduces the precision used to represent model values. Distillation trains a smaller model to reproduce useful behavior from a larger one. Speculative decoding uses a cheaper proposal mechanism to reduce serial work by the main model. They intervene at different places and require different evidence of success.
The useful objective is a service that meets a quality and latency requirement at an acceptable cost. Tokens per second is one measurement inside that objective. It should not replace the user’s experience, the workload’s completion rate, or the resources consumed by retries and failed answers.
Separate the phases before selecting a technique
Autoregressive text generation commonly has a prefill phase, in which the model processes the input, and a decode phase, in which it generates successive tokens. The relative cost depends on prompt length, output length, architecture, hardware, and serving implementation.
Time to first token includes work before the first generated token reaches the user. Inter-token latency concerns the pace of subsequent output. End-to-end completion time also includes queueing, retrieval, networking, and application logic. A faster decoding kernel does not necessarily shorten a long wait in the request queue.
Model weights occupy memory, but they are not the only consumer. Runtime buffers and the attention key-value cache can be important, especially with long contexts or many concurrent requests. A configuration that fits one short request can exhaust memory under the actual traffic pattern.
Start by recording the shape of the workload: input and output length distributions, concurrency, arrival bursts, task categories, and response deadlines. Preserve an unoptimized baseline with the same conditions so later measurements compare the intended change rather than a different test.
LLM quantization: represent values with fewer bits
Quantization maps numerical values to a smaller set of representable levels. For weights, this can reduce memory traffic and storage. The representation usually includes scales and sometimes other metadata, so the total footprint does not equal the nominal bit width multiplied by the parameter count.
As an illustrative calculation, seven billion values at two bytes each require about fourteen billion bytes before overhead. At four bits each, the raw packed values require about three and a half billion bytes. This arithmetic estimates weight storage only; it is not a promise about application memory or a fourfold speedup.
Weight-only quantization changes how weights are stored and used while other calculations may retain higher precision. Weight-and-activation quantization also reduces activation precision. Different hardware and kernels support these arrangements differently, so the format must match the intended runtime.
A smaller representation can save bandwidth yet add unpacking or conversion work. The net result depends on implementation and workload. Measure the actual backend, device, and batch regime instead of assuming that the smallest format is always the fastest.
Post-training quantization and calibration
Post-training quantization transforms a trained model without retraining it through its original learning process. Some methods use calibration examples to estimate how quantization affects important computations. Calibration should resemble relevant deployment inputs closely enough to reveal sensitive behavior.
GPTQ proposes a post-training approach that approximates weight quantization effects using second-order information. It is one influential method, not a generic name for every low-bit model. Its research results apply to the tested arrangements and should not be read as universal deployment guarantees.
Activation-aware Weight Quantization, or AWQ, uses activation statistics to help protect important weight channels in a low-bit representation. The paper illustrates why the distribution of actual computation matters, rather than only the range of weight values.
The practical lesson is to test calibration coverage. A model calibrated mainly on conversational text may need additional scrutiny on code, numerical data, multilingual input, or long documents. A favorable average score can hide failures in a small but important task family.
What can go wrong after quantization?
Quality loss may appear as a broad decline or as a narrow change in behavior. Structured output can become less reliable, a rare label can become harder to distinguish, or a model can fail a task near its original capability boundary. The test set needs enough resolution to find those changes.
Quantizing the key-value cache is a separate choice from quantizing the weights. It can address memory pressure associated with context and concurrency, while introducing its own numerical and implementation tradeoffs. Record which components are changed in every candidate configuration.
Verify format support, tokenizer compatibility, and model architecture support in the chosen runtime. A quantized checkpoint that loads successfully is only the start. Run representative tasks and measure memory under realistic concurrency before calling the deployment ready.
Knowledge distillation: change the model doing the work
Distillation trains a student model using information from a teacher. The classic work by Hinton, Vinyals and Dean describes transferring knowledge through softened output distributions. Contemporary language-model workflows also use teacher-generated responses or other supervision, depending on what access and training resources are available.
These variants should be distinguished. Training on final answers is not identical to matching the teacher’s probability distribution over tokens. Both may be useful, while they convey different information and impose different requirements.
The student can be smaller or specialized for a narrower task. For example, a routing model may need to choose among a small set of workflows rather than reproduce every capability of the teacher. Narrowing the contract can produce a more practical deployment than trying to compress a general assistant without changing its responsibilities.
Distillation moves work into training and data preparation. The teacher’s generation cost, data review, student training, and evaluation all belong in the economic comparison. A smaller model can be cheap per request while still being the wrong investment for a low-volume application.
Build the student around a task contract
Consider an illustrative document-routing service. Its contract is to select a destination, extract a few fields, and defer ambiguous cases. The teacher can generate candidate labels and explanations, but the training program should verify labels against trustworthy records rather than assume every teacher response is correct.
Include difficult boundaries: documents with missing fields, contradictory labels, unfamiliar layouts, and content that looks authoritative but is irrelevant. A student trained only on easy examples may appear efficient while handing the most costly mistakes to downstream teams.
Keep a separate evaluation set drawn from the intended workflow. Compare correct routing, unnecessary deferral, invalid output, and serious misclassification. If the student is weaker on a high-cost category, the service may need a fallback model or a narrower scope.
The student can inherit the teacher’s errors and preferences. Distillation is a transfer process, not a factual verification procedure. Data provenance, permission to use the teacher’s outputs, and retention requirements also need to fit the organization’s operating arrangements.
Speculative decoding explained without the shortcut myth
Ordinary autoregressive decoding asks the main model to produce the next token repeatedly. Speculative decoding introduces a cheaper draft process that proposes several tokens, then lets the target model evaluate those proposals together. When enough proposed tokens are accepted, the target performs less serial work per emitted token.
The speculative decoding algorithm from Leviathan and colleagues uses an acceptance-and-correction procedure that preserves the target distribution under the method’s assumptions. It is more than asking the target whether a draft looks plausible. The probability handling is central to the exactness claim.
A compatible implementation does not require the draft model to be as capable as the target. It needs to be cheap enough and accurate enough in its proposals for verification to pay off. If the draft frequently disagrees with the target, proposal work can become overhead.
Some systems use different proposal arrangements, including methods tied more closely to the target model. The general engineering question remains the same: how much useful output is accepted for the total cost of proposing, verifying, and scheduling the work?
When speculation helps and when it disappoints
Speculation can help when target decoding is constrained by serial execution and there is room to verify multiple proposed positions efficiently. Its benefit depends on draft cost, acceptance behavior, output characteristics, and serving load. A benchmark on one model pair is not a forecast for every application.
A long technical answer with uncommon tokens may produce a different acceptance pattern from a conventional customer-support reply. Measure by task family. Otherwise, an aggregate improvement can conceal a regression in the very requests that dominate peak demand.
Additional draft-model memory matters too. If the draft displaces useful cache or reduces concurrency, a single-request speedup may coexist with worse service throughput. Evaluate the whole server under realistic traffic, including the resources devoted to the draft process.
Separate exact distribution-preserving methods from heuristics that merely resemble speculation. If a serving option changes the sampling behavior or uses an approximate shortcut, document it as a distinct candidate and verify quality accordingly.
Combine techniques only after understanding them separately
Quantization, distillation, and speculation can be combined, but the effects do not simply multiply. A smaller student may no longer have the same bottleneck as the original teacher. A quantized target may change the cost balance between draft and verification. A memory saving may enable a different concurrency level.
Run controlled comparisons. First establish a baseline, then change one major feature at a time. Once the individual behavior is understood, evaluate combinations under the same workload and quality gates. This makes a surprising regression easier to explain.
Architecture still matters. The companion article on Mixture of Experts architecture explains how sparse activation changes computation without eliminating all parameter storage and communication costs. An optimization suitable for a dense model may behave differently in an expert-based deployment.
Likewise, hardware changes the available options. The discussion of neuromorphic computing and specialized AI hardware provides related context about matching computation to a device. It should not be read as a claim that a conventional quantized transformer automatically runs efficiently on any specialized chip.
A worked decision process for a production service
Imagine a service that summarizes long support cases and produces a structured handoff. Users care about a complete, accurate handoff within a deadline. The engineering team suspects the model is slow, but baseline measurement shows that long prompts and queueing account for much of the wait.
First, remove irrelevant duplicated input through an application change that preserves the evidence required for the task. Evaluate summary completeness to make sure the shorter prompt did not discard critical facts. This may improve the service before any model-level optimization is attempted.
Next, compare a quantized version of the same model. Inspect peak memory, time to first token, completion latency, and structured-output validity under expected concurrency. Pay particular attention to cases containing dates, account identifiers, and conflicting statements.
Then train or evaluate a smaller candidate for the narrow handoff task, using verified examples. If it meets the quality contract, compare its total cost at expected request volume, including training and maintenance. If it fails only on an identifiable difficult class, test a routing policy with explicit fallback.
Finally, evaluate speculation on the candidate that still needs faster decoding. Record acceptance behavior and performance under peak load. This sequence avoids adding serving complexity to a model that the application may no longer need.
Build an optimization scorecard that resists misleading wins
Record quality measures by task, latency distributions, throughput at a defined concurrency, memory, and resource cost. Include invalid responses and timeouts in the denominator. Reporting only successfully completed requests can make an unstable configuration look efficient.
Measure warm and cold behavior separately when both occur in production. State whether model loading, compilation, and cache preparation are included. Use repeated trials or sufficiently long traffic windows to distinguish a consistent change from a favorable run.
Keep prompts, model versions, generation settings, hardware, runtime, and measurement code fixed or explicitly versioned. A different output length can dominate a latency comparison. If a candidate becomes faster because it answers less completely, the quality review must expose that tradeoff.
For energy or cost claims, specify the boundary. Device power is not the same as total facility energy. A quoted accelerator price is not the same as the service’s fully allocated operating cost. Useful estimates are transparent about what they include.
Calculate cost per accepted result, not per generated token
Suppose a hypothetical summarization service generates a handoff, checks its required fields, and asks for a retry when the output is invalid. A candidate model uses fewer resources per generation but fails validation more often. Its apparent saving may disappear after repeated generations, additional checks, and human repair are included.
Define an accepted result in terms of the task contract. For the handoff service, that could mean a complete account of the issue, supported dates, correct routing, and valid structure. Count all attempts required to obtain that result. This makes the comparison sensitive to useful work rather than raw output volume.
Apply the same reasoning to a fallback arrangement. A small model may handle most requests cheaply while escalating difficult cases to a larger model. The route is valuable only if the system can identify those difficult cases with adequate reliability. A routing error that silently accepts a bad answer has a different cost from a deliberate escalation.
Evaluate the router as a component. Measure which cases it sends to the large model, which failures it misses, and whether its own latency erases part of the benefit. Include the possibility that the incoming workload shifts toward harder cases. A cost estimate based on an easy historical mix may not hold during a new product launch.
Make the benchmark resemble the service’s queue
A single request running alone can reveal kernel behavior, but a production service shares resources. Use an arrival pattern that resembles the expected workload, including bursts and variation in sequence lengths. Record the offered load and the proportion of requests that finish within the required deadline.
Compare configurations at more than one operating point. A candidate may win at low concurrency and lose near saturation. Another may maintain throughput while producing an unacceptable tail of slow requests. These differences matter more than one favorable throughput number.
Include waiting time in user-facing latency. If measurement begins only when a worker starts generation, the benchmark can conceal overload in the admission queue. Also record rejected or cancelled requests. A system that looks fast because it discards difficult work needs that behavior reflected in the scorecard.
Finally, keep output quality checks active during load tests. Capacity pressure can expose timeouts, truncation, or fallback paths that were absent in a quiet quality evaluation. The configuration should meet its quality contract under the conditions for which the team is claiming performance.
For a rollback, preserve the previous model artifact, serving configuration, and evaluation record together. Reverting only the weights may leave an incompatible quantization format, tokenizer, or routing policy in place. Practice the complete configuration change in a controlled environment so the team knows how long recovery takes and which requests can be affected during the transition.
Operational safeguards belong in the optimization
Plan for fallback, rollback, and capacity limits. A model that barely fits memory under a synthetic workload can fail during a burst of longer requests. Admission control and explicit context limits make the service more predictable than hoping the average request stays small.
Monitor quality after deployment, particularly when inputs change. A new language, document template, or product policy can invalidate the assumptions used in calibration or distillation. Store enough version information to associate failures with the configuration that produced them.
Prefer the simplest configuration that meets the actual contract. Additional runtimes, draft models, routing rules, and conversion formats create maintenance obligations. An optimization earns its place when its sustained benefit exceeds both its runtime cost and its operational complexity.
Fast and cheap inference is a system property. It comes from matching the task, model, numerical representation, hardware, and serving policy, then verifying the result against user outcomes. Quantization, distillation, and speculative decoding are powerful tools when that chain remains visible.