Chapter 2: Microservices to Micromodels

Microservices taught us to decompose applications into independently deployable units aligned to business capabilities. That shift delivered organizational agility: smaller codebases, faster releases, clearer ownership boundaries. Yet as AI became central to user experience and operations, a new form of decomposition emerged. Instead of only splitting by domain logic, teams also split by learned behavior, placing small, task-specific models where decisions are made. This chapter explores the journey from microservices to micromodels: compact, purpose-built models embedded in or adjacent to services, versioned and governed like code, and orchestrated as first-class runtime components. The aim is not to replace foundational models, but to complement them with local intelligence that meets tight latency budgets, reduces cost, and respects privacy constraints while keeping global capabilities available through shared platforms. 
Micromodels change the unit of design.  

Where microservices encapsulated functions such as “checkout” or “profile,” micromodels encapsulate decisions such as “is this session risky,” “which variant to rank first,” or “does this payload look anomalous.” They are smaller than general-purpose LLMs, often distilled or quantified, and they operate within narrow contexts with explicit inputs and outputs. This narrower scope reduces inference latency and model size, enabling placement directly on the hot path without excessive hardware demands. It also clarifies accountability: a team owns not only the service, but the decision policy encoded in the model, with measurable SLOs for accuracy, drift, and responsiveness. Designing with micromodels shifts contracts from purely syntactic to semantic. Traditional APIs specify fields and types; micromodel contracts add the meaning of features, the embedding spaces used, and the acceptable ranges of behavior. A fraud-precheck micromodel, for example, declares its feature schema, calibration targets, and fail-safe behavior when features are missing or stale. Versioning expands beyond image tags to include model weights, tokenizer/prompt variants, and training data lineage. Every decision should be stamp able with a tuple, (model version, data snapshot, prompt, or feature spec), so that rollbacks, audits, and A/B analyses are deterministic rather than anecdotal.  

In practice, this requires a model registry, feature store, and policy-as-code to gate promotions just as CI/CD gates code. Placement becomes a first-order architectural decision. In-process micromodels minimize hops and maximize tail latency performance but couple model updates to the service lifecycle and increase blast radius. Sidecar micromodels preserve locality with separate scaling and release cadence, at the cost of replicating runtime overhead next to each service replica. Remote platform endpoints consolidate heavier models behind network calls, improving GPU utilization and governance, but they introduce variability and tail risk across tenants. Mature systems mix these: small classifiers and gatekeepers stay close to the code path; embedding generators and rankers live in sidecars; retrieval-augmented or foundation capabilities are accessed via platform services; and nightly or streaming re-ranking runs out-of-band. The guiding principle is to place intelligence as close as necessary to minimize end-user latency while keeping expensive or shared capabilities centralized for efficiency and control. Observability evolves to include model-centric signals alongside service health. Latency and error rates remain essential, but now you also track calibration curves, drift metrics, feature freshness, retrieval hit rates, and safety flags. Dashboards report not only p95 latency but also the fraction of requests that used cache, a fallback heuristic, or a smaller model due to budget guardrails.  

Traces propagate model version metadata so an incident can reveal whether a regression correlates with a weight update, a prompt tweak, or a feature schema change. Postmortems feed evaluation datasets that harden future releases. The operational muscle memory SREs built for microservices, canaries, circuit breakers, chaos tests, extends to models: shadow deployments score in parallel before receiving traffic, chaos experiments simulate feature corruption or vector index loss, and rollback playbooks include prompt and feature-store reverts. Cost and sustainability pressures intensify with AI. Micromodels offer a path to rational unit economics by aligning model size to task value. Teams prune, distill, or quantize to run on commodity CPU or small NPUs, reserving GPUs for bursts or shared endpoints. KV caches, embedding caches, and approximate search reduce repeated computation. Autoscaling shifts from reactive thresholds to predictive signals that anticipate load; budgets become SLOs enforced by policy, triggering graceful degradation (smaller model, cached answer, deterministic fallback) when spend or carbon intensity crosses thresholds. FinOps dashboards show cost per thousand decisions, tokens per second, and cache hit rates alongside user outcomes, enabling pragmatic trade-offs that keep experiences fast and affordable at scale. 
Security and privacy acquire semantic depth. Zero-trust boundaries now account for model artifacts, embedding indexes, and tool credentials.  

Micromodels help by keeping sensitive decisions and features local to the service boundary, minimizing cross-network exposure of raw data. Data minimization and redaction occur at ingress, with classifiers labeling sensitivity and routing accordingly. Supply-chain assurance extends to weights and datasets with signatures and attestations; policy-as-code blocks unverified artifacts from deployment. For LLM-integrated flows, guardrails constrain tool use and redact prompts; micromodel gatekeepers can pre-filter inputs to reduce injection and exfiltration risk. Audit trails tie user-visible outcomes to the exact model and data versions involved, converting compliance queries into reproducible evidence. Team topology adapts to technology. Product teams become stewards of decision quality, not just API correctness. Platform teams provide shared rails: model registries, evaluation harnesses, feature stores, vector databases, and inference infrastructure. Collaboration resembles DevOps, but with “ModelOps” disciplines: golden datasets, continuous evaluation, fairness and safety scorecards, and automatic report generation for releases. Documentation shifts from “what the endpoint does” to “what the model promises,” including known limitations and out-of-distribution behaviors. This shared vocabulary reduces friction and supports faster, safer iteration when business requirements or data distributions change. Migration from microservices to micromodels is evolutionary, not disruptive. Start by identifying high-impact decision points where latency is acute or cost is outsized. Replace heuristic rules with small, evaluated models that can run in-process or as a sidecar, and instrument them thoroughly. Establish versioned artifacts and promotion gates before expanding to retrieval or agent patterns.  

Service decomposition with AI boundaries: domain vs. model context 

Service decomposition with AI boundaries asks a simple but consequential question: should we slice systems purely by business capability, or also by the context a model needs to make a good decision? Domain boundaries are stable and aligned to product concepts, orders, billing, identity. Model contexts are shaped by data availability, latency budgets, and evaluation loops, fraud risk, ranking, anomaly scores. When intelligence becomes core to experience and operations, both forces matter. 
Decomposing only by domain can bury decision logic inside services, making models hard to upgrade or compare. Decomposing only by models can fragment ownership and inflate hops. The art is to let domain services own business invariants while extracting “decision seams” where inputs, features, and quality metrics are well-defined.  

1. Defining the seam: domain capability vs. decision context  

A clean seam begins by naming the decision separate from the domain workflow: “should we approve this session” is not the same artifact as “place an order.” Domain services encapsulate business rules, data integrity, and user-facing APIs; decision components encapsulate learned behavior with explicit inputs/outputs and confidence signals. Start by listing high-value, high-frequency judgments, risk prechecks, personalization, routing, content safety, and write their question/answer schemas before choosing a model. Then map constraints: latency budget, allowable staleness, expected error costs, and fallback behavior. Where budgets are tight, prefer micromodels placed in-process or as sidecars with deterministic escape hatches. Where reuse dominates, prefer platform endpoints with strong SLAs. The seam should expose evaluation hooks (labels or proxy metrics), so outcomes feed continuous learning. Finally, draw the blast radius: if the model fails or drifts, what business invariants must the domain still enforce? This clarity prevents accidental coupling and turns decisions into swappable, testable modules. 

2. Contracts and versioning: from syntax to semantics  

Classic service contracts stop at fields and types; AI boundaries extend contracts to semantics: feature definitions, embedding spaces, calibration targets, and safety constraints. Each decision interface should declare required features (with lineage), acceptable null rates, and invariants on ranges or units. Responses must carry versioned metadata, model hash, prompt/tokenizer revision, dataset snapshot ID, and calibration bucket, so incidents and audits can trace outcomes to artifacts. Promotion gates in CI/CD enforce quality thresholds (AUC, ECE, hit-rate), fairness checks, and robustness tests before a version is eligible for traffic. Shadowing and canaries are first-class: score in parallel, compare distributions, and gate cutovers on counterfactual win rates, not just latency. Provide migration playbooks for breaking changes in features or schemas, including dual-read periods and backfills. Document known failure modes and out-of-distribution detectors with recommended fallbacks. This “semantic API” mindset keeps domain services stable while decision components evolve safely, enabling rapid model iteration without churn across the broader microservice graph. 

3. Placement and latency: choosing proximity without coupling  

Where the decision runs is as important as what it decides. In-process micromodels minimize hops and tail latency, ideal for gating, ranking, or safety filters that must answer in single-digit milliseconds. Sidecars maintain locality but decouple lifecycles, letting teams hot-swap weights and adjust runtimes independently; they suit embedding generation, feature extraction, and pre/post-processing. Platform endpoints centralize heavier models and accelerators for efficiency, governance, and reuse at the cost of network variance and multi-tenant tails, mitigate with locality-aware routing, budget SLOs, and caches (KV, embedding, response). Out-of-band pipelines compute recommendations or risk scores ahead of time when freshness budgets allow trading immediacy for cost efficiency. The rule of thumb: place intelligence as close as necessary to meet user SLOs, but as centralized as possible to maximize utilization and control. Always model downgrade ladders, small model → cache → deterministic logic, and encode them as policies so services fail fast and safe when budgets, safety checks, or dependencies degrade. 

4. Data and feature ownership: preventing training–serving skew  

AI boundaries fail without disciplined data ownership. Domain teams own authoritative events and schemas; decision components own feature derivations and freshness SLOs. Use feature stores to ensure offline/online parties and to socialize reusable transformations across teams. Tag features with residency, sensitivity, and retention so routing respects privacy and compliance from ingress to inference. Monitor quality continuously: schema drift, null spikes, population shift, and leakage. When drift triggers, domain services must have defined fallbacks rather than silently accepting degraded judgments. For retrieval and RAG patterns, treat embedding generation and vector indexes as shared assets with explicit rebuild and TTL policies; stamp responses with index versions to debug inconsistencies. Close the loop by collecting labels or proxies (clicks, chargebacks, escalations) and feeding them back to evaluation jobs. Clear data contracts and lineage turn “it worked in training” into “it is reproducible in production,” shrinking the gap between experimentation and dependable decisions at runtime.