An AI service is the application-owned layer that turns a product request into a controlled model operation and returns a result the rest of the system can understand. Its purpose is not to hide a model call behind a thin function. Its purpose is to make prompts, retrieval, provider choice, output validation, safety policy, cost limits, and diagnostics explicit enough that the application can evolve without every feature depending on one vendor-specific API call.
In this course’s AI Application Patterns section, this lesson sits between model capability and product architecture. The pattern matters because generative AI behavior is shaped by many inputs: instructions, retrieved evidence, model parameters, tool definitions, policies, and response schemas. A well-designed AI service names those parts, versions them, and exposes a stable application interface such as draft_reply, classify_ticket, or extract_invoice_fields. Product code should ask for a business operation; the AI service should decide how to assemble and verify the model interaction.
Purpose and Outcome
Plainly, an AI service gives the rest of your application a dependable place to ask for AI-assisted work. After this lesson, you should be able to design that service with clear boundaries: a request schema at the edge, a prompt registry, optional retrieval, a policy gate, a provider adapter, an output parser, telemetry events, and failure categories that callers can act on. The outcome is a service that can swap models, roll back prompts, reject unsafe requests, and diagnose bad answers without rewriting every product workflow.
Internal Mechanism
A typical request enters through an application-facing method, not a raw chat-completion call. The service first normalizes user and domain input into a typed request. Normalization may trim text, cap document count, attach the caller’s tenant or role, and select a task name. Next, the service resolves a prompt version. A prompt version is a named artifact such as support_reply:v3 that contains system instructions, message templates, allowed tools, model settings, and the output contract expected by downstream code.
If the task needs private or changing knowledge, the service performs retrieval before generation. Retrieval should be scoped by identity, tenant, data source, freshness, and document type. The AI service should pass the model only the evidence that the caller is allowed to use, usually with source identifiers so the answer can cite or trace its claims. Retrieval is part of the service design rather than a convenience function because it affects correctness, privacy, latency, and cost.
The policy gate then checks the assembled request. Some checks happen before the model call, such as rejecting missing authorization, oversized input, or a request to produce disallowed content. Other checks happen after generation, such as validating that an output schema was followed or that required citations refer to supplied evidence. These checks are different from vague trust language: they are concrete functions that decide whether the service may call the model, return the result, retry with repair instructions, or return a typed refusal.
The provider adapter is the narrow layer that knows how to call a specific model provider. It translates your internal request into provider parameters, handles provider error codes, records token and latency metadata, and converts provider responses into your internal AIResult. Keeping this adapter narrow prevents application logic from depending on provider-specific message formats, finish reasons, retry headers, or streaming event shapes.
API Anatomy
A useful AI service API normally has four visible parts. The input contract describes what the caller can ask for. The task configuration names the prompt version, model family, output schema, temperature or sampling policy, token budget, retrieval policy, and retry policy. The result contract returns either validated data or a structured failure. The telemetry contract records enough detail to explain behavior without storing secrets or unnecessary personal data.
Configuration should be versioned separately from code when teams need fast prompt rollback, but the application should still treat configuration as a deployable artifact: reviewed, tested, and tied to evaluation results. A prompt edit is a behavior change. A model switch is a behavior change. A retrieval filter change is a behavior change. The service design should make those changes visible.
Example 1: Request Normalization
The first example creates an application-facing request for a support reply. The caller supplies product data, but the service decides the task name, prompt version, and length budget. The expected behavior is deterministic: blank messages fail, valid messages become immutable request objects.
from dataclasses import dataclass
@dataclass(frozen=True)
class SupportReplyRequest:
tenant_id: str
user_message: str
priority: str
prompt_version: str = "support_reply:v1"
max_output_tokens: int = 220
def build_support_reply_request(tenant_id: str, user_message: str, priority: str) -> SupportReplyRequest:
message = user_message.strip()
if not tenant_id.strip():
raise ValueError("tenant_id is required")
if not message:
raise ValueError("user_message is required")
if priority not in {"normal", "urgent"}:
raise ValueError("priority must be normal or urgent")
return SupportReplyRequest(tenant_id=tenant_id, user_message=message, priority=priority)
request = build_support_reply_request("acme", " My invoice is wrong. ", "normal")
print(request)The service has already made several design choices. It does not let the caller choose arbitrary prompt versions. It limits the output budget before reaching the provider. It validates priority as a product concept, not as model text. If this function raises, the caller can return a normal validation error without spending model tokens.
Example 2: Provider Adapter
The second example shows why an adapter is useful. The fake provider stands in for a network model call, but the service still converts a provider-shaped response into an application-shaped result. Expected output is deterministic because the provider is stubbed.
from dataclasses import dataclass
from typing import Protocol
@dataclass(frozen=True)
class AIResult:
text: str
provider: str
prompt_version: str
input_tokens: int
output_tokens: int
class TextProvider(Protocol):
def complete(self, messages: list[dict[str, str]], max_output_tokens: int) -> dict[str, object]:
...
class FakeProvider:
def complete(self, messages: list[dict[str, str]], max_output_tokens: int) -> dict[str, object]:
return {"text": "Please send the invoice number so I can check the charge.", "usage": {"input": 31, "output": 13}}
def generate_support_reply(provider: TextProvider, request: SupportReplyRequest) -> AIResult:
messages = [
{"role": "system", "content": "Write concise, factual customer support replies."},
{"role": "user", "content": request.user_message},
]
raw = provider.complete(messages, request.max_output_tokens)
usage = raw["usage"]
return AIResult(
text=str(raw["text"]),
provider=provider.__class__.__name__,
prompt_version=request.prompt_version,
input_tokens=int(usage["input"]),
output_tokens=int(usage["output"]),
)
result = generate_support_reply(FakeProvider(), request)
print(result.text)
print(result.provider)The adapter boundary lets tests verify product behavior without a live provider. In production, a real adapter would also map timeout errors, rate limits, content-filter stops, and malformed responses into service-level error categories such as dependency_timeout, throttled, or invalid_provider_response.
Example 3: Retrieval, Policy, and Schema
The third example adds a simple retrieval step and a schema check. The service requires the generated answer to mention one retrieved source identifier. This is not a complete citation system, but it demonstrates the mechanism: retrieve allowed evidence, build the prompt from that evidence, then validate that the answer obeys the contract.
from dataclasses import dataclass
@dataclass(frozen=True)
class Evidence:
source_id: str
text: str
def retrieve_allowed_evidence(tenant_id: str) -> list[Evidence]:
catalog = {
"acme": [Evidence("kb-17", "Invoice corrections require the invoice number and billing email.")]
}
return catalog.get(tenant_id, [])
def validate_cited_answer(answer: str, evidence: list[Evidence]) -> bool:
return any(item.source_id in answer for item in evidence)
def draft_grounded_reply(request: SupportReplyRequest) -> str:
evidence = retrieve_allowed_evidence(request.tenant_id)
if not evidence:
return "POLICY_REFUSAL: no permitted evidence"
answer = "Please send the invoice number and billing email so I can review this. Source: kb-17"
if not validate_cited_answer(answer, evidence):
return "INVALID_OUTPUT: missing source citation"
return answer
print(draft_grounded_reply(request))The expected result is a grounded support reply that includes kb-17. If the tenant has no permitted evidence, the service returns a policy refusal instead of inventing a billing rule. If the generated answer omits the source, the service treats the output as invalid and can retry, repair, or fail closed depending on the product requirement.
Design Choices and Trade-offs
The first major choice is how wide the AI service should be. A single generic generate method is easy to start with, but it pushes task-specific policy and schemas into callers. Separate task methods are more verbose, but they let each operation define its own prompt version, retrieval policy, and output contract. For product systems, task-specific methods usually age better.
The second choice is where to store prompts and model settings. Keeping them in code makes review and testing straightforward. Keeping them in a registry supports rollback and experimentation. The trade-off is control: registry-driven prompts need permissions, audit history, validation, and evaluation gates so a text edit cannot silently change production behavior.
The third choice is how strict output validation should be. Free-form text is useful for drafts and summaries. Structured JSON is better when downstream code will route tickets, update records, or trigger actions. Strict schemas reduce ambiguity but increase the chance of repair loops or user-visible failures. The safest design is to match strictness to impact: higher-impact workflows need tighter schemas and more conservative failure handling.
Failure Modes and Troubleshooting
Symptom: replies mention policies from another customer. Cause: retrieval was filtered by topic but not by tenant or caller permission. Diagnostic steps: inspect the retrieval query, logged source identifiers, tenant id, and authorization context for the request. Reproduce with two tenants that share similar vocabulary. Correction: enforce tenant and permission filters inside retrieval, add a test that cross-tenant documents are excluded, and fail closed when the authorization context is missing.
Symptom: a deployment suddenly returns malformed JSON. Cause: a prompt version or model setting changed without an output-schema evaluation. Diagnostic steps: compare prompt version, model parameters, and validation error rates before and after the deployment. Sample invalid responses and classify whether they are missing fields, wrong types, or extra prose. Correction: roll back the prompt version, add schema-specific examples, and require evaluation pass rates before promotion.
Symptom: latency spikes during peak traffic. Cause: the service performs retrieval, long prompts, and retries synchronously for every request. Diagnostic steps: break latency into retrieval time, provider time, validation time, and retry count. Check token counts and cache hit rate. Correction: cap retrieved passages, cache stable evidence, stream low-risk drafts, shorten prompt templates, and use bounded retries with a clear degraded response.
Security, Performance, and Reliability
Security depends on concrete enforcement points. The AI service should never rely on the model to decide which private records a caller may see. Authorization must happen before evidence is placed in context. Prompt templates should treat user and document text as untrusted input; retrieved documents can contain instructions that conflict with system policy. The service should delimit evidence, state how evidence may be used, and validate outputs before performing actions.
Performance is mostly controlled by token volume, retrieval fan-out, provider latency, retries, and streaming strategy. A shorter prompt is not automatically better if it removes necessary constraints, but repeated boilerplate and irrelevant evidence are real costs. Reliability comes from typed failures, idempotent retries for safe operations, prompt rollback, provider fallback where behavior remains acceptable, and evaluation sets built from real incidents.
Hands-on Lab
Prerequisites: Python 3, a terminal, and no provider account. This lab uses the examples above as a local simulation so you can verify the service boundaries before adding a real model adapter.
- Create a file named
ai_service_lab.pyand paste the three code examples in order. - Run
python ai_service_lab.py. Verify that the first printed object contains tenantacme, prompt versionsupport_reply:v1, and the trimmed invoice message. - Verify that the fake provider prints the reply asking for the invoice number and then prints
FakeProvider. - Verify that the grounded reply includes
Source: kb-17. - Change the tenant id from
acmetounknownand run the file again. The grounded step should returnPOLICY_REFUSAL: no permitted evidence. - Change the hard-coded answer in
draft_grounded_replyso it omitskb-17. Run again and verifyINVALID_OUTPUT: missing source citation. - Cleanup is simple: remove the temporary file after the lab. If you adapted the pattern in an application branch, roll back by removing the provider adapter registration and restoring the previous prompt version.
The verification matters more than the toy data. You have tested three service behaviors: boundary validation, provider isolation, and output validation against allowed evidence. Those are the same behaviors you need before connecting a production provider.
Assessment Exercises
- Design a task-specific API for an AI service that extracts renewal dates from contracts. What should the input schema, output schema, and failure categories be?
- A model response is fluent but cites no retrieved source. Should the service return it, retry it, repair it, or fail? Explain how the answer changes for a casual summary versus an automated billing action.
- Given a latency trace with 80 ms retrieval, 4 seconds provider time, and two validation retries, which design change would you try first and how would you prove it helped?
- Write a test case that prevents cross-tenant retrieval leakage. What fixture data would make the test meaningful?
- When would you choose a prompt registry over prompts in code, and what controls must exist before non-engineers can edit production prompts?
Summary
Designing an AI service means turning model use into an application component with named tasks, versioned prompts, scoped retrieval, explicit provider adapters, validated outputs, and typed failures. The central mechanism is assembly and verification: normalize the request, resolve configuration, retrieve permitted evidence, apply policy, call the model through an adapter, validate the response, and record diagnostic metadata. The trade-offs are practical: generic APIs are fast to start but leak responsibility; strict schemas are safer but less forgiving; registry-driven prompts are flexible but need governance. A good AI service makes those choices visible so the rest of the product can depend on AI behavior without being coupled to every prompt and provider detail.
