AI is not one service or one kind of model. A useful design starts with the task, the available evidence, and a measurable definition of success. Azure offers model-training platforms, hosted model endpoints, search and retrieval, and prebuilt AI services; none removes the need to protect data or evaluate the result.
What you’ll learn
- Distinguish model training, inference, and generative AI workloads.
- Explain tokens, embeddings, vector retrieval, and retrieval-augmented generation (RAG).
- Select task-appropriate quality and safety evaluations.
- Identify sensitive-data, prompt-injection, latency, and cost risks.
- Decide when a model is unnecessary or a simpler search/rules solution is better.
Prerequisites
- Basic Azure identity, networking, data, and monitoring concepts.
- No subscription is required for this overview.
- Model inference, training compute, search capacity, and safety services can incur charges.
Start with the problem, not the model
Write the task as an input, an expected output, and a measurable acceptance criterion. “Classify an incoming support request into one of six queues” is different from “summarize the request” or “answer questions from a controlled policy library.” A database query, deterministic rule, or conventional search may be more accurate, cheaper, and easier to audit than an LLM.
Training and inference are different workloads
Training uses examples to fit model parameters. Inference applies a trained model to new inputs. Many teams consume a pretrained model through an endpoint and do not train a foundation model themselves. They may instead use prompt design, retrieval, or a supported fine-tuning workflow.
Machine-learning tasks include:
- Supervised learning: learn from labeled examples, such as a known class or numeric target.
- Unsupervised learning: find structure in data without a supplied target, such as clusters or anomalies.
- Generative AI: produce or transform text, images, audio, or other content based on a prompt and model context.
Training requires representative, governed data and a repeatable evaluation process. Inference requires an endpoint, capacity, identity, monitoring, and an operational response when outputs are wrong or unavailable. Azure Machine Learning supports custom model development and lifecycle operations; Microsoft Foundry provides model and application experiences for supported hosted models and tools.
Tokens, context, and embeddings
Language models process text as tokens, not as words or bytes. Tokenization affects context-window use, latency, and usage charges. A prompt that includes long chat history or entire documents can be expensive and may crowd out the evidence needed for the current question.
An embedding is a numeric representation of content that can support similarity search. An embedding model maps text into a vector space; a vector index retrieves items that are close under a chosen similarity method. The index does not understand truth or permissions. Chunk size, metadata, model choice, and query formulation all affect retrieval quality.
Retrieval-augmented generation (RAG) retrieves relevant content and supplies it to a generative model along with the user’s question. The model can then produce an answer grounded in that context. RAG can improve access to changing or private information without embedding every fact into model weights, but it does not guarantee correctness, citations, or authorization. Retrieval, source permissions, prompt construction, and output validation remain application responsibilities.
Evaluate the whole system
Evaluate the task and its failure modes, not just whether a demo response sounds plausible:
- For classification, measure class-specific precision, recall, and confusion patterns on held-out examples.
- For generated answers, check relevance, factual support, citation correctness, refusal behavior, and consistency against representative questions.
- For RAG, separately measure retrieval recall and ranking, then answer quality with the retrieved evidence.
- Test boundary cases: ambiguous requests, missing evidence, stale content, conflicting sources, malformed input, and prompt injection.
- Measure latency, token consumption, search capacity, error rates, and cost per useful result under expected concurrency.
Keep a small, versioned evaluation set that reflects real task categories without exposing unnecessary personal or confidential data. Review model, prompt, index, and source changes against it before release. Human review remains appropriate for consequential decisions.
Responsible AI and security boundaries
Treat prompts and retrieved documents as untrusted input. A malicious instruction embedded in a document can attempt to override the application’s instructions. Tell the model how to treat source text, test attacks, and use suitable filtering or prompt-shield capabilities where they fit; do not treat a system prompt or safety filter as an authorization boundary.
Enforce access in the application and data services before content reaches a model. Do not put secrets in prompts, logs, evaluation datasets, or source documents. Minimize retained prompts and responses, set an explicit retention policy, and review the service’s data handling for the deployed model and endpoint. Separate the model’s ability to generate text from any ability to call tools or perform actions.
✅ Checkpoint: Pick one task and name its input, success metric, baseline without AI, model or retrieval components, sensitive-data boundary, expected latency, and cost unit. Identify one test where the system must refuse or escalate.
Next: Choose Azure AI services for training, inference, and retrieval.
Series navigation: Previous: Cost and performance architecture · Current: Phase 11, Part 1 · Next: Azure AI services