Microsoft Azure Learning Journey

AI Without the Hype: Models, Embeddings, and RAG

Understand what machine learning and generative AI do, how to evaluate their outputs, and where Azure services fit.

5 min read · 910 words
Microsoft AzureAzure AIMachine LearningGenerative AICloud Computing
AI Without the Hype: Models, Embeddings, and RAG cover image

AI is not one service or one kind of model. A useful design starts with the task, the available evidence, and a measurable definition of success. Azure offers model-training platforms, hosted model endpoints, search and retrieval, and prebuilt AI services; none removes the need to protect data or evaluate the result.

What you’ll learn

Prerequisites

Start with the problem, not the model

Write the task as an input, an expected output, and a measurable acceptance criterion. “Classify an incoming support request into one of six queues” is different from “summarize the request” or “answer questions from a controlled policy library.” A database query, deterministic rule, or conventional search may be more accurate, cheaper, and easier to audit than an LLM.

Training and inference are different workloads

Training uses examples to fit model parameters. Inference applies a trained model to new inputs. Many teams consume a pretrained model through an endpoint and do not train a foundation model themselves. They may instead use prompt design, retrieval, or a supported fine-tuning workflow.

Machine-learning tasks include:

Training requires representative, governed data and a repeatable evaluation process. Inference requires an endpoint, capacity, identity, monitoring, and an operational response when outputs are wrong or unavailable. Azure Machine Learning supports custom model development and lifecycle operations; Microsoft Foundry provides model and application experiences for supported hosted models and tools.

Tokens, context, and embeddings

Language models process text as tokens, not as words or bytes. Tokenization affects context-window use, latency, and usage charges. A prompt that includes long chat history or entire documents can be expensive and may crowd out the evidence needed for the current question.

An embedding is a numeric representation of content that can support similarity search. An embedding model maps text into a vector space; a vector index retrieves items that are close under a chosen similarity method. The index does not understand truth or permissions. Chunk size, metadata, model choice, and query formulation all affect retrieval quality.

Retrieval-augmented generation (RAG) retrieves relevant content and supplies it to a generative model along with the user’s question. The model can then produce an answer grounded in that context. RAG can improve access to changing or private information without embedding every fact into model weights, but it does not guarantee correctness, citations, or authorization. Retrieval, source permissions, prompt construction, and output validation remain application responsibilities.

Evaluate the whole system

Evaluate the task and its failure modes, not just whether a demo response sounds plausible:

Keep a small, versioned evaluation set that reflects real task categories without exposing unnecessary personal or confidential data. Review model, prompt, index, and source changes against it before release. Human review remains appropriate for consequential decisions.

Responsible AI and security boundaries

Treat prompts and retrieved documents as untrusted input. A malicious instruction embedded in a document can attempt to override the application’s instructions. Tell the model how to treat source text, test attacks, and use suitable filtering or prompt-shield capabilities where they fit; do not treat a system prompt or safety filter as an authorization boundary.

Enforce access in the application and data services before content reaches a model. Do not put secrets in prompts, logs, evaluation datasets, or source documents. Minimize retained prompts and responses, set an explicit retention policy, and review the service’s data handling for the deployed model and endpoint. Separate the model’s ability to generate text from any ability to call tools or perform actions.

✅ Checkpoint: Pick one task and name its input, success metric, baseline without AI, model or retrieval components, sensitive-data boundary, expected latency, and cost unit. Identify one test where the system must refuse or escalate.

Next: Choose Azure AI services for training, inference, and retrieval.

Series navigation: Previous: Cost and performance architecture · Current: Phase 11, Part 1 · Next: Azure AI services

References

Share this article

LinkedIn post draft