Generative AI & agentic solutions · 30–35%
Build generative AI apps
Endpoint + SDK decision
| You need | Use |
|---|---|
| Agents, evaluations, tracing, connections, Foundry direct models, governance | Foundry SDK (azure-ai-projects) + project endpoint (…services.ai.azure.com/api/projects/<name>) |
| Max OpenAI compatibility, portability, plain inference | OpenAI SDK + Azure OpenAI endpoint (…openai.azure.com/openai/v1) |
Both can coexist in one app. Foundry SDK chat client = project_client.get_openai_client() (still OpenAI under the hood).
Responses API vs ChatCompletions
Both are OpenAI chat APIs: you send messages, the model answers. The difference is who keeps the conversation state.
| Responses (recommended) | ChatCompletions (legacy/compat) | |
|---|---|---|
| State | Stateful — previous_response_id chains turns | Stateless — you resend full messages history |
| Models | Azure OpenAI + Foundry direct models | Broad ecosystem compatibility |
| Merges | ChatCompletions + Assistants patterns | — |
r1 = client.responses.create(model="gpt-4.1", instructions="You are…", input="What is ML?")
r2 = client.responses.create(model="gpt-4.1", input="Example?", previous_response_id=r1.id)
- Key params:
instructions(system prompt),temperature,max_output_tokens,top_p,stream=True. - Responsiveness: streaming (partial output) + AsyncOpenAI (non-blocking).
- Context window per call = instructions + history + tool schemas + tool outputs + retrieved docs — all tokenized every request.
Built-in tools (Responses API)
A tool is a capability you list in the request that the model may decide to call — running code, searching, or invoking your functions — before it answers.
| Tool | Does | Trigger phrase in questions |
|---|---|---|
code_interpreter | Runs Python in sandbox (pandas/numpy, no network) | “calculate / analyze CSV / chart” |
web_search | Live internet info | “current / latest / after training cutoff” |
file_search | Searches vector store of your uploaded files | “answer from our PDFs” |
function | Model emits a call, your code runs it, you send back function_call_output with the call_id | “call our API / take action” |
function_call_output + call_id). Validate arguments — don’t trust them blindly.RAG in an app
RAG (retrieval-augmented generation) is a pattern that fetches relevant data at question time and injects it into the prompt, so the model answers from your facts instead of its training memory. The three steps: retrieve matching content, augment the prompt with it, generate the answer. Embeddings (vectors) plus cosine similarity find meaning-matched content; Azure AI Search hosts the index; hybrid search recommended. For enterprise-scale agent knowledge, use Foundry IQ instead of building your own.
Evaluate models & apps
- AI-assisted quality: groundedness (fabrication check; Groundedness Pro = binary), relevance, coherence, fluency.
- Safety: defect rate = % responses over severity threshold; includes protected material + indirect attack (jailbreak).
- NLP metrics (need ground truth): F1 classification/retrieval, BLEU translation, ROUGE summarization, METEOR (synonyms), GLEU (sentence-level).
- Evaluation targets: model, agent, or pre-generated dataset (CSV/JSONL, or synthetic generation).
- When scores are low, try fixes in this order: 1) prompt engineering, 2) a different model, 3) RAG, 4) fine-tuning. Cost rises at each step.