Information extraction solutions · 10–15%

Retrieval & grounding pipelines

~2 min read

Azure AI Search is a cloud search service: it indexes your content, then answers keyword, vector, and hybrid queries over it. It powers both classic search and RAG grounding.

Azure AI Search — indexing flow

The flow, in order: a data source holds your files; an indexer pulls them in; document cracking opens each file and extracts its content; the enrichment pipeline (skillset) adds AI-derived fields; everything lands in the index (and, optionally, a knowledge store).

An indexer is the process that pulls documents from the data source and builds the index. A skillset is the ordered list of AI skills it runs on each document to add derived fields.

• Documents built hierarchically; images land in normalized_images; each skill adds fields (e.g. OCR text per image); merge skill combines content + image text into merged_content.
• Field mappings: source fields implicit or explicit; skill outputs always explicit.
• Built-in skills (need attached Foundry Tools resource, same region; free tier ≤20 docs): language detection, entities, key phrases, translate, PII, OCR, image captions/tags.
• Custom skills: your logic (Azure Function) — e.g. wrap Document Intelligence for form fields.

Index field attributes — memorize

Querying

• Lucene syntax: simple (basic terms) vs full (regex, complex filters via $filter OData — case-sensitive!).
• searchMode: Any = match any term · All = match every term.
• A query runs in four stages: 1) parsing, 2) lexical analysis (lowercase, drop stopwords, stem words), 3) document retrieval, 4) scoring (TF/IDF).
• Facets take two queries: first query with facet=field to get the possible values and show them as clickable filters; then filter with $filter=field eq 'value'.
• Sort: relevance by default; $orderby=field desc overrides.

Vector / semantic / hybrid (RAG grounding)

• Vector search = embeddings + cosine similarity; semantic ranking = re-rank by meaning; hybrid (keyword+vector+semantic) = recommended for gen-AI.
• RAG ingestion: chunk the documents, embed each chunk, then index the vectors. The OCR skill turns scanned docs and images into text; enrichment adds structure.
• Connect retrieval to agents: AI Search tool, file_search vector stores, or Foundry IQ knowledge bases (MCP).

Knowledge store

A knowledge store saves the enriched data somewhere other than the index, so analytics tools can use it too. It persists data as projections: JSON objects (ETL), tables (relational analytics), image files. Defined in the skillset.