## TL;DR
Multimodal search enables "find me products that look like this photo" or "find videos about this topic" -- bridging the gap between different media types through a shared embedding space. From e-commerce product search to enterprise knowledge retrieval, multimodal embeddings let users search across text, images, video, and audio with a single query, in any modality.
## Core Explanation
Traditional search: text query matches text documents (TF-IDF, BM25). Multimodal search: any modality -> any modality. Example: upload a photo of a dress -> find similar products in catalog (image-to-image); describe an outfit -> find matching items (text-to-image); hum a tune -> find the song (audio-to-audio). Architecture: (1) Multimodal embedding -- separate encoders for each modality (CLIP: ViT image encoder + transformer text encoder) trained with contrastive loss to align embeddings in a shared space. Cosine similarity between any pair of embeddings measures relevance; (2) Two-stage retrieval -- Stage 1: approximate nearest neighbor (ANN) search over embeddings (FAISS, Milvus) retrieving top-100 candidates; Stage 2: cross-modal reranker (cross-attention between query and candidate) scoring the top-K, improving precision; (3) LLM-based search -- LLM interprets natural language queries, decomposes them into sub-queries, and synthesizes results from multiple modalities.
## Detailed Analysis
CLIP (2021, OpenAI): 400M image-text pairs from web -> contrastive pretraining -> zero-shot image classification and cross-modal retrieval. Follow-ups: SigLIP (sigmoid loss), EVA-CLIP, OpenCLIP. Multimodal search stacks: embedding (encode items offline) -> vector database (Milvus, Qdrant, Elastic) -> retrieval (ANN) -> reranking (cross-encoder) -> serving. Alibaba Qwen3-VL-Embedding (2026): Matryoshka Representation Learning enables nested embeddings -- a single model can produce embeddings at different dimensionalities (64, 128, 256,..., 4096d). At 64d, retrieval is fast but lower recall; at 4096d, recall is highest. Applications: e-commerce (Amazon, Shopify -- visual product search), enterprise (searching across documents, presentations, and images), and media (stock photo/video search). Key challenges: fusion of structured filters (price, category, date) with embedding similarity; freshness (new items need real-time embedding); and personalized search adapting to user preferences.