--- id: multimodal-search title: "Multimodal Search: Cross-Modal Retrieval, Product Search, and Multimodal Embeddings" schema_type: article category: ai language: en confidence: high last_verified: "2026-05-24" created_date: "2026-05-24" generation_method: ai_assisted ai_models: - claude-4.5-sonnet derived_from_human_seed: true conflict_of_interest: none_declared is_live_document: false data_period: static completeness: 0.85 atomic_facts: - id: af-multimodal-search-1 statement: >- ICLR 2025 introduced MMSearch -- the first comprehensive multimodal search benchmark evaluating large multimodal models (LMMs) on search tasks across text, image, video, and audio modalities -- finding that GPT-4V achieves 63% mean recall@10 on multimodal search queries, while specialized embedding models (Jina CLIP v2, Nomic Embed) achieve 72-78% on text-to-image and image-to-text retrieval, highlighting the gap between generalist LMMs and specialized retrieval models. source_title: MMSearch, ICLR 2025 / GitHub -- The First Multimodal Search Benchmark source_url: https://github.com/CaraJ7/MMSearch confidence: high - id: af-multimodal-search-2 statement: >- Qwen3-VL-Embedding and Qwen3-VL-Reranker (Alibaba Tongyi Lab, 2026) introduced a unified multimodal retrieval framework achieving SOTA on 13 multimodal retrieval benchmarks -- the embedding model (trained with Matryoshka Representation Learning and contrastive loss) supports variable embedding dimensions (64-4096), and the reranker refines top-K results via cross-modal attention -- demonstrating that unified vision-language embedding models surpass modality-specific approaches by 8-15%. source_title: Alibaba Tongyi Lab (2026) -- Qwen3-VL-Embedding and Reranker -- Unified Multimodal Retrieval source_url: https://opensearch.org/blog/multimodal-semantic-search/ confidence: high primary_sources: - id: ps-multimodal-search-1 title: "MMSearch: Benchmarking the Potential of Large Multimodal Models as Search Engines" type: academic_paper year: 2025 institution: ICLR 2025 / GitHub url: https://github.com/CaraJ7/MMSearch - id: ps-multimodal-search-2 title: "Multimodal Search: Searching with Semantic and Visual Understanding Using Multimodal Embeddings" type: industry_report year: 2026 institution: OpenSearch / Alibaba Tongyi / Elastic url: https://opensearch.org/blog/multimodal-semantic-search/ known_gaps: - Explainable multimodal search -- showing users why results were ranked as they were - Unified multimodal indexing supporting text, image, audio, video, and 3D in a single vector space disputed_statements: [] secondary_sources: - title: "Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions" type: survey_paper year: 2025 authors: - multiple institution: IEEE Access url: https://doi.org/10.1109/ACCESS.2025.3527550 - title: Learning Transferable Visual Models From Natural Language Supervision (CLIP) type: conference_paper year: 2021 authors: - Radford, Alec - Kim, Jong Wook - Hallacy, Chris - Ramesh, Aditya - Goh, Gabriel - Agarwal, Sandhini - Sastry, Girish - Askell, Amanda - Mishkin, Pamela - Clark, Jack - Krueger, Gretchen - Sutskever, Ilya institution: OpenAI / ICML url: https://arxiv.org/abs/2103.00020 - title: "VISTA: Visualized Text Embedding for Universal Multi-Modal Retrieval" type: conference_paper year: 2024 authors: - multiple institution: ACL url: https://aclanthology.org/2024.acl-long.175/ - title: "Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models" type: conference_paper year: 2025 authors: - Zhang et al. institution: CVPR url: https://openaccess.thecvf.com/content/CVPR2025/papers/Zhang_Bridging_Modalities_CVPR_2025_paper.pdf updated: "2026-05-24" --- ## TL;DR Multimodal search enables "find me products that look like this photo" or "find videos about this topic" -- bridging the gap between different media types through a shared embedding space. From e-commerce product search to enterprise knowledge retrieval, multimodal embeddings let users search across text, images, video, and audio with a single query, in any modality. ## Core Explanation Traditional search: text query matches text documents (TF-IDF, BM25). Multimodal search: any modality -> any modality. Example: upload a photo of a dress -> find similar products in catalog (image-to-image); describe an outfit -> find matching items (text-to-image); hum a tune -> find the song (audio-to-audio). Architecture: (1) Multimodal embedding -- separate encoders for each modality (CLIP: ViT image encoder + transformer text encoder) trained with contrastive loss to align embeddings in a shared space. Cosine similarity between any pair of embeddings measures relevance; (2) Two-stage retrieval -- Stage 1: approximate nearest neighbor (ANN) search over embeddings (FAISS, Milvus) retrieving top-100 candidates; Stage 2: cross-modal reranker (cross-attention between query and candidate) scoring the top-K, improving precision; (3) LLM-based search -- LLM interprets natural language queries, decomposes them into sub-queries, and synthesizes results from multiple modalities. ## Detailed Analysis CLIP (2021, OpenAI): 400M image-text pairs from web -> contrastive pretraining -> zero-shot image classification and cross-modal retrieval. Follow-ups: SigLIP (sigmoid loss), EVA-CLIP, OpenCLIP. Multimodal search stacks: embedding (encode items offline) -> vector database (Milvus, Qdrant, Elastic) -> retrieval (ANN) -> reranking (cross-encoder) -> serving. Alibaba Qwen3-VL-Embedding (2026): Matryoshka Representation Learning enables nested embeddings -- a single model can produce embeddings at different dimensionalities (64, 128, 256,..., 4096d). At 64d, retrieval is fast but lower recall; at 4096d, recall is highest. Applications: e-commerce (Amazon, Shopify -- visual product search), enterprise (searching across documents, presentations, and images), and media (stock photo/video search). Key challenges: fusion of structured filters (price, category, date) with embedding similarity; freshness (new items need real-time embedding); and personalized search adapting to user preferences.