--- id: neural-architecture-search title: "Neural Architecture Search: Automated Design of Deep Neural Networks" schema_type: article category: ai language: en confidence: high last_verified: "2026-05-24" created_date: "2026-05-24" generation_method: ai_assisted ai_models: - claude-4.5-sonnet derived_from_human_seed: true conflict_of_interest: none_declared is_live_document: false data_period: static completeness: 0.85 atomic_facts: - id: f1 statement: >- NAS with Reinforcement Learning (Zoph & Le 2017, Google Brain) pioneered automated architecture search, using an RNN controller to generate network architectures trained and evaluated on a proxy task. source_title: Zoph, Barret, and Quoc V. Le. Neural Architecture Search with Reinforcement Learning. ICLR 2017 source_url: https://arxiv.org/abs/1611.01578 confidence: high - id: f2 statement: >- DARTS (Liu et al. 2019, CMU/DeepMind) introduced differentiable architecture search, reducing search cost from thousands of GPU-days to a few GPU-days by relaxing the discrete architecture space to continuous optimization. source_title: "Liu, Hanxiao, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. ICLR 2019" source_url: https://arxiv.org/abs/1806.09055 confidence: high - id: f3 statement: >- EfficientNet (Tan & Le 2019, Google) used NAS to discover a family of models that achieved state-of-the-art accuracy while being 5-10× smaller than previous models, through compound scaling of depth, width, and resolution. source_title: "Tan, Mingxing, and Quoc V. Le. EfficientNet: Rethinking Model Scaling for CNNs. ICML 2019" source_url: https://arxiv.org/abs/1905.11946 confidence: high primary_sources: - id: ps-neural-architecture-search-1 title: "Neural Architecture Search: Methods, Applications, and Theory (Nature Collection)" type: academic_paper year: 2025 institution: Nature / Springer url: https://www.nature.com/collections/adjaeijhja - id: ps-neural-architecture-search-2 title: Generative neural architecture search (GNAS) type: academic_paper year: 2025 institution: Neurocomputing / Elsevier url: https://www.sciencedirect.com/science/article/pii/S092523122501032X known_gaps: - NAS for transformer and LLM architectures (vs. CNN-focused methods) - Hardware-aware NAS incorporating energy and carbon cost constraints disputed_statements: [] secondary_sources: - title: "Neural Architecture Search: A Comprehensive Survey" type: survey_paper year: 2023 authors: - Ren, Pengzhen - Xiao, Yun - Chang, Xiaojun - et al. institution: ACM Computing Surveys url: https://doi.org/10.1145/3447582 - title: "DARTS: Differentiable Architecture Search" type: conference_paper year: 2019 authors: - Liu, Hanxiao - Simonyan, Karen - Yang, Yiming institution: CMU / Google DeepMind / ICLR url: https://arxiv.org/abs/1806.09055 - title: "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" type: conference_paper year: 2019 authors: - Tan, Mingxing - Le, Quoc V. institution: Google Research / ICML url: https://arxiv.org/abs/1905.11946 - title: NAS with Reinforcement Learning (NASNet — Zoph & Le) type: conference_paper year: 2017 authors: - Zoph, Barret - Le, Quoc V. institution: Google Brain / ICLR url: https://arxiv.org/abs/1611.01578 updated: "2026-05-24" --- ## TL;DR Neural Architecture Search (NAS) automates the design of neural network architectures — replacing years of human intuition and trial-and-error with algorithmic search. From RL-based methods consuming thousands of GPU-hours to modern supernet-based approaches finding architectures in hours, NAS is democratizing optimal network design. ## Core Explanation NAS pipeline: (1) Search space — define the space of possible architectures (layer types, kernel sizes, channel counts, skip connections, depth). Cell-based spaces define reusable building blocks (normal cell, reduction cell); macro spaces define overall network structure; (2) Search strategy — how to explore the space: reinforcement learning (controller RNN samples architectures, reward = validation accuracy), evolutionary algorithms (mutation & crossover), Bayesian optimization, differentiable search (DARTS — relax discrete architecture choices to continuous, optimize via gradient descent); (3) Performance estimation — evaluating candidates: full training (too slow), proxy tasks (smaller dataset, fewer epochs), weight sharing (train one supernet containing all possible sub-networks). Modern weight-sharing NAS (Once-for-All, OFA) trains a single large supernet, then extracts specialized sub-networks for different hardware targets (latency, memory, FLOPs constraints) without re-training. ## Detailed Analysis Key milestones: NASNet (Zoph et al., 2018) — RL-discovered architecture achieving SOTA on ImageNet but requiring 1,800 GPU-days. DARTS (Liu et al., ICLR 2019) reduced search to ~1 GPU-day via differentiable optimization over a continuous relaxation of the architecture space. Once-for-All (Cai et al., ICLR 2020) trained a 40M-parameter supernet spanning 10^19 sub-networks, deployable on cloud TPUs down to mobile CPUs. Generative NAS (GNAS, 2025): learns the distribution of high-performing architectures, generating novel candidates that inherit successful design patterns. Nature's 2025 NAS collection highlights practical adoption: NAS-designed EfficientNet variants power real-world mobile vision systems; hardware-aware NAS optimizes for specific accelerators (NPU, TPU, GPU). Limitations: (1) CNNs dominated NAS research — transformer architecture search (AutoFormer, HAT) is less mature; (2) Multi-objective NAS (accuracy + latency + energy + memory) requires careful Pareto optimization; (3) Architecture transferability — architectures optimized for ImageNet may not transfer to medical imaging or satellite imagery; (4) Reproducibility — small changes in training hyperparameters can overwhelm architecture effects. ## Further Reading - AutoML.org: Neural Architecture Search Overview - Once-for-All: Train One Network and Specialize it for Efficient Deployment - DARTS: Differentiable Architecture Search (Liu et al., ICLR 2019)