---
id: data-column-pruning-and-file-statistics
title: 'Data Column Pruning and File Statistics'
schema_type: TechArticle
category: computer-science
language: en
confidence: medium
last_verified: '2026-06-03'
created_date: '2026-06-03'
generation_method: ai_structured
derived_from_human_seed: true
conflict_of_interest: none_declared
is_live_document: false
data_period: static
atomic_facts:
  - id: fact-cs-data-column-pruning-and-file-statistics-1
    statement: >-
      BigQuery performance documentation recommends querying only the columns needed instead of
      using SELECT *.
    source_title: BigQuery Performance Best Practices
    source_url: https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute
    confidence: medium
  - id: fact-cs-data-column-pruning-and-file-statistics-2
    statement: >-
      Apache Parquet documentation describes file metadata that includes row groups and column
      chunks.
    source_title: Apache Parquet Metadata
    source_url: https://parquet.apache.org/docs/file-format/metadata/
    confidence: medium
  - id: fact-cs-data-column-pruning-and-file-statistics-3
    statement: >-
      Apache Iceberg documentation says table metadata can be used to plan efficient scans and avoid
      reading unnecessary files.
    source_title: Apache Iceberg Performance
    source_url: https://iceberg.apache.org/docs/latest/performance/
    confidence: medium
completeness: 0.82
known_gaps:
  - Column pruning and file skipping depend on selected columns, predicate pushdown, file format, table statistics freshness, partition transforms, projection expressions, nested columns, and the query engine's optimizer.
disputed_statements: []
primary_sources:
  - title: BigQuery Performance Best Practices
    type: documentation
    year: 2026
    url: https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute
    institution: Google Cloud
  - title: Apache Parquet Metadata
    type: documentation
    year: 2026
    url: https://parquet.apache.org/docs/file-format/metadata/
    institution: Apache Parquet
  - title: Apache Iceberg Performance
    type: documentation
    year: 2026
    url: https://iceberg.apache.org/docs/latest/performance/
    institution: Apache Iceberg
secondary_sources: []
updated: '2026-06-03'
ai_models:
  - gpt-5-codex
---

## TL;DR

Data agents should check projected columns and file statistics before recommending more compute for a slow analytical query.

## Core Explanation

Columnar engines can avoid reading columns that a query does not need. Table and file statistics can also help engines skip row groups, files, or partitions that cannot match predicates. These optimizations depend on the physical layout and whether the generated SQL preserves filter and projection opportunities.

Agents generating warehouse SQL should record selected columns, scanned bytes, file format, table metadata age, and explain-plan evidence. A query that uses `SELECT *` or wraps partition columns in unoptimizable expressions may defeat pruning and make the data system look slower than it is.

## Source-Mapped Facts

- BigQuery performance documentation recommends querying only the columns needed instead of using SELECT *. ([source](https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute))
- Apache Parquet documentation describes file metadata that includes row groups and column chunks. ([source](https://parquet.apache.org/docs/file-format/metadata/))
- Apache Iceberg documentation says table metadata can be used to plan efficient scans and avoid reading unnecessary files. ([source](https://iceberg.apache.org/docs/latest/performance/))

## Further Reading

- [BigQuery Performance Best Practices](https://docs.cloud.google.com/bigquery/docs/best-practices-performance-compute)
- [Apache Parquet Metadata](https://parquet.apache.org/docs/file-format/metadata/)
- [Apache Iceberg Performance](https://iceberg.apache.org/docs/latest/performance/)