What Is the Parquet File Format? Complete Guide

When your analytics bill depends on bytes scanned, your storage layout can determine whether you read a few relevant chunks or an entire dataset. Apache Parquet is an open source column-oriented format built for efficient storage and retrieval, and it groups values by column so your query engine can read only the fields a query needs.

The Parquet file format uses a binary layout with built-in compression and encoding, and the Apache Software Foundation governs it under Apache License v2.0.

This guide covers how Parquet organizes files, how compression and encoding reduce storage and scan costs, and when you should choose another format.

A Brief History of Apache Parquet

Twitter and Cloudera started Parquet in 2013 and used the record shredding and assembly algorithm from Google’s Dremel paper. Parquet became a top-level Apache project on April 27, 2015. Netflix, Stripe, and the National Aeronautics and Space Administration (NASA) had already adopted it in production, and its open, self-describing layout gave different engines a common analytics format.

How Parquet Stores Data (Inside the File Format)

A Parquet file follows a file hierarchy of row groups, column chunks, and pages. A binary footer maps the chunks. Row groups control parallel work, column chunks limit projection reads, and pages provide compression and encoding units.

Row Groups

A row group is a horizontal partition, and your engine can read each group independently. The recommended row-group sizes of 512 MB to one GB keep sequential reads large. In Hadoop Distributed File System (HDFS) deployments, a row group fits within one block when you configure the block size accordingly.

Column Chunks

Within each row group, Parquet stores a column’s values in one contiguous chunk, while the footer records chunk metadata such as its offset and size. Your query engine reads the footer, seeks to the chunks a query requests, and skips the rest through projection pushdown or column pruning. A Structured Query Language (SQL) query such as SELECT B FROM table WHERE A > 35 needs only the chunks for columns A and B.

Pages

Each column chunk divides into pages, the smallest compression and encoding units. A page can carry repetition levels, definition levels, and encoded values. These elements allow Parquet to represent nested and null values, while page indexes let compatible readers skip unneeded pages instead of decoding the full chunk.

The Footer and Predicate Pushdown

The footer follows the data, so a writer (the software library producing the file, such as pyarrow, fastparquet, or Spark’s Parquet writer) can produce the file in one pass. It stores the schema, row counts, chunk locations, codecs, and per-row-group metadata. Writers may also include minimum values, maximum values, and null counts.

Your engine can use predicate pushdown to compare available statistics with a query’s WHERE clause. If a filter asks for amount > 35 and the metadata lists a row-group maximum of 20, the engine can skip that group.

Columnar vs. Row-Based Storage: Where the Read Savings Come From

Row-based formats such as comma-separated values (CSV) and JavaScript Object Notation (JSON) store complete records together, while columnar formats place each field’s values together. On a 50-column table, a query naming two columns can read only two column chunks. One comparison reported 1,800 milliseconds against CSV, 90 milliseconds against Parquet, and 20 milliseconds against Hive-partitioned Parquet in its DuckDB benchmark results.

Compression and Encoding in Parquet

Parquet applies two compression layers: encodings exploit patterns within each column at page level, then codecs such as Snappy and Gzip compress the encoded pages. Zstd provides another codec option. These encoding layers compound because the codec processes encoded output. Parquet therefore uses less storage than uncompressed CSV when column values compress well.

Dictionary Encoding

Dictionary encoding stores each distinct value once per chunk and replaces repeated values with integer indexes. If the dictionary becomes too large, the writer can fall back to plain encoding. This approach fits low-cardinality columns such as status codes and regions.

Run-Length Encoding

Run-length encoding (RLE) collapses consecutive repeats into value-and-count pairs. The sequence 4, 4, 4, 4, 4, 1, 2, 2, 2, 2 becomes three entries: 4 five times, 1 once, and 2 four times. Parquet combines RLE with bit packing for boolean values and dictionary indexes.

Delta Encoding

Delta encoding stores the first value in full and then records differences between consecutive values. For example, 10, 12, 14, 16 becomes 10, 2, 2, 2. It works well for Unix timestamps and increasing identifiers (IDs), but poorly ordered data can produce larger deltas.

Bit Packing

Bit packing stores small integers with only the bits their range requires. A standard 32-bit integer reserves 32 bits per value regardless of how small the actual numbers are, so a column that only ever holds values from 0 to 10 wastes most of that space. Since values up to 10 fit in four bits (2⁴ = 16 possible values), bit packing stores each one using four bits instead of 32.

Dictionary indexes and boolean columns often benefit, and RLE can compress repeated packed values.

Snappy vs. Gzip vs. Zstd: Which Codec Should You Choose?

The codec choice sets your storage and latency tradeoff. Your benchmark should compare Snappy, Gzip, and Zstd against representative data and query patterns. Results can guide your standard codec while allowing exceptions for hot query paths or archives.

Codec	Compression profile	Decompression profile	Best fit
Snappy	Speed-oriented	Low-latency reads	Hot query paths
Zstd	Size-speed balance	General analytical reads	General-purpose production use
Gzip (zlib)	Size-oriented	Archive reads	Write-once cold archives

Benefits of the Parquet File Format

The layout gives you smaller files, cheaper scans, and data that carries its own schema. These properties reduce coordination overhead when engines share data in object storage.

• Smaller files: A Parquet size benchmark found files often five to six times smaller than uncompressed CSV. The smaller files cut storage and pay-per-scan query bills.
• Faster queries: Column pruning and predicate pushdown skip columns and row groups that your query never touches.
• Embedded schema: The footer records schema and types, while optional statistics support pruning. A catalog or table format must enforce table-level rules across files.
• Broad tool support: Spark, DuckDB, pandas, Apache Arrow, and other analytics tools provide tool support.
• Language independence: Implementations span Java, Rust, Go, and Python.

Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi commonly use Parquet as their data file format. Their metadata layers coordinate files and add table behavior. Broad implementation support makes Parquet a shared data layer across engines without requiring teams to treat one file as a complete table.

Parquet vs. CSV, JSON, Avro, and ORC

Orientation affects whether a workload favors record access or analytical scans. Human readability, schema behavior, and tool availability also shape the choice. The Avro specification defines its row-oriented layout.

Format	Orientation	Human-readable	Compression	Embedded schema	Best for
CSV	Row	Yes	None built in	None (readers infer types)	Small interchange files, spreadsheets, manual inspection
JSON	Row	Yes	None built in	Flexible or none	Application programming interfaces (APIs), ingestion layers, nested exchange data
Avro	Row	No (binary)	Yes	Yes (schema travels with data)	Streaming ingestion, Kafka pipelines, write-heavy workloads
ORC	Columnar	No (binary)	Yes	Yes	Hive and Hadoop batch processing
Feather (Arrow IPC)	Columnar	No (binary)	Yes	Yes (Arrow schema)	Fast local reads, Python-to-R interchange
Parquet	Columnar	No (binary)	Yes (Snappy, Zstd, Gzip, more)	Yes (schema plus optional statistics)	Data lakes, cross-engine analytics

Avro’s row orientation suits record-at-a-time writes, while Parquet’s columnar layout suits analytics. Apache Hive’s developers created ORC for Hive workloads, while Parquet works across a wide range of query engines and analytics tools. Feather V2 uses the Arrow interprocess communication (IPC) file format for fast local reads.

Reading and Writing Parquet Files

pandas, Apache Spark, DuckDB, and the R {arrow} package all read and write Parquet. Each can read the footer first, so you can request columns or filters without changing the file. Together, they cover Python, R, Java Virtual Machine (JVM), and SQL workflows.

Python and R

pandas Parquet support uses the pyarrow or fastparquet engine. Version 3.1 of pandas deprecates fastparquet. You can standardize on pyarrow for both directions:

import pandas as pd

df = pd.read_parquet("events.parquet", engine="pyarrow")

df.to_parquet("events_out.parquet", engine="pyarrow")

read_parquet returns a DataFrame, and a columns list limits the fields you read. In R, the {arrow} package’s read_parquet() returns a tibble. Its write_parquet() function uses Snappy compression by default.

Apache Spark

Spark provides Spark Parquet support for files and partitioned directories:

# Read a Parquet file
DF = spark.read.parquet("people.parquet")
# Write a Parquet file
df.write.parquet("people_out.parquet")

Both statements use the schema that the footer records. Spark marks every column nullable when it reads the file for compatibility. Your table or application can apply stricter constraints.

DuckDB and No-Code Viewers

DuckDB treats a Parquet file as a table that you can query directly. It requires no import step.

-- Query a Parquet file
SELECT * FROM 'events.parquet';
-- Export a Zstd-compressed Parquet file
COPY (SELECT * FROM tbl) TO 'out.parquet' (FORMAT parquet, COMPRESSION zstd);

The statements query the file in place and write a Zstd-compressed copy. The DuckDB command-line client and Apache’s parquet-cli support command-line inspection. Tad and ParquetViewer can also render the binary schema and rows.

Parquet in Modern Data Infrastructure: Lakehouses, Iceberg, and Delta Lake

Parquet is a file format, while Apache Iceberg, Delta Lake, and Apache Hudi are table formats. Their metadata layers add transactions with atomicity, consistency, isolation, and durability (ACID) and schema evolution over Parquet files. They also support time travel, which lets you query the state of a table as it existed at an earlier point in time by referencing an older snapshot or transaction version. Delta Lake adds a file-based transaction log, while Iceberg and Hudi also coordinate Parquet data files.

Teams store these files in Amazon Simple Storage Service (S3) and Azure Data Lake Storage. Google Cloud Storage provides another option, and teams query the files in place. Coralogix’s Remote Query reads telemetry data from customer-owned object storage. Coralogix writes the data in open Parquet format, so compatible Parquet engines can read the same files.

When Should You Not Use Parquet?

Parquet is a poor default when applications write or retrieve records one at a time. The file configuration guidance describes metadata and a columnar layout that favor analytical scans over frequent row mutations. Four workload patterns are a poor fit:

• Streaming, row-by-row appends: Micro-batch sinks commonly create new Parquet files instead of appending to an existing file, which can cause problems in high-volume log management pipelines. Land events in Avro and compact them into Parquet later.
• Small datasets: Footer and statistics overhead provides the most value on files of meaningful size, while CSV remains simpler for small files.
• Human-readable workflows: A text editor can’t render Parquet as readable rows or edit it directly, so CSV or JSON better supports manual work.
• Online transaction processing (OLTP): Materializing one row touches a chunk from every column, while updating one row can require rewriting the file.

Buffering events in a row format and compacting them into large Parquet files covers the first two cases. Transactional databases are a better fit for frequent single-row lookups and updates. This split keeps streaming writes responsive and analytical storage efficient.

What to Do Next

Parquet earns its default status through the Parquet file hierarchy: row groups for parallelism, column chunks for pruning, encoded pages for compression, and footer metadata for predicate pushdown. You will get the most value by buffering streams in a row format, keeping Parquet files large, and leaving transactional rows in a database. Open storage also lets your team change query engines without converting every file.

Frequently Asked Questions About the Parquet File Format

How much smaller is Parquet than CSV?

Results from the Parquet size benchmark show that Parquet files are typically five to six times smaller than uncompressed CSV, and roughly 30 to 35 percent smaller than gzip-compressed CSV. The ratio depends on cardinality, data types, encoding, codec, and sort order.

What compression should I use with Parquet: Snappy, Gzip, or Zstd?

Zstd is a strong general-purpose candidate, but you should test it against Snappy and Gzip using your data and query patterns. Hot query paths and write-once archives can require different tradeoffs.

How do I open a .parquet file?

DuckDB opens one in place with SELECT * FROM 'events.parquet‘. For visual inspection, desktop viewers such as Tad or ParquetViewer and Apache’s parquet-cli display the schema and rows.

How do I read a Parquet file in Python?

pd.read_parquet("events.parquet", engine="pyarrow") returns a pandas DataFrame. Passing a columns list reads only the fields you need, while PyArrow and Polars can read the same files directly.

Is Parquet the same as Iceberg or Delta Lake?

Parquet stores column data in files. Iceberg and Delta Lake add transaction metadata and schema evolution to collections of those files. They also add time travel.

If you are auditing your telemetry data storage format, you can assess whether it creates vendor lock-in. Coralogix’s unified observability platform keeps telemetry data as open Parquet files in your own cloud storage, and a free 14-day Coralogix trial lets you query them with the Parquet tools you already run.