Parquet and Arrow: What Is Actually Inside the File

ParquetPyArrowData Engineering

TL;DR: Parquet is not a box that magically shrinks your data. I tested a 500,000-row table with pyarrow 26.0.0 on this Windows machine and found that the default use_dictionary=True wraps integers and floats in a dictionary too, making the whole table 15.0% larger than with it off (15,697,403 vs 13,650,207 bytes); switching to compression="zstd" is 33x faster to write than gzip and still 12.4% smaller; reading a single column touches only 19.6% of the file's bytes; and sorting the data by region shrinks that column 80.6x and drops filter IO to 20.5%, at the cost of a file that is 3.5% bigger and an id-range filter that can no longer prune anything.

No affiliate links appear in this article. Every number below was produced on this machine.

What Is Actually Inside a Parquet File

The Apache Parquet format documentation gives this layout: a 4-byte magic PAR1, then the data area as column chunks grouped by row group (N column chunks for row group 1, then N for row group 2, up to row group M), then the file metadata, then a 4-byte little-endian length of that metadata, then another 4-byte PAR1 at the very end.

The important line in the official docs is "File metadata is written after the data to allow for single pass writing" — the metadata lives at the tail. Reading a Parquet file always starts with a seek to the end, grabbing the footer, and walking back to the column chunk offsets. That is the physical reason column projection really saves IO, which I measure further down: the footer carries the start offset of every column chunk, so a reader can seek directly instead of scanning.

Test Setup

Item	Value
Libraries	pyarrow 26.0.0 (writes `created_by` as `parquet-cpp-arrow version 26.0.0`), pandas 3.0.6, numpy 2.5.4
Data	500,000 rows x 5 columns, synthetic, seed 20261021
Columns	id (sequential int64), region (8 distinct strings), url (~500,000 distinct strings), value (random float64), flag (bool)
Machine	Windows, local SSD, single process
Sources	parquet.apache.org file format and configuration pages, Apache Arrow 26.0.0 `write_table` reference

Reproducible script (comments removed so it runs as-is):

import numpy as np, pyarrow as pa, pyarrow.parquet as pq
rng = np.random.default_rng(20261021)
N = 500_000
regions = np.array(["us-east-1", "us-west-2", "eu-central-1", "ap-southeast-1",
                    "ap-northeast-1", "eu-west-1", "sa-east-1", "us-east-2"])
tbl = pa.table({
    "id":     pa.array(np.arange(N, dtype="int64")),
    "region": pa.array(rng.choice(regions, N)),
    "url":    pa.array([f"/api/v1/items/{i}/detail?q=abc{i%97}" for i in range(N)]),
    "value":  pa.array(rng.random(N)),
    "flag":   pa.array(rng.random(N) > 0.5),
})
pq.write_table(tbl, "base.parquet", row_group_size=100_000)
print(pq.ParquetFile("base.parquet").metadata)

1. The Default Dictionary Makes the Whole Table 15.0% Bigger

write_table defaults use_dictionary to True, and it applies to every column. Flipping the switch one column at a time (500,000 rows, snappy, row groups of 100,000):

Column	Dictionary on	Dictionary off	Verdict
region	191,631	1,424,216	dictionary is 7.43x smaller
url	3,724,421	3,475,834	dictionary is 7.1% bigger
id	3,017,665	2,004,753	dictionary is 50.5% bigger
value	5,016,860	4,003,111	dictionary is 25.3% bigger
flag	64,522	64,522	identical

flag being identical is not a coincidence. The pyarrow docs state plainly that BOOLEAN type doesn't support dictionary encoding; the column uses RLE, and its metadata lists only RLE and PLAIN, while every other column also carries RLE_DICTIONARY.

Put together, the table written with defaults is 15,697,403 bytes versus 13,650,207 with use_dictionary=False — the default is 15.0% fatter for nothing. I ran a third variant with the dictionary limited to low-cardinality columns (use_dictionary=["region"] style) and got 13,430,547 bytes: 14.4% smaller than the default and still 1.6% smaller than switching it off entirely. The fix is not to disable the dictionary, it is to scope it.

2. The Dictionary Sweet Spot Sits Near 65% Cardinality

Dictionary encoding does not win just because a column holds strings. I wrote 100,000 rows of a fixed 25-character token and varied only the number of distinct values:

Distinct / rows	Dictionary on	Dictionary off	Winner
8 / 100,000	38,675	269,801	dictionary by 6.98x
512 / 100,000	116,151	347,693	dictionary by 2.99x
32,768 / 100,000	378,387	608,500	dictionary by 1.61x
50,000 / 100,000 (50.0%)	580,532	622,368	dictionary by 1.07x
65,536 / 100,000 (65.5%)	633,942	632,424	dead heat (0.998)
100,000 / 100,000 (100%)	675,487	643,385	PLAIN by 5.0%

Numeric columns are blunter: with 100,000 sequential int64 values, dictionary on is 603,746 versus 401,768 off (50.3% bigger); sequential int32 comes out at 1.506x; random float64 at 1.253x. So skip the dictionary above roughly 65% cardinality, and skip it on numeric columns almost always.

3. Stop Reaching for gzip: zstd Is Both Smaller and Faster

Same table, same parameters, only the codec changes (row groups of 100,000, dictionary on):

Codec	File size	vs snappy	Write time	Full read time
none	35,880,270	128.6% bigger	0.299s	0.459s
snappy (default)	15,697,403	1.00	0.595s	0.062s
lz4	16,004,470	2.0% bigger	0.400s	0.041s
zstd	10,216,336	34.9% smaller	0.455s	0.053s
gzip	11,657,380	25.7% smaller	15.125s	0.065s
brotli	9,921,421	36.8% smaller	4.247s	0.077s

Two conclusions you can copy directly. First, zstd beats gzip on both axes: 12.4% smaller and 33.2x faster to write (0.455s vs 15.125s). Only brotli is smaller, and it is 9.3x slower to write. Second, zstd's default level is 1: the default output of 10,216,336 bytes is byte-identical to compression_level=1. Level 9 saves another 8,580 bytes for 2.4x the time, and level 17 takes 8.043s while producing a file 1,661 bytes *larger* than level 1.

4. Column Projection Really Does Read Only Those Bytes

Reading base.parquet through a file wrapper that counts every byte handed to it:

Columns read	Bytes actually read	Share of file	Time
all 5	15,758,717	100.4%	0.082s
id only	3,082,270	19.6%	0.008s
value only	5,081,446	32.4%	0.005s
url only	3,788,773	24.1%	0.016s
id + value	8,098,180	51.6%	0.010s

This is what the footer pays for: seek to the chunk you need. value is random float64 and barely compresses, so even one column costs 32.4% of the file — which also tells you that splitting a wide table, or parking the fat column elsewhere, pays out in literal bytes.

5. Sorting Buys Pruning and Costs Size

I sorted the same data by region before writing (row groups of 50,000, ten of them) and then filtered:

Scenario	Bytes read	Share of file	Rows matched
unsorted, filter region="eu-west-1"	12,254,333	100.5%	62,704
sorted, filter region="eu-west-1"	2,587,292	20.5%	62,704
unsorted, filter id in [0,100000)	2,524,399	20.7%	100,000
sorted, filter id in [0,100000)	12,676,244	100.5%	100,000

You can only optimize for one filter column. Unsorted, every row group's region statistics run from ap-northeast-1 to us-west-2, so none can be skipped; sorted, only 2 of 10 row groups fall in range. But sorting by region scatters id, and the id range filter degrades from 20.7% to 100.5% of the file.

The size cost is real: per-column compressed bytes show region dropping from 191,587 to 2,376 (80.6x smaller), while the correlated url column grows from 3,974,336 to 4,570,381 (15.0% bigger, because url embeds the sequential id and sorting destroys that pattern). The whole file goes from 12,195,324 to 12,617,166, 3.5% bigger. You sort for pruning, not for size.

Row group sizing is not monotonic either: at 1,000 rows per group the file is 15,783,076 bytes (500 row groups, 340,426 bytes of footer), at 10,000 it is *larger* (15,979,172), and it only drops to 13,275,918 at 500,000 rows per group (1 row group, 1,391 bytes of footer). The footer grows linearly with the number of row groups, so very fine grouping gives back in metadata what it saves in compression.

6. The Official Recommendation and the pyarrow Default Do Not Agree

The Parquet configuration page says of row groups: "We recommend large row groups (512MB - 1GB)." Of pages: "We recommend 8KB for page sizes."

pyarrow 26.0.0 ships something else. Both the local docstring and the Apache Arrow 26.0.0 reference say row_group_size defaults to None, meaning "the minimum of the Table size (in rows) and 1024 * 1024", and data_page_size defaults to None, meaning "use the default data page size of 1MByte". I wrote a 1,200,000-row single-column table and got [1,048,576, 151,424] — two row groups, exactly 1024x1024, not 512MB–1GB.

Neither is wrong; they just use different units. The recommendation is in bytes, the default is in rows. The wider your columns, the further 1,048,576 rows drifts from the official advice — which makes this a parameter worth setting by hand on wide tables.

Nobody Verifies a Declared Sort Order

The sorting_columns documentation says outright: "The writer does not sort the data nor does it verify that the data is sorted." I tested it by writing a thoroughly shuffled table with sorting_columns=[SortingColumn(1)]. pyarrow accepted it without a sound, and reading it back gives row_group(0).sorting_columns equal to (SortingColumn(column_index=1, descending=False, nulls_first=False),) — while every row group's region statistics still span ap-northeast-1 to us-west-2.

The good news is that pyarrow's own filters uses the real statistics, so results stay correct (12,481 rows, matching a brute-force filter of 12,481). The risk sits elsewhere: any engine that trusts the declaration and skips row groups on that basis will return wrong answers, and whoever wrote the file sees no error at all. Declaring a sort order is not sorting.

Two more fields I wanted and could not get: statistics.distinct_count is None (pyarrow writes no cardinality statistics, so an engine cannot tell from metadata whether a column deserves a dictionary), and write_page_index defaults to False — the docs note that turning it on "disables the old method of writing statistics to each data page header", meaning page-level pruning is off by default.

Troubleshooting: Six Real Errors

Error 1: reading a non-Parquet file as Parquet

ArrowInvalid: Could not open Parquet input source 'notparquet.bin': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

Usually a truncated download or something else sitting in your object store. File extensions prove nothing — check whether the last 4 bytes are PAR1.

Error 2: a zero-byte file

ArrowInvalid: Parquet file size is 0 bytes

A killed write job, a full disk, or an upstream filter that produced no rows but still created the file. Note the difference: writing a genuinely typed empty table with pq.write_table() produces a valid 467-byte file (0 rows, complete schema) that reads back fine. The 0-byte one never finished and cannot be opened.

Error 3: mixed types in one column

ArrowInvalid: Could not convert 'three' with type str: tried to convert to double

Handing [1, 2.5, "three"] to pa.array() blows up. Parquet gives you one type per column and no lenient mode; clean the data before you build the table.

Error 4: writing a different schema to the same ParquetWriter

ValueError: Table schema does not match schema used to create file:
table:
a: double vs.
file:
a: int64

The second batch turned a into a float. cast() every batch to one schema before appending, or write separate fragment files with write_to_dataset.

Error 5: a filter naming a column that does not exist

ArrowInvalid: No match for FieldRef.Name(nope) in id: int64
v: double

A typo'd column name does not give you an empty table, it raises. Column names are case-sensitive, and nested columns are referenced differently.

Error 6: mismatched schemas inside one directory

ArrowInvalid: Failed to parse string: 'y' as a scalar of type int64

pq.read_table("directory") treats every parquet file there as one dataset and parses the rest with the first file's schema. One fragment with a different schema fails during parsing, not merging, which makes it easy to mistake for dirty data.

One more boundary: a Python int past the int64 ceiling gives OverflowError: int too big to convert from pa.array([2**63 + 1]), not an Arrow error type.

How to Turn the Six Knobs

Knob	Default	What these measurements suggest
`use_dictionary`	True (all columns)	Pass a column list, low-cardinality only; leave numeric columns off
`compression`	snappy	Use zstd for archival or cold data; lz4 only when read speed is everything
`compression_level`	None (= zstd level 1)	Do not raise it: level 9 saves 0.08% for 2.4x the time
`row_group_size`	None (= 1,048,576 rows)	On wide tables, recompute against the official 512MB–1GB byte target
`write_statistics`	True	Keep it — it is the only basis for row group pruning
`sorting_columns`	None	Declare it only after you have actually sorted

Where This Stops

Everything here comes from one Windows machine running pyarrow 26.0.0 single-processed, on synthetic data: 500,000 rows, fixed-length string tokens, uniformly distributed floats. Real-world string length distributions, time locality, and null ratios all move these numbers, especially the 65% crossover in section 2 — that figure belongs to one combination (25-character tokens plus snappy) and will drift with a different codec or a different string shape. I did not test Spark, Trino, or DuckDB read behavior, and I did not test remote reads against object storage, where random access costs nothing like a local SSD and column projection usually pays off far more than the 19.6% measured here. For page-level pruning (write_page_index) I only confirmed it is off by default; I did not run an on/off comparison.

FAQ

Should I use dictionary encoding or not?

It depends on cardinality. On this data, dictionary encoding is a net win below roughly 65% distinct values and a net loss above; for numeric columns (integers and floats) it loses almost always, and leaving the default on made this table 15.0% bigger.

Which compression codec?

Default snappy is the safe choice for speed. For size, pick zstd: here it is 12.4% smaller than gzip and 33.2x faster to write. lz4 gave the fastest full read on this machine (0.041s) but landed 2.0% larger than snappy.

I sorted by timestamp and now id lookups got slower. Why?

Because sorting serves exactly one filter column. After sorting by region, the region filter dropped from 100.5% to 20.5% of the file, while the id range filter went from 20.7% to 100.5%. Decide your most frequent filter first, then pick the sort key.

Does setting sorting_columns sort my data?

No. The official docstring says the writer neither sorts nor verifies. I wrote shuffled data with a sort declaration and got no error at all.

A file will not open. Truncated, or never Parquet to begin with?

Look at the last 4 bytes for PAR1. A 0-byte file raises Parquet file size is 0 bytes; a file with content but a broken tail raises Parquet magic bytes not found in footer. Different failures, different fixes.