Apache Parquet for Observability Data: Benefits and Limits

Observability data grows quietly until retention becomes expensive or a long-range query starts reading far more data than expected. At that point, the file format underneath the system matters.

Apache Parquet is an open, column-oriented file format for analytical data. It can store logs, metrics and traces as typed columns, let query engines skip fields and data ranges they do not need and keep historical telemetry readable by a broad data ecosystem.

Parquet supplies the file format, not the complete observability system. Databases and indexes provide the serving layer. File size, sorting, partitioning, metadata, caching and query execution determine performance.

This guide explains the Apache Parquet file format, why Parquet storage fits many observability workloads, what it does not solve and how to evaluate a Parquet-based telemetry architecture.

In this guide

• What Apache Parquet is
• How the Parquet file format works
• Why Parquet fits observability data
• What Parquet does not solve
• Parquet compared with other storage formats
• How Parseable uses Parquet
• How to evaluate Parquet for telemetry

What is Apache Parquet?

Apache Parquet is an open-source, column-oriented data file format designed for efficient storage and retrieval. It records schema, typed values, encodings, compression metadata and statistics that analytical engines can inspect before reading column data.

“Parquet storage format” is common shorthand, but Parquet defines files rather than a complete storage system. Query engines such as DataFusion, DuckDB, Spark or Trino read those files and execute queries. Table formats such as Apache Iceberg can add metadata, snapshots and transaction semantics across many Parquet files.

Columnar storage versus row storage

Consider three structured log events:

A row-oriented format such as JSON Lines keeps the fields for each event together:

[timestamp, service, severity, message]
[timestamp, service, severity, message]
[timestamp, service, severity, message]

A columnar layout groups values from the same field:

timestamp: [10:00:00, 10:00:01, 10:00:02]
service:   [checkout, payments, checkout]
severity:  [INFO, ERROR, INFO]
message:   [order accepted, gateway timeout, order completed]

This physical arrangement benefits queries that touch a small subset of a wide record. A query counting errors by service needs timestamp, service and severity; it does not need to read every message, stack trace, request header or deployment attribute.

Row-oriented storage remains useful when an application frequently reads or writes complete records. Columnar storage is a better match for analytical queries that scan many records but select relatively few fields.

How the Parquet file format works

The official Parquet concepts documentation describes a hierarchy of row groups, column chunks and pages.

Parquet keeps columns together inside each row group; it does not create one independent file per column. This preserves the rows needed to reconstruct records while retaining column-oriented reads.

Each row group contains the same set of columns for a different horizontal slice of rows. The footer records the schema, row-group locations and statistics that a query engine uses to plan reads.

Encoding and compression

Parquet separates encoding from compression. Encoding represents values efficiently before a compression codec processes the page.

The specification includes dictionary, run-length, delta and byte-stream-split encodings. Repeated values such as INFO, checkout or a cloud region may use a compact dictionary. Ordered timestamps may benefit from delta encoding. Parquet then supports compression codecs with different CPU and size trade-offs.

Compression is data-dependent. Repeated structured fields often compress well. Unique random identifiers, encrypted values, stack traces and already-compressed payloads may not. Measure encoded file size with representative telemetry instead of assuming a fixed compression ratio.

Projection and predicate pruning

Two optimizations sound similar but reduce different kinds of work:

• Projection pruning avoids reading columns that a query does not reference.
• Predicate pruning uses partition values, row-group statistics, page indexes or Bloom filters to avoid reading data that cannot match a filter.

For example:

SELECT service_name, count(*) AS errors
FROM application_logs
WHERE severity = 'ERROR'
  AND timestamp >= now() - interval '15 minutes'
GROUP BY service_name;

Projection pruning can skip fields such as message, trace_id and request_headers. Predicate pruning can skip files, row groups or pages outside the time range or without matching severity values. DataFusion's Parquet pruning explanation shows why sorting, statistics, page indexes and query predicates must align for this to work well.

The query engine first eliminates row groups that cannot match the filter. It then reads only the requested columns from the remaining data. These are separate decisions: predicate pruning removes irrelevant data ranges, while projection pruning removes unused fields.

Why Parquet fits observability data

Observability data is append-heavy, time-oriented, wide and commonly queried through filters and aggregations. Those properties align with Parquet for telemetry storage when the surrounding system manages ingestion and file layout carefully.

Selective queries read less data

OpenTelemetry logs and spans can carry dozens of resource, scope and event attributes. Incident queries typically select a time window, one or more services and a few fields. Column and predicate pruning reduce the bytes that must be fetched, decompressed and processed.

The engine reads relevant metadata and encoded pages, not only the rows returned. File organization and filter selectivity determine how much work it avoids.

Structured fields retain their types

JSON text requires parsing before a query engine can treat timestamps, numbers, arrays and nested values as typed data. Parquet stores physical and logical types in its schema. Typed columns make filtering, aggregation and compression more predictable.

Telemetry still needs a schema strategy. Dynamic attributes can produce wide or sparse data, and different writers can disagree about a field's type. The file format records each file's schema; the reader or table layer must reconcile compatible schemas across files.

Historical data remains accessible to multiple engines

Parquet is supported across the analytical ecosystem. DuckDB can query Parquet directly, while DataFusion, Spark, Trino, Athena, Polars and other tools provide their own readers.

Open files reduce dependence on a single query engine, but they do not make migration work disappear. External readers still need storage credentials, schema conventions, the partition layout, encryption configuration and any catalog metadata required to interpret the dataset.

Object storage separates durable data from compute

Parquet is well suited to immutable files on S3-compatible object storage. Durable history can remain on the storage layer while query capacity scales separately. This pattern supports longer retention without requiring every byte to remain on local SSD.

Object storage has higher request latency than local memory or NVMe. A production system usually adds buffering, metadata caches, data caches, compaction and a hot path for recent telemetry. The observability data lake architecture guide explains how those layers fit together.

High-cardinality fields do not require one index entry per value

A Parquet file can store fields such as trace_id, user_id or pod name as column values without requiring a global inverted index for every distinct value. That changes the ingestion and storage cost model.

The trade-off moves to reads. Exact lookup by an unsorted random identifier may still scan significant data unless the system supplies suitable sorting, Bloom filters, secondary indexes or cached metadata. The high-cardinality observability solution describes how Parseable applies this architecture at the product level. For metric-specific consequences, see high-cardinality metrics in columnar time-series storage.

What Parquet does not solve

Parquet is strongest as an analytical file format. A low-latency observability product still needs a serving system around it.

Fresh data needs an ingestion layer

Parquet files are immutable after they are written. An observability system cannot efficiently create a complete file for every incoming event. It must buffer records, write useful row groups, close files and compact undersized files while keeping recent data queryable.

Small files can erase the benefits

Thousands of tiny files increase object-store requests, metadata work and query-planning overhead. Very large files can reduce parallelism and make compaction expensive. The right target depends on ingestion rate, query concurrency, partition width and the object store.

Full-text search and point lookups need indexes or layout

Parquet statistics are valuable for selective analytical filters, but they are not a replacement for a full-text index. Searching arbitrary message substrings or one random trace ID requires an additional index, useful Bloom filters, a suitable sort order or more scanning.

Schema evolution requires coordination

Adding a compatible field to later files does not rewrite earlier files, but readers still need rules for merging schemas. Renaming a field, changing its type or interpreting nested structures can require explicit migration or a table/catalog layer. New format features can also have reader compatibility implications, as the Parquet format-version documentation explains.

Transactions and retention are outside the file format

Parquet does not provide ACID transactions, snapshots, deletion tracking or table-level retention. Object-store lifecycle policies, a catalog or an open table format must manage those concerns. Compliance also requires access control, encryption, retention enforcement and, where needed, object locking. A Parquet file alone is not an immutable audit trail.

Parquet compared with other storage formats

This is not a choice between “open but slow” and “proprietary but fast.” Databases can use open files underneath and Parquet-backed systems can add indexes, caches and execution layers. Evaluate the complete architecture rather than the filename on object storage.

How Parseable uses Parquet

Parseable uses Parquet on object storage as its durable telemetry layer, as described in the Parseable storage documentation. The ingestion path accepts telemetry, buffers records, organizes them for storage and writes Parquet files rather than asking applications to create files directly.

The Parseable query layer uses Apache DataFusion. Reading Parquet still requires metadata access, decompression, decoding and execution; there is no literal “zero-cost” conversion. The advantage is that DataFusion can decode selected Parquet columns into Arrow batches and apply pruning before processing unrelated data.

Parseable combines that storage layer with the parts Parquet does not provide by itself: ingestion protocols, schema handling, query planning, caching, SQL and PromQL access, dashboards, alerts and lifecycle management.

With Bring Your Own Bucket, the durable files live in object storage controlled by the customer. Compatible tools can read those files independently when they have the required schema, layout and access information. This improves data ownership without pretending that a file format alone replaces an observability platform.

The same architecture supports logs, metrics and traces, but their access patterns differ. Logs often need message search. Traces need identifier lookups. Metrics need time-series functions. The storage layer must support those serving paths rather than forcing every signal through one undifferentiated scan. For a signal-specific example, see how OpenTelemetry traces are stored in Parquet.

How to evaluate Parquet for telemetry

A smaller file is useful, but it does not tell you whether the system will work in production. Test the complete workload:

1. Use representative telemetry. Include structured fields, free-form messages, sparse attributes and high-cardinality identifiers.
2. Measure ingest delay. Track how quickly fresh data becomes queryable while the system buffers and writes files.
3. Inspect file layout. Check file size, row-group size, partition count, sorting, statistics and small-file accumulation.
4. Run real queries. Test dashboards, long-range aggregations, full-text searches and point lookups separately.
5. Separate storage and compute costs. Compare encoded bytes, object-store requests, cache resources, query CPU and compaction work.
6. Test failure behavior. Confirm what happens during partial writes, schema conflicts, compaction failures and query-memory exhaustion.
7. Verify portability. Read a sample directly with a second engine and document the metadata or catalog information it needs.

Parquet is a strong foundation for append-heavy telemetry when historical analysis and open storage matter. It is a poor substitute for a complete serving layer when workloads require constant row-level updates, arbitrary low-latency lookups or transaction semantics across files.

Conclusion

Apache Parquet fits observability because it stores typed telemetry in columns, supports selective reads and data pruning and remains accessible to multiple analytical engines. Paired with object storage, it also separates durable history from query compute.

Those benefits depend on the system around the files. Buffering, compaction, partitioning, sorting, indexes, caching, schema coordination and resource controls determine the user experience. The useful question is whether an open columnar storage layer, combined with the right serving architecture, matches your telemetry workload.

Try Parseable with representative logs, metrics or traces and compare ingest delay, query latency, encoded storage and external data access.