6 years of fuzzing Arrow and Parquet C++

Antoine Pitrou

October 12, 2026

Introduction

Author presentation

  • Contributor to Apache Arrow since 2018
  • Apache Arrow and Parquet committer and PMC member
  • ASF member since 2026
  • CPython core developer
  • Technical director at QuantStack

Why give this talk

  • Fuzzing is a useful software quality technique
  • Both widely-used and little-used
    • Python, Angular, Apache HTTPd, OpenSSL, OpenSSH, etc.
    • Most OSS projects don’t use it
    • Most developers wouldn’t know how to set it up
  • I have set up fuzzing for Arrow C++ in 2018 and been its main maintainer ever since
    • Developing most improvements and additions
    • The one who usually fixes issues found by fuzzing

Why I got interested in fuzzing

  • Being a CPython contributor has been a foundational experience
    • Large, high quality, mixed C and Python codebase
    • Detailed human code reviews and extensive unit testing
    • Still, we would regular find latent bugs and potential crashes
  • An appreciation that a widely-used code base
    • tends to be used in very unexpected ways
    • are attractive targets for attacks and exploitation
    • any situation that you don’t plan and test for can potentially hide an issue

Fuzzing

What is fuzzing, 1: unit testing

  • Human-written inputs and/or parametrization: test cases are curated by a human reasoning about the code’s intended purpose.
  • Inputs tend to be “reasonable”, they make sense to the person writing them, even when testing for errors.
  • For a developer, it is not intuitive to think about pathological cases (but for an attacker, it can be).
  • Unit test suites need to be deterministic, for the sake of reproducibility (any exceptions to this?).

What is fuzzing, 2: fuzzing

  • Fuzzing is a machine-driven process than generates random inputs to a function under test.
  • Fuzzing is not human-supervised, but it needs an oracle: the fuzzer must know whether an input led to misbehavior.
    • The crudest oracle: the process crashed/panicked/etc.!
    • Other oracles possible, such as idempotency or reprocity (for example: encode-then-decode roundtrip).
  • Generation is stochastic: inputs are different most of the time.
  • Generation uses heuristics to try to find interesting inputs: those that uncover a bug or a piece of pathological behavior.
  • The inputs generated might be “reasonable”, but they do not need to. The most interesting ones are usually not!

How efficient is fuzzing?

  • On any non-trivial function, the search space is huge (virtually infinite)!
  • Countless unsuccessful attempts are needed before finding bugs.
    • Often the first bugs come quickly, then it may take days… months…
  • Fuzzing is usually iterative: mutate past inputs, with an emphasis on the “interesting” ones.
    • What is an interesting input? Bugs, but not only!
  • Fuzzing is usually stateful: record past knowledge to avoid starting a from blank slate every time.
  • Fuzzing benefits massively from function-specific heuristics for input selection… which need maintenance to stay relevant.

Fuzzing “vs.” unit testing

Unit testing Fuzzing
Human-written inputs and/or parametrization Machine-generated inputs
Emphasize “reasonable” inputs (even error cases) Accidentally produces “pathological” inputs… by design
Deterministic (ideally, otherwise a bug) Stochastic by design
Stateless Stateful to avoid searching from scratch every time
Easy to set up (wide-spread, mature tooling, well-known best practices) Difficult to set up
Finds bugs immediately ⇒ can synchronously block PRs Can take a lot of time (potentially infinite) ⇒ bugs often found long after merging
100% coverage a (theoretical) goal Opportunistic: only fuzz the most sensitive code paths
Needs maintenance to stay relevant Needs maintenance to stay relevant

⇒ The two techniques are complementary!

Why fuzz C/C++ projects

  • C/C++ are low-level languages with many pitfalls.
    • Dereferencing null pointers
    • Stray pointers and references (use-after-free)
    • Out-of-bound accesses
    • Integer overflow is silent unless you explicitly test for it
  • Many of these issues are categorized “Undefined Behavior” (UB): a C/C++ language concept that means that compiled code need not care about them, and can assume it never happens.
  • C/C++ codebases often have an emphasis on performance, encouraging unsafe constructs.
  • Safe constructs are most of the time more verbose and come less naturally.

C/C++ trap: memory lifetime

  • When is it safe to access an object that lives in memory?
    • If passed by value: ✔️ always
    • If passed using an owning pointer (unique_ptr, shared_ptr): ✔️ always
    • Otherwise: who knows? it’s contextual
      • raw pointer: 💣 potentially unsafe
      • raw reference: 💣 potentially unsafe
      • this: 💣 potentially unsafe (except in constructor & destructor)
  • Straight-line synchronous code can seem easy, but multi-threaded or async programming makes this a hairy problem (especially on error).
    • even this can suddenly become invalid
  • The safe constructs (shared_ptr, enable_shared_from_this, weak_ptr) require more typing, more boilerplate, and are generally less easy to use.

General needs for setting up fuzzing

  • Summary of required items:

    • An input generator that generates random inputs for a function under test
    • Some feedback from the executed function to guide input generation
    • Orchestration to handle persistent state (past knowledge), alert about found issues

  • You can try to do all this by yourself, or use off-the-shelf facilities.

  • Enter OSS-Fuzz.

OSS-Fuzz

Overview

  • A Google-initiated and Google-hosted project that provides fuzzing infrastructure for selected open source projects (since 2016?).
  • Runs fuzzing continuously on Google infrastructure for the configured projects.
  • Supports (at least) C/C++, Go, JavaScript, JVM-based languages, Python, Ruby, Rust, Swift.
  • Based on various OSS building blocks:
    • Fuzzing engines (for C/C++): libFuzzer, AFL++, Honggfuzz, Centipede
    • Orchestration: ClusterFuzz, also a Google project.

Caveats

  • Project approval is opaque.

    To be accepted to OSS-Fuzz, an open-source project must have a significant user base and/or be critical to the global IT infrastructure.

    (you could set up ClusterFuzz or ClusterFuzzLite on your own infrastructure)

  • Project governance is unclear.

  • Maintenance status is unclear (is Google still engaged?).

  • Some tooling a bit clunky.

  • Not much interaction between OSS-Fuzz maintainers and project authors.

(if you are a OSS-Fuzz maintainer and this doesn’t feel right, please tell me)

Statistics

OSS-Fuzz projects by language (as of 22-09-2026)
Language Number of projects
C/C++ 592
Go 150
JavaScript 27
JVM-based 231
Python 278
Ruby 2
Rust 89
Swift 3

Setting up a project

  • Executive summary: 3 files required in the OSS-Fuzz repository

    • project.yaml: (short) metadata about the project
    • Dockerfile: environment in which the project is built for fuzzing
    • build.sh: build script for the project

  • The Docker environment uses custom OSS-Fuzz base images with wrapper scripts for building the project.

  • Please read the fine docs: “Setting up a new project”

How C/C++ projects get built

  • The project will be built with the following features enabled:
    • Code coverage (to guide random generation)
    • Address Sanitizer and/or Undefined Behavior Sanitizer (for more thorough bug detection)
      ⇒ recommend to have a ASan / UBSan build and tests on CI already.
  • The project must configure some fuzz targets (one per function under test).

    • A fuzz target is an executable made by linking the fuzzing engine (e.g. libFuzzer) with the code you want to fuzz.

    • The fuzzing engine defines its own main function that exposes a CLI for fuzzing. It calls into a predefined entry point each time it wants to execute the function under test.

      // (from libFuzzer documentation)
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
  DoSomethingInterestingWithMyAPI(Data, Size);
  return 0;
}
  • Project executable should be linked statically, including non-system dependencies.
    • Non-system dependencies should be built using the same setup (avoid using non-instrumented upstream binaries).

How OSS-Fuzz reports issues

  • When an issue is detected while fuzzing, project maintainers are privately notified by e-mail.

  • Issue details can be browsed on a dedicated issue tracker with restricted access; reproducers can be downloaded for local investigation.

    • Just feed the reproducer to the corresponding fuzz target, and use your tools of choice (such as GDB).
    • A helper script allows reproducing against the exact fuzzing environment through Docker.

  • If you push a fix to your codebase, later OSS-Fuzz builds of your project (presumably the day after) will pick it up the and detect that the issue cannot be reproduced anymore; the issue will then be closed.

How to help OSS-Fuzz find issues

  • OSS-Fuzz uses generic fuzzing engines (libFuzzer, etc.). It does not know anything about your code’s semantics or expectations.
    • A few basic configuration options are offered, such as max input size.

  • It is critical to help the fuzzing engine generate potentially interesting inputs:

    1. By defining a seed corpus: a ZIP file of payloads that serve as a starting point for input generation.
      • Inputs do not have to be “valid” for your function.
      • Be sure to include all input features that your function supports.
      • A seed corpus can be static or generated during build.
    2. By defining a dictionary: a list of binary fragments that are combined to produce inputs for your function.
  • (Why seed corpus? Because fuzz targets accrue their own corpus of interesting payloads when they are ran)

Fuzzing Arrow C++

Fuzz targets

  • A fuzz target is an executable that combines the fuzzing engine and the function under test.

  • Arrow C++ defines fuzz targets for these components:

    • Arrow IPC stream and file readers
    • Parquet file reader
    • Parquet encoding/decoding subsystem
    • CSV file reader
    • (Arrow IPC tensor stream reader – deprecated functionality!)

  • Components selected because of:

    • The fact that they may receive untrusted input
    • Their criticity for the project
    • Their assumed usage (are they widely used?)
    • Their complexity and therefore potential fragility

Build setup

  • On the Arrow C++ side:
    • A CMake option ARROW_FUZZING to enable fuzz targets and utilities
    • Target definitions for fuzz targets and utilities that generate the seed corpuses.
  • On the OSS-Fuzz side:
    • Dockerfile downloads fresh Boost
    • build.sh
      1. downloads and builds OpenSSL from source
      2. compiles Arrow C++ and its fuzzing executables
      3. generates the seed corpuses
  • Most build complexity remains in the Arrow repository.

Continuous Integration

  • Fuzz-related builds on the mainline Arrow CI:

    • A Docker-based build replicating the OSS-Fuzz build step (not the fuzzing).

    • A ASan/UBSan-enabled build that checks for regressions against all OSS-Fuzz issue reproducers (+ runs the regular C++ test suite).

Assorted notes

  • Arrow C++ validation APIs are designed to accept untrusted inputs, they are exercised by the fuzzers.

  • LibFuzzer crashes when a fixed memory ceiling is reached

    • this makes fuzzing less efficient and should be avoided
    • ⇒ solution: use custom memory pool implementation that refuses to allocate past a given amount

  • OSS-Fuzz does not support fuzz target parametrization

    • ⇒ solution: reserve payload start to encode fuzz target parameters and ensure the seed corpus encodes all supported parameter combinations

Fuzzers: Arrow IPC readers

  • An Arrow IPC stream or file represents a sequence of record batches with associated metadata.
    • Two variations over the same underlying protocol / format:
      IPC file = magic number + IPC stream + special footer for random access.
  • IPC stream fuzz target tries to parse its payload as an IPC stream
    1. Reads and validates each record batch sequentially
    2. If valid, a record batch is pretty-printed
  • IPC file fuzz target tries to parse its payload as an IPC file
    1. Reads, validates and pretty-prints each record batch sequentially, with pre-buffering disabled
    2. Same, with pre-buffering enabled
    3. Reads and validates the IPC stream embedded in the IPC file
    4. Checks that all three streams of record batches are equal (“differential fuzzing”)

Fuzzers: Arrow IPC readers (continued)

  • For both IPC stream and file fuzz targets, the seed corpus consists of:
    1. generated random data (during OSS-Fuzz build step)
      • for all Arrow data types
      • both compressed and uncompressed
    2. “golden” integration files generated at various points in Arrow’s history

Fuzzers: Parquet file reader

  • Parquet has many optional features and we strive to exercise most of them.

  • The Parquet fuzzer tries to read its payload as an entire Parquet file.

    1. opens the file, reads and deserializes all metadata
    2. reads, decodes and exercises all bloom filters
    3. reads, decodes and exercises all page indexes
    4. reads all file data, decodes it to Arrow, using several batch sizes
      • Arrow data is validated

Fuzzers: Parquet file reader (continued)

  • The seed corpus consists of:

    1. generated random data (during OSS-Fuzz build step)
      • for all Arrow data types that we allow writing to Parquet
      • for all Parquet encodings supported for each data type
      • for all compression algorithms supported by Parquet C++
      • using “V1” and “V2” data pages
    2. generated encrypted Parquet files (during OSS-Fuzz build step)
      • using various configurations of Parquet encryption
      • adding metadata that allows recovering the encryption keys from the file
    3. all test files (good and bad) in the official parquet-testing repository

Fuzzers: Parquet encoder / decoder

  • Goal: exercise the Parquet encoders and decoders specifically.

  • A parametric fuzzer with the following parameters:

    • source encoding
    • roundtrip encoding
    • Parquet physical type to decode for
    • number of decoded values to expect

  • Runs the following steps:

    1. decodes its payload using the source encoding
    2. encodes and re-decodes it using the roundtrip encoding
    3. compares the values decoded in both steps

Fuzzers: Parquet encoder / decoder (continued)

  • The seed corpus consists of generated random data
    • for all supported Parquet physical types
    • for all Parquet encodings supported for each type
    • using different numbers of values

Fuzzers: CSV reader

  • Goal: exercise various flavors of the Arrow CSV reader (serial, multi-threaded, streaming)

  • Fuzzer tries to read its payload as a CSV file, using different CSV reader APIs; validates the results.

    • type inference enabled
    • automatic dict-encoding enabled
    • small block size to exercise multi-threading

  • Seed corpus consists of:

    • generated CSV files written from a range of Arrow data types
    • example CSV files from the arrow-testing repository
    • example CSV files from the Pandas repository

Statistics

Aggregate fuzzer statistics (as of 2026-09-30). Only “verified” bugs shown.
Fuzz target First run First bug Last bug # bugs found Seed corpus OSS-Fuzz corpus
arrow-ipc-stream-fuzz 2020-01-15 2020-01-16 2026-03-04 68 133 files, 2 MB 35854 files, 93 MB
arrow-ipc-file-fuzz 2020-01-15 2020-01-16 2026-04-24 55 133 files, 2.1 MB 43357 files, 98 MB
arrow-csv-fuzz 2025-10-13 2025-10-14 2025-12-10 8 53 files, 3.9 MB 7305 files, 169 MB
parquet-arrow-fuzz 2020-02-12 2020-02-13 2026-01-13 109 571 files, 8.9 MB 31257 files, 302 MB
parquet-encoding-fuzz 2026-03-03 2026-03-05 2026-06-07 7 144 files, 2.6 MB 9716 files, 149 MB

Takeaways from fuzzing Arrow

  • Bug occurrence very correlated to changes in fuzzing or upstream code base
    • First bugs usually come quickly
    • A long tail of issues that can detected after weeks or months of running
    • Improving the seed corpus or making the fuzz target more thorough often pays off quickly
  • Fuzzing is not always successful
    • Need to design the fuzz target and its seed corpus carefully
    • OSS-Fuzz dashboard not easy to navigate but provides data about fuzzer runs (fuzzing speed, coverage, etc.)
  • Fuzzing itself needs maintenance
    • Handle build failures etc.
    • Improve seed corpus and fuzz targets as the functionality under test accrues more features
  • Fuzzing helps most on deterministic bugs. If a bug is not reproducible reliably, then all you know is that an issue might exist (!).

Is OSS-Fuzz necessary?

  • Understaffed
  • Infrastructure seems stagnating
  • Project direction unclear
  • Does fuzzing really need to be continuous? can it be run “from time to time” (when?)