Independent Proof That Hardened Containers Don't Cost Speed: 8 Databases Benchmarked for Chainguard

Sona Hakobyan

Author

Sona Hakobyan

Sona Hakobyan is a Senior Copywriter at PFLB. She writes and edits content for websites, blogs, and internal platforms. Sona participates in cross-functional content planning and production. Her experience includes work on international content teams and B2B communications.

Senior Copywriter

Reviewed by Boris Seleznev

Reviewed by

Boris Seleznev

Boris Seleznev is a seasoned performance engineer with over 10 years of experience in the field. Throughout his career, he has successfully delivered more than 200 load testing projects, both as an engineer and in managerial roles. Currently, Boris serves as the Professional Services Director at PFLB, where he leads a team of 150 skilled performance engineers.

Security that can't be paid for in latency

Chainguard's reputation is built on container images that are hard to attack: minimal, tightly managed, updated constantly, engineered to keep the vulnerability count down. That reputation was already established. The open question was the one every security-first vendor eventually has to answer in front of an engineering buyer - does the hardening cost anything at runtime?

Strong security matters. It also cannot come at the price of being slower than the image the developer would otherwise have pulled without thinking about it. Chainguard needed clear, objective proof that their containers do not compromise on speed or efficiency, expressed as transparent benchmark testing against popular, widely-used images.

The credibility constraint shaped who could do the work. Numbers produced by the vendor about the vendor are marketing. To be worth publishing, the testing had to be done by someone unaffiliated - trusted, independent, and willing to report what the measurements said rather than what would be convenient.

Why Chainguard chose PFLB

Three properties, and they are the same three any buyer should be checking for when the output of an engagement is going to be quoted publicly:

• Complete impartiality. PFLB does not play favourites. Testing is transparent, neutral, and not influenced by vendor interests. When the conclusion says something performs well, it is backed purely by measurement.
• Robust and repeatable testing. Methods are built so that anyone can reproduce them. Consistency across runs is what separates a benchmark from an anecdote.
• Clear, actionable insight. Numbers on their own do not settle anything. Reports have to say plainly where the strengths are, where the weaknesses are, and what the result means for a decision.

Scope of work

PFLB benchmarked a range of database Docker containers against their counterparts from DockerHub, selecting databases that represent genuinely different real-world workloads and usage patterns:

• PostgreSQL - tested on both ARM and x86 architectures, so the findings would hold across different infrastructure setups rather than one
• CockroachDB
• MariaDB
• MySQL
• Redis
• Memcached
• ClickHouse
• MongoDB

Methodology

Testing environment. Everything ran on consistent, cloud-based infrastructure using AWS virtual machines. Standardising the environment is the whole point of a comparative benchmark: every container got the same conditions, with no hidden variables between one image and the next. Docker configurations were uniform and the benchmarking and monitoring setup was standardised across all runs.

Benchmarking tools, chosen per database type rather than forced into one harness:

• BenchBase for the relational engines - PostgreSQL, CockroachDB, MariaDB, MySQL and ClickHouse - running the recognised TPC benchmarks:
• - TPC-C for transactional workloads, measuring real-world responsiveness
• - TPC-H for analytical workloads, evaluating complex query performance
• memtier_benchmark for Redis and Memcached, driving realistic read-heavy scenarios that mirror how caches are actually used in production

Execution. Each test was run four to five times per scenario to establish accuracy and statistical reliability. Workloads and scenarios were varied to cover a wide spectrum of realistic demand: transactional (TPC-C), analytical (TPC-H), and high-volume read-heavy caching.

Monitoring, across three levels at once:

• System-level metrics - CPU usage, RAM consumption, disk I/O performance, network throughput
• Docker-specific metrics - per-container CPU and memory utilisation, so resource use could be attributed to the container rather than the host
• Latency and throughput - raw throughput in operations per second *and* goodput, the successful transactions only, alongside detailed latency distribution

That last distinction matters more than it sounds. A container that posts a high operations-per-second figure while quietly failing transactions is not fast; it is loud. Separating throughput from goodput is what stops a benchmark from flattering the wrong image.

Reporting

• Dedicated reports. Each database received its own detailed performance report, with clear comparison tables and precise visualisations, so Chainguard could see strengths and weaknesses per image against the standard Docker equivalent.
• A unified heatmap. Alongside the detailed documents, one visual summary spanning every tested database and scenario - the view that makes strengths, weaknesses and optimisation opportunities visible at a glance.
• A consistent methodology throughout, which is what makes cross-database comparison legitimate: results could be read against each other, not just each against its own baseline.

Results

• Chainguard received clear, unbiased benchmarking data covering 8 databases and two CPU architectures, usable directly in product decisions and optimisation work.
• Reliable proof points to support performance claims - produced by a party with no stake in the outcome, which is what makes them quotable.
• Per-image and cross-image visibility, through per-database reports plus the unified heatmap, so optimisation effort and resource allocation could be aimed at the specific containers that needed it.
• A demonstration of the commitment itself. By commissioning neutral validation rather than self-reporting, Chainguard showed the kind of evidence standard its own customers expect.

The unified heatmap: every tested database against every scenario, grouped by latency, CPU, RAM, network and disk, with outliers colour-coded. This single view is what made strengths and weaknesses visible across 8 databases at once.

Questions this engagement answers

What should I look for when choosing a performance testing vendor?

The three criteria this engagement was awarded on: impartiality (no vendor stake in the result), a repeatable methodology anyone can reproduce, and reporting that states weaknesses as plainly as strengths. If a vendor's report only ever contains good news, it is not measurement.

Can benchmark results be trusted when a vendor commissions them?

They can when the testing party is unaffiliated and the method is published in enough detail to be reproduced - identical AWS virtual machines, uniform Docker configuration, named tools, standard TPC workloads, four to five iterations per scenario. Everything that could differ between the two images was held constant except the image.

How do you compare containers fairly across different database types?

By selecting the benchmark to the workload rather than the other way round: BenchBase with TPC-C and TPC-H for the relational engines, memtier_benchmark for the caches - then keeping the environment, monitoring and iteration count identical across all of them.

Why measure goodput and not just throughput?

Because raw operations per second counts attempts, and goodput counts successes. Reporting both, with latency distribution, is what keeps a high-throughput result from hiding a failure rate.