2024-03-29
Apache Benchmark tool

A Simple Review of HTTP Benchmark/Loading Tools in 2021

https://sylvainleroy.com/wp-admin/options-general.php?page=ad-inserter.php#tab-2

I tested a list of Benchmarking / Loading test tools and offer you some feedback about the tools listed on the excellent page https://gist.github.com/denji/8333630.

For 2021, I am curious to test and discover my new favorite HTTP Benchmarking tool. I am will test it on a new Java Backend that I am migrating to Spring 2.x

If you cannot read the whole test, I recommend the tool autocannon or cassowary to launch a simple load test from your CLI  and as a fallback the traditional ab. And check https://github.com/hatoo/oha since it looks promising.

Contents

Apache Benchmark tool

I know fairly well this tool since it is globally available everywhere on the Unix system. It is easy to use, understand but a bit slow and the results are not so easy to use.

This tool contains enough options to make a simple /Endpoint benchmark test easily: number of requests, concurrency, and time-limited test.

The output format is limited (Console or HTML).

Globally available, dead simple to use but a lack of reporting functionalities

Personal review:

star3

 

Apache Benchmark tool
Apache Benchmark tool

 

Apigee Benchmark tool

Link: https://github.com/apigee/apib

This tool is not friendly to use if you are on a Linux or Windows System. You will have to compile the tool to use it.

Annoying to compile and an additional set of features rather limited in to ab. I will totally ignore it.

Personal review:

star1

 

Apigee Benchmark
Apigee Benchmark

 

 

Autocannon

Link : https://github.com/mcollina/autocannon

This tool is written in offers a decent interface and a JSON output for your benchmarks. The tool is working well.

Some drawbacks the tool is a bit slow to launch and especially slow to stop when you are trying to interrupt the tool.

I find the reporting a bit lightweight and it misses the HTTP Status drill down (% of requests KO)

Console colors are a nice plus.

Personal review:

star3

 

autocannon
autocannon

 

Baloo

Link: https://github.com/h2non/baloo

This tool is written in GO. There is no update for 2 or 3 years.

I am not fluent in GO, therefore I did not give it a try.

Personal review: Only for GO developers

Baton

Link: https://github.com/americanexpress/baton

This tool is written in GO. It is really fast however the reporting and the output are too limited.

However, I like the idea to use a file with requests to load your tests.

Personal review: Much better tools exist.

star2

baton

Bombardier

Link: https://github.com/codesenberg/bombardier

This tool is written in GO. The error message from the console is really poor when your command is invalid.

The functionalities however are interesting :

  • Duration
  • HTTP 1 and 2 support
  • Disable KeepAlive
  • Output JSON / Text

The console output is a bit disappointing though, I prefer the autocannon’s one.

This tool is one of the fastest ( I mean the request/second rate produced).

bombardier
bombardier

 

 

Personal review: A nice tool with decent output functionalities and a big set of features regarding the communication protocol and some HTTP important tweaks. But probably abandoned or poorly maintained.

star3

Cassowary

Link: https://github.com/rogerwelin/cassowary

This tool is written in GO. Not available through packages (deb), but an RPM is offered.

The console help is good even I personally prefer  Bombardier.

The functionalities however are interesting :

  • Duration
  • CloudWatch metrics export
  • JSON Metric exports
  • Color consoled

The tool seems to be really slow. The parameters may be incompatible together as in the screenshot.

The tool has several features to allow to embed the tool into a script or smoke tests in a DevOps pipeline.

I cannot output the percentile metrics in the console, a NOGO for me.

personally

 

Personal review: Interesting if you want to export to CloudWatch or requires an script, otherwise I would rather skip it.

star3

FastHttpLoader

Link: https://github.com/hagen177/fasthttploader

Interesting functionalities :

  • Web reporting
  • Graphs
  • No JSON output, Text output is mediocre

FastHttpLoader
FastHttpLoader

star3

Vegeta

Link: https://github.com/hagen1778/fasthttploader

Interesting functionalities :

  • Reporting, graphs
  • HTTP 1, HTTP2 Protocol
  • Scripting

Drawbacks :

star2

K6

Link: https://github.com/loadimpact/k6#install

Interesting functionalities :

  • Scripting in ES6 JS: support for modules to aid code reusability across an organization
  • Everything as code: test logic and configuration options are both in JS for version friendliness
  • Automation-friendly: checks (like asserts) and thresholds for easy and flexible CI configuration!
  • HTTP/1.1, HTTP/2, WebSocket and gRPC protocol support
  • TLS features: client certificates, configurable SSL/TLS versions and ciphers
  • Batteries included: Cookies, Crypto, Custom metrics, Encodings, Environment variables, JSON, HTML forms, files, flexible execution control, and more.
  • Built-in HAR converter: record browser sessions as .har files and directly convert them to k6 scripts
  • Flexible metrics storage and visualization: InfluxDB (+Grafana), JSON or k6

Positive aspects :

  • DEB packages and Docker image
  • Great documentation

Drawbacks :

  • Requires to pass the args through the PIPE STDIN or with an external file
  • Tests are JS files.
  • Not Opensource for the UI and the running.

star2

ALI

Link: https://github.com/loadimpact/k6#install

Interesting functionalities :

  • Real-time funny and easy to use CLI/UI Benchmarking tool

star4

Funny, easy to use but somewhat limited.

Other tools tested or reviewed :

Drill : https://github.com/fcsonline/drill

star3

I don’t have RUST and Cargo, boring to compile ( I removed a start). However, the idea to provide a DSL (YML Format ) is good and allows interesting possibilities to write load tests.

HttpStat : https://github.com/reorx/httpstat

star1

Useless

Slapper: https://github.com/ikruglov/slapper

star1

Not working. Not maintained.

Locust:  star3

Easy to use, annoying to install (Python). Works well.

Yandex-Tank: https://yandextank.readthedocs.io/en/latest/index.html

star3

Plenty of features but overkill for daily usage. Great to use in a Software Factory.

FBender:  https://github.com/facebookincubator/fbender

star3

Multi-protocol tool DHCP, DNS, HTTP …

 

Not Tested

Fortio : seems to be complex to set up and bloated for a benchmarking/load test tool.

It has a ton load of features and I would recommend it for GRPC load testing.

 

 

 

 

 

Sylvain Leroy

Senior Software Quality Manager and Solution Architect in Switzerland, I have previously created my own company, Tocea, in Software Quality Assurance. Now I am offering my knowledge and services in a small IT Consulting company : Byoskill and a website www.byoskill.com Currently living in Lausanne (CH)

View all posts by Sylvain Leroy →