I tested a list of Benchmarking / Loading test tools and offer you some feedback about the tools listed on the excellent page https://gist.github.com/denji/8333630.
For 2021, I am curious to test and discover my new favorite HTTP Benchmarking tool. I am will test it on a new Java Backend that I am migrating to Spring Boot 2.x
If you cannot read the whole test, I recommend the tool autocannon or cassowary to launch a simple load test from your CLI and as a fallback the traditional ab. And check https://github.com/hatoo/oha since it looks promising.
Apache Benchmark tool
I know fairly well this tool since it is globally available everywhere on the Unix system. It is easy to use, understand but a bit slow and the results are not so easy to use.
This tool contains enough options to make a simple URL/Endpoint benchmark test easily: number of requests, concurrency, and time-limited test.
The output format is limited (Console or HTML).
Globally available, dead simple to use but a lack of reporting functionalities
Personal review:
Apigee Benchmark tool
Link: https://github.com/apigee/apib
This tool is not friendly to use if you are on a Linux or Windows System. You will have to compile the tool to use it.
Annoying to compile and an additional set of features rather limited in comparison to ab. I will totally ignore it.
Personal review:
Autocannon
Link : https://github.com/mcollina/autocannon
This tool is written in Node.JS offers a decent interface and a JSON output for your benchmarks. The tool is working well.
Some drawbacks the tool is a bit slow to launch and especially slow to stop when you are trying to interrupt the tool.
I find the reporting a bit lightweight and it misses the HTTP Status drill down (% of requests KO)
Console colors are a nice plus.
Personal review:
Baloo
Link: https://github.com/h2non/baloo
This tool is written in GO. There is no update for 2 or 3 years.
I am not fluent in GO, therefore I did not give it a try.
Personal review: Only for GO developers
Baton
Link: https://github.com/americanexpress/baton
This tool is written in GO. It is really fast however the reporting and the output are too limited.
However, I like the idea to use a file with requests to load your tests.
Personal review: Much better tools exist.
Bombardier
Link: https://github.com/codesenberg/bombardier
This tool is written in GO. The error message from the console is really poor when your command is invalid.
The functionalities however are interesting :
- Duration
- HTTP 1 and 2 support
- Disable KeepAlive
- Output JSON / Text
The console output is a bit disappointing though, I prefer the autocannon’s one.
This tool is one of the fastest ( I mean the request/second rate produced).
Personal review: A nice tool with decent output functionalities and a big set of features regarding the communication protocol and some HTTP important tweaks. But probably abandoned or poorly maintained.
Cassowary
Link: https://github.com/rogerwelin/cassowary
This tool is written in GO. Not available through packages (deb), but an RPM is offered.
The console help is good even I personally prefer Bombardier.
The functionalities however are interesting :
- Duration
- CloudWatch metrics export
- JSON Metric exports
- Color consoled
The tool seems to be really slow. The parameters may be incompatible together as in the screenshot.
The tool has several features to allow to embed the tool into a script or smoke tests in a DevOps pipeline.
I cannot output the percentile metrics in the console, a NOGO for me.
Personal review: Interesting if you want to export to CloudWatch or requires an automated script, otherwise I would rather skip it.
FastHttpLoader
Link: https://github.com/hagen1778/fasthttploader
Interesting functionalities :
- Web reporting
- Graphs
- No JSON output, Text output is mediocre
Vegeta
Link: https://github.com/hagen1778/fasthttploader
Interesting functionalities :
- Reporting, graphs
- HTTP 1, HTTP2 Protocol
- Scripting
Drawbacks :
- Really cumbersome use of the CLI see https://geshan.com.np/blog/2020/09/vegeta-load-testing-primer-with-examples/
- Basically, I cannot remember how to use it, I have to browse the (somewhat obscure) documentation
- Requires to pass the args through the PIPE STDIN
K6
Link: https://github.com/loadimpact/k6#install
Interesting functionalities :
- Scripting in ES6 JS: support for modules to aid code reusability across an organization
- Everything as code: test logic and configuration options are both in JS for version control friendliness
- Automation-friendly: checks (like asserts) and thresholds for easy and flexible CI configuration!
- HTTP/1.1, HTTP/2, WebSocket and gRPC protocol support
- TLS features: client certificates, configurable SSL/TLS versions and ciphers
- Batteries included: Cookies, Crypto, Custom metrics, Encodings, Environment variables, JSON, HTML forms, files, flexible execution control, and more.
- Built-in HAR converter: record browser sessions as .har files and directly convert them to k6 scripts
- Flexible metrics storage and visualization: InfluxDB (+Grafana), JSON or k6 Cloud
Positive aspects :
- DEB packages and Docker image
- Great documentation
Drawbacks :
- Requires to pass the args through the PIPE STDIN or with an external file
- Tests are JS files.
- Not Opensource for the UI and the Cloud running.
ALI
Link: https://github.com/loadimpact/k6#install
Interesting functionalities :
- Real-time funny and easy to use CLI/UI Benchmarking tool
Funny, easy to use but somewhat limited.
Other tools tested or reviewed :
Drill : https://github.com/fcsonline/drill
I don’t have RUST and Cargo, boring to compile ( I removed a start). However, the idea to provide a DSL (YML Format ) is good and allows interesting possibilities to write load tests.
HttpStat : https://github.com/reorx/httpstat
Useless
Slapper: https://github.com/ikruglov/slapper
Not working. Not maintained.
Locust:
Easy to use, annoying to install (Python). Works well.
Yandex-Tank: https://yandextank.readthedocs.io/en/latest/index.html
Plenty of features but overkill for daily usage. Great to use in a Software Factory.
FBender: https://github.com/facebookincubator/fbender
Multi-protocol tool DHCP, DNS, HTTP …
Not Tested
Fortio : seems to be complex to set up and bloated for a benchmarking/load test tool.
It has a ton load of features and I would recommend it for GRPC load testing.