HTTP client to generate traffic for testing Flume

Question

I am testing a flume setup. I am willing to generate some HTTP traffic using some Traffic generators which can be implemented/integrated with Java and Flume for load testing of my Flume and HDFS setup.

Any Suggestions?

Thanks in advance!

Answer 1

Did you look at apache ab. ab is one of the widely used apache benchmarking tool for http. This can be used to create http load and simulate concurrent users.

http://httpd.apache.org/docs/2.2/programs/ab.html

Answer 2

Another alternatives are

HTTPERF

http://www.hpl.hp.com/research/linux/httperf/

Apache JMeter

http://jmeter.apache.org/

Answer 3

I think that Apache JMeter is the best option as

1. It is from the same vendor as Flume and Hadoop
2. It's Java based
3. Supports record-and-replay for HTTP Protocol via embedded HTTP Proxy
4. Can be easily extended via JSR223 scripting or implementing a custom plugin in Java
5. Hadoop Set of JMeter Plugins in general and HDFS Operations in particular seems to be what you're looking for.
6. JMeter can be executed in distributed clustered mode if resources of a single machine doesn't provide enough load to stress your server.

If for some reasons JMeter is not acceptable there are following free and open source alternatives:

• Java-based
  • Grinder
  • Gatling
• Non-Java-based
  • Tsung

For main features of the tools listed above highlighted and compared see Open Source Load Testing Tools: Which One Should You Use? post.