Load Testing and Setting Benchmarks

A lot has been said about load testing and performance testing already — I do not intend to repeat the dose. However, I have found that there is indeed very little information about what is a good test for load and performance.  Services like NewRelic are awesome but they will monitor performance and the ideal scenario is supposed to be:

• check for slowest running request in NewRelic
• optimize and test again.

What intrigued us was to have a client set benchmarks on an existing rebuild ! So here we were supposed to FIRST give guarantees about how much we can improve the performance after rebuilding their portal. Can one be so sure of to give metrics on what has not been built yet?

We decided to take up the challenge — benchmark their existing system and then set our targets. The current system is built on Joomla but its hardly a content management system that the client requires. Apparently what started off an exercise in education, transformed into huge dynamic analysis and calculations. So Joomla got left behind and a whole lot of php code was patched up – this was 2 years ago !

Now, JOSH comes into the picture with trying to set guarantees about speed. I started off with installing httperf and autobench on the clients production system. Both these tools are awesome and I find them more flexible than apache bench.

The first question which arises when trying to test load — should I run httperf and autobench on the production server or on a remote server. Answer (after some searching) is that you should install on production server to avoid issues like network latency and other routing issues.

The first test I ran was simple enough:

autobench --single_host --host1=<server> --uri1=/ --low_rate 10 --high_rate 100 --rate_step 10 --num_conn 400 --num_call 1

Lo-and-behold , the server crashed at first and I reduced the num_conn field and things at least worked. However, the rude  shock was to see all 4xx errors in the response. I knew something was wrong and it was with my configuration.  Further logs research said  ‘403- Forbidden Error’ — WTF! Further googling told me that I was to turn off anti-flooding setting in SH404SEF !! (whatever that meant – LoL)

Finally I was on my way now and could test some basic URLs. However, soon I realized that I required to test with sessions (that is logged in users) and though there were a lot of tutorials on httperf and -wsesslog flag, I could not find any autobench setting for this! The man page (the last place I decided to look) said clearly that if you wanted to send additional httperf parameters use the prefix httperf_ — voila

# Add this line in the ~/.autobench.conf
httperf_wsesslog = "100,2,path.txt"

Note: the ” (double quotes) is important otherwise autobench throws a bunch of errors. Also you cannot set this via command line.. that is you cannot issue an –httperf_wsesslog=100,2,path.txt as a command line parameter to autobench !

The mysterious path.txt has content like this:

/Log-yourself-in    method=POST  contents='username=somename&password=somepassword'

After running autobench — you get LOADS of data. I realized that there are 2 or 3 columns which are very important.

• The Requests / second (demo_req_rate)
• Avg. Reponse Time (resp_time_localhost)
• Errors (errors_localhost)

It makes sense to generate graphs from these few columns for easy reading. Check out this one:

As you can see the current system is quite slow — almost 5-6 seconds per request (CRAZY). After this, we are pretty sure we can re-architect this as a Rails application and get the following committed benchmarks:

• Average Response time: ~ 800 ms
• Average Page Load: under 8 seconds
• Average failure rate on concurrency: max 2 for every 10 requests / sec

Wish us luck!