Memcache Client API performance (version 2)

Written by Brian Moon, dealnews.com developer, on June 29, 2006

After writing up my findings in my last memcached test, I posted a link to it on the memcached mailing list. Some users had some useful questions and suggestions. So, I basically started from scratch and ran the tests again.

The Setup

I started by installing a fresh build (from Gentoo's Portage) of Apache 1.3.34, PHP 5.1.4, mod_perl 1.29 and mod_python 2.7.11. The previous tests had been run using Apache 2 and versions of the modules built for it. At the time I did those tests, we were moving to Apache 2. However, some problems we encountered led us back to Apache 1.3. I thought it best to test using what we will be using in production. I also installed fresh copies of all the memcached clients, including the rewritten Perl client mentioned on the list.

I then configured a virtual host for running the perl and python scripts. If someone more familiar with mod_perl and mod_python and sees some huge flaw in my config that would affect these tests, please let me know and I will correct it.

Perl

# without these, I got errors in my Apache error log
# even though the pages returned data
PerlModule ("/var/www/localhost/htdocs/perl/Hello.pm")
PerlModule ("/var/www/localhost/htdocs/perl/Large.pm")
PerlModule ("/var/www/localhost/htdocs/perl/Small.pm")
PerlModule ("/var/www/localhost/htdocs/perl/SetsGets.pm")
PerlModule ("/var/www/localhost/htdocs/perl/Gets.pm")
PerlModule ("/var/www/localhost/htdocs/perl/LargeMemcache.pm")
PerlModule ("/var/www/localhost/htdocs/perl/SmallMemcache.pm")
PerlModule ("/var/www/localhost/htdocs/perl/LargeMemcacheNoOutput.pm")
PerlModule ("/var/www/localhost/htdocs/perl/SmallMemcacheNoOutput.pm")
Listen 8080
NameVirtualHost *:8080
<VirtualHost *:8080>
DocumentRoot "/var/www/localhost/htdocs"
<Location /perl/hello>
SetHandler perl-script
PerlHandler Hello
</Location>
<Location /perl/large>
SetHandler perl-script
PerlHandler Large
</Location>
<Location /perl/small>
SetHandler perl-script
PerlHandler Small
</Location>
<Location /perl/sets-gets>
SetHandler perl-script
PerlHandler SetsGets
</Location>
<Location /perl/gets>
SetHandler perl-script
PerlHandler Gets
</Location>
<Location /perl/large-memcache>
SetHandler perl-script
PerlHandler LargeMemcache
</Location>
<Location /perl/small-memcache>
SetHandler perl-script
PerlHandler SmallMemcache
</Location>
<Location /perl/large-memcache-no-output>
SetHandler perl-script
PerlHandler LargeMemcacheNoOutput
</Location>
<Location /perl/small-memcache-no-output>
SetHandler perl-script
PerlHandler SmallMemcacheNoOutput
</Location>
</VirtualHost>

Python

Listen 8081
NameVirtualHost *:8081
<VirtualHost *:8081>
DocumentRoot "/var/www/localhost/htdocs"
PythonPath "sys.path+['/var/www/localhost/htdocs/python']"
<Location /python/hello>
SetHandler python-program
PythonHandler hello
</Location>
<Location /python/large>
SetHandler python-program
PythonHandler large
</Location>
<Location /python/small>
SetHandler python-program
PythonHandler small
</Location>
<Location /python/sets-gets>
SetHandler python-program
PythonHandler sets-gets
</Location>
<Location /python/gets>
SetHandler python-program
PythonHandler gets
</Location>
<Location /python/large-memcache>
SetHandler python-program
PythonHandler large-memcache
</Location>
<Location /python/small-memcache>
SetHandler python-program
PythonHandler small-memcache
</Location>
<Location /python/large-memcache-no-output>
SetHandler python-program
PythonHandler large-memcache-no-output
</Location>
<Location /python/small-memcache-no-output>
SetHandler python-program
PythonHandler small-memcache-no-output
</Location>
</VirtualHost>

I do want to note that when I ran each test, I only loaded the needed module into Apache.

The Tests

In the last tests, there were some questsions unanswered. For example, was the Perl and Python performance on the large memcached request slower due to its memcached client or some other factor. So, this time I ran nine tests to help answer some of those questions.

Hello World

The old reliable Hello World test was run so we can again establish a base line for how each language peforms within Apache. Again a static file was used as a base line comparison

Small File

In this test, a small file (10k, the HTML from http://dealnews.com/developers/) is read from disk and sent back to the browser.

Large File

In this test, a large file (266k, the HTML from http://dealnews.com/) is read from disk and sent back to the browser.

100 Sets/Gets

In this test, 100 memcached sets are performed, each time setting a new key with a new, small value. Then those 100 keys are fetched. No output is sent to the browser.

100 Gets

In this test, only the 100 gets of the same keys from above are performed.

Large Memcache

In this test, the large file from above is retrieved from memcached and sent back to the browser. The first request would fill the cache. Given that I ran a test on the scripts before starting the benchmark, no sets should have been performed.

Large Memcache - No Output

This is the same test as above, but the contents of the cache are not sent back to the browser.

Small Memcache

In this test, the small file from above is retrieved from memcached and sent back to the browser. The first request would fill the cache. Given that I ran a test on the scripts before starting the benchmark, no sets should have been performed.

Small Memcache - No Output

This is the same test as above, but the contents of the cache are not sent back to the browser.

The Results

(numbers are requests per second)	Perl	Perl (new)	Python	PHP	Static File
Hello World	478.67	—	399.11	461.66	786.90
Small File	239.33	—	259.08	259.34	602.77
Large File	36.41	—	37.98	36.23	42.84
100 Sets/Gets	5.14	5.18	7.75	42.07	—
100 Gets	7.13	7.38	13.38	63.47	—
Large Memcache	30.64	29.78	32.96	43.05	—
— No Output	47.01	45.88	51.23	103.24	—
Small Memcache	197.93	200.11	201.48	262.19	—
— No Output	214.25	226.53	263.56	292.83	—

So, this looks a lot like the last tests. There is a little change here and there. One surprise is that the new Perl client did not perform that well for me. I could have done something wrong, but I don't know what. The new perl client seems to really not like large chunks of data from memcached. I suppose for many users, this is a non-issue. 266k is a lot to stick in one cache. Reducing the size of the HTML on our front page is on my list of things to do.

There was a question on the list about compression. The PHP client uses compression by default. The Perl client used compression (see source). I could find no mention of compression in the Python client code.

Here is the static files, Perl, Python and PHP code I used:

memcached-tests-062006.tar.gz

Addendum

I was asked on the list to remove Apache from the equation. Frankly, Apache is part of my application so, its not valid for my needs to do this. However, I ran the following tests. Their output follows them.

Perl (current CPAN client)

# cat gets.pl
use strict;
use warnings;
use Time::HiRes qw( usleep ualarm gettimeofday tv_interval );
# timing example from http://perl.active-venture.com/lib/Time/HiRes.html
my $start = [gettimeofday];
use Cache::Memcached;
my $memd = new Cache::Memcached {
'servers' => [ "127.0.0.1:11211" ],
'debug' => 0,
'compress_threshold' => 10_000,
};
my $content = "";
my $x = 0;
for (0..99) {
$content = $memd->get("item$_");
}
my $end = [gettimeofday];
my $elapsed = tv_interval($start, $end);
print "$elapsed\n";
# perl gets.pl
0.110083
# perl gets.pl
0.107015
# perl gets.pl
0.107577
# perl gets.pl
0.106776
# perl gets.pl
0.105072
# perl gets.pl
0.103913
# perl gets.pl
0.108042

Perl (new client)

# cat gets.pl
use strict;
use warnings;
use Time::HiRes qw( usleep ualarm gettimeofday tv_interval );
# timing example from http://perl.active-venture.com/lib/Time/HiRes.html
my $start = [gettimeofday];
use Cache::Memcached;
my $memd = new Cache::Memcached {
'servers' => [ "127.0.0.1:11211" ],
'debug' => 0,
'compress_threshold' => 10_000,
};
my $content = "";
my $x = 0;
for (0..99) {
$content = $memd->get("item$_");
}
my $end = [gettimeofday];
my $elapsed = tv_interval($start, $end);
print "$elapsed\n";
# perl gets.pl
0.065681
# perl gets.pl
0.065462
# perl gets.pl
0.067464
# perl gets.pl
0.067719
# perl gets.pl
0.066819
# perl gets.pl
0.066642
# perl gets.pl
0.065069

Python

# cat getscl.py
import time
time.time()
start = time.time()
import memcache
mc = memcache.Client(['127.0.0.1:11211'], debug=0)
for i in range(100):
key = "thing" + str(i)
rc = mc.get(key)
end = time.time()
elapsed = end - start
print str(elapsed)
# python getscl.py
0.084823846817
# python getscl.py
0.0830268859863
# python getscl.py
0.085088968277
# python getscl.py
0.0827698707581
# python getscl.py
0.0841782093048
# python getscl.py
0.0829510688782
# python getscl.py
0.0845789909363

PHP

# cat gets.php
<?php
$start = microtime(true);
$MEMCACHE = new Memcache();
$MEMCACHE->addServer ( "127.0.0.1" );
$MEMCACHE->setCompressThreshold(10240);
for($x=0;$x<100;$x++){
$var = $MEMCACHE->get("thing$x");
}
$end = microtime(true);
$elapsed = $end - $start;
echo "$elapsed\n";
?>
# /usr/bin/php gets.php
0.0155520439148
# /usr/bin/php gets.php
0.0157949924469
# /usr/bin/php gets.php
0.0156910419464
# /usr/bin/php gets.php
0.0163218975067
# /usr/bin/php gets.php
0.0159170627594
# /usr/bin/php gets.php
0.0155100822449
# /usr/bin/php gets.php
0.015743970871