The great web technology shootout - Round 2: PHP deserves a helping hand

A lot of the information below is out of date. Please see the new framework shootout page for the latest benchmarks.

This post is the continuation of a series. Please read Round 1 first if you are just now joining.

In Round 1, PHP was looking like quite the tortoise of the group. However, if you're familiar with some of the core differences between Python & PHP, you'll know that Python has been "cheating" slightly.

Let me explain: By default, Python compiles each script into bytecode on its first execution, allowing this bottleneck to be skipped on subsequent runs. PHP, however does not perform this type of optimization by default (in the 5.x line at least), so the PHP interpreter must re-compile each file every time it is run. As you can imagine, this can give PHP (without an accelerator) a huge disadvantage when compared to languages such as Python.

With this in mind, I have decided to take Round 2 to focus solely PHP. This will hopefully provide a clear picture of the benefits of PHP bytecode caching (at least when it comes to page-views — the memory benefits are a whole other story), and give you an idea of PHP's performance with the help of an accelerator.

There are many PHP accelerators available, but I have chosen APC for use here (mostly due to its inclusion in the upcoming PHP 6 core).

What you should know about Round 2:

  1. The hardware/software platform is the same as in Round 1.
  2. PHP has been upgraded to v5.2.9.
  3. The APC tests are using APC v3.0.19.
  4. ApacheBench was run with -n # -c 10, with -n usually being 5000 or 10000, depending on the test subject's speed.
  5. Each test was run several times to make sure that there were no anomalies, with the "optimum average" chosen as the numbers represented here.
  6. In an attempt to try to make these tests more scientific, I am now including the source code for each test (minus the CMS tests, which were performed on a standard install).

My apc.ini settings:

extension = apc.so
apc.enabled = 1
apc.shm_size = 96
apc.include_once_override = 1
apc.mmap_file_mask = /tmp/apc.XXXXXX

Raw PHP - php_test_src.zip

To investigate the speed of "raw PHP", I created a couple of minimal PHP scripts that printed out a simple "hello world", and read a few lines of text from a database. Take a look at the test source code for reference.

APC Disabled

The 'hello world' test:

Document Path:          /hello.php
Document Length:        12 bytes
Requests per second:    4199.43 [#/sec] (mean)

The simple database query test:

Document Path:          /hello_db.php
Document Length:        2524 bytes
Requests per second:    2797.54 [#/sec] (mean)

APC Enabled

The 'hello world' test:

Document Path:          /hello.php
Document Length:        12 bytes
Requests per second:    6618.42 [#/sec] (mean)

Simple database query test:

Document Path:          /hello_db.php
Document Length:        2524 bytes
Requests per second:    3713.99 [#/sec] (mean)

Right out of the gate, we can see that APC gives PHP a significant boost.

Of course, a script that simply does a "echo 'hello world'" isn't really practical, so let's take a look at a couple of popular PHP frameworks to see what a full stack looks like.

CakePHP v1.2.5 - cakephp_test_src.zip

CakePHP was the slowest test subject by a large margin in Round 1, but if you take a look at the caching section of its core configuration file you can see that the developers have written it with acceleration in mind. It includes special configuration directives for not just APC, but also Xcache and Memcache.

Still though, without acceleration CakePHP is a sad story indeed:

APC Disabled - File Caching Engine (default)

The return 'hello world' test:

Document Path:          /hellos/
Document Length:        13 bytes
Requests per second:    45.25 [#/sec] (mean)

The template hello world test:

Document Path:          /hellos/hello
Document Length:        931 bytes
Requests per second:    39.90 [#/sec] (mean)

The simple database query test (using the ORM):

Document Path:          /hellos/hello_db
Document Length:        2469 bytes
Requests per second:    38.80 [#/sec] (mean)

However, with the help of APC things finally begin to look reasonable:

APC Enabled - APC Caching Engine (in core.php)

The return 'hello world' test:

Document Path:          /hellos/
Document Length:        13 bytes
Requests per second:    140.98 [#/sec] (mean)

The template hello world test:

Document Path:          /hellos/hello
Document Length:        931 bytes
Requests per second:    120.04 [#/sec] (mean)

The simple database query test (using the ORM):

Document Path:          /hellos/hello_db
Document Length:        2469 bytes
Requests per second:    117.56 [#/sec] (mean)

As you can see, APC increases CakePHP's speed about threefold in these examples. These results may or may not be acceptable to you (remember that we are running these tests with a concurrency of 10), but I felt much better about CakePHP after pairing it with APC.

Kohana v3.0 - kohana3_test_src.zip

Kohana (a divergent of CodeIgnitor) is a much more lightweight PHP framework than CakePHP, but still gives you a pretty full featureset. Since v3.0 was released shortly after Round 1, I decided to use the new version for the tests here in Round 2.

It should be noted that although file caching is enabled in a default CakePHP install, Kohana's caching is disabled by default. I have included a couple of tests with caching disabled here to exhibit the performance of a default Kohana install.

APC Disabled - Caching Disabled (default)

The return 'hello world' test:

Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    86.73 [#/sec] (mean)

APC Disabled - Caching Enabled (bootstrap.php)

The return 'hello world' test:

Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    180.50 [#/sec] (mean)

The template hello world test:

Document Path:          /hellos/index
Document Length:        930 bytes
Requests per second:    162.27 [#/sec] (mean)

The simple database query test (using the Database module):

Document Path:          /hellodb/index
Document Length:        2525 bytes
Requests per second:    113.92 [#/sec] (mean)

The simple database query test (using the ORM module):

Document Path:          /helloorm/index
Document Length:        2525 bytes
Requests per second:    99.73 [#/sec] (mean)

As you can see, even without APC Kohana is quite a bit faster than CakePHP.

APC Enabled - Caching Disabled

The return 'hello world' test:

Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    125.93 [#/sec] (mean)

APC Enabled - Caching Enabled (bootstrap.php)

The return 'hello world' test:

Document Path:          /hello/index
Document Length:        12 bytes
Requests per second:    280.67 [#/sec] (mean)

The template hello world test:

Document Path:          /hellos/index
Document Length:        930 bytes
Requests per second:    253.50 [#/sec] (mean)

The simple database query test (using the Database module):

Document Path:          /hellodb/index
Document Length:        2525 bytes
Requests per second:    173.28 [#/sec] (mean)

The simple database query test (using the ORM module):

Document Path:          /helloorm/index
Document Length:        2525 bytes
Requests per second:    153.90 [#/sec] (mean)

Although not a threefold gain as with CakePHP, Kohana with APC+caching definitely makes a good attempt at shortening the gap seen in Round 1 between the PHP frameworks and the offerings from other languages. Of course, if raw speed is what you're looking for, there are faster PHP frameworks available (note that the benchmarks on that page were run on a completely different setup and should not be compared with any of the numbers here).

Next, let's take a look at how APC enhances the performance of a few of PHP's heavyweights.

WordPress v2.8.4 (default install)

Although I've heard that WordPress has a built-in caching setting, a quick glance at wp-settings.php did not reveal anything. So, there are simply two WordPress tests—with APC, and without:

APC Disabled

Document Path:          /
Document Length:        5519 bytes
Requests per second:    24.80 [#/sec] (mean)

APC Enabled

Document Path:          /
Document Length:        5519 bytes
Requests per second:    74.00 [#/sec] (mean)

Joomla v1.5.14 (default install, without sample data)

The results of these tests were quite disappointing to me, and made me wonder if I was doing something wrong. Sadly, it looks as though Joomla is quite the slow one of the bunch.

APC Disabled

Joomla Caching Disabled (default):

Document Path:          /
Document Length:        4006 bytes
Requests per second:    27.90 [#/sec] (mean)

Joomla Caching Enabled (File):

Document Path:          /
Document Length:        4006 bytes
Requests per second:    31.46 [#/sec] (mean)

APC Enabled

Joomla Caching Disabled:

Document Path:          /
Document Length:        4006 bytes
Requests per second:    41.27 [#/sec] (mean)

Joomla Caching Enabled (APC):

Document Path:          /
Document Length:        4006 bytes
Requests per second:    45.01 [#/sec] (mean)

Drupal v6.14 (default install, clean URLs disabled)

Drupal has several caching settings, so I decided to see what each one would look like and how much of a difference "aggressive" caching made.

APC Disabled

Drupal Caching Disabled (default):

Document Path:          /
Document Length:        5548 bytes
Requests per second:    36.72 [#/sec] (mean)

"Normal" Drupal Caching:

Document Path:          /
Document Length:        5548 bytes
Requests per second:    289.69 [#/sec] (mean)

"Aggressive" Drupal Caching:

Document Path:          /
Document Length:        5548 bytes
Requests per second:    335.68 [#/sec] (mean)

APC Enabled

Drupal Caching Disabled:

Document Path:          /
Document Length:        5548 bytes
Requests per second:    83.93 [#/sec] (mean)

"Normal" Drupal Caching:

Document Path:          /
Document Length:        5548 bytes
Requests per second:    498.37 [#/sec] (mean)

"Aggressive" Drupal Caching:

Document Path:          /
Document Length:        5548 bytes
Requests per second:    564.48 [#/sec] (mean)

As you can see, Drupal's caching (especially when combined with APC) changes things drastically. These results seem to indicate that Drupal's caching does a great job of allowing the CMS to get out of the way.

Closing thoughts

  • The benefits of a PHP accelerator are really too good to pass up. Even without written-in caching, an accelerator can give your code a significant boost.
  • While these tests have only really explored the impact on website page-load times, a PHP accelerator typically also cuts down on the amount of memory used by your PHP scripts.
  • PHP 6 has promised to fix a lot of things, including the lack of built-in bytecode caching. It will be interesting to revisit these tests in a few years (?) to see how much optimization the new version brings to the table.

Can't get enough of this madness? Just wait until you read Round 3

Comments

comments powered by Disqus