[Solar-talk] Framework Benchmarks

[Paul M Jones]

Hi James -- combining replies here.

> Am I reading the benchmarks correctly Paul on your atlanta talk. At  
> best(solar) a framework comes out to only be about 12% as efficient  
> as writing the PHP code myself and doing some simplistic seperation  
> of business logic and layout?
Not exactly.  The 6-12% comparison is for writing "hello world" as a  
framework view rendered from an empty action method, vs. `echo "hello  
world"` in plain PHP.

My standard response to this is that the "performance drop" is likely  
to be similar in any system that provides features similar to the  
tested frameworks.

Even if you roll-your-own from existing libraries and glue code, I  
assert (without proof) that you will see responsiveness drop to  
around those levels when compared to the absolute max that PHP can do  
when it does nothing (i.e., `echo "hello world"`).  Every line of  
"real" application code you add will only cause execution to slow  
down, and when you add the set of features that most generic  
application structures need (config, controller, view, action  
processing, etc) you'll reduce responsiveness to the levels shown in  
the charts.  Even Code Igniter, the fastest of the bunch with  
anything remotely like a reasonable architecture (damning with faint  
praise ;-) only hits 18%, and it's not more than a stack of includes.

So one caveat is that percentages are for comparing frameworks to  
each other, not frameworks to PHP proper.  Additionally, the  
percentage figure helps when comparing the frameworks on different  
hardware configurations: 10% of the max on system A should still show  
10% of the max on system B, even if system A runs at 1000 req/sec max  
and system B runs at 2000 req/sec max.

I hope this helps to show the right mindset to use when approaching  
these numbers for decision-making; please let me know if I have  
missed the point, or failed to address something else you're thinking  
about.


...

> Basically it is, has anybody started looking at, from an economics  
> viewpoint, at which point in scaling an application up does  
> hardware costs/maintenance costs become more expensive than the  
> developer costs? Is it possible for that to occur? Is it possible  
> that sometimes we gain productivity on the front end and pay for it  
> on the back end?
>
> Does anybody actually look at that kind of information(this  
> application cost us $100,000 to write but we've spent 2 million on  
> server costs over the last 3 years.. etc..)
>
> I wonder if there is even any concern that could be a problem?  
> Could we quantify the 'cost per request' in some way? given a  
> specific page load or such? "It costs us .0005 cents to show the  
> customer the product"

I am not aware of any such formal research or reporting.  However, in  
"The Inmates Are Running The Asylum", Alan Cooper does hint toward  
those ideas.  IIRC, his line is that properly written software has an  
infinite cost/request before deployed (because there have been no  
requests on it yet), but the cost/request rapidly drops to asymptotic- 
zero when it goes into production (because the software keeps running  
without further intervention).


> I liked your supply demand curves btw in the atlanta talk. And I've  
> started bouncing back through some of the rest of the topics/posts.  
> Very good work on that

Thanks very much for saying.  :-)


> and it's amazing to me how easily people ignore the underlying  
> 'truth' of the statements and ignore that you start with a simple  
> case before arguing a complex case. And yet all of them want to  
> somehow jump straight into the complex case right away.. It makes  
> no sense to me.

Makes little sense to me from a technical perspective, but from a  
social/psychological one I totally get it (although I was not  
preprared for the intensity of it).


> Of course, listening to people rant and rave over the last decade  
> about Intel/AMD, NVidia/Whoever the ohter one is.. and so one with  
> benchmark after benchmark and how this benchmark shows the clear  
> winner..
>
> sigh, maybe it comes of having taught college courses in computer  
> architectures and showing how you can specialize things and how  
> benchmarks only tell a small part of the story(important but must  
> be taken in context) that it drives me nuts when people do nothing  
> but ignore the forest for the tree that is falling on top of them  
> while they argue about whether it's falling or simply swaying in  
> the wind.

I completely agree.


> Now, off to test, 'ease of use' and 'responsiveness' of solar by  
> trying to get it up and running with zend core.

Good luck with benching the "ease of use" portion.  ;-)  That's  
another one that I just don't get from the responses to the benching:  
"show much more productive you with one than another!"  As if you  
could properly control for all the variables in that kind of  
experiment.  Oh well, its good information for those who know how to  
use it.  :-)


--

Paul M. Jones  <http://paul-m-jones.com>

Solar: Simple Object Library and Application Repository
for PHP5.  <http://solarphp.com>

Join the Solar community wiki!  <http://solarphp.org>

Savant: The simple, elegant, and powerful solution for
templates in PHP.  <http://phpsavant.com>