[Solar-talk] Framework Benchmarks

Wed Oct 3 09:31:56 CDT 2007

Clay, Ray,

I am a project manager and do program management activities. Last thing
I want to have is a senior program manager ask me how much extra we had
to pay in servers because we are using a framework and not have 'real'
numbers about cost. The navy doesn't simply accept, "It's commodity
don't worry about it." 

You've made the statement that 'server hardware is a commodity resource'
but I keep seeing things talking about the cost of power, administrators
and other things similar to this so I wondered(as would program
managers) if that's as true as we assume it to be. Especially with long
lifetime applications. They've shown that software costs are 80% in the
maintenance phase of software, which is why frameworks are considered
good because they reduce maintenance costs but what about the cost of
actually running the application? Is it really as low as we have always
assumed it to be? Your comments about the EC2 boxes is well taken and
that's kind of what I wanted to get a feel for, so you are paying around
400-600 a month in hardware costs, how much are you paying per request?
Should be easy to figure out right? Doesn't sound too bad and like it
should be an issue but WITHOUT numbers like that we can't simply assume
that it works out okay because it feels right. 

That's why I asked the question. I've never looked at it from that
viewpoint, and most of the work I've done is not at the enterprise
level, it's at a different level entirely and a lot of it was done,
quite frankly, years before any of the current technologies were at
their current level of usage. 

It never hurts to go back and review things to ask the question and make
sure we haven't gone back down the path of 'use it because it's neat'
rather than for the right reasons. I suspected people would give me the
data showing it I was just curious if anybody had ever quantified the
'real' costs of a request.(btw, the .0005 number was simply meant as
let's have a real number it was not an attempt to quantify if the number
was accurate). 

Also, not all web applications, even big ones, can be hosted on
companies who just do that for a living. Some of us have to deal with
the servers, that setup, configuration, maintenace, power, backup, and
other issues when thinking about the applications as well. Another
server is more time and not always trivial either(try dealing with
security requirements for classifying the server and you'll see the
'server' starts to cost more and more).

Thanks guys. 

James Kilbride 

-----Original Message-----
From: solar-talk-bounces at lists.solarphp.com
[mailto:solar-talk-bounces at lists.solarphp.com] On Behalf Of Clay
Loveless
Sent: Wednesday, October 03, 2007 10:14 AM
To: solar-talk at lists.solarphp.com
Subject: Re: [Solar-talk] Framework Benchmarks

James,

If the performance problem you're raising a a concern about (12% as
efficient as a hello-world one-liner) is actually significant in some
scenario where you're paying anything close to $0.0005/request, I
presume the servers you're running on are hand-carved from blocks of
solid gold. :)

Look at it this way: server hardware is a commodity resource. Web
developers and web development productivity are not.

You'll spend more money just trying to *hire* a quality team -- forget
about their salaries and benefits -- than you'll spend in a decade of
handling web requests at 12% efficiency of "Hello World".

I'm a co-founder of a web services company called Mashery that processes
millions of very complex requests per day through Solar, and I could not
be happier. We do it with a team of 1-2 developers, and six plain-jane
Amazon EC2 instances. The cost of one Amazon EC2 box is in the
neighborhood of $70-100/month.

With that set-up, we have withstood traffic originating from several
articles about us on TechCrunch over the past year -- while handling our
normal day-to-day millions of requests -- without breaking a sweat.

We're now in the process of ramping up to handle requests at the rate of
thousands *per second* with Solar as the underlying core.

Additional costs to scale from millions to hundreds-of-millions per day?
We'll throw another dozen or so EC2 boxes online. No additional staff.

My point: servers are commodity, code quality and developer productivity
are not. Comparing them is not apples-to-oranges, it's more like
apples-to-yaks.

-Clay

On Oct 3, 2007, at 5:50 AM, Kilbride, James P. wrote:

> Btw, Paul, my brain pulled a distraction on me while writing that and 
> I forgot to finish the full conceptual thought in there.
>
> Basically it is, has anybody started looking at, from an economics 
> viewpoint, at which point in scaling an application up does hardware 
> costs/maintenance costs become more expensive than the developer 
> costs? Is it possible for that to occur? Is it possible that sometimes

> we gain productivity on the front end and pay for it on the back end?
>
> Does anybody actually look at that kind of information(this 
> application cost us $100,000 to write but we've spent 2 million on 
> server costs over the last 3 years.. etc..)
>
> I wonder if there is even any concern that could be a problem?  
> Could we quantify the 'cost per request' in some way? given a specific

> page load or such? "It costs us .0005 cents to show the customer the 
> product"
>
> I liked your supply demand curves btw in the atlanta talk. And I've 
> started bouncing back through some of the rest of the topics/posts.
> Very good work on that and it's amazing to me how easily people ignore

> the underlying 'truth' of the statements and ignore that you start 
> with a simple case before arguing a complex case. And yet all of them 
> want to somehow jump straight into the complex case right away.. It 
> makes no sense to me.
>
> Of course, listening to people rant and rave over the last decade 
> about Intel/AMD, NVidia/Whoever the ohter one is.. and so one with 
> benchmark after benchmark and how this benchmark shows the clear 
> winner..
>
> sigh, maybe it comes of having taught college courses in computer 
> architectures and showing how you can specialize things and how 
> benchmarks only tell a small part of the story(important but must be 
> taken in context) that it drives me nuts when people do nothing but 
> ignore the forest for the tree that is falling on top of them while 
> they argue about whether it's falling or simply swaying in the wind.
>
> Now, off to test, 'ease of use' and 'responsiveness' of solar by 
> trying to get it up and running with zend core.
>
> James Kilbride
>
> From: solar-talk-bounces at lists.solarphp.com [mailto:solar-talk- 
> bounces at lists.solarphp.com] On Behalf Of Kilbride, James P.
> Sent: Wednesday, October 03, 2007 8:12 AM
> To: solar-talk at lists.solarphp.com
> Subject: [Solar-talk] Framework Benchmarks
>
> Am I reading the benchmarks correctly Paul on your atlanta talk. At
> best(solar) a framework comes out to only be about 12% as efficient as

> writing the PHP code myself and doing some simplistic seperation of 
> business logic and layout?
>
> Ouch.. And some of those are on the 6% order. That's some serious 
> degredation of performance. Not saying I won't use them, since you get

> some good productivity gains but numbers like that do hurt the cause a

> bit. If I've got to spend 8-20 times as much on hardware to have the 
> same performance programmer productivity may become cheaper than 
> hardware.
>
> James Kilbride
>
> _______________________________________________
> Solar-talk mailing list
> Solar-talk at lists.solarphp.com
> http://mailman-mail3.webfaction.com/listinfo/solar-talk

--
Killersoft.com

_______________________________________________
Solar-talk mailing list
Solar-talk at lists.solarphp.com
http://mailman-mail3.webfaction.com/listinfo/solar-talk