[Solar-talk] Framework Benchmarks

[Clay Loveless]

On Oct 3, 2007, at 7:31 AM, Kilbride, James P. wrote:

> Clay, Ray,
>
> I am a project manager and do program management activities. Last  
> thing
> I want to have is a senior program manager ask me how much extra we  
> had
> to pay in servers because we are using a framework and not have 'real'
> numbers about cost. The navy doesn't simply accept, "It's commodity
> don't worry about it."

Ah, I didn't realize this was a military-related question. In that  
case, the answer would be "That question is regarding concerns that  
are above your pay grade, soldier." :)

Seriously, it's not the job of the developer to worry about the cost  
of the hardware. They just need to know what the constraints are that  
the hardware puts on them, and write good code. Only in a very, very  
small shop is it *also* the developer's job to worry about the  
monthly server fees.

> You've made the statement that 'server hardware is a commodity  
> resource
> but I keep seeing things talking about the cost of power,  
> administrators
> and other things similar to this so I wondered(as would program
> managers) if that's as true as we assume it to be. Especially with  
> long
> lifetime applications. They've shown that software costs are 80% in  
> the
> maintenance phase of software, which is why frameworks are considered
> good because they reduce maintenance costs but what about the cost of
> actually running the application? Is it really as low as we have  
> always
> assumed it to be? Your comments about the EC2 boxes is well taken and
> that's kind of what I wanted to get a feel for, so you are paying  
> around
> 400-600 a month in hardware costs, how much are you paying per  
> request?
> Should be easy to figure out right? Doesn't sound too bad and like it
> should be an issue but WITHOUT numbers like that we can't simply  
> assume
> that it works out okay because it feels right.

 From the sound of your situation, what I'm paying per request isn't  
relevant, since your scenario is so much different than mine.

If you've got to deal with an existing hardware infrastructure, and  
the associated costs that go with that (such as electricity and sys  
admin costs, hardware replacement/redundancy costs, etc.), then your  
cost picture is very different from that of a newly minted internet  
company. :) You've got to factor in *your* costs -- my numbers are  
meaningless.

> It never hurts to go back and review things to ask the question and  
> make
> sure we haven't gone back down the path of 'use it because it's neat'
> rather than for the right reasons.

The right reasons for using any MVC framework are:

- Maintainability

- Common problems solved for you, so your team doesn't need to  
reinvent the wheel. Easy database access/usage, web app security  
issues, etc. -- the list of problems a decent, semi-mature framework  
solves is very long.

- Maintainability

- Third-party support available (time permitting) at no cost, in the  
case of open source frameworks.

- Maintainability

If you've ever been up to your eyeballs in spaghetti code that grew  
out of a non-framework approach, it's easy to see the benefits of a  
well-organized framework. It's not "because it's neat", it's "because  
it's sane."

With the abundance of excellent frameworks out there in nearly every  
language, its insanity to go the roll-your-own route *unless* you're  
committed to using a language that has no framework that suits your  
needs.

I see people do this from time to time. They figure that the overhead  
of the learning curve to A) make the decision on the best framework  
for them and B) learn how to use that framework to their advantage is  
too much to bear, and instead choose to "get coding right away"  
hacking it out from scratch.

Imagine that the guys who hack it out from scratch aren't working for  
you anymore. A legitimate concern if you're dealing in the "long  
lifecycle" world. So those guys who wrote it from scratch are now  
gone, and they didn't document their work very well. Or, they used a  
few techniques that are about to become obsolete in the next version  
of PHP.

Now you've got to bring in a new team. They have to learn all this  
crazy stuff, and there's no roadmap to guide them. Pitfalls are  
everywhere. There's no one they can ask for help who has insight into  
the system, since the old guys are gone.

But hey, it's running on cheap hardware. :)

By using a framework, you're buying into a community of users that  
have built documentation, examples, blog posts, etc -- so your team  
isn't an island unto itself. There are resources who can help. New  
team members have some place to go to get up to speed on the basic  
approaches utilized by the project. Etc., etc.

It's a decision based on wisdom and experience of having done things  
the old/hard way. It's not about "neat" in my opinion.


> Also, not all web applications, even big ones, can be hosted on
> companies who just do that for a living. Some of us have to deal with
> the servers, that setup, configuration, maintenace, power, backup, and
> other issues when thinking about the applications as well. Another
> server is more time and not always trivial either(try dealing with
> security requirements for classifying the server and you'll see the
> 'server' starts to cost more and more).

One thing you haven't mentioned is the number of requests you're  
expecting. Rather than focusing on the percentage difference of a  
framework versus a "Hello World" one-liner and what the relative  
costs are to serve them, think about what kind of load you're  
anticipating.

Paul's benchmarks indicate that the best you can expect with a Solar- 
based application on a single server configured like an EC2 box is  
about 142 requests/second.

142 requests/second. That's 8,520 requests/minute. 12,268,800  
requests/day.

That's some *major* traffic. Are you building something that needs to  
serve 12 million pages a day? Make sure you're framing your concerns  
in a realistic estimate of what your application will need to handle.  
If you're building something that's internal -- sounds like you are,  
if you're concerned with things like classifying a server -- it's  
unlikely that the use of any framework is going to be bottleneck for  
you.

A much more likely source of concern, cost, hardware requirements,  
etc. is data access. Most expenses in scaling web apps comes from  
scaling data storage/retrieval, not in serving the code that hooks  
into that data. Every chokepoint I've seen lately, in my own stuff  
and that of colleagues, has been at the database server, not the web  
application server.

Again, your specific requirements/needs/environment quite clearly  
vary from mine. I just mention these things to suggest that the costs  
of *any* framework (even Symfony (!!)) is likely to be much less than  
the cost of making your data efficiently accessible.

-Clay

--
Killersoft.com