[Solar-talk] Framework Benchmarks

[Kilbride, James P.]

I had a rather lenthy response planned and realized.. You know, it
really doesn't matter or even necessary belong here. I was simply
curious and doing a little bit 'talking' out loud on the list about
thoughts that were popping into my head from a program management
viewpoint. Good to see some thought by people on it though. Even if my
costs were orders of magnitude higher, they still run out to be
relatively low. I've never done the math, I wondered if anybody else had
and realized it was stupid to waste their time worrying.

And with that. Back to poking solar until it squeaks. Trying to decide
if it's prettier than Zend. At least it's got a cuter lord and master,
who uses lolcats in his presentations, since Zend doesn't really seem to
have a talking head to put to it.

James Kilbride

-----Original Message-----
From: solar-talk-bounces at lists.solarphp.com
[mailto:solar-talk-bounces at lists.solarphp.com] On Behalf Of Clay
Loveless
Sent: Wednesday, October 03, 2007 1:14 PM
To: solar-talk at lists.solarphp.com
Subject: Re: [Solar-talk] Framework Benchmarks


On Oct 3, 2007, at 7:31 AM, Kilbride, James P. wrote:

> Clay, Ray,
>
> I am a project manager and do program management activities. Last 
> thing I want to have is a senior program manager ask me how much extra

> we had to pay in servers because we are using a framework and not have

> 'real'
> numbers about cost. The navy doesn't simply accept, "It's commodity 
> don't worry about it."

Ah, I didn't realize this was a military-related question. In that case,
the answer would be "That question is regarding concerns that are above
your pay grade, soldier." :)

Seriously, it's not the job of the developer to worry about the cost of
the hardware. They just need to know what the constraints are that the
hardware puts on them, and write good code. Only in a very, very small
shop is it *also* the developer's job to worry about the monthly server
fees.

> You've made the statement that 'server hardware is a commodity 
> resource but I keep seeing things talking about the cost of power, 
> administrators and other things similar to this so I wondered(as would

> program
> managers) if that's as true as we assume it to be. Especially with 
> long lifetime applications. They've shown that software costs are 80% 
> in the maintenance phase of software, which is why frameworks are 
> considered good because they reduce maintenance costs but what about 
> the cost of actually running the application? Is it really as low as 
> we have always assumed it to be? Your comments about the EC2 boxes is 
> well taken and that's kind of what I wanted to get a feel for, so you 
> are paying around 400-600 a month in hardware costs, how much are you 
> paying per request?
> Should be easy to figure out right? Doesn't sound too bad and like it 
> should be an issue but WITHOUT numbers like that we can't simply 
> assume that it works out okay because it feels right.

 From the sound of your situation, what I'm paying per request isn't
relevant, since your scenario is so much different than mine.

If you've got to deal with an existing hardware infrastructure, and the
associated costs that go with that (such as electricity and sys admin
costs, hardware replacement/redundancy costs, etc.), then your cost
picture is very different from that of a newly minted internet company.
:) You've got to factor in *your* costs -- my numbers are meaningless.

> It never hurts to go back and review things to ask the question and 
> make sure we haven't gone back down the path of 'use it because it's 
> neat'
> rather than for the right reasons.

The right reasons for using any MVC framework are:

- Maintainability

- Common problems solved for you, so your team doesn't need to reinvent
the wheel. Easy database access/usage, web app security issues, etc. --
the list of problems a decent, semi-mature framework solves is very
long.

- Maintainability

- Third-party support available (time permitting) at no cost, in the
case of open source frameworks.

- Maintainability

If you've ever been up to your eyeballs in spaghetti code that grew out
of a non-framework approach, it's easy to see the benefits of a
well-organized framework. It's not "because it's neat", it's "because
it's sane."

With the abundance of excellent frameworks out there in nearly every
language, its insanity to go the roll-your-own route *unless* you're
committed to using a language that has no framework that suits your
needs.

I see people do this from time to time. They figure that the overhead of
the learning curve to A) make the decision on the best framework for
them and B) learn how to use that framework to their advantage is too
much to bear, and instead choose to "get coding right away"  
hacking it out from scratch.

Imagine that the guys who hack it out from scratch aren't working for
you anymore. A legitimate concern if you're dealing in the "long
lifecycle" world. So those guys who wrote it from scratch are now gone,
and they didn't document their work very well. Or, they used a few
techniques that are about to become obsolete in the next version of PHP.

Now you've got to bring in a new team. They have to learn all this crazy
stuff, and there's no roadmap to guide them. Pitfalls are everywhere.
There's no one they can ask for help who has insight into the system,
since the old guys are gone.

But hey, it's running on cheap hardware. :)

By using a framework, you're buying into a community of users that have
built documentation, examples, blog posts, etc -- so your team isn't an
island unto itself. There are resources who can help. New team members
have some place to go to get up to speed on the basic approaches
utilized by the project. Etc., etc.

It's a decision based on wisdom and experience of having done things the
old/hard way. It's not about "neat" in my opinion.


> Also, not all web applications, even big ones, can be hosted on 
> companies who just do that for a living. Some of us have to deal with 
> the servers, that setup, configuration, maintenace, power, backup, and

> other issues when thinking about the applications as well. Another 
> server is more time and not always trivial either(try dealing with 
> security requirements for classifying the server and you'll see the 
> 'server' starts to cost more and more).

One thing you haven't mentioned is the number of requests you're
expecting. Rather than focusing on the percentage difference of a
framework versus a "Hello World" one-liner and what the relative costs
are to serve them, think about what kind of load you're anticipating.

Paul's benchmarks indicate that the best you can expect with a Solar-
based application on a single server configured like an EC2 box is about
142 requests/second.

142 requests/second. That's 8,520 requests/minute. 12,268,800
requests/day.

That's some *major* traffic. Are you building something that needs to
serve 12 million pages a day? Make sure you're framing your concerns in
a realistic estimate of what your application will need to handle.  
If you're building something that's internal -- sounds like you are, if
you're concerned with things like classifying a server -- it's unlikely
that the use of any framework is going to be bottleneck for you.

A much more likely source of concern, cost, hardware requirements, etc.
is data access. Most expenses in scaling web apps comes from scaling
data storage/retrieval, not in serving the code that hooks into that
data. Every chokepoint I've seen lately, in my own stuff and that of
colleagues, has been at the database server, not the web application
server.

Again, your specific requirements/needs/environment quite clearly vary
from mine. I just mention these things to suggest that the costs of
*any* framework (even Symfony (!!)) is likely to be much less than the
cost of making your data efficiently accessible.

-Clay

--
Killersoft.com



_______________________________________________
Solar-talk mailing list
Solar-talk at lists.solarphp.com
http://mailman-mail3.webfaction.com/listinfo/solar-talk