Direction of Zend Framework?
Paul Jones was so gracious as to mention my name in a presentation he gave in Atlanta, a couple of my coworkers where there but I wasn't able to make it, instead I am going to the ZendCon.
Anyway since it was about benchmarking it made me think back to my orignal tests where the Zend Framework came out around 18% the performance of a plain html and Paul claimed Solar was 4 times faster then ZF.
Why test against plain html or plain php? To get a baseline of the fastest possible performance you can get. It can give you a reasonable goal, for example if a plain page gets 100rps and a php page gets 50rps you know there is room up to 50rps.. What you have to do is decide what your reasonable goal is. Its sort of like weight loss, If you set unreasonable goals your going to lose. If you set a goal of 10% or 5rps improvement thats a goal that you can reach.
So Zend Framework is now at 1.0.1, what did benchmarking find? The goal is to just output Hello World with the most basic set of files possible for each case.
ab -c 10 -t 60 http://zend.cyberlot.net/
Plain HTML
------------------------------------------
Concurrency Level: 10
Time taken for tests: 14.553157 seconds
Complete requests: 50000
Requests per second: 3435.68 [#/sec] (mean)
Time per request: 2.911 [ms] (mean)
Time per request: 0.291 [ms] (mean, across all concurrent requests)
Transfer rate: 915.95 [Kbytes/sec] received
Pure PHP
------------------------------------------
Concurrency Level: 10
Time taken for tests: 17.330895 seconds
Complete requests: 50000
Requests per second: 2885.02 [#/sec] (mean)
Time per request: 3.466 [ms] (mean)
Time per request: 0.347 [ms] (mean, across all concurrent requests)
Transfer rate: 571.93 [Kbytes/sec] received
Zend Framework 1.0.1
-------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.13438 seconds
Complete requests: 3633
Requests per second: 60.54 [#/sec] (mean)
Time per request: 165.190 [ms] (mean)
Time per request: 16.519 [ms] (mean, across all concurrent requests)
Transfer rate: 12.00 [Kbytes/sec] received
That sucked pretty bad had to figure out why, I know they have added a lot but not that much. Well what about our original tests.
Zend Framework 0.2.0
----------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.39417 seconds
Complete requests: 6420
Requests per second: 106.93 [#/sec] (mean)
Time per request: 93.519 [ms] (mean)
Time per request: 9.352 [ms] (mean, across all concurrent requests)
Transfer rate: 21.20 [Kbytes/sec] received
Better but still pretty poor compared to my old results, the basic php and html results are faster then before, framework is slower?
APC is whats different, a recent recompile left my apc disabled, Fixed this and reran
Zend Framework 1.0.1
------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.7181 seconds
Complete requests: 13347
Requests per second: 222.42 [#/sec] (mean)
Time per request: 44.959 [ms] (mean)
Time per request: 4.496 [ms] (mean, across all concurrent requests)
Transfer rate: 44.09 [Kbytes/sec] received
Zend Framework 0.2.0
-------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.119 seconds
Complete requests: 22109
Requests per second: 368.48 [#/sec] (mean)
Time per request: 27.138 [ms] (mean)
Time per request: 2.714 [ms] (mean, across all concurrent requests)
Transfer rate: 73.05 [Kbytes/sec] received
What does this tell you? If your not running apc, zend or another package your shooting yourself in the foot.
It also shows a nice a pretty unhealthy drop between 0.2.0 and 1.0.1 for something as simple as "Hello World". While it took me fewer lines and is easier to setup is it worth it?
A nod to Paul
Solar 0.28
--------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 60.080 seconds
Complete requests: 18841
Requests per second: 314.02 [#/sec] (mean)
Time per request: 31.845 [ms] (mean)
Time per request: 3.185 [ms] (mean, across all concurrent requests)
Transfer rate: 122.35 [Kbytes/sec] received
Its late and I didn't have time to customize Solar, It has a "hello-mini" built in so I used that and all the default settings based on Getting Started. Hats off to Paul, back in the 0.2.0 days the whole reason I ran my benchmarks is I couldn't understand why his results where so slow and found ZF to be just as fast but now Solar has a clear lead.
"Cyberlot Framework" Alpha Stripped down Zend Framework + Savant 3 (Can't help it I just like pauls work)
-----------------------------------------------------------
Concurrency Level: 10
Time taken for tests: 34.435357 seconds
Complete requests: 50000
Requests per second: 1452.00 [#/sec] (mean)
Time per request: 6.887 [ms] (mean)
Time per request: 0.689 [ms] (mean, across all concurrent requests)
Transfer rate: 287.84 [Kbytes/sec] received
Of course my controller meets my needs and doesn't include the kitchen sink ;) But it does what I need.. just food for thought, Yes a Framework will speed up development but if specific application code will be faster.
What would I change in the Zend Framework? For development all the checks and exceptions they have are great and make debugging easier but in the real world for example my class layout and files are already decided and not moving, I don't need the Framework to check 5 different ways to make sure the file and class exists! By making these checks optional its possible to double the speed of the loader for example.
Anyway thats all for tonight, Later.
Try to remove
The Zend Framework uses **waayyyy** too many require_once() calls. Just write a tool to remove all of those from the framework and use __autoload() to find them instead. Your ZF app will go much faster, especially at high concurrency levels. It beats me why the ZF guys still use require_once()...
Thijs,
www.startinchina.com
Reflection after a nights sleep.
After a night of sleep I would like to expand on something real quick.
During development its great to have detailed errors that tell me right away that I did something stupid, For example ZF will let me know that it found file Class1.php but there was no Class1 class inside. Thats great, development in ZF is extremely user friendly and easy.
That same check though is the problem, when I launch my code Class1.php has Class1 class inside it, Its not moving, its not changing and well to put it all out there, if it does "I SCREWED UP" and there are very likely worse problems then a single file missing.
To me Frameworks need to have 2 frames of mind, because they exist with 2 real purposes and 2 life cycles, Development and Production. Both of these life cycles have vastly different needs and a Framework has to be flexible enough to meet both these needs otherwise its either hard to develop in, or its going to be used for Prototyping only.
Yes meeting that goal is high task but its a task worth approaching in my mind.
Performance counterpoint
While intuitively I want to agree that Zend Framework seems like way too much of a resource hog to be in production for a site with even moderate amounts of traffic, I read an article earlier today from Joel Spolsky about performance vs. features which was good food for thought (http://www.joelonsoftware.com/items/2007/09/18.html).
Based on that read, I tend to think in the immediate future of ZF should concentrate on ease of development, but of course, there needs to be balance in the longer run.
I agree that it would be excellent to have two versions of the framework. Perhaps something like the Smarty "compiler" would be a good approach for production use, where code generation is used to streamline the original ZF source files.
How about Cachegrind?
Have you tried using xdebug and looking at the cachegrind output to find where the bottlenecks are? This speed drop does bring some serious concerns and I know I would like to find the points in the Zend framework that are actually causing the slowdowns...
I enabled xdebug
I enabled xdebug
http://zend.cyberlot.net/ for anyone that wants to look at the output, loadFile loadClass _includeFile rule the roost
Feels like Assertions
I agree with your assessment -- these checks are invaluable during development and initial QA, but once your code has stabilized, they're superfluous and just waste time and resources. These checks would be probably be implemented as assertions in other languages, which would be compiled away in production builds.
What would you think about retrofitting most of the ZF plumbing (controller, dispatcher, action base classes, etc.) so these checks would behave more like assertions which could easily be disabled? Something like:
if (! isset(ZEND_DISABLE_ASSERTIONS)) {
// do assertion-style checking
}
// do important code
And then in my bootstrap file, I could globally disable these expensive checks with a one-liner:
define(ZEND_DISABLE_ASSERTIONS, 1);
Sure, there would be a marginal bit of overhead with all of the isset() checks, but they're language constructs, so they're really fast. The difference for development environments would probably be hard to measure, but for production, it should make a noticeable difference.
Thoughts?
Can't Wait
I think the idea that Jester suggested to have the option to disable all the slow memory intensive aspects of the Zend Framework for production is a great idea.
Has anyone seen a solution that does this? Has anyone heard if the Framework has any plans to add it to the next release?
I agree, thats what I have
I agree, thats what I have played with in my stripped down version except I right now I have DEBUG enabled or disabled in a config I load in the registry, Regardless of how its done it needs to be done.