Comparing PHP VOMS-Admin and Java VOMS-Admin performance

On testbed server installation of nginx, apache and apache tomcat (just "tomcat" for simplisity) has been performed. Standard JAVA-based VOMS-Admin v.2.0.18 installed an configured to serve VO "testbed" with MySQL backend for database. VOMS-Admin naturally served with tomcat and referenced in tests below as tomcat. PHP VOMS-Admin (PVA) v.0.3 has been installed to serve the same VO "testbed" using the same database. There are two instances of PVA - referenced below with the same name as the frontend that it is served by - apache and nginx correspondingly.

Testbed for tesing was testbed32.cluster.univ.kiev.ua machine in internal KNU Cluster network with the following characteristics:

Intel SE7501HG2
2
3.06GHz
Intel(R) Xeon(TM) Family 15 Model 2
1024 KB
2 GB

Main characteristic that was investigated - the server response time. To determine response time we have used cURL tool. This was due to requirement for test suite to support clien X.509 certificate authorization. Standard web-server benchmarking tools (such as Apache Benchmark) does not support this option, but curl does and provite mechanism for time mesurement (with 1ms accuracy). You may find our test suite here.

Following you can find graphs that represent response time distribution for each solution. Why distribution? Measurement of response time demonstrate a huge results dispersion. We proceeded with 1000 and 10000 measurement cycles getting the same distribution results. Measurements werw performed with different number of concurrent connections (represented by different colours), modelling the situation in which many resources has the same default cron time without jitter.

The first series of tests performed on empty VOMS database:

apache with empty db nginx with empty db tomcat with empty db

The result of PHP VOMS-Admin: peak of apache distribution is 1-2 ms, nginx is sligthly wider and most probable value 3 ms. Java VOMS-Admin on tomcat showing the most wide range of values and peak time - 17 ms

The reason why apache is faster than nginx is that apache inteprete PHP in the same process as opposed to nginx using socket to PHP-FPM, and static content absence (where nginx is much faster!). Java VOMS-Admin is definitely slower 6-10 times on empty database, first of all because using Java for concerned tasks is defenitely slow.

The next series of tests performed on database with 1000 VOMS-users:

apache with 1000 users nginx with 1000 users tomcat with 1000 users

PVA on apache has the same good results on 1000 users due to execution in the same process and apache's internal caching mechanisms (1-2 ms !!!). PVA on nginx show slightly worse results (7-9 ms), but still good enough to fullfit all the distribution in 50-100 ms in the worse situation with concurent connections. As we can see, increasing number of concurent connections almost not beat apache performance, but beat nginx near 10 times.

And what we are able to see about Java VOMS-Admin performance during our investigation - performance downgrading to 47-55 ms when using only one connection at time, and unpredictably slow results when concurrent connections occur. Server response time increase to more than 100 ms in the best case! On the production database PVA is at least 30 times faster!

Following graph represent the percent of connections to Java VOMS-Admin on tomcat, that have server response time bigger than 1 second with concurent connections increasing:

tomcat with empty db

Performance downgrade may be compared by looking at the following graphs, that contain distibutions for all 3 solutions same time. One for empty db, and second one for 1000 users.

25 concurent (empty db) 25 concurent (1000 users)

Comparing this different solutions response time we get the following conclusions:

  • PHP VOMS-Admin running apache backend have response time 1-2 ms even if 50 concurrent connections are made;
  • PHP VOMS-Admin performance degradation was not inspected on apache when number of users in database increase;
  • PHP VOMS-Admin running nginx backend have response time near 5-10 ms and slightly degraded with increasing concurent connections (worst result near 100 ms);
  • PHP VOMS-Admin performance degradation on nginx is significant due to PHP in another process, but still remain worst results near 100-200 ms;
  • Java VOMS-Admin on tomcat backend have response time up to 50 ms without concurrent connections;
  • Java VOMS-Admin performance degradation on increasing users number is dramatical, especially when concurent connections are made and response time grow upward of 1 second!

And the last one expectation is about resource usage. With Apache Tomcat/5.5.27 on JDK 1.6.0_16 (one WepAPP for VO testbed running) memory utilization was:

  • Just after running: approximately 800MB VmSize including 180MB VmRSS
  • Under the tests workload: approximately 830MB VmSize including 300MB VmRSS

Comparing to PHP VOMS-Admin solution this is beyond belief:

  • Apache: approximately 90MB VmSize including 30MB VmRSS per worker process even on workload
  • Nginx: approximately 12MB VmSize including 4MB VmRSS + PHP-FPM: approximately 70MB VmSize including 6MB VmRSS pre worker process

And this resource usage is just only for single VO!!! To serve several VOs you must have separate Java WebApp threads for each VO, when using Java VOMS-Admin solution, thus even more increasing memory and processor usage. Opposite to this, PHP VOMS-Admin may serve hundreeds of VOs, and does not increase memory usage, just working with the same PHP application that utilize different databases.

PHP VOMS-Admin is many times faster and have comparely insignificant resourse requirements! The reason why PVA much better is just because using accountable approach when solving such simple kind of tasks.