We set out on this mission to uncover how Magento sites in the wild are performing. We have observed that many Magento sites provide a slow end user experience, and that performance issues often are sought solved with various cache methods.

Is Full Page Caching a viable way of solving Magento performance issues?
Will replacing the cache backend improve Magento performance?
Are there other better ways to solve Magento hosting performance issues?

Magento Response Time, Server Latency and Time To First Byte

The response time of websites matter for website visitors, for customers and for conversion and is popularly referred to as the Time to First Byte. Contrary to what most believe, caching is not what determines the performance of your website – or at least not the way you expect it to.

The Time to First Byte (TTFB), or server latency, is the name of the delay starting when you click on a webpage, and until the first byte is received over the network. A low TTFB gives the site visitor the feeling of instant delivery of a web page when something is clicked.

This test is focused on TTFB, and the results do not tell anything about the scalability and max capacity, or the front-end performance of the tested sites. However, a high TTFB tells us, as hosting professionals, that scaling the site will be much harder – and that the performance and experience for the end user is bad.

Common for all the stores we have tested is that they have quite a bit of traffic. Therefore we would expect the sites to be properly set up and perform well when tested for performance.

Magento Performance Test Objective

The main aim of the test is to collect response times, distribution of response times and information about how well full page caching is working for performance. The results will give an interesting perspective on the user experience for store visitors.

We have tested the performance of 15 popular Magento E-commerce sites, hosted by different companies on various locations in Northern Europe.

Wanted behaviour

The following graph is an example that shows the distribution of response times like one would like them to be (KariTraa.com). The majority of requests in the initial test are delivered under 1 second, and a few requests within 2 seconds.

The consecutive cache test shows that full page caching is working for close to all requests, which tells us that sudden high traffic would be dealt with smoothly.

Non-wanted behaviour

The graph below is an example of an equivalent test as above (Hoyer.no), but with results that tell us that users are getting a very bad experience. Only a small percentage of requests are served under 1 second, and the large majority is served in three seconds or more.

The distribution in the cache test also suggest that full page caching is in use, but for the large majority of requests – not working. Sudden high traffic to this site would most likely render it unavailable, because the backend resources would be spent delivering mostly uncached requests.

Results Overview

We have measured the time you have to wait before the browser starts rendering the webpage. This is the time everyone has to wait before any other optimisations come in play.  It does not matter how the front-end is optimised – smaller images, aggregating css and javascript, reducing domain name lookups – this waiting time applies to everyone.

The user experience is what matters for conversion, and we use the following reference values when we measure and evaluate the Time to First Byte:

• Average Response time less than 0.8 seconds generally gives good performance
• Average Response time over 0.8 seconds gives stores a somewhat sluggish feeling
• Average Response time over 1.2 seconds feels sluggish, and risks that people start double-clicking
• Average Response time over 1.5 seconds suggests annoyed and struggling visitors
• Anything over 2 seconds guarantees a poor user experience

How to read and analyse this diagram

• The red response time bars show the response time most visitors will experience
• Only the first four sites have acceptable and consistent average response times below 0.8 seconds
• The blue response time bars may occur for visitors during heavy traffic events, hitting the full page cache
• Response times less than a second for Full Page Cache Tests indicate effective use of Full Page Caching for Scaling (FPC Test response times less than 0.5 seconds should not be compared because network latency is the main source of variation)
• Large absolute differences between the initial run and the confirmation runs suggest that performance randomly varies (Hellyhansen.com, Eplehuset.no, Hoyer.no)

Some extra info on the three Servebolt-hosted sites: YMEUniverse.com, Coverbrands.no and KariTraa.com

• None of these use REDIS or any other alternative caching backends
• Coverbrands.no delivers the second best response times, and does not use full page caching. It is proven to scale to more than 2000 simultaneous users without a cluster setup
• KariTraa.com runs on PHP 5.6. Upgrading the site to PHP 7 would slice of another 35-50% of the average response time

Enklereliv.no is the only non-Servebolt site that delivers super performance in this test. The numbers, and our from-the-outside inspection of the setup suggests that their full page cache is primed at all times.

Chance of Poor Performance

The following graph includes all sites were more than 3.5% of pages on average respond in more than 2 seconds. Site visitors usually visit more than 1 page, but this graph displays the chance per click of hitting a page 2 second+ page.

Full Page Caching Results

These charts include only the sites that effectively implement full page caching for scaling. The blue bars show the general performance.

Coverbrands.no does not use full page caching, but is included to emphasise that it is possible to achieve outstanding performance, without utilising full page caching.

There were more sites that implemented full page caching, but where the amount of requests served from the cache were too few for it to be effective for scaling, these sites are not listed.

Helly Hansen and Stormberg have somewhat sluggish performance, but will scale with traffic. Fjellsport.no is generally slow, and Voorwinden.nl and Eplehuset.no generally deliver a bad user experience – unless visitors are lucky and hit a cached page.

Performance Score

We have made an an index that reflects the TTFB performance random visitors can expect, giving results on a scale between 0 and 100, where 100 is the best possible score. The index consists of the following elements;

• Response time (up to 30 points)
• Cached response time (up to 20 points)
• Chance of hitting < 1s page (up to 30 points)
• Chance of hitting < 2s page (up to 20 points)

Bauhaus.se

Eplehuset.no

HellyHansen.com

Gsport.no

Coverbrands.no

Hoyer.no

Tilbords.no

YMEUniverse.com

KariTraa.com

Stormberg.com

Fjellsport.no

Enklereliv.no

Voorwinden.nl

Pietzoomers.nl

Vita.no

Reflection on Magento and Caching

Caching is a software or hardware component that stores data so future requests for that same data can be served faster. It is vital to speed up computing, and saves resources and time used to access data. Caches are implemented in most layers of a web server software stack, in the file system, database, the web server and web applications.

A common perception is that alternative caching backends and full page caching will solve store performance problems. This is rarely true.

From experience, we know that Magento developers start asking for a faster caching backends like REDIS and a full page cache solutions like Varnish when challenged with performance problems. The general expectation is that installing and configuring this will solve the performance issues – it won’t!

Magento E-commerce and Full Page Caching

This test demonstrates that full page caching usually does not deliver the results developers are looking for to achieve. And here is why;

• The Full Page Cache only caches recently accessed pages
  Most stores have many hundred, or often thousands, of different pages. Full page caches usually do not cache everything, and cache items are frequently expired.
• Site visitors access random pages
  Okay, they usually come by the front page – but the pages a customer vists are in general a random selection. For a full page cache to be effective, the pages need to be ready in the cache when asked for, and if they are not – full page caching is not helping on performance.
• Magento admins clear caches, all the time
  Full Page Caching is contradicting the ordinary day-to-day work flow for Store Owners and Admins. 
• Integrations force cache refreshes, all the time
  Many stores synchronise their products from other sources, and to get changes online – caches are cleared continuously.
  
• Full Page Caching does never apply to backend performance
  Order lists, order management, product management, category management and all other day-to-day tasks in the backend will remain slow and uncached.

Given all these factors, what performance gains can one really expect to get by implementing full page caching? When would the cache work? The answer is that full page caching gives performance improvements only for very frequently accessed pages, and when experiencing traffic spikes. 

The test results suggest that the performance gains for the day-to-day performance are very small, and that full page caching primarily is a tool for scaling.

Alternative Caching Backends – REDIS & MemCached

It is also common to switch out the default file or database cache backend, with REDIS or MemCached. Both of these are excellent applications, but will usually not make much of a difference on performance for the users of the website.

The reason for this is that alternative caching backends only provide faster read and write access to the cache.

The most common problem in Magento is not cache read or write access, but the actual creation of cache elements – which is done in MySQL and PHP. An alternative caching backend will not speed up the time it takes to make cache elements, it will only provide a faster way to save and read data from the cache.

Also, the I/O performance gains from memory based cache backends are generally overstated for Magento. If you have a correctly configured software stack, running on SSD hard drives with enough RAM, the elements will be cached in the file access system – or in RAM anyway.

Test Method

Magento Performance Test Objective

The main aim of the test is to collect response times, distribution of response times and information about how well full page caching is working for performance. The results will give an interesting perspective on the user experience for store visitors.

Tests were conducted during week 49 and 50 2017 from the Netherlands.

Magento Performance Test Method

Data was collected by using Screaming Frog SEO Spider, with a configuration that only fetched the HTML pages (no CSS, jpg, js etc), and prevented testing of pages using filters, and other non-relevant GET parameters.

For the three indexing runs per site, we recorded the average response time for all requests and a table of response times ordered per second.

Run 1 – Initial test
Run indexer for approx 5 minutes (indexing between 350 and 750 pages).

Run 2 – Full Page Cache test
Run approx the same amount of pageviews, directly after executing run 1

Run 3 – Confirmation test
Index an equivalent number of page views, a few hours after Run 1 and 2.

The principle is to test how real users are experiencing a visit to the site. Test Run 1 is expected to provide a fair snapshot of the site’s overall performance. Run 2 should hit the full page cache, if it is in use and correctly configured, and would simulate the performance when experiencing sudden traffic spikes (like sites get when sending newsletters, campaigns etc). For sites using a full page cache we expect the average response time, and distribution to improve. The confirmation test, Run 3, is to confirm that the results from Run 1 are fair, and to observe how sticky the full page cache is.

All tests are run against live production sites, like they are available to everyone on the internet. This means that sites making proper use of caching and having a lot of traffic ought to have an overall upper hand compared to the lower traffic sites, and those not using full page caching.

Summary and Recommendations

We started out on this research project to see how Magento sites in general are performing for the end users, and the results are in line with what we expected. The questions we wanted to answer and enlighten were;

• Is Full Page Caching a viable way of solving Magento performance issues?
• Will replacing the cache backend improve Magento performance?
• Are there other better ways to solve Magento performance issues?

Performance problems for Magento sites are often attempted solved by installing more applications, more extensions and more code. The results demonstrate that this usually is the wrong way to address performance problems. Read the reflection to get insights into why this is so.

Is Full Page Caching a viable way of solving Magento performance issues?

It is possible to achieve good front end performance by implementing full page caching, however only one of the 15 sites tested provided an example of effective use of full page caching for performance. For all stores with average response times higher than 1 second, results show that visitors usually do not hit the cached pages. Also, Full Page Caching will never speed up back-end performance.

Will replacing the cache backend improve Magento performance?

None of the Servebolt hosted sites use alternative caching backends. The reason for that is that cache backend I/O normally is not what is causing performance problems. If cache I/O is the problem, switching to a memory based cache backend will help – but for most Magento sites, the PHP/MySQL performance is what slows the site down.

The only medicine for slow PHP/MySQL is faster and better hosting.

Are there other ways to achieve exceptional Magento performance?

To achieve good performance, our main recommendation is to focus all efforts on speeding up the non-cached performance. This can often prove to be easier said than done, but to sum up the basics:

• Use a fast host with a properly configured OS and application stack
  The standard setups you get on VPSes and cloud services do not deliver good performance.
• Fix all known bugs
  Developers tend to ignore errors in the webserver logs, and Magento logs. Every bug makes your application work slower, and fixing real problems is much harder when error logs are big.
• Test all pages for performance before deploying new code
  When implementing changes, make sure your front, category, product and checkout pages still respond in a timely matter. Also always keep an eye on your back end performance.
• Select extensions to use with care
  A lot of extensions hurt performance. Uninstall extensions you don’t use.
• Test  your Magento with Full Page Caching disabled
  Accept the fact that most users will hit pages that are not cached. Full page caching is for scaling, not for performance. The uncached performance is what you need to keep an eye on.

The reward for working like this will result in better performance, in both the front-end and backend.

When you encounter performance problems, the way of dealing with them is to pinpoint exactly what the problem is. This is usually easiest done by using the Magento Profiler and database query logging.

Elaborate!

Thank you for checking out this study. If you have comments, questions or specific cases you would like to discuss – feel free to participate below, or contact us on chat, phone or by mail!

– The Servebolt Team