Magento Performance Test Objective
The main aim of the test is to collect response times, distribution of response times and information about how well full page caching is working for performance. The results will give an interesting perspective on the user experience for store visitors.
Tests were conducted during week 49 and 50 2017 from the Netherlands.
Magento Performance Test Method
Data was collected by using Screaming Frog SEO Spider, with a configuration that only fetched the HTML pages (no CSS, jpg, js etc), and prevented testing of pages using filters, and other non-relevant GET parameters.
For the three indexing runs per site, we recorded the average response time for all requests and a table of response times ordered per second.
Run 1 – Initial test
Run indexer for approx 5 minutes (indexing between 350 and 750 pages).
Run 2 – Full Page Cache test
Run approx the same amount of pageviews, directly after executing run 1
Run 3 – Confirmation test
Index an equivalent number of page views, a few hours after Run 1 and 2.
The principle is to test how real users are experiencing a visit to the site. Test Run 1 is expected to provide a fair snapshot of the site’s overall performance. Run 2 should hit the full page cache, if it is in use and correctly configured, and would simulate the performance when experiencing sudden traffic spikes (like sites get when sending newsletters, campaigns etc). For sites using a full page cache we expect the average response time, and distribution to improve. The confirmation test, Run 3, is to confirm that the results from Run 1 are fair, and to observe how sticky the full page cache is.
All tests are run against live production sites, like they are available to everyone on the internet. This means that sites making proper use of caching and having a lot of traffic ought to have an overall upper hand compared to the lower traffic sites, and those not using full page caching.