Divante.com Blog

Magento platform is known as well-designed piece of software. Elastic architecture gives engineers opprtunities to build almost every kind of transactional platform based on this framework.

Core developers of Magento choosen mechanism like EAV (Entity – Attribute -Value) structure of databse, data indexation, XML configs. They also build very deep objects tree to avoid constraints of relational databases (MySQL in this case) and allow full-OOP approach.

That’s true – you could add everything and change everything in Magento without changing core-code. You could use observers, modules, change admin panel panes and so on. Which means – Your code will be easy upgradeable and loose-coupled.

The main weakness is the main strength – which is: complexity. In many cases, as Your store will grow You’ll see that probably some optimizations have to be done to perform better and serve more customers.

Last year we’ve worked on few technical-demanding cases for optimizations and scalability. I thought of describing few techniques we used to show what to do when Your store meet performance problems.

Case

Most interesting case we have on some B2B platform. In our case B2B means mainly – non-standard customer traffic habits. Customers log on morning, all day collects products to basket (in some cases 200-300 positions per quote!) and then – save order. In database we have – many or not so many ? – 37 800 stock items. That’s was not the main problem. Main problem was count of attributes of products – about 5500. In our case this causes that Magento “flat tables” mechanism stop works (Inno DB limitations in MySQL allows us to create tables which only about 1000 columns).

Without flat tables – Magento EAV mechanism work very, very slowly. In this deployment also non standard was integration architecture. Prices are computed in CRM system in almost-realtime and via WebServices send to store. Baskets are also in realtime send to CRM and back. This causes few houndrets WebService calls per one customer visit.

The problem

For some time – when user traffic was not so high store works well. We’ve of course done almost everything what is recommended at start for optimizations of Magento. Thanks to our prior experiences and to Ivan Chempurnyi great presentation we have done at start:

  • code was optimized – we were using collections and models properly (according to Ivan’s recommendations), we use caching where it was possible,
  • we have separate database server (which after all was not so great idea – but about this later)<
  • we’ve used SSD disks for database and application servers,
  • we’ve tunned MySQL (InnoDB buffers, IO buffers in Operating System),
  • we’ve tunned app server (apache2 + FPM).

We couldn’t use flat tables and we’ve a lot of writes (B2B traffic habits).

First of all – know Your enemy/bottlenecks. Closely.

To see that site is running rather slow is easy. But what causes problem? I was always fan of principle that before any change You’ve to have evidences that change give You any benefit.

Key is to measure and profile software and hardware. You could use following tools:

  • Collectd – installed on all servers. Using Collectd we discovered bottlenecks – I/O, database locks (on MySQL) on indexing Magento data,
  • Logs – we recorded duration of all web-service calls and queue execution times (we use Gearman queues to perform Web-Services communication). We save also stack-traces in logs to see where – exactly bottlenecks are hidden,
  • htop, iotop – linux commands used to check what generates CPU and I/O load – what process (which could be debuged later),
  • Xdebug  – a PHP debugger (integrated into many IDE’s – for example phpStorm that we’re using ). We use it on test-servers to profile our oucde and find bottlenecks.
  • JMeter, siege – stress-test tools to check performance after and before changes. Those tools give You evidence if after changes applications is performing better or not.
  • New Relic – is very complex tool – which also easy interface and integration. You put newrelic-agent on Your servers and compile extension for PHP. After that in one-simple-dashboard You get everything. Slow queries, slow piece of code (which stack-traces and detailed statistics), alerts and so on. New Reluc could work on production servers (we’re using it on them) because of very low overhead and high stability.

That’s all for now. In next post we show You what we’ve optimized on Application and then Database layer of Magento.

 

Share your comment

Comments

  1. Avatar
  2. Avatar

    Reply

    Please clarify, what the Logs are ?
    How to did you setup logging of all queries to troubleshoot bottlenecks?
    Thanks!

  3. Avatar

    Reply

    You could setup slow queries log in MySQL. We’ve added programmaticaly logging to piece of application code, where we were expecting bottlenecks.