Rolling updates: What does it mean?

Rolling Updates: The What & Why

We do slow rolling updates for all the servers in operation. In most simplest terms this means only some of the servers are updated at a single time.

Updating process depends on how major an update is and how critical it is. If it’s just a tiny update and most servers are already on the latest major revision we might do a big bulk update of most of the servers. On the other hand, if it’s a major update then we do only a small number first.

It all boils down to Quality Assurance (QA)

Small updates have no, or very small chance of regressions, ie. new bugs introduced to the system, while major updates have a big chance. This means on big updates we need to be careful and slow on the rollout.

Same reason is why we try to avoid doing major updates too frequently. There is always something that breaks when you do an major update. For example, we lately updated ruTorrent and this has resulted in hundreds upon hundreds of tickets, and even some service cancellations. Reason? Despite our best efforts to force browsers to revise their cache, most stick to the cached versions resulting in ruTorrent javascript failing. Despite using every method to tell the browser update the jquery javascript library.

QA is only as good as the testers

No developer can achieve full QA themselves, there is simply too varied use of any system. With slow rollout we gain invaluable feedback and catch potential usability issues before it is affected by the vast majority.

Every user is different, that’s why the same person developing the software may not notice even the most obvious issues. What works on Windows/Chrome combination may not work on Ubuntu/Firefox.

Backwards compatibility

Our software backdates to when Debian 5 was new and fancy. There are still servers out there running Debian 5, despite, support for Deb 5 was dropped a long time ago. Same goes for Debian 6, and soon Debian 7 will move to the history as well. So far, there’s a lot of things which needs to be tweaked for Debian 8 before it becomes supported, and at the same time we need to also support Debian 6 and 7.

This means we need to support a variety of versions, configurations etc. out of the box. Quite simply, it is impossible to update every and single server at the same time for Debian 8 – Especially those out of our control but using our software package.