Rolling Updates: The What & Why
We do slow rolling updates for all the servers in operation. In most simplest terms this means only some of the servers are updated at a single time.
Updating process depends on how major an update is and how critical it is. If it’s just a tiny update and most servers are already on the latest major revision we might do a big bulk update of most of the servers. On the other hand, if it’s a major update then we do only a small number first.
It all boils down to Quality Assurance (QA)
Small updates have no, or very small chance of regressions, ie. new bugs introduced to the system, while major updates have a big chance. This means on big updates we need to be careful and slow on the rollout.
QA is only as good as the testers
No developer can achieve full QA themselves, there is simply too varied use of any system. With slow rollout we gain invaluable feedback and catch potential usability issues before it is affected by the vast majority.
Every user is different, that’s why the same person developing the software may not notice even the most obvious issues. What works on Windows/Chrome combination may not work on Ubuntu/Firefox.
Our software backdates to when Debian 5 was new and fancy. There are still servers out there running Debian 5, despite, support for Deb 5 was dropped a long time ago. Same goes for Debian 6, and soon Debian 7 will move to the history as well. So far, there’s a lot of things which needs to be tweaked for Debian 8 before it becomes supported, and at the same time we need to also support Debian 6 and 7.
This means we need to support a variety of versions, configurations etc. out of the box. Quite simply, it is impossible to update every and single server at the same time for Debian 8 – Especially those out of our control but using our software package.