At LRobwe are prioritizing three objectives: safety, availability, performance. And that's why our update policy is deliberate. simple, readable, repeatable... and above all frequent.
We are convinced that up-to-date servers protect against attacks and enable more sustainable infrastructure.
At LRob, we don't beat around the bush, either in practice or in explanation. So hang in there and find out how we keep our Linux server park clean and free of surprises.
⚠️ Please note: This article is our example, our opinion, and not an absolute truth or a guide to be blindly copied for any company. Every company is different, your choices are your own and LRob cannot be held responsible for the consequences of your choices. Also, some statements may be difficult to bear in cases of cognitive dissonance - we hope this won't be too difficult to endure.
Contents
Prerequisite: making the right choices upstream for optimal maintenance
Choosing a future-proof Linux distribution: Debian
Let's be clear: a policy of upgrading to the wrong OS is pointless. The OS must be the most stable and predictable possible, including when applying updates. So choose wisely.
Our choice won't be unanimous, but it's definitely a safe bet. We think it's the safest distribution of all: We standardize our servers on Debian.
Why? For its simplicityhis stabilityhis predictability and its community governance. Debian is a sober basewho was able to show reliable over the long term. Debian also allows major distribution version upgrades, which can be useful, although we prefer reinstallation on a "fresh", newer and more powerful server. At LRob, major releases are seen as an opportunity to update the installed base.
We believe that Debian is much more reliable for production than its "forks" (derivatives) like Ubuntu, whose policy seems less stable to us (as do the latest versions of the packages often offered), or even than some pay distributions whose pricing policy can change and hold you prisoner, ruining, in our view, much of the appeal of Linux.
Because Debian also has the advantage of being entirely free and open-source and therefore free of charge. The money saved can then be used for maintain properly its fleet in-house... Or if your structure is big enough, why not go as far as improving Debian and Linux. Because "open-source" also means "community", and that's never stopped private individuals from improving it for themselves and others. It also provides visibility and a positive image. It's a win-win situation.
Of course, standardization is a key factor in efficient maintenance. We therefore believe that the chosen OS should be standardized across almost all servers.
Software and hardware lifecycles and durations
At LRob, a server does not remain in production beyond 5 years (OS and/or hardware). Our forecasts even include an average of 2 to 3 yearsThis average is only possible because we lease the servers... So we're not prisoners of a 5-year amortization period like many others. Think about it too, to get a technological lead.
This means that the duration of an "LTS" is generally not decisive for us, particularly with regard to the long lifespan of recent Debian releases (~10 years).
A server is replaced according to :
- visit major releases OS and kernel, and their ease of upgrading Linux distributions,
- visit hardware evolutions (CPU, Storage, RAM) and available prices,
- visit requirements of the load (CPU, storage, RAM, here too).
Two options:
- OS upgrade when relevant;
- Often, clean reinstallation on newer material to stay in the top performance.
Please note: The servers we stop using are then reused by other, less demanding users. The quest for performance should not prevent us from being eco-responsible. If you own your equipment and wish to renew it more regularly, there are second-hand resellers available.
Avoid technical debt with simple architectures
Multiplying software dependenciesis as much risks that one of them becomes incompatibleno maintainedor requires the creation of a new repository, or plant to update.
When designing your software, you must therefore choose reliable, time-tested technologies. Don't give in to the first fashionable language or framework that may no longer be maintained. So that the servers running this software are perennialcan be updated efficiently, with their applications, so that you don't have to redo everything in 2 years' time...
Because we all know how a company reacts when everything becomes impossible to update: it stops updating altogether, thus dramatically increasing its technical debt.
Beyond the server aspect, you also need to be ready to maintain your applications to keep up with version upgrades. If only for PHP version upgrades. But the same applies to MySQL/MariaDB, NodeJS and so on.
Ultimately, apply the principles of "KISS": "Keep It Stupid Simple".
Maintenance policy
Scary update many system administrators. Entire teams make endless, pointless meetings to plan each package update... And in the end lose weeks, even monthswith late versions piling up and security holes that slip through. Linux, in our experience, this method is absolutely counter-productive and a much more down-to-earth approach is much more relevant.
Basic principles of good maintenance
In the Linux environment, we have made three simple observations:
- Keeping up to date improves security by correcting vulnerabilities discovered over time.
- Infrequent updating means many changes during updates, increasing the risk of multi-causal bugs, which are much harder to understand and correct than a single bug.
- More frequent updating reduces the number of simultaneous changes, decreases the number of potential bugs and greatly simplifies their resolution.
Consequently, we deduce the following principle, which seems obvious to us:
Frequent updating is safer, more stable and more worry-free.
We therefore categorically believe that delaying updates is a double fault: strategic and technical.
Incremental approach
The idea: Go for it straightforwardly, but surely. Yes, it's antinomic, but understandable:
We update a first serverWe then observe any changes and adjustments that may need to be made. We can then de repeat the procedure on the rest of the park... Which is normally ISO (identical) if no disparate choices were made at the design stage, and if monitoring is correct.
Process foreseeable, reproducible, documented. In short, the definition of efficiency.
Invalid reasons to delay updates
Every day, we see fleets that are insufficiently maintained, sometimes within large companies.
The result: every day, companies of all sizes are hacked as a result of known security flaws.
The cause? Barriers to updating. Excessive caution and precaution, leading to inaction. When it comes to maintaining a proper level of security, how far should caution go? Not too far, of course: the priority should always be to secure.
What's worse? The reasons given are always the same... Here is our top list of those famous phrases we don't want to hear anymore, as well as the answers to push for real action:
- It works, so we don't touch
- -> If you think like that, women still wouldn't have the right to vote.
- Update may cause bugs
- -> We'll take as long as it takes to resolve them.
- Update will cause downtime
- -> Plan this maintenance and downtime, and no one will hold it against you.
- The update is not compatible with our technical debt
- -> Let's fix or redo this old app' without delay.
- You can't back out of a problem easily
- -> Learn how to downgrade a system package.
- We don't have backups, or they can't be restored easily enough.
- -> Take immediate action and draw up a disaster recovery plan.
Lessons learned:
- Excessive "caution" regarding security updates becomes a risk.
- A functional risk is better than a security risk... A hack is a stain.
- Knowing the risks and doing nothing has a name: irresponsibility.
- Seeing problems is good, finding solutions is better.
If that wasn't clear enough, King Arthur has a message for you:
Update frequency: automatic vs. manual, standby, major decisions
As we've seen, frequent updating solves a number of problems in its own right.
The update frequency should therefore be : as often as possible. However, there is a balance to avoid spending a lifetime on the subject, and in the end really save time, security and peace of mind.
In concrete terms, here's how LRob works to achieve the best compromise between safety, time spent and reliability.
- Every night on automatic : safe" updates small application servers are configured with
unattended-upgradeswhile Plesk web servers perform safe upgrades automatically. - Every 7 days max: Reading of changelogs of the main software used.
- Each month, manually : checkup server by server. First check the list of changes, then apply, clean up and decide whether or not to reboot for Kernel Updates.
- Every 1 to 6 months: assessment and planning of major version changes (PHP, MySQL) which have a specific update or addition process.
- 24/7 monitoringalert and immediate intervention if necessary.
The result: almost no functional changes or bugs.
About 1x/year, we come across a "breaking change": each time, it's a minor fix, like removing or adapting an obsolete config line.
And also 1x/year, a service doesn't restart correctly after an update... 24/7 monitoring is also used for this.
The $1 million question: Would you rather take emergency action for 5 minutes once a year, or spend hours, weeks or months planning overdue updates or doing them needlessly by hand?
For LRob, the answer is obvious: simplify, automate, control, monitor.
The manual upgrade command : apt (and apt-get)
A server can of course be updated from a terminal in most cases.
Under Debian, we use apt (Advanced Package Tool) for set up, update and delete software.
NB: The apt has gradually replaced apt-get.
Our standard sequence is done in 1x so as not to forget anything:
apt update && apt upgrade && apt autoremove && uptime && uname -aapt updaterefreshes the list of packages. On reads what will change (release notes, sensitive packages) to detect any breaking change.apt upgrade: applies updates, since everything is fine 99.99% of the time.apt autoremovecleans up old kernels and other dependencies no longer needed.uptimetime since last restart.uname -atechnical identity (kernel, architecture, etc.).
Visit && means : the sequence is executed only if the previous step was successful.
Fewer chain errors, more control.
Reboot or not reboot?
Unless the OS crashes completely, there's never any need to reboot a Linux machine... Except for upgrading the kernel, if you're not using KernelCare or similar for a hot upgrade. For the rest, all OS services can be restarted independently and all configurations are editable. So if you think you need a reboot, it's probably because you haven't found the right program to restart... Or would you like to update kernel.
Thus, without KernelCare, depending on the kernel in place, changelogs on new kernels, and in particular the security patchesand from uptime, you decide whether or not to reboot the machine and switch to the new kernel.
So should we reboot or not? Here's an example of our decision scale:
- On a non-critical or redundant VPS server like a DNS → The reboot takes a few seconds, so we reboot every time a new kernel is available.
- On a dedicated web server → The reboot takes ~8 minutes, so it's best done for a major upgrade or security patch. And above all: at night for less disturbance.
- Server not rebooted for +6 months → The new kernel surely has things to bring us, we'll plan the reboot.
What you gain with frequent scheduled updates
- Sustainable performance smart equipment renewal, controlled reuse.
- Stability no big bang, just small, controlled steps.
- Security minimum exposure window, fast patches.
- Transparency you know what, when, why.
Hosted customer: the right questions to ask your hosting provider
- What is automated on a daily basis?
- How does the incident management ?
- What is the reboot policy (windows, redundancy, rollback)?
- Who bed sensitive changes every month, and how it's layout ?
- What is the hardware lifecycle strategy (perf vs. sobriety, reuse)?
The LRob approach, in a nutshell
Automation security patches + monthly human checkup + 24/7 monitoringwith reboots tailored to service and equipment that's always levelwithout unnecessary e-waste.
Tired of struggling with a poorly maintained fleet? Migrating to LRob is simple and transparent.
-> Because when you're proud of your work, you show it!
-> ? and you become a sure thing!
Leave a Reply