…for beginners using Linux/BSD.
You’ve found your provider, picked the package that suits your needs and now you’re ready to purchase it. But before you do, there are two things to consider; your current needs and your future needs. Although you could currently get away with running a server with bargain hardware, how well will it work when capacity needs to increase? Or a similar question, will you really need that dual quadcore processor when the system is going to be sitting idle most of the time? Running something like BOINC/SETI@Home for a day and looking at the “% of time BOINC client is running” statistic, you’ll be able to see how often you don’t need that extremely powerful processor*. Be as frugal as possible. Instead of that expensive processor, invest in RAM upgrades and fast disks, or even (KVM over IP, etc…) remote console access. The biggest benefits are usually found there.
Find out if your provider is able to upgrade your hardware or bandwidth plans at a reasonable price with minimal downtime or if there is a possibility of switching hosting companies without a hassle. You will need to plan what your needs are in terms of hardware and bandwidth now and your reasonable expectations on requirements for the future.
The hassle of planning your infrastructure now will save you a headache later.
Now for the “meat” of the article. Two of the most important sections are security and reliability. Performance is not as important and that topic is well covered elsewhere, so I will not emphasize or cover it. Although security is as equally covered, I’ll only go over the general techniques that are worthwhile for the beginner, not those that are done by the severely paranoid. Since a beginner to system administration in this context probably isn’t doing it as his main job, he most likely doesn’t have the time to research and stay up to date with mailing lists. For that reason, I prefer to focus on passive security tools as they’re the easiest to understand and implement.
The best way to go about securing your server is with the mindset of “I’ll eventually get hacked and there is nothing I can do it to prevent it.” There is no point arguing it**, no system is 100% safe and since that’s the case, any system will eventually get hacked. Proceeding with that mindset, the thing you can do and what everyone tries to do at some level is to give the hacker a hard time. You can do that by keeping your system up to date with patches, not running unnecessary services (cups, inetd in most cases) or services with a bad security record (bind/wu-ftpd) and separating as many services from the rest of the system. By separation, I mean FreeBSD jails or other virtualization techniques. The benefit of this is when one service is compromised, your other services are still safe.
Firewalls are useful but I don’t bother with them right away since they get in the way. A good firewall policy can prevent something like a rootkit or backdoor from getting network access but there are arguably better ways of dealing with those things. My priority is fast detection and removal rather than paranoid prevention. I run chkrootkit, rkhunter, and tripwire as well as a network intrusion detection system. The first three require no knowledge of any firewall software specific syntax to use and you don’t ever have to modify any policies when you add a new service. The fourth, it’s pretty simple to understand and requires uncommenting a few lines in a configuration file and you’re done. Save the firewall policy configuration for last, when you’re done with everything else and have a bit of free time. An admin who doesn’t understand what’s going on behind the scenes of a service he’s running is a useless admin. Invest some time in understanding your firewall software, what you’re blocking and allowing before implementing it in a production environment.
I may be contradicting myself when I mention passive security and things that are easy to understand but then go on to suggest setting up virtual machines. Well, I’m not the one to deny being a hypocrite but it’s well worth the effort if you follow the same security considerations*** for the guest as you do for the host.
Reliability and uptime is something you can have issues if you don’t have an plan. I suggest installing all of the servers, their patches, all of the software you’ll need and your kernel upgrades right away. When you first get access to your server, chances are it wont be patched up and running the latest kernel so you need to fix that. The best time to do is before you move your traffic to the new server so you can avoid downtime. Kernel upgrades are especially dangerous to reliability for the inexperienced. Even a small mistake or misstep during a kernel upgrade could render your system unbootable. In certain cases, even a Grub fallback (or your boot loaders equivalent) might not work. Fallback is a feature to boot into another predefined kernel if booting into another one fails. Some people swear by it, I don’t. Fallback will not help you if you made a mistake and didn’t install your network drivers. If you’re not fully confident with installing your own kernels, just stick with the kernels your distribution provides. Although they’re not optimized for your system, they’re almost guaranteed to work. If you don’t have a choice and need to upgrade your kernel while lacking confidence, there is something an alternative. The alternative of having to wait for your hosting support staff to reboot your server and boot into a working kernel (it can take hours) is remote console access such as KVM over IP. Remote console access in invaluable during a kernel upgrade since you can reboot the server yourself and boot into a working kernel. For BSD upgrades, it’s even more useful since it’s recommended to boot into single user mode before doing an ‘make installworld’. Some hosts offer this service for free while most charge for it.
The natural transition from remote consoles is the topic of remote management access. How many different ways can you log in to completely manage your server? Most people are happy only with SSH but I think it’s a good idea to have more than just SSH. If you were convinced to get remote console access just because of the kernel upgrade issue or you already have it, pat yourself on the back and skip these next few paragraphs. However, since most of the time I don’t have remote console for budget issues, I need two different services that allow root access to the system. My recommendation is SSH and Webmin. Although I dislike control panels because they promote bad laziness and breed admins who don’t understand their services, Webmin is a bit better since most modules are only front-ends to configuration files. Chances are that nothing will happen to SSHd to prevent remote logins but it’s a good idea to have backups. Don’t go overboard with backups though since running more services is proportional to how many security vulnerabilities your server contains.
One topic that’s bound to come up is automated patches. It’s a good idea for a anti-virus scanner but for your services, it isn’t. You should read the upgrade notes before upgrading, such is in ports after a cvsup or your distribution’s equivalent. The reason for this is sometimes, important bits of a service change and some specific configuration of a service will not work in its new version. The result is that your service will fail to run. A real world example is my clamav anti-virus software. I use it clamav to scan incoming e-mails but since clamav wasn’t configured properly after an upgrade and I didn’t have a backup anti-virus scanner, my anti-virus e-mail scanning software didn’t work and no mail was delivered for about a night before I caught and fixed the issue. Take that as a warning to read upgrade logs and to test after an upgrade to make sure everything is working.
You also have to ask yourself, other than providing network conncectivity, power and the hardware itself, how else are you depending on your hosts? Usually, either they add their in house name servers (to handle your DNS lookups) or they also provide OS patched to download locally. If the OS patch server fails, it’s not a big issue since it’s likely that your package manager has been configured to move on to the default update site provided by the distributor. However, if the hosts nameservers fail, you’re out of luck. If you take a look in your /etc/resolv.conf, you’ll see at least two nameservers listed with some local IP address. If they’re all your hosts nameservers, add a third that’s off site. I suggest the range from 4.2.2.1 to 4.2.2.6 as they’re reliable and fairly quick all over the world. Also, OpenDNS has good nameservers but with limited latency if you’re not near one of their nameservers. I suggest you review their web site first as they offer other features related to name resolution. Do latency testing on the name servers before adding them. If they’re greater than 70 ms, find another one but if your network’s latency is in general quite poor, use your own threshold. NOTE: /etc/resolv.conf can only handle 3 nameserver entries, others will be ignored and removed.
Finally, the stability of specific services is often directly related to performance type settings. Since the only general recommendation I can give is “trial and error,” a better option would first reviewing multiple optimization HOWTOs for that service and then proceeding to make the chances. Ideally, you would do your performance optimization as soon as possible to avoid downtime. Since going further into this topic is not in the scope of this article, I will not cover it. If there are questions, suggestions or more helpful tips to contribute, leave them in the comments and I’ll adjust this article appropriately.
* Of course, I’m not advocating that you get the cheapest processor and then have your services run without an issue. A slow processor will run your services slower but you need to find the right balance of processor speed (price) and performance gains. Burstable loads also should be considered.
** But you probably will anyway with something along the lines of ”unplugging your computer from the network will make it pretty safe” but then you need to think about physical security and I’d say most systems are more secure remotely than they are with local, physical access.
*** All of the aforementioned tips are useless if you don’t follow the basic principles; don’t use easy passwords, disallow (direct) remote root login for every service, don’t use the same password for multiple accounts and across multiple systems (same root password for all virtual machines and the host), use encryption for all authentication and traffic you reasonably and legally can.