InvXim

For the Moodle scripts, it’s free for personal use. Including it with any sort of package without my permission is forbidden (subject to change). Update to date license is in http://blog.ods.org/mood/license

Exactly like the previous article of finding inactive sites, the script sends a CSV file to the browser with the columns DaysInactive,URL to the course site, Instructor, and the intstructors e-mail address. For courses with multiple instructors, each instructor gets a row in the CSV instead of putting all instructors and their e-mail addresses in one row.

Inactive Moodle Site CSV Output Script

This script outputs inactive courses based on mdl_log table from the database, so you must have that logging on.

Inactive Moodle Sites Script

The following script outputs a table with file sizes using du (YOU MUST HAVE IT) for each course in moodledata. As Wordpress cannot correctly display HTML as text, I’ve linked to it. Drop it in your main moodle directory, the same one that has config.php

Moodle Site Size Script

Currently, OpenMPI is the best option for Cell blades due to its loopback communication performance. Loopback communiction is used when a node is sending data to another node on the same system, which on a Cell blade is likely due to it having dual Cell Broadband Engines.  OpenMPI already implements shared memory for loopback communication, but MPICH2 uses less effecient means for this type of loopback communication. That means MPICH2 has lower performance in the average case until the the communcation module “Nemesis” is released for the Cell/PowerPC.

To test these conjectures, take a look at the classic MPI Ping-pong example (C or Fortran77). The output from a Cell QS20 running OpenMPI version 1.2.1 is below -

Hello from 0 of 2
Hello from 1 of 2
Timer accuracy of ~4.053116 usecs

8 bytes took        70 usec (   0.228 MB/sec)
16 bytes took        13 usec (   2.440 MB/sec)
32 bytes took        11 usec (   5.836 MB/sec)
64 bytes took        11 usec (  11.671 MB/sec)
128 bytes took        10 usec (  25.565 MB/sec)
256 bytes took        12 usec (  42.108 MB/sec)
512 bytes took        13 usec (  78.090 MB/sec)
1024 bytes took        19 usec ( 108.733 MB/sec)
2048 bytes took        29 usec ( 141.982 MB/sec)
4096 bytes took       382 usec (  21.448 MB/sec)
8192 bytes took        85 usec ( 192.492 MB/sec)
16384 bytes took       138 usec ( 237.784 MB/sec)
32768 bytes took       570 usec ( 114.964 MB/sec)
65536 bytes took       433 usec ( 302.562 MB/sec)
131072 bytes took       912 usec ( 287.454 MB/sec)
262144 bytes took      1599 usec ( 327.919 MB/sec)
524288 bytes took      3019 usec ( 347.315 MB/sec)
1048576 bytes took      5971 usec ( 351.212 MB/sec)

Asynchronous ping-pong

8 bytes took        36 usec (   0.444 MB/sec)
16 bytes took         8 usec (   3.948 MB/sec)
32 bytes took         9 usec (   7.064 MB/sec)
64 bytes took        11 usec (  11.423 MB/sec)
128 bytes took        11 usec (  23.342 MB/sec)
256 bytes took        12 usec (  42.108 MB/sec)
512 bytes took        15 usec (  68.174 MB/sec)
1024 bytes took        20 usec ( 102.261 MB/sec)
2048 bytes took        25 usec ( 163.618 MB/sec)
4096 bytes took        64 usec ( 128.208 MB/sec)
8192 bytes took        71 usec ( 230.602 MB/sec)
16384 bytes took       140 usec ( 233.740 MB/sec)
32768 bytes took       290 usec ( 225.865 MB/sec)
65536 bytes took       434 usec ( 302.064 MB/sec)
131072 bytes took       807 usec ( 324.915 MB/sec)
262144 bytes took      1531 usec ( 342.474 MB/sec)
524288 bytes took      3104 usec ( 337.818 MB/sec)
1048576 bytes took      5987 usec ( 350.274 MB/sec)

Bi-directional asynchronous ping-pong

8 bytes took        25 usec (   0.645 MB/sec)
16 bytes took        11 usec (   2.918 MB/sec)
32 bytes took        12 usec (   5.263 MB/sec)
64 bytes took        11 usec (  11.671 MB/sec)
128 bytes took        13 usec (  19.884 MB/sec)
256 bytes took        15 usec (  34.087 MB/sec)
512 bytes took        17 usec (  60.492 MB/sec)
1024 bytes took        20 usec ( 102.261 MB/sec)
2048 bytes took        35 usec ( 117.670 MB/sec)
4096 bytes took        85 usec (  96.516 MB/sec)
8192 bytes took       100 usec ( 163.618 MB/sec)
16384 bytes took       181 usec ( 181.079 MB/sec)
32768 bytes took       365 usec ( 179.541 MB/sec)
65536 bytes took       522 usec ( 251.145 MB/sec)
131072 bytes took      1108 usec ( 236.607 MB/sec)
262144 bytes took      2870 usec ( 182.689 MB/sec)
524288 bytes took      5892 usec ( 177.965 MB/sec)
1048576 bytes took     11677 usec ( 179.596 MB/sec)

Max rate = 351.211540 MB/sec  Min latency = 4.053116 usec

This ping-pong example will not give you accurate results and the page that hosts it recommends NetPIPE. I suggest both, but ping-pong tends to produce more reliable results as I have encountered some inconsistency with NetPIPE’s measurements.

Scale:

1000 nanoseconds = 1 microsecond (usec), 1000 microseconds (usec) = 1 millisecond, 1000 milliseconds = 1 second

There is a patch for 1.9 that adds the functionality of changing settings to enable/disable emoticons, but it doesn’t quite work on 1.8. The best way to do it in Moodle 1.8 is going to your moodle directory, then lib/weblib.php and look for the function replace_smilies (line with “function replace_smilies(&$text)”) and comment them out until after the very last one, which would be the ‘( )’ => ‘egg’ entry.

The benefit of this is that during parsing of text emoticons will not get automatically added, although they can still be used if selected from the menu of an editor. A note is that you should keep at least one valid emoticon entry in the $emoticons array, just to make sure that it doesn’t break anything by making $emoticons empty.

…for beginners using Linux/BSD.

 You’ve found your provider, picked the package that suits your needs and now you’re ready to purchase it. But before you do, there are two things to consider; your current needs and your future needs. Although you could currently get away with running a server with bargain hardware, how well will it work when capacity needs to increase? Or a similar question, will you really need that dual quadcore processor when the system is going to be sitting idle most of the time? Running something like BOINC/SETI@Home for a day and looking at the “% of time BOINC client is running” statistic, you’ll be able to see how often you don’t need that extremely powerful processor*. Be as frugal as possible. Instead of that expensive processor, invest in RAM upgrades and fast disks, or even (KVM over IP, etc…) remote console access. The biggest benefits are usually found there.

Find out if your provider is able to upgrade your hardware or bandwidth plans at a reasonable price with minimal downtime or if there is a possibility of switching hosting companies without a hassle. You will need to plan what your needs are in terms of hardware and bandwidth now and your reasonable expectations on requirements for the future. 

The hassle of planning your infrastructure now will save you a headache later.

Now for the “meat” of the article. Two of the most important sections are security and reliability. Performance is not as important and that topic is well covered elsewhere, so I will not emphasize or cover it. Although security is as equally covered, I’ll only go over the general techniques that are worthwhile for the beginner, not those that are done by the severely paranoid. Since a beginner to system administration in this context probably isn’t doing it as his main job, he most likely doesn’t have the time to research and stay up to date with mailing lists. For that reason, I prefer to focus on passive security tools as they’re the easiest to understand and implement.

The best way to go about securing your server is with the mindset of “I’ll eventually get hacked and there is nothing I can do it to prevent it.” There is no point arguing it**, no system is 100% safe and since that’s the case, any system will eventually get hacked. Proceeding with that mindset, the thing you can do and what everyone tries to do at some level is to give the hacker a hard time. You can do that by keeping your system up to date with patches, not running unnecessary services (cups, inetd in most cases) or services with a bad security record (bind/wu-ftpd) and separating as many services from the rest of the system. By separation, I mean FreeBSD jails or other virtualization techniques. The benefit of this is when one service is compromised, your other services are still safe.

Firewalls are useful but I don’t bother with them right away since they get in the way. A good firewall policy can prevent something like a rootkit or backdoor from getting network access but there are arguably better ways of dealing with those things. My priority is fast detection and removal rather than paranoid prevention. I run chkrootkit, rkhunter, and tripwire as well as a network intrusion detection system. The first three require no knowledge of any firewall software specific syntax to use and you don’t ever have to modify any policies when you add a new service. The fourth, it’s pretty simple to understand and requires uncommenting a few lines in a configuration file and you’re done. Save the firewall policy configuration for last, when you’re done with everything else and have a bit of free time. An admin who doesn’t understand what’s going on behind the scenes of a service he’s running is a useless admin. Invest some time in understanding your firewall software, what you’re blocking and allowing before implementing it in a production environment.

I may be contradicting myself when I mention passive security and things that are easy to understand but then go on to suggest setting up virtual machines. Well, I’m not the one to deny being a hypocrite but it’s well worth the effort if you follow the same security considerations*** for the guest as you do for the host.

Reliability and uptime is something you can have issues if you don’t have an plan. I suggest installing all of the servers, their patches, all of the software you’ll need and your kernel upgrades right away. When you first get access to your server, chances are it wont be patched up and running the latest kernel so you need to fix that. The best time to do is before you move your traffic to the new server so you can avoid downtime. Kernel upgrades are especially dangerous to reliability for the inexperienced. Even a small mistake or misstep during a kernel upgrade could render your system unbootable. In certain cases, even a Grub fallback (or your boot loaders equivalent) might not work. Fallback is a feature to boot into another predefined kernel if booting into another one fails. Some people swear by it, I don’t. Fallback will not help you if you made a mistake and didn’t install your network drivers. If you’re not fully confident with installing your own kernels, just stick with the kernels your distribution provides. Although they’re not optimized for your system, they’re almost guaranteed to work. If you don’t have a choice and need to upgrade your kernel while lacking confidence, there is something an alternative. The alternative of having to wait for your hosting support staff to reboot your server and boot into a working kernel (it can take hours) is remote console access such as KVM over IP. Remote console access in invaluable during a kernel upgrade since you can reboot the server yourself and boot into a working kernel. For BSD upgrades, it’s even more useful since it’s recommended to boot into single user mode before doing an ‘make installworld’. Some hosts offer this service for free while most charge for it.

The natural transition from remote consoles is the topic of remote management access. How many different ways can you log in to completely manage your server? Most people are happy only with SSH but I think it’s a good idea to have more than just SSH. If you were convinced to get remote console access just because of the kernel upgrade issue or you already have it, pat yourself on the back and skip these next few paragraphs. However, since most of the time I don’t have remote console for budget issues, I need two different services that allow root access to the system. My recommendation is SSH and Webmin. Although I dislike control panels because they promote bad laziness and breed admins who don’t understand their services, Webmin is a bit better since most modules are only front-ends to configuration files. Chances are that nothing will happen to SSHd to prevent remote logins but it’s a good idea to have backups. Don’t go overboard with backups though since running more services is proportional to how many security vulnerabilities your server contains. 

One topic that’s bound to come up is automated patches. It’s a good idea for a anti-virus scanner but for your services, it isn’t. You should read the upgrade notes before upgrading, such is in ports after a cvsup or your distribution’s equivalent. The reason for this is sometimes, important bits of a service change and some specific configuration of a service will not work in its new version. The result is that your service will fail to run. A real world example is my clamav anti-virus software. I use it clamav to scan incoming e-mails but since clamav wasn’t configured properly after an upgrade and I didn’t have a backup anti-virus scanner, my anti-virus e-mail scanning software didn’t work and no mail was delivered for about a night before I caught and fixed the issue. Take that as a warning to read upgrade logs and to test after an upgrade to make sure everything is working.

You also have to ask yourself, other than providing network conncectivity, power and the hardware itself, how else are you depending on your hosts? Usually, either they add their in house name servers (to handle your DNS lookups) or they also provide OS patched to download locally. If the OS patch server fails, it’s not a big issue since it’s likely that your package manager has been configured to move on to the default update site provided by the distributor. However, if the hosts nameservers fail, you’re out of luck. If you take a look in your /etc/resolv.conf, you’ll see at least two nameservers listed with some local IP address. If they’re all your hosts nameservers, add a third that’s off site. I suggest the range from 4.2.2.1 to 4.2.2.6 as they’re reliable and fairly quick all over the world. Also, OpenDNS has good nameservers but with limited latency if you’re not near one of their nameservers. I suggest you review their web site first as they offer other features related to name resolution. Do latency testing on the name servers before adding them. If they’re greater than 70 ms, find another one but if your network’s latency is in general quite poor, use your own threshold. NOTE: /etc/resolv.conf can only handle 3 nameserver entries, others will be ignored and removed.

Finally, the stability of specific services is often directly related to performance type settings. Since the only general recommendation I can give is “trial and error,” a better option would first reviewing multiple optimization HOWTOs for that service and then proceeding to make the chances. Ideally, you would do your performance optimization as soon as possible to avoid downtime. Since going further into this topic is not in the scope of this article, I will not cover it. If there are questions, suggestions or more helpful tips to contribute, leave them in the comments and I’ll adjust this article appropriately.

 * Of course, I’m not advocating that you get the cheapest processor and then have your services run without an issue. A slow processor will run your services slower but you need to find the right balance of processor speed (price) and performance gains. Burstable loads also should be considered.

** But you probably will anyway with something along the lines of ”unplugging your computer from the network will make it pretty safe” but then you need to think about physical security and I’d say most systems are more secure remotely than they are with local, physical access.

*** All of the aforementioned tips are useless if you don’t follow the basic principles; don’t use easy passwords, disallow (direct) remote root login for every service, don’t use the same password for multiple accounts and across multiple systems (same root password for all virtual machines and the host), use encryption for all authentication and traffic you reasonably and legally can. 

Seems that VMWare on my computer will not boot 2.6 kernels although 2.4 works just fine. Also, I have network connectivity issues when working with FreeBSD in Workstation.  My earlier project is scrapped for now and I’ll find something else to do.

This is a brief explanation of how to use ISPAdmin.py for “everyday” administration after setting up the ISP style e-mail servers. An even more brief explanation that will get you started on using the tool is on the original page but there are a few things on using it that are left unmentioned. The first thing you should do however after downloading ISPAdmin.py from the original source or from my mirror, is editing the following line -

db_uri = ‘mysql://root:root2007@127.0.0.1:3306/mailtest’

Change the username (root) and password (root2007) accordingly as well as the database (mailtest) where the settings are stored.  When that’s done correctly, you can start ISPAdmin.py. Since I already have domains in there, it’ll look slightly different that when you’re starting it up for the first time.

Connected to database (2 domains, 15 users, 2 aliases)
Welcome to the ispmail console!
=>> d
 [  1]  domain1.com
 [  2]  domain2.com

The ‘d’ command shows a list of all domains. If you want to add a new user to domain1.com, you would type in ‘d 1′ or ‘d domain1.com’. The number 1 comes from the domain ID, which is given by the SQL database when the domain is added for the first time. The ID is sequential for each domain added but if you remove the last one (ID #2) and add another domain, the new domain wont get ID #2 but rather ID #3. This detail isn’t terribly important but I thought I should mention it as the same property follows for users. I suppose one area where this has an impact is if you’re counting the number of users a certain domain has or how many domains are hosted by the largest ID number, you could get inaccurate results. Moving on…

=>> help

Documented commands (type help <topic>):
========================================
EOF  aliases  da  delalias   deluser  du    help  na  newalias   newuser  u
a    d        dd  deldomain  domains  exit  hist  nd  newdomain  nu       users

That is a list of all the commands available to you. We’ll start with users. To add users (or aliases), you need to switch to that domain, as mentioned earlier.

=>> d 1
[domain1.com] =>> newuser username asdf
New user: username (password: asdf)

Notice that since we’re under domain1.com, we don’t need to specify the domain part for the newuser command. This command creates the necessary entries in the SQL database but if you go to /home/vmail/domain1.com, you’ll notice the users you added aren’t listed in there. Don’t worry about this because once mail is received, Dovecot will create the file structure so you don’t have to.

Adding new aliases is exactly the same but you might be tricked into doing something like -

[domain1.com] =>> newalias elvedin@domain1.com elvedin@domain0.com
New alias: elvedin@domain1.com  -> elvedin@domain0.com

Don’t do this as it will NOT work. What this is expecting is to forward the mail from the user ”elvedin@domain1.com” to the new address and not for the user “elvedin”. The correct way of adding a new alias is -

[domain1.com] =>> newalias elvedin elvedin@domain0.com
New alias: elvedin  -> elvedin@domain0.com

The ‘users’ command will give you a list of users sorted alphabetically. Obviously, there is not much to it.

To change a users password, you could edit the SQL database yourself and and encrypt a new password but that’s too much work. The easy way is to delete the user ‘deluser username’ and then add the user again with the new password. The one problem with this is that during the few seconds the user’s account is deleted, the mail they receive will be rejected with a friendly message from Postfix to the sender mail server saying something along the lines of “That account doesn’t exist, please don’t e-mail me anymore.” Unlikely that the user is going to receive the mail over the few seconds it takes you to make the changes but you can play it safe if you want to. Modifying the password field in the database directly wouldn’t have this issue.

The final note that should be mentioned is that when logging in, the username is the full e-mail address. It looks weird in certain mail clients (such as Thunderbird) when it says it’s logging into “username@domain1.com@domain1.com” but that’s not a huge deal. This little detail can also save you lots of troubleshooting time as it took me a while to figure out why my logins were failing even though I thought I used the right username and password.

Hope this more concise documentation of ISPAdmin.py helped you as it would have helped me. I know I didn’t read everything Christoph Haas wrote thoroughly the first time around.

I couldn’t decide on which topic to write a “HOWTO” so I decided to do both. I’ll start with making a VMWare appliance based on Debian etch (and this VMWare appliance ), Postfix, Dovecot IMAP/POP3, with MySQL authentication and virtual users and domains (based on this walk through). I chose doing this first because it’s easier but also because the Java project isn’t even close to finished.

 The Java project is a GUI for a dynamic DNS update client made specifically for ODS to fix the shortcomings in some of the current clients used.   while providing the source code for everyone to improve and contribute back.

 The other project I was working on was an rFactor game server manager in PHP. As the server is Windows only, there’s only a handful of tools I could really work with, like the “PSTools” port to control (start and stop) the game server process. The server itself will run from a premade configuration file when starting it with specific parameters. If those parameters aren’t passed, the server loads its own GUI for configuration. The other thing this server manager was supposed to do is provide a friendly interface for setting the server configuration while first loading it from an existing file. After being overwhelmed with managing all the file input/output and more system command calls than any PHP script should, I decided to quit. There is no way such a thing should be made.

 My last side project, I haven’t really thought about starting but it shouldn’t take long at all is a web interface to maintain that ISP style e-mail setup. Basic stuff like adding and removing domains, users, aliases or changing passwords or allowing users to change their own passwords. Christoph Haas (the author of the HOWTO) has written a Python program which does all of the necessary things but it’s not quite a web front end and it isn’t well documented. I’ve written a few things documenting the setup and the use of ISPAdmin.py.