Dr Fragen in the operating room


  • Fail2ban Problems and Solutions

    If you use Fail2ban then you are probably aware of the fact that you must add a rule number to the ipfw deny rule for actionban in ipfw.conf. If you don’t add a rule number then there is no way for fail2ban to delete the rule after it expires. The problem lies in that you can easily set a different rule number for each filter but if the filter adds many rules within it’s ban time then when that first actionunban gets triggered all rules with the same number are removed, even if there full ban time has not transpired.
    I was looking for an elegant solution to this and finally figured out how to do it myself. What I’ve done is in the ipfw.conf file I’ve added a variable that will create a random number between 10000 and 12000 to use as the rule number.
    The code is pretty simple.

    echo $((RANDOM%2000+10000))

    There needs to be an extra % in there for it to work. I think it has something to do with python. So far it seems to be working pretty good here. While it is possible that I could get a duplicate rule number, it’s unlikely.
    I’ve modified my installation of Fail2ban significantly; but only by adding filters, jails, etc. Here’s a bundled version of all of my modifications. Here are instructions for using my modifications. So far everything seems to be working great. I’ve had to add a few items to ignoreregex so I don’t ban people using their iPhones on 3G or at home from certain dynamic IP cable providers.
    What I’ve done is a host lookup on the IP that’s banned and if I find it’s a local ISP, like Verizon or Time Warner Cable, I add part of their host lookup to the ignoreregex list. So far it seems to be doing the trick.

  • Checking Fail2ban regex

    I’ve just stumbled across a great command in Fail2ban to check whether or not your filter will actually score a hit from your log file.
    From the command line.
    [code lang=bash]
    $ fail2ban-regex /path/to/logfile /etc/fail2ban/filter.d/myfilter.conf regex_to_ignore
    As an example.
    [code lang=bash]
    $ fail2ban-regex /var/log/secure.log /etc/fail2ban/filter.d/sshd.conf (myusername|myIPaddress)
    This seems like a great way to test whether changes to your filters are correct, rather than just waiting to see if anything shows up in fail2ban.log.

  • Fail2ban and OS X Server, Part Deux

    As some of you might know I run my own installation of OS X Server. I’ve since updated it to Snow Leopard Server and I think I’ve got most of it running well. As I check my server logs frequently I find that there are all sorts of script kiddies attempting to log in to my server in various ways.
    The traditional method was to simply try an ssh session with a username and password combination. Unfortunately now I see more attempts to log in via VNC or in attempts to check or send email. It’s amazing to see 10 – 15 login attempts in a second. There’s a real motivating force to stop that kind of attention my poor little server is getting.
    As I’ve written before, I’ve found the Fail2ban scripts to to be a perfect solution. I have had to make a number of additions and improvements along the way and now I thought I’d share.
    I’ve created a couple of new jails and improved upon a couple of the distribution jails so they work better with Snow Leopard. I’ve packaged up all my modifications. Here’s how to install them for yourself.
    Download the modifications tarbell.
    Then you’ll need to issue the following commands from Terminal.

    sudo tar xzf install_fail2ban_mods.tar.gz

    This will create a folder containing all the modifications. At this point you can manually put all the files in the proper folders or you can use my installation script. The installation script, install_fail2ban_mod.sh shouldn’t delete anything. I only use the cp command. If you already have a jail.local file that is backed up. You may also need to modify the jail.local file that I have.
    Additionally, I’ve found that sometimes the fail2ban server might have hung or its process has stopped. I’ve also written a script and a couple of plists for /Library/LaunchDaemons/ that periodically check to make sure fail2ban continues to hum along. You’ll have to load these plists manually or simply restart.
    The jails that I’ve added check for SMTP, POP, IMAP, VNC and non-existant web pages. These, in addition to monitoring SSH, seem to cover most of it. Please remember that some of these filters are somewhat specific to Snow Leopard so they check against a Dovecot mail server.
    So far my only problem has been when a user has changed their password but not correctly transferred these changes to Mail.app. What happens is fail2ban sees them as a break-in attempt and bans their IP for 10 minutes or so. I’m sure it can be frustrating. Sorry, I’m doing my best to fix it for you.
    By all means, let me know how you’ve improved Fail2ban for your server.

  • Fail2ban on Leopard Server

    So here I am running my own server — for almost 2 years now. It’s been a real learning experience and I’ve tried to share. My latest add-on has been fail2ban. I got tired of looking into my logs and seeing where script kiddies or bots were trying to take control of my server. Fortunately, none have.
    After a bit of googling, I found fail2ban. It’s a collection of python scripts.

    Fail2ban scans log files like /var/log/pwdfail or /var/log/apache/error_log and bans IP that makes too many password failures. It updates firewall rules to reject the IP address.

    There are a few tricks I’ve discovered along the way to make it work on my installation and likely on Mac OS X Server in general.
    First is that fail2ban creates a PID and socket file in a directory that it fails to create. Yeah, that’s a bug. Since I didn’t want to mess around with the actual scripts in the program, I created a plist that issues the mkdir /var/run/fail2ban command. I placed this in /System/Library/LaunchDaemons and set it to Run at Load. Lingon is your friend, but’s now inactive. 馃檨
    After creating the file you have to use the command line to move it to the /System/Library/LaunchDaemons directory. I also created another launchd plist to reload fail2ban every day. I did this because I run multiple virtual websites and the error logs for those sites get rotated and the names have some time code or something tacked on the end of the filename.
    OK, problem 1 solved. Next I discovered that since fail2ban is really running on a multitude of linux boxes all the different methods of IP tracking, sorting etc. were really useless on my OS X Server. I run ipfw firewall and fortunately there’s a module for that in fail2ban. Unfortunately it’s not quite set up correctly, at least it wasn’t for me. I had to tweak it a bit.
    What this means is that your action is always going to be ipfw. I tweaked the ipfw.conf file a bit. Now it does the following.

    1. Logs it’s action to ipfw.log
    2. Adds a rulenum to the ipfw command. I did this because some other rule in my setup was allowing the IP before my deny could take effect. By lowering the rulenum my deny now fires off first.
    3. Abstracted the protocol (tcp, udp) to pass as a variable. Just in case something you want to block isn’t tcp.

    I also created another filter as I found many times some machine would excessively hit my Apache server looking for nonexistent files. Since it sounds like something a bot would do I decided to ban it. This was the simple creation of a new filter.
    I created a jail.local file to hold all my prefs and through trial and error discovered that the examples of how to call for a jail weren’t working for me. Perhaps I just didn’t understand the examples. I soon discovered that parameters for the jail action needed to be passed inside of square brackets in the prefs.
    I’m sure, if you’ve gotten this far that you’re either very confused by this whole post or that you’ve had an epiphany. To further the epiphany along I’ve uploaded my file changes.
    You should be able to figure out what file goes where from the folder structure of the upload.
    A couple of things in summary to remember. First, turn on your server’s firewall. Then make sure you change your server’s local IP address in the files to match your own. That’s the setting for localhost.
    Good luck. If you have any questions leave a comment.