“It was a cold and windy night, but the breeze of ill omen blowing across the ‘net was colder. The regular trickle of login attempts suddenly became a torrent of IP addresses, all trying to break into the back-end of the Joomla site I host. I poured another cup of joe, it was gonna be a long night.”
Tech noir aside, there was something odd going on. I get an email from that web-site each time there is a failed login. The occasional login attempt isn’t surprising, but this was multiple attempts per minute, all from different IP addresses. Looking at the logs, I got the feeling they were pulling usernames and passwords from one of the various database dumps, probably also randomly seeding information from the Whois database on my domain.
Whois Sluething
Speaking of which, “Whois” is a useful tool for finding information about a domain name or IP address. It’s a digital directory containing the company responsible for the IP or domain, and contact information. I selected a few of the attacking IP addresses and ran the look-up. These IPs were from Digital Ocean, (a virtual server hosting company), Google cloud services, an ISP in Thailand, etc. Many of addresses seem to belong to servers. Pulling up a few of the IPs in a web browser seemed to confirm that it was primarily Linux machines. The only plausible explanation is that this web site had fallen into the sites of a botnet running on compromised Linux servers.
Botnets have two general operating modes. The most obvious mode is when they attack a site or service to knock it offline. This is usually a Denial of Service (DoS) attack, intended to exhaust bandwidth or other computer resources. There are companies like Cloudflare, who are built around the goal of thwarting such attacks. The other operating mode is that of spreading — infecting more computers. Many botnets are viral in nature, using the existing botnet to try to compromise other machines. I believe this is what I was seeing.
So how to get out of the cross-hairs and avoid my box being cracked? First, if you wait till you’re getting attacked to think about security, it’s already too late. Some basic hygiene goes a long way– keep your OS up to date, Keep Joomla/WordPress/etc up to date, and keep any plugins or extensions updated as well. Hardly a week goes by before the next vulnerability is found, and running old software makes your server low hanging fruit.
Disabling the default “admin” account makes a huge difference. Running through a public password list against the admin account can be done pretty quickly. Most frameworks allow renaming or replacing the built-in admin account. If you allow SSH logins at all, root login can be disabled. Make an attacker’s life as hard as you can, don’t leave obvious usernames to be attacked.
Terminating the Problem
I had done my best to follow the above guidelines, so I wasn’t too worried about being compromised. The flood of notification emails was still a pain, and there’s always the remote chance an attacker could get lucky with a login guess or targeted fishing email. I chose a sort of nuclear option: using the web server’s configuration to restrict access to the admin interface. In Apache, within the “VirtualHost” stanza for that website, I added a “Directory” statement pointing at the administrative interface. In that section, a Require ip statement allows me to block anyone else from even trying to log in to the site’s administration portal.
The module at work here is “mod_authz_host”, and it will take a network/netmask as well. If you don’t have a static IP, another option would be a VPN. If you use 10.0.1.x addresses, the statement would look like Require ip 10.0.1.0/24, ensuring that your protected interface is only available to VPN clients.
Finding yourself the target of a botnet is never pleasant, but there are at least steps that can be taken to solve the problem, particularly when it’s as simple as trying to guess the admin password of a Joomla website.
Once, long ago, working a sysadmin gig I noticed that one large network (turned out to be a college in mainland China somewhere) and they had messed up entering their whois information such that they’d left what was obviously a crypt()’d password where something else was supposed to be. I had to try, out of curiosity, and set up a spare server I had at home running John the Ripper on it and five weeks later BAM: a totally believable leetspeak contortion of the English acronym for the school with a traditionally lucky number appended. (I didn’t try using my newly discovered credentials but I also wouldn’t be surprised if the lax security that made it trivial for me to get the credentials for administering their network block with ICANN also is what got them flooded with botnets).
I almost felt bad for the password guessing bot given that our ssh server required public key auth so no amount of password guessing was gonna let them in
You can also implement Fail2Ban[1], which watches log files for failed login attempts and ban them or execute some action if they match conditions.
Typical usage is to watch mail and auth log files and drop connection from failed logins, but it can be configured to monitor other log files and evaluate regular expressions.
Also, check CSF: http://configserver.com/cp/csf.html
[1] https://www.fail2ban.org/wiki/index.php/Main_Page
I’ve used fail2ban, love it. In this case, the IP addresses we’re never getting repeated, and the username guests were pretty random too. Fail2ban excells at locking out IPs that make multiple attempts.
were not we’re
What are you talking about?
We’re is a contraction of “we are”
Oh the joy of writing comments on a touch screen with over-zealous auto-correct.
I set up a script like fail2ban that looks for any username failures that aren’t obviously me (SMTP,SSH,IMAP, anything really) and automatically add them to an ipset. Reports them to ipabusedb. At the same time once a day downloading the latest list of the top probers and populating a block list ipset.
Cuts down on the noise significantly.
drenehstral buried the lede: turn off ssh password auth and use public key auth.
`PasswordAuthentication no` in `sshd_config`, except that the OP talks of attempts to sign in to a web site, not SSH.
“The only plausible explanation is that this web site had fallen into the sites of a botnet running on compromised Linux servers.”
Sites should be spelled sights, as in gun sights.
IOW, Botnets have sites in their sights!
B^)
Password authentication?
[youtube https://www.youtube.com/watch?v=k7OLCbjuidE&w=560&h=315%5D
As was mentioned above, key-pair authentication and strict control over those keys is a good step. I finally gave up on WordPress myself, it’s just a bit too swiss-cheesey for my liking. Turns out, it’s not terribly hard to build a simple procedural page doodad atop really any database engine with a bit of php and python. The upside with this method is you get exactly what you want, no more, no less.
WordPress is only Swiss cheese if you drop loads of dodgy plugins or themes into it, or don’t keep it updated. Sadly many popular themes and plugins are Swiss cheese, often due to doing too many things instead of doing one thing and doing it well. But the WP core is pretty solid.
Security must be a multi-layer approach. And it’s a process, not a state.
My server does have SSH, but on a random port. The default port runs a fake ssh daemon that rejects every and any password, and disconnects after 3 attempts. If you portscan the server and hit 3 closed ports, fail2ban asks iptables to tell you every single port is closed. If you guess the correct port, you must have my password plus TOTP token, or have my key.
As far as I can tell, almost nobody got past the first layer, a few got firewalled, and only me ever connected to the real SSH port, in almost a decade.
Same. for most of that.
Key only login to SSH (I could do password plus token but I’ve never so desperately needed to login that it made it worthwhile.) Fail2Ban to ban failures. No one’s ever tried my non-standard port as far as I can tell, which is weird – i’d expect bots that just sweep whole port ranges to hit on it sooner than later (it’s not even a particularly random port).
And even if the machine was comprimised, I have airlocked backups.
Fail2ban is essential in systems where you can afford it. Locking down as much ports as you can, and yes: key only login is a must. Nevertheless, your very front end app (e.g. WordPress) could make your life hard. Especially if it’s a well-known app.
I use password-authentication on the moments I am away from my computer, and have to login on my server to some maintenance. I’ve used it a couple times. Leaving it key-auth only would lock me out when I don’t have my key with me. I know it reduces the overall security, but it’s a small risk when protected by all the other layers.
Pop an SSH key on your phone, then you can temporarily let whatever computer you’re on in via your phone.
Set it up so ports understand knock, knock jokes. ;-)
Fwknop for the win.
Is the fake ssh server publicly available, or did you write it? I suppose one could write one easily enough, perhaps even a shell script would suffice.
You just need to run an openssl running its own configuration file pointing to empty/invalid passwd/shadow files (you could do a chroot, but that would be overkill).
Ask Google for sshpot. You can even log the usernames/passwords if you want. I just throw them away.
What might be a bit of fun is a fake SSH server that pretends to be vulnerable, except that when it is asked to run the payload, it instead spews a bunch of junk back at the sender in order to try to crash it.
couldnt you set up the logins to after so many attempts the account is locked down?
you mis enter your pin at the atm 3 times and the machine takes the card and the account gets locked so you have to go to the bank and get your card re pinned and prove your identity.
so shouldnt the same be done with the admin, root and user accounts?
That leaves an easy way for someone else to lock your account.
I use this f2bloop, helps with changing IP’s and really slows down attempts.
http://blog.shanock.com/fail2ban-increased-ban-times-for-repeat-offenders/
Fail2Ban has saved many of my servers. basically 3 failed logins from the same ip earns you a 10 ban (firewall drops packets) and 3 10-min bans gets you perma-banned. Most real users become careful typers after the first 10 min ban, otherwise they can always request an unban through the proper channels
If I were you, I would do something like add a “forgiveness” factor, where after X many logins, (from the same or similar range of IP addresses, or same geolocation, etc.) , strikes are forgiven, and / or the number of strikes increases
If it works in your case, I’d also suggest adding geoblock rules to one’s host or firewall. For instance, I know there’s no reason for anybody outside the US to reach my network, so I block non-US addresses (i.e., anything that’s not in http://www.ipdeny.com/ipblocks/data/aggregated/us-aggregated.zone). That cut down my door knock traffic by about 90% in my wild estimation. I used to have hundreds, if not over a thousand, of fail2ban automatically generated rules on my hosts. With the geoblock rules upfront I get a couple of dozens.
FWIW. Thanks.
“You seem to have scripting disabled”.
“To see the non script version, click the button/link below”
aannnnd of course the button won’t fucking work without scripting enabled.
(?seriously?)
Or you get that “wait a moment” page with nothing else on it. (Incapsula)
You people/Admins should know by now, that many people block scripts (and cookies) until we trust a site.
So you need a minimal landing page with something other than a captcha or redirect to that damned “wait a moment” , page.
Just stonewalling us with a challenge screen (and no idea of the site content) assures that most of us will never come back.
Seriously guys, all sites need a simple home page (at minimum) that gives us a chance to decide if it’s worth allowing a script.
We (the web citizens) are, also, tired of having our equipment “attacked”.
You can consider to block visitors from certain countries. In Linux, you can use iptables or ipsec as firewall. For the country list, you can get it free from https://www.ip2location.com/free/visitor-blocker
Sleuthing is spelled with the e before the u.