FuckUps

So, for a little while now, one of the sites I host on my VPS has had some sort of malware.  I have no idea how it got there but I have ideas about fixing it if I could find it.  There are plenty of sites that will tell you that your website is infected.  There don’t seem to be any that will also tell you “It’s probably in this file here, go look there.”  Instead it’s all “We’ll fix it for a small monthly subscription of $50/month.”

I think I’ll pass on that one.

Instead, I opted to simply rebuild the website from the ground up.  It’s a simple process really, set up a fresh WordPress install, ad the appropriate plug ins, copy the images in the Uploads folder, and do a quick export/import of the database.  I also wanted to make sure I got the permissions right, to avoid any future malware issues, since this was the likely culprit for how the malware got there.

This is where I screwed up.  Instead of doing a CHMOD on the local directory, I mistakenly did a CHMOD on /* -R.  Or in other words, everything in the root directory, Recursively through each directory.  Or in other, other words, “everything”.  It actually failed to run on a bunch of files, likely because they were in use.  It did however break SUDO, which meant I couldn’t easily try to change anything back.  It also immediately killed every website I host since they all use MySQL which could no longer use it’s databases, because it didn’t have permissions.

I don’t host anything major at least.  A couple of personal blogs, my wife’s two blogs, some side projects like TinyTinyRSS.  My main concern were my wife’s blogs, frankly, no one reads my shit at all anymore (why are you here???), lots of people read both of her blogs.

If this were a physical server, I’d load a recovery CD and backup or even just reinstall from there.  This is a VPS though.  There isn’t a physical machine I can access and really, there probably isn’t even a physical machine at all, not a dedicated one.  There may be a dozen other servers on the same physical machine as my VPS.  Fortunately, with the use of a support ticket, Digital ocean will mount a virtual recovery disc to your virtual server.

So I managed to get access to the server files.  I set about with two plans at this point.  Worst case scenario, I would need to reimage the server and rebuild everything.  I’ve done this sort of thing many times over the years moving from server to server, I’m actually pretty good at it.  Getting the data was the important part, so I started some downloads of the data.  Honestly, this was always the only option, but I was hoping I could get the old set up running because it would make my life easier.  If I could get MySQL working I could make proper back ups instead of trying to use the raw files, something I’ve never done (it wasn’t hard in the end).  So, 50,000 files later, I had all of the needed files downloaded.  I probably could have saved some time and just reinstalled the core WordPress files but I wanted to keep things as pain free as possible to avoid any more screw ups.

How to restore the server.  The problems stem from permissions, as in, nothing has permissions on anything.  So the simplest solution seemed to be to set the files all to 777, or open access to every user, group and everything.  This is absolutely horrible practice for a live server and should not be done.  However, I needed ten minutes or so to dump some SQL files and a few other proper back ups that would be much easier in a live environment.

Setting everything to 777 didn’t work, for starters, all those system files that were previously inaccessible, were now accessible, since the recovery CD wasn’t using them.  So now EVERYTHING became 777.  I don’t know much beyond that other than it flat out refused to work at all now.

Fortunately, I had my files, the important stuff.  The next few steps were simple, re-image the server with a clean install, sudo apt-get on apache2, php5, mysql-server, proftpd, ftp the files in the appropriate places.

This is also where I did right on permissions, like I should have done to start with.  Instead of screwing with permissions themselves using CHMOD, I set the appropriate ownership with CHOWN.  This was partially necessary, for example, the files created by MySQL normally own and belong to the mysql user and group.  The ones I restored, were all owned by root.

I also took this opportunity to pair down some of the cruft I’d accumulated.  I kept a copy, but it all doesn’t need to go back.

I feel like the end result worked out well, everything is mostly back on line.  I found later that something had gone wrong in backing up the SQL files for both Joshmiller.net and Blogging Intensifies.  Fortunately there isn’t anything on JoshMiller.net since I had purged it all.  And I don’t post here super often so I only came up missing 2 posts from my last back up, I was able to recover both posts from Google’s Cache pages.  Everything for both The Zippy Zebra and Treasured Tidbits came over though, which was my main concern.