Server

Purging WordPress .ico Malware… (Hopefully)

So, this is a hopefully, because it’s been a bit since I have done this, and things seem to be clean. So, there is a reasonably common bit of Malware out there that seems to affect WordPress sites, I say reasonably common, because in my time looking for a solution, I have come across a fair number of others with the issue, but no solutions. And I have tries several solutions. As of now, I have been a few months clean, and without hacky work-arounds. I’m going to attempt to run through what I did that held it at bay, and what seems to have finally managed to purge it, in hopes of helping others.

The Malware itself basically would occasionally redirect the blog domain to a spam website. I say occasionally, because it’s not all the time, and with enough anti-advertising stuff in your browser, you may never see it happen. I have personally, never once seen it happen, on any of the sites I run on this web space. I first found it was infected because occasionally, my wife would mention that someone she had linked her blog to, was getting sent to a spam website. Initially I thought maybe someone was mistyping the domain along the line. My wife also said it would happen occasionally. In my work combating this malware, it seems like the actual redirect occurs slowly over time, as the infections spreads.

It also will spread across sites hosted on the same server. Which made it extra tricky to fight because I had to juggle several sites at once.

Part 1 – Keeping It at Bay and How it Spreads

I have no idea how the infection was initially started, which is rough, because that would be key to KNOWING it’s gone. As near as I can tell, the initial source of the infection is int he uploads directory of a blog. It eventually starts to add “gibberish code” to files like wp-config.php and settings.php. I say “Gibberish Code” because it’s actual PHP, but it’s very messy in it’s design and encoding to make it hard to read to locate files. The gibberish code would generally show up at the top of the files, but could be elsewhere.

Eventually, random folders would start showing up in the root WordPress directory, sometimes with gibberish names, sometimes with specific spammy sounding names, sometimes with names that appear to be part of the blog (like ‘site’ or ‘blog’).

The first step in holding this at bay was so dump all write permissions for several critical WordPress files that kept being infected. This seemed to only sort of help, the problem was more that the owner, www-data, still could write to the files.

The next step was to convert all of the web files to an alternative user account as the owner, then set the files so www-data could only read. This created a new problem, it meant I could not easily update anything or upload images for blog posts easily. Since www-data had no permissions to write anywhere. If I was making a new post, I would have to SSH into the server, temporarily change the permissions, then change things back.

Pain in the ass.

My temporary fix there, was to keep the current year as writable, and run a script that would probe for malware files and delete them. There were two scripts, one for the hidden .ico files that would crop up and one for any .php files that were in the Uploads folders. Both run with a cron job.

#!bin/sh
/usr/bin/find /var/www/html -name ".*.ico" -exec rm {} +

This is, admittedly, an EXTREMELY Hack way to correct this problem. Hack less in the sense of “computer hacker”, hack in the sense of “jankey or shoddy”. But it worked, while I figured out the root issues.

Part 2 – Fixing the Issue

Eventually I sat down and just sort of rebooted everything, all at once. I started with everything set up as an alternative user permissions and locked down. I then scrubbed out all infected php and ico files from the upload folders. I then thoroughly scrubbed out the wp-config files. Basically, any files I would need to moved to a fresh WordPress install, which was the uploaded images, and the configurations, were completely sanitized.

Next, I downloaded a fresh copy of WordPress, expanded it out and made copies for each site folder with websitename_new. After that, I copied the uploads folders and wp-config files to the new copies. Then i renamed each current folder as websitename_old, and renamed the new ones as simply websitename. (I actually did this and the subsequent steps once at a time for each site). This made the new, fresh installed copies live.

Except they have no plug ins and no themes. I did not transfer any old theme files or plug in files, for worry of infection. Instead I went into the old folders to get a list, then redownloaded each theme and plug in to the fresh copies. This meant doing some reconfiguring but it was worth it for clean copies. I also left out anything that wasn’t absolutely essential to the basic look and operation.

Site note, when a fresh install copy is made live, it may not load until you go into /wp-admin, and change the theme to literally anything else (Generally, the current year WordPress Default works). The config files will still be looking for the non-existent old theme.

With everything fresh and ready to go, I deleted the old potentially infected copies, to ensure the infection was now completely purged. After that, I created a backup folder, and copied all of the current fresh versions of the site folders to the back ups. This way, in the event of a reinfection, I could simply, slap a fresh back up in place easily. It might be missing a few recent images, but it would be way less work.

Still worried, I then reverted the permissions for the sites back to www-data, but I did them one at a time, roughly a week apart. Carefuly checking for reinfection with each change.

So far so good, I have not had any problems. Here is hoping it stays.

Migrating Mail-In-A-Box to a New VPS

A few years ago, I started running my own mail server using Mail-In-A-Box. Four years or so actually, if the age of my old server was accurate. I have several different email addresses, mostly to better segment out content. I have done this with Reddit, and Twitter, and TT-RSS, and probably other things. In my Mail-In-A-Box I run email for 3 domains, two of mine, one for my wife’s. Overtime I may eventually migrate all of my email to it, at this point, I am a little worried about being blacklisted, so I mostly use it for secondary, receive only, email aggregation.

For a while I’ve been putting off migrating the system to a new VPS. It’s been running on Ubuntu 14.04 since it was created. Newer MiaB won’t run on 14.04 and I can’t distro update the machine. The only choice is to roll a new VPS and migrate the mail.

I use Digital Ocean for my online services, feel free to sign up with the link in the side bar if you want, I get a little kickback if you do. It’s easy to use and affordable. Plus in cases like this, I can spin up an extra VPS, then easily destroy it and spin up a new one, when I discover that MiaB only works up through 18.04, so 20.04, which I used initially, won’t work. Also having the extra server just means a temporary bump in my billing for the month.

The basic process for migrating Mail-In-A-Box is here, in the official documentation. I had a few hiccups along the way but I got them ironed out.

First step was creating the new machine. I mentioned above, I first made a 20.04 machine, but found that doesn’t work, so I killed that and made a new 18.04 machine. Before anything else, I did a few security based housecleaning tasks. The server was creating with Shared Keys log in set up, but it only had a root account. So I created a new user and made them a sudoer. I also copied the SSH keys from root to the user.

adduser Username
usermod -aG sudo Username
cp ~/.ssh /home/Username
chown Username:Username /home/Username/.ssh -R

Next step was to add the new user to the SSH users and secure up that access.

sudo pico /etc/ssh/sshd_config

Then edit:

#Port 22

To a custom port and change:

PermitRootLogin no

Finally add:

AllowUsers Username

Lastly restart the ssh server with sudo service sshd restart. Then test the connection using the regular user. If that works, then disconnect from the root session and continue on the regular user.

I was doing an upgrade but the fresh install guide is here. All I needed was the set up line really, which takes a minute to run but does an initial set up of Mail-in-a-Box.

curl -s https://mailinabox.email/setup.sh | sudo -E bash

The next part was the trickiest bit. I linked the migration article above but I ended up trying to simplify things a bit. On the old machine, I stopped the mailinabox service, so no new mail would come in, then ran the backup python script as described int he article above. I found it was easiest to just connect to the server using Filezilla using SSH FTP, which meant importing my keys to Filezilla. It’s in the settings under SFTP. Something to keep in mind if you set a custom port is you’ll need to add sftp:// before the IP address.

Things are a little tricky here, since root owns the backup folder. I ended up doing a sudo copy into my user home directory, then a chown on the folder to give my user account access to the folder. This meant Filezilla could see the folder and download it to my local machine. There are way to directly transfer between the new and old server, but between custom ports and SSH keys and permissions, I found it was easiest just to download to my local laptop. Afterwards, I connected with SFTP to the NEW server, and pushed the backup folder to the new server. You need the whole folder with the “secret_key” text file and the encrypted folder and files. Basically, this is all the settings and emails.

Next step was to ssh into the New Server, go to the freshly uploaded backup directory, and import the old files, as described in the link. This is two commands run, separately.

export PASSPHRASE=$(cat secret_key.txt)

sudo -E duplicity restore --force file:///home/Username/backup/encrypted /home/user-data/

This takes a minute to run. The next step listed is to rerun the mailinabox set up with “sudo mailinabox”.

I had trouble here. Nginx would not restart. After sound troubleshooting I found it was an issue with SSL. Basically what seemed to happen was the restore, pulled the old SSL certs. Or maybe it was looking for the old SSL certs. Whatever the case, the fix was this process.

rm -rf /home/user-data/ssl/*

The fix was to delete the SSL certificates. then run “sudo mailinabox”. Everything started up. I verified I could log into the control panel and the mailbox using the UP address of the new server. I verified that all my custom DNS records existed, these are needed since the Glue Records point to the Mail-In-A-Box machine but because I host my websites on a separate machine, I have to have DNS records set up appropriately.

One thing I noticed was the SSL Certificates seemed to be wrong, which meant things worked, but would cause annoying security messages. I am not sure if this was related to deleting the certs above, or just that it was still looking for the old IP address. Whatever the case, I did a manual update with certbox for my MiaB Subdomain using

sudo certbot certonly --force-renewal -d Subdomain.Domain.comHere

Another minor issue I ran into, doing this needs to drop a file either in the webroot folder, or spin up a temporary web server to host it’s own file. I couldn’t find the webroot for the custom MiaB set up (it was not /var/www/html) so I temporarily ran “sudo service nginx stop”, then ran the above certbox command, using a temporary webserver option, then “sudo service nginx start” to restart Nginx. NGinx had to be stopped since otherwise it is using Port 80, and the temporary server can’t start to runt he certificate verification process.

Another note, I am not sure if the –force-renewal option is needed above. It didn’t throw out any errors and it fixed the issue, so I left it.

The final step was to go to my Domain Registrar and update the name servers and Glue Records to point to the new Server IP. After a short bit of waiting, eventually the mail server URL connected to the admin and web consoles. I did some test send and receive of emails between my server and gmail to verify everything was working properly. One nice bit, the newer MiaB has a different interface for Roundcube webmail, so I could easily tell if I was going to the new or old server.

Once everything was satisfactory, i went back to Digital Ocean and powered down the old server. If everything is still working in a few days, I will destroy the old server, so I don’t have to keep paying upkeep on it. One thing to keep in mind, both the old and new servers require a specific hostname, so they will be named the same, so double check that you are powering down and deleting the correct server. some easy ways to verify are IP address, or server age. The old server is several years old but the new server is several days old.