VPS

Migrating Mail-In-A-Box to a New VPS

A few years ago, I started running my own mail server using Mail-In-A-Box. Four years or so actually, if the age of my old server was accurate. I have several different email addresses, mostly to better segment out content. I have done this with Reddit, and Twitter, and TT-RSS, and probably other things. In my Mail-In-A-Box I run email for 3 domains, two of mine, one for my wife’s. Overtime I may eventually migrate all of my email to it, at this point, I am a little worried about being blacklisted, so I mostly use it for secondary, receive only, email aggregation.

For a while I’ve been putting off migrating the system to a new VPS. It’s been running on Ubuntu 14.04 since it was created. Newer MiaB won’t run on 14.04 and I can’t distro update the machine. The only choice is to roll a new VPS and migrate the mail.

I use Digital Ocean for my online services, feel free to sign up with the link in the side bar if you want, I get a little kickback if you do. It’s easy to use and affordable. Plus in cases like this, I can spin up an extra VPS, then easily destroy it and spin up a new one, when I discover that MiaB only works up through 18.04, so 20.04, which I used initially, won’t work. Also having the extra server just means a temporary bump in my billing for the month.

The basic process for migrating Mail-In-A-Box is here, in the official documentation. I had a few hiccups along the way but I got them ironed out.

First step was creating the new machine. I mentioned above, I first made a 20.04 machine, but found that doesn’t work, so I killed that and made a new 18.04 machine. Before anything else, I did a few security based housecleaning tasks. The server was creating with Shared Keys log in set up, but it only had a root account. So I created a new user and made them a sudoer. I also copied the SSH keys from root to the user.

adduser Username
usermod -aG sudo Username
cp ~/.ssh /home/Username
chown Username:Username /home/Username/.ssh -R

Next step was to add the new user to the SSH users and secure up that access.

sudo pico /etc/ssh/sshd_config

Then edit:

#Port 22

To a custom port and change:

PermitRootLogin no

Finally add:

AllowUsers Username

Lastly restart the ssh server with sudo service sshd restart. Then test the connection using the regular user. If that works, then disconnect from the root session and continue on the regular user.

I was doing an upgrade but the fresh install guide is here. All I needed was the set up line really, which takes a minute to run but does an initial set up of Mail-in-a-Box.

curl -s https://mailinabox.email/setup.sh | sudo -E bash

The next part was the trickiest bit. I linked the migration article above but I ended up trying to simplify things a bit. On the old machine, I stopped the mailinabox service, so no new mail would come in, then ran the backup python script as described int he article above. I found it was easiest to just connect to the server using Filezilla using SSH FTP, which meant importing my keys to Filezilla. It’s in the settings under SFTP. Something to keep in mind if you set a custom port is you’ll need to add sftp:// before the IP address.

Things are a little tricky here, since root owns the backup folder. I ended up doing a sudo copy into my user home directory, then a chown on the folder to give my user account access to the folder. This meant Filezilla could see the folder and download it to my local machine. There are way to directly transfer between the new and old server, but between custom ports and SSH keys and permissions, I found it was easiest just to download to my local laptop. Afterwards, I connected with SFTP to the NEW server, and pushed the backup folder to the new server. You need the whole folder with the “secret_key” text file and the encrypted folder and files. Basically, this is all the settings and emails.

Next step was to ssh into the New Server, go to the freshly uploaded backup directory, and import the old files, as described in the link. This is two commands run, separately.

export PASSPHRASE=$(cat secret_key.txt)

sudo -E duplicity restore --force file:///home/Username/backup/encrypted /home/user-data/

This takes a minute to run. The next step listed is to rerun the mailinabox set up with “sudo mailinabox”.

I had trouble here. Nginx would not restart. After sound troubleshooting I found it was an issue with SSL. Basically what seemed to happen was the restore, pulled the old SSL certs. Or maybe it was looking for the old SSL certs. Whatever the case, the fix was this process.

rm -rf /home/user-data/ssl/*

The fix was to delete the SSL certificates. then run “sudo mailinabox”. Everything started up. I verified I could log into the control panel and the mailbox using the UP address of the new server. I verified that all my custom DNS records existed, these are needed since the Glue Records point to the Mail-In-A-Box machine but because I host my websites on a separate machine, I have to have DNS records set up appropriately.

One thing I noticed was the SSL Certificates seemed to be wrong, which meant things worked, but would cause annoying security messages. I am not sure if this was related to deleting the certs above, or just that it was still looking for the old IP address. Whatever the case, I did a manual update with certbox for my MiaB Subdomain using

sudo certbot certonly --force-renewal -d Subdomain.Domain.comHere

Another minor issue I ran into, doing this needs to drop a file either in the webroot folder, or spin up a temporary web server to host it’s own file. I couldn’t find the webroot for the custom MiaB set up (it was not /var/www/html) so I temporarily ran “sudo service nginx stop”, then ran the above certbox command, using a temporary webserver option, then “sudo service nginx start” to restart Nginx. NGinx had to be stopped since otherwise it is using Port 80, and the temporary server can’t start to runt he certificate verification process.

Another note, I am not sure if the –force-renewal option is needed above. It didn’t throw out any errors and it fixed the issue, so I left it.

The final step was to go to my Domain Registrar and update the name servers and Glue Records to point to the new Server IP. After a short bit of waiting, eventually the mail server URL connected to the admin and web consoles. I did some test send and receive of emails between my server and gmail to verify everything was working properly. One nice bit, the newer MiaB has a different interface for Roundcube webmail, so I could easily tell if I was going to the new or old server.

Once everything was satisfactory, i went back to Digital Ocean and powered down the old server. If everything is still working in a few days, I will destroy the old server, so I don’t have to keep paying upkeep on it. One thing to keep in mind, both the old and new servers require a specific hostname, so they will be named the same, so double check that you are powering down and deleting the correct server. some easy ways to verify are IP address, or server age. The old server is several years old but the new server is several days old.

My Experience with Cloud At Cost VPS

I’ll try to remember to do an update post if things iron out but I wanted to throw out my experience so far with Cloud At Cost, a VPS provider that charges a one time upfront fee for their VPS, as opposed to a monthly subscription or other reoccurring cost.  They had a sale for Black Friday, 90% off (later 80% off).  This meant that for $7, you could buy a VPS, for life.

This seems really really “too good to be true”.  So I did some research.  First, apparently Cloud at Cost has some horrible uptime and shouldn’t be used for anything production level.  I can live with that, it’s cheaper than buying another Pi for a project server, and this one’s in the cloud!

I also read one report that the servers are insecure and CloudAtCost has cancelled accounts for lifetime subscribers for hosting malicious software, after the servers were hacked due to C@C’s poor server security.  This mostly means, nothing production or crucial.  It’s something I can accept the risk of for the cost.

I opted for two of the tier 3 servers at $14 each.  This got me 2x servers with the following specs, 4 cores each, 2GB of RAM each, and 40GB of space each.  What’s notable though is that as near as I can tell, you pay for e cores, RAM, and HD space, it all just pools together in the end and you can build as many servers as you’d like.

That’s what I chose, and paid for.  On my user dashboard it shows that I have 2 products (the ones I chose listed).  Weirdly, I received several additional invoice emails, though I never ordered more, nor was I charged for more nor does my account say I have more.  When I log into the control panel to manage my servers, It shows I have available 28 cores, 14GB of RAM, and 240 GB of hard drive space.  Basically, it’s like I have 6 servers, instead of 2.  I have no idea if they will realize what happened and scale my resources back or not but I went ahead and kept things within the range I paid for, for now.

It took forever to get the initial purchase set up and available as well, though I’m blaming that on the Black Friday sale.

The interface of the site leaves a LOT to be desired.  There is a user dashboard that tells what you’ve purchased at members.cloudatcost.com.  Managing the servers actually occurs at panel.cloudatcost.com.  There isn’t an obvious link between these two sites that I could find.  Also, the panel site said it used your same password but it didn’t work and I had to do a reset to access the control panel.

The servers also took a good while to get spun up, though I’m blaming Black Friday again.  I created one Windows 7 Ultimate server and one Ubuntu 14.04 server.  I was thinking of using the Windows server to run some Corrade bots in Second Life.  Not sure yet on the Ubuntu server.  I could access both servers via the console but networking seems to be offline on both since I couldn’t ping out to anything.  I tried to set up Remote Desktop on the Windows 7 server but since networking is not working, I couldn’t access it.  The same for SSH on the Ubuntu server.

I checked around and others on Reddit had the issue and it seems to be part of the problems this provider has and should resolve itself… eventually.

So the initial experience has been interesting, for sure, and I can definitely say I won’t be putting anything that NEEDS to be up on these servers, like any sort of website for example.

 

 

I Screwed Up

So, for a little while now, one of the sites I host on my VPS has had some sort of malware.  I have no idea how it got there but I have ideas about fixing it if I could find it.  There are plenty of sites that will tell you that your website is infected.  There don’t seem to be any that will also tell you “It’s probably in this file here, go look there.”  Instead it’s all “We’ll fix it for a small monthly subscription of $50/month.”

I think I’ll pass on that one.

Instead, I opted to simply rebuild the website from the ground up.  It’s a simple process really, set up a fresh WordPress install, ad the appropriate plug ins, copy the images in the Uploads folder, and do a quick export/import of the database.  I also wanted to make sure I got the permissions right, to avoid any future malware issues, since this was the likely culprit for how the malware got there.

This is where I screwed up.  Instead of doing a CHMOD on the local directory, I mistakenly did a CHMOD on /* -R.  Or in other words, everything in the root directory, Recursively through each directory.  Or in other, other words, “everything”.  It actually failed to run on a bunch of files, likely because they were in use.  It did however break SUDO, which meant I couldn’t easily try to change anything back.  It also immediately killed every website I host since they all use MySQL which could no longer use it’s databases, because it didn’t have permissions.

I don’t host anything major at least.  A couple of personal blogs, my wife’s two blogs, some side projects like TinyTinyRSS.  My main concern were my wife’s blogs, frankly, no one reads my shit at all anymore (why are you here???), lots of people read both of her blogs.

If this were a physical server, I’d load a recovery CD and backup or even just reinstall from there.  This is a VPS though.  There isn’t a physical machine I can access and really, there probably isn’t even a physical machine at all, not a dedicated one.  There may be a dozen other servers on the same physical machine as my VPS.  Fortunately, with the use of a support ticket, Digital ocean will mount a virtual recovery disc to your virtual server.

So I managed to get access to the server files.  I set about with two plans at this point.  Worst case scenario, I would need to reimage the server and rebuild everything.  I’ve done this sort of thing many times over the years moving from server to server, I’m actually pretty good at it.  Getting the data was the important part, so I started some downloads of the data.  Honestly, this was always the only option, but I was hoping I could get the old set up running because it would make my life easier.  If I could get MySQL working I could make proper back ups instead of trying to use the raw files, something I’ve never done (it wasn’t hard in the end).  So, 50,000 files later, I had all of the needed files downloaded.  I probably could have saved some time and just reinstalled the core WordPress files but I wanted to keep things as pain free as possible to avoid any more screw ups.

How to restore the server.  The problems stem from permissions, as in, nothing has permissions on anything.  So the simplest solution seemed to be to set the files all to 777, or open access to every user, group and everything.  This is absolutely horrible practice for a live server and should not be done.  However, I needed ten minutes or so to dump some SQL files and a few other proper back ups that would be much easier in a live environment.

Setting everything to 777 didn’t work, for starters, all those system files that were previously inaccessible, were now accessible, since the recovery CD wasn’t using them.  So now EVERYTHING became 777.  I don’t know much beyond that other than it flat out refused to work at all now.

Fortunately, I had my files, the important stuff.  The next few steps were simple, re-image the server with a clean install, sudo apt-get on apache2, php5, mysql-server, proftpd, ftp the files in the appropriate places.

This is also where I did right on permissions, like I should have done to start with.  Instead of screwing with permissions themselves using CHMOD, I set the appropriate ownership with CHOWN.  This was partially necessary, for example, the files created by MySQL normally own and belong to the mysql user and group.  The ones I restored, were all owned by root.

I also took this opportunity to pair down some of the cruft I’d accumulated.  I kept a copy, but it all doesn’t need to go back.

I feel like the end result worked out well, everything is mostly back on line.  I found later that something had gone wrong in backing up the SQL files for both Joshmiller.net and Blogging Intensifies.  Fortunately there isn’t anything on JoshMiller.net since I had purged it all.  And I don’t post here super often so I only came up missing 2 posts from my last back up, I was able to recover both posts from Google’s Cache pages.  Everything for both The Zippy Zebra and Treasured Tidbits came over though, which was my main concern.