Synology

Synology NAS

NAS Recovery

Ramen Junkie 0Comment

What a fun time it’s been with my Synology NAS lately. And before I get going here, I want to make it clear, nothing here is a knock against Synology, or WD for that matter. The NAS I have is like ten years old, if it had failed, I was already pricing out a new, up-to-date, Synology. Heck, I may still get one anyway.

But for now, it seems to be working fine again.

As I mentioned, it’s been like ten years or so. I ran on one 4TB WD Red drive for a long time. Eventually, I did add the second drive to make things RAID and redundant. Sometimes last year, my original WD drive died on me, I ordered a replacement and swapped it out, and everything was fine.

Sometime, maybe a month ago now, I received an error about a drive failure. The newer drive was already showing bad. I made up an RMA request with Western Digital, wiped the drive, and then sent it in. They sent me a replacement.

A short time before the replacement arrived, I found another error, “Volume has crashed”. It showed that the, at the time, one drive was “Healthy”, and I could still read all of my data. This was starting to feel a bit suspect. I have everything important backed up online to OneDrive, but just in case, I started pulling things off to other storage as a secondary backup. This basically involved eating up all the spare space on my project server (temporarily) and using a USB enclosure and an old 2TB drive that, seems to be failing, but would work well enough for short-term storage. The point was, if I had to rebuild things, I would not have to download mountains of data off OneDrive. USB transfer is much easier and faster.

With everything backed up, I received the replacement for my RMA drive. My hope was, I could attach the replacement drive, and whatever was causing this Volume to show as crashed would clear itself out. Unfortunately, I could not really interact with the volume at all.

After several attempts at various workarounds, I gave up on recovering the Volume. I had the data, that is what matters.

I pulled the crashed drive out, which allowed me to create a new volume using the new drive. I then recreated the set of shared network folders, Books, Video, Music, Photo, General Files, as well as reestablished the home folders for users.

Fortunately, because I kept the same base names, all of my Network Mapped drives to the NAS, just worked. Fixing my own connections would be easy, hassling with connections on my wife and kids’ laptops, would be a pain. They get all annoyed when I do anything on their laptops.

Unfortunately, the crashed volume seems to have killed all of the apps I had set up. This is not a huge loss honestly, I don’t actually use most of the built-in Synology apps anymore beyond Cloud Sync and the Torrent client. The main one I need to reconfigure is the VPN client. I may just move that to a docker instance on my project PC. Fortunately, last year, I pulled both my email and blog archives off of the NAS. All my email is consolidated again in Outlook, and my blog archive is in a Docker container now. This means I can just remove all of these apps instead of reinstalling them.

I did find that I had failed to do a fresh local backup of my “Family Videos” folder, but I was able to resync that down from the One Drive backup. Speaking of which rebuilding all those sync connections was a little tedious since they are spread across two One Drive accounts, but I got them worked out and thankfully, everything recognized existing files and called it good. While I didn’t put everything back on the NAS, I have a few things that are less important that I’m just going to store on the file server/project server, I somehow gained about 1.5TB of space. I’ve repeatedly checked and everything is there as it should be. I can only speculate that there was some sort of residual cruft that had built up over time in logs or something somewhere. I also used to use Surveillance station, so it’s possible I had a mountain of useless videos stored on it.

In general, it’s actually been a bit of an excuse to clean up a few things. I had some folders in there that used to sync my DropBox and Google Drive, neither of which I use anymore, for example.

I am 99% sure everything is back in working order, and the last step I keep putting off is to whip the drive from the crashed volume (it still reads healthy) and read it to the current, new volume.

It’s been a hassle, but not really that bad. The main hassle is because it’s large amounts of data, it often means starting a copy and just, letting it run for hours.

Organizing Synology NAS

Organizing Digitally – The NAS

Ramen Junkie 0Comment

I want to do a sort of series about how I have my digital world organized but I was sort of trying to decide the best place to start. I wanted to run down some file structure methods, and I want to run down Office 365 use, and previous backup methods, but ultimately, the core of everything, is my NAS.

So this is also sort of a followup to that last set of articles about my Synology NAS. I am sure there are other ways to do a lot of what the Synology does, but there are a lot of simple to use built in features that are nicely integrated into my workflow. It’s a little pricey to set up initially, with the box and the drives, but the reality it, any good solution will be.

Features I Use

These aren’t in any particular order, but I wanted to touch on the aspects of the NAS that I use pretty regularly.

  • OpenVPN – I used to go to a lot of hassle opening up firewall ports on my home network to different devices and machines, so I could access web cams or SSH to different servers and blah blah blah. This is a bit of a security problem, since it means lots of open target points as well. I’ve long since dumped that in favor of OpenVPN, which is built into the NAS. I connect through my laptop or my phone to my home network, then I connect to whatever network drive or SSH connection I need to. It works perfectly and requires way less hassling with the firewall.
  • Download Station – This is essentially a Tor downloader, though I think it can handle a lot of other url types. I don’t really directly interact with this, I keep a folder for incoming files that I occasionally sort and a watch folder for Torrent files that it pulls from. The fun part is syncing the watch folder using One Drive, so I can dump Torrent files to it from anywhere. And for what it’s worth, I don’t use this for piracy, primarily I use it for downloading Humble Bundle purchases. A bundle often has 20+ items, so I will bulk download the torrents (to save HB some bandwidth) and then dump them into the watch folder.
  • Video Station/DS Video – I tried running Plex for watching digital movies from the NAS but it was flaky as hell since there isn’t an official Synology app and Plex is increasingly pushing their subscription nonsense instead of just being a client/server self hosted application. Fortunately, there are Synology Apps for Fire TV (Which I use for streaming on both TVs). So I’ve sorted all of my home movies into the Videos folder and (for a future blog post) encoded them to be easily accessible and compatible.
  • Photo Station – Ok, I don’t actually use this… yet… but I want to revisit it going forward. I want to do a separate post on photos with more details, but basically, I wasn’t using the Photos folder for backup purposes, and that situation has changes recently.
  • Audio Station – I have a ton of music from different sources compiled and sorted together. It’s not my primary GoTo for music, but I want to get more organized playlists going so I can more easily use this for playing my music. For the most part, I am fine with just sticking music ON my phone though.
  • Mail Station – I don’t use Mail Station for actually sending emails, but I did set up the Mail Station server and I use it as a deep archive of emails. I essentially have all my email I have ever sent, going back to the 90s, pulled forward through various email clients, and now it’s all dumped into a Mail Server in a sorted, searchable archive.
  • Cloud Sync – Cloud Sync lets you hook your Synology to various cloud drive services and sync them to your local drives. I’ve got several Dropbox accounts that I have used in the past (Personal, server syncing, each family member) and now a couple of One Drive accounts for backup and personal document sync all linked. It even does Google Drive.

Features I Stopped Using

There aren’t a lot of features I have stopped using, but there are a couple.

  • Web Station – The Synology comes with an optional Webserver and a weird WordPress system that can be enabled. This has been weirdly buggy since day one and I already have plenty of experience managing LAMP stack servers. I recently disted off one of my older Pis, set it up with WordPress and moved the primary use I was using the Synology Web Station for to the Pi. Mostly, It was just a WordPress Archive of all of my old blog posts from various blogs. The links were weird and didn’t work properly because it didn’t quite understand subdirectories or something. The images were present but they didn’t always work because they pointed to old URLs and working the SQL system to change them always came off as wonky. Basically, I didn’t need this archive to be on the NAS and it was an easy thing to just offload to another device.
  • Cloud Station Server – This is a back up system for devices and computers. It will sync specific local folders to a folder on the NAS as a backup. Maybe I was doing something wrong but it always felt really flaky as well, so I just sort of stopped using it. I had it on every laptop in the family for a while but as laptops were replaced, then things started getting weird and getting others to grok how to pull back their files wasn’t super easy either. The better solution I have found is to just give everyone a shared folder specific to them that they can shove files they want to keep into. For my personal use it was just redundant because my entire workflow for years has essentially been cloud based with Dropbox or One Drive keeping everything backed up by default.
  • Surveillance Station – I still sort of use this, but all of my webcams died except one, which doesn’t have night mode anymore. So, it exists and I would use it, but I don’t really use it much anymore. Also. frankly, there was never anything worth seeing on the recordings.

Workflow

The real workflow from the NAS comes from shared folders. Everyone has access to the Family Photos folder mapped to their laptops. I created a shared folder for all of the Blog graphics my wife was using for her blog work that everyone can access since my daughters both helped her with that. They use a shared drive for all of the Ebay and Mercari photos they work on.

I keep folders for photos, and videos and ebooks. I keep folders for important family documents like Tax Returns. All of this can easily be synced to a backup in the cloud and I have a couple of USB keys and loose drives that I do periodic manual backups to, that get stuck in a fire proof safe.

It also lets me map other network drives in as well, for shuffling files around. I have a whole second Linux box set up that has another 4TB or so space in it across several drives, that I use to store less important files like Installable programs and games, ISOs, temporary files for video editing projects, a mountain of internet memes and images saved over the years, music concerts I’ve downloaded, etc. Plus I can map things like, the web root for my Raspberry Pi, or set up a one way(ish) SSH tunnel to my Webserver for pulling backups through.

The box itself sits behind the TV upstairs, and if there ever was a fire or something, it’s likely one of the things I might try to grab on the way out the door, but I’d like to thing my system is robust enough that even if it were lost anything important would be recoverable.

Synology NAS

Synology Phase 03 – The Apps

Ramen Junkie 0Comment

ScreenShot253 The key component of the Synology is the Software.  You can buy cheaper NAs devices if you just want a network storage device.  Honestly, the price justification is almost entirely in the software, though the Hardware RAID (as opposed to Software RAID) is a partial factor as well.  The box itself isn’t all that sophisticated or exciting honestly.

I don’t plan to cover every available app by any means this is just sort of a run down of some of the apps I find useful, and probably the ones that are most commonly used.

Photo Station

The main reason I even bothered with investing in a NAS at all was because of my Photo collection.  Everything else is a bonus.  I’ve got 250GB of family photos I’ve taken over the years, plus a whole mess of other photos from Blog Posts etc.  I’ve gone through a drive crash on them and the drives I’ve been using are aging rapidly.  The wife doesn’t care to lose photos either.

Photostation is of course, made for Photos, and it’s pretty great.  The interface is very similar to OneDrive, which is is probably my favorite interface of all the photo services I’ve used.  Everything flows together to make a wall of photos, real great for easy navigation.  There are also tagging functions which I plan to use later once everything is loaded.

Audio Station

I’ve tried several solutions for streaming my audio collection to my phone.  I had sort of mostly settled on Google Play Music since it’s the only service that would let me upload my entire collection of some 20,000 songs at once.  I don’t really NEED all these songs at once but I do like having the option.  I buy most music these days via Amazon and so streaming via Amazon is always an alternative as well.

Unfortunately, my recent conversion to Windows Phone from Android means neither service is an option.  I’ve been back to putting music on my device like a caveman.  Fortunately, there is a DSMusic app available for Windows Phone, and I don’t have to worry about any limitations of any service, it’s just all there.  Now I just need to build some good playlists.

A secondary benefit here, Google Play Music was one of the few Google Services I still sometimes used.  I’ve worked pretty hard to divorce myself from all things Google for a variety of reasons, primarily privacy concerns and secondary they are starting to push their own semi proprietary services on the web over long standing more open ones.  Basically, they are using their considerable size to bully everyone to their methods.  “Don’t Be Evil” doesn’t seem to be a thing anymore, but anyway, Google isn’t the topic here.  Having a good Google Music alternative that works on WP is.

Download Station

I don’t use Torrents too often, mostly for my Humble Bundle downloads, but the Synology has a really nice built in Torrent client.  I don’t have to worry about keeping a program running elsewhere or drive space on my PC in use, it just downloads them right to the NAS.  There is even some auto extraction settings, though I have not looked into those yet.

Web Station

A nice little bonus here I wasn’t quite aware existed before buying this NAS, it can function as a web server.  I’ve long given up hosting my own web sites from my house but I do keep some wordpress and other files on an internal web server for archival purposes.  The web server has allowed me to archive these files off my Ubuntu Server to the NAS, which also means I get the backup functionality of the NAS itself.

Note Station

Another unexpected surprise, though I have not explored it completely yet, the Synology includes an app called Note Station, which can sync (or at least download) from Evernote.  I’ve been racking my brain for a while on a good way to backup my Evernote notes, with ideas ranging from Print to PDF using some script to just pulling it weekly to a PC client.  Problem solved.

Cloud Sync

Another nice backup feature, The Synology can hook into and sync with both Dropbox and One Drive.  I use One Drive for some backups and Dropbox for some phone syncing so pulling both to an internal local storage is a plus.  I may even look into using One Drive as a secondary backup like I had originally planned.  Office 365 now includes unlimited storage on One drive, in addition to client licenses for 5 copies of Office (there are conveniently 5 people in my family all with PCs).  It’s a really tempting offer, and with it I could set up the Synology to start pushing all (or select) data to the cloud for an offsite backup.

Surveillance Station

Last, and the ONLY feature I have been disappointed with so far is the Surveillance Station.  I mentioned recently setting up cameras for monitoring and security.  I currently have three cameras and may install a few more.  The Synology only allows the use of 2 cameras before needing to purchase additional licenses.  I’m not super irritated about the additional cost, I get the whole “It supports dev costs”” thing.  My problem is that licenses are $60+, EACH.  If I wanted to add another 3 cameras like I am considering, I’d need 4 more licenses, or $240.  I’d be alright with maybe $10-$15 per seat, or even $60 for “unlimited” (within the capacity of the device) but $60 each is a little ridiculous.

I suspect there is some lame ass license fees Synology has to pay to someone involved but that is also kind of giving them the benefit of the doubt.

I’m still super satisfied with the box, but having better/cheaper access to Surveillance station seats would be eliminating my Ubuntu server completely.

Synology NAS

Synology Phase 02 – The Drive

Ramen Junkie 0Comment

4TB WD RedSo I’ve been sitting on my Synology NAS for not quite a month now.  I had already stuck a spare 250gb drive in it just for the sake of using the thing some and getting to play around with some of the software.  I’ll touch on the software more in a future post, though the short is, I’m pretty impressed with everything except the Surveillance Station.

I was going to wait a bit longer to pick up the drives but “fake” using it really had me wanting to REALLY use it.

To really use a NAS, especially as a RAID, you can’t just put any drive into the enclosure, RAID drives are designed to withstand a much more frequent number of reads to help keep them alive for longer periods.  They are designed to be more robust in general since chances are the user will really be relying on them for potentially sensitive data (say, a business keeping business records).  My only apprehension about this is that I replace an awful lot of drives in the servers at work, especially the HUGE arrays that serve video.  On the other hand, these are machines that are pushing content to hundreds or thousands of people at once so the usage on these drives is probably pretty astronomical.  It’s not really apples to apples.

I’ve had a bit of an internal debate on which drives to get, and it generally comes down to price, reliability, and size.  Price is a factor f the other two points, more expensive tends to be better.  Size is a mater of needs, so that really just leaves reliability.  Everything I have seen, and I mean EVERYTHING from recommendations on Amazon to 4chan threads on /g/ to Reddit search results and posts and random message boards says, “Western Digital Reds”.  My only apprehension on the Reds is that Newegg’s reviews section is riddled with people complaining of failures.  Everything on New Egg goes counter to what I see elsewhere on WD Red drives.  I almost went with another brand but I decided to go with the WD Red drives.

I also almost cut down to 3TB drives.  They are $50-$60 cheaper and do I REALLY need 4TB.  I had to remind myself that yes, I really need the 4TB and the reality is, for the long haul, I SHOULD be buying 6TB drives, which is the largest this unit will support (I think).

So then there is the matter of the drive swap itself.  This didn’t quite work out as smoothly as expected, though if I had been putting two drives in instead of just trying to transition to one larger drive things would have gone better.  I had already been using one drive and had set up several things on this drive.  I inserted the new drive in the empty bay and then set both drives up as a mirrored array.  Wait a few minutes for the sync to happen, then remove the small older drive.

This is where there was some hang up.  Removing the small drive caused the NAS to believe a drive had failed.  i didn’t want to leave it like this and I don’t have a second 4TB drive to stick in there yet.  Unfortunately, there also is no way to break a 2 disk array back down into a single disk configuration.  This mean that as far as the NAS was concerned, there was a 250gb drive involved, which was the max size it would let me use.

I ended up having to dump all the files off to a separate storage and reinitialize the array with the single 4TB drive, then re copy and reconfigure everything.  Not a horrible pain but kind of annoying.

Synology NAS

Synology Phase 01 – The Box

Ramen Junkie 2Comments

IMGP2353

 

I finally got something I’ve wanted for a while, a sexy new NAS box.  Unfortunately, I don’t have drives for it yet.  I went ahead and ordered the box because in helps discourage me from accidentally spending the money I’ve been saving if it’s not there to spend.  I plan to stick a pair of 4tb WD Red drives in it with a mirrored RAID.

I’ve wanted a dedicated NAS for a while.  I have a lot of data.  Like, a “fuckton” of it.  I’ve been a digital packrat since I’ve been using computers and it shows.  I’m honestly not sure if 4tb will be enough.  A general run down of some of the data I’m dealing with.

225gb of Family Photos from the past ten+ years.

129gb of music files

22gb of “Personal Files” from writings to web projects to drawings etc

226gb of Program and game install files from Bundles and such

37gb of ISO files, a lot of Linux distros, some backups of my discs

256gb of Videography projects from the past 8 years

A Plus a whole mess of other files from saved image files to ebooks

      to news articles saved to PDF

I also have a massive CD wallet of old files I’d like to start sorting through

I want to rip all of my DVDs to some digital video format for safekeeping

      and private video streaming

Currently most of the data is crammed across a couple of machines, though most of it is in what I generally refer to as the “Family PC” in the basement.  I really want to get the data off of this machine, and off any machine that is actually used by anyone.  One, the machine itself is over 5 years old now.  The drive isn’t but this is starting to reach a point of potential failure.  I went through one drive failure and lost some data, I’m not doing it again.  Two, I want it off a machine people use due to malware issues.  It’s not been a problem recently but on more than one occasion I have found that the kids went and downloaded some hack client for Runescape or Minecraft and gummed up the machine a bit.  It’s never been anything irreversible but the potential is there.

Mostly it’s the data loss potential, which is why I want a RAID.  I have know about and been aware of RAID for a long time now but never really appreciated it until I was replacing drives at work.  Every server we have is at least a Mirrored RAID, the larger ones have a pair of mirrored striped raids with redundant servers.  Redundancy on redundancy, and redundancy is good.  Which is why Next year when my current storage tier runs out I’ll probably also sign up for Office 365, because it will give me a Terabyte of online storage to backup my photos and video files too, the real important and hard to replace stuff.

As for the Synology itself, I looked at quite a few models of NAS before settling for the ds213j.  A friend of mine has a Synology and loves it.  In fact everything I found online suggested the Synology was the best NAS you could get.  I looked into some Buffalo drives and while they were half the price they lacked almost any features besides being a network drive.  The Synology has a lot of nice features including the ability to run web and email servers as well as stream video and music.  If it works, I may dump the Linux box I’ve been using as a web dev server, for power savings.

I also looked into building a FreeNAS box, which would be more powerful and have even more features, except the minimal cost for the FreeNAs box would have been as much as the Synology and the 2x4tb drives, and that was not putting any drives in the FreeNAS.  Honestly the FreeNAS is still really appealing and maybe in a few years I’ll build one, but I really need a solid solution now, I’ve been putting off the NAS for too long.

So after settling on the Synology it was down to which model.  Partially for cost, partially because I don’t feel like I need more, I went with a two bay device.  Honestly after looking at this device construction and design wise there isn’t much reason the larger devices should cost more than like 20-50 dollars more than a 2 bay system but they do, and I don’t care to drop $700 on one of these, I can build a FreeNAS box for that.  So which 2 bay model?  I went with the 213j, the 13 designates this as the 2013 model, the 2 is the 2 bay model.  The 214se version has half the RAM and a smaller processor, I want to be able to stream video off this thing so less power is not better.

Anyway, it may be a month or more before I manage to put some drives in it but you can be sure I’ll come back to discussing this device at that time.