Organizing

Sorting Out all My Writing

Coding Python isn’t the only project I’ve been working on recently, though it IS the major one.  Another project I’ve been working on, that is at least tangential to “modernizing how I code” is organizing all of my writing.  I write a LOT.  I sometimes list “writing” as a hobby, but I almost never list it as a “Primary Hobby” but it’s arguably the one hobby I have done the longest, even longer than collecting toys, and that I would like to think I do, pretty well.  Ok, no scratch that, I’ve been a “Gamer” since before I could really write.  Actually, it seems like all of my “major hobbies” started when I was like 5-10, so I guess those “formative years” really do matter.  My first programming was on the family’s old Franklin PC with two 5/25 floppy drives, writing BASIC that my dad had taught me.  He had been going to college for Computer Science at the time.

Anyway, writing.

I write, a lot.  I write about all sorts of topics.  Sometimes I write technical write ups, sometimes I write (purposely) shitty Final Fantasy VII Fan Fiction. I write casual blog posts about music, and movies and toys, I write detailed instructions for work or FAQs for Video Games. They aren’t all “winners” but I have gotten a lot of compliments of the years for my writing style and methods.  i also save everything.  I mean, literally EVERYTHING I create.  There are a few things I no longer have and I still think about them sometimes, and wish I had copies.  A few years ago I even started transposing some of my old paper journals and stories into digital text.  

The end result is that I have a lot of files in a lot of formats. Some are text files, some are Word Files, some are exported XML archive files.  A few are PDF based exports as well as some olf “Windows Live Writer” files.

As part of my personal journey to “level up” a bit on my computer skills (which are already pretty great), I have been working on getting more accustom to using Markdown.  Markdown is essentially “Fancy Text Files”. They are plain text files, with special symbols inserted occasionally to make things look prettier in a Markdown reader.  The thing is, this means they are very compact in size and can still be read by even the most basic reader (albeit with the random symbols inserted sometimes).

Most of this effort involves a LOT of copy and pasting.  I’ve converted a bunch of Word Docs I had over to Markdown files. Text docs aren’t generally huge to start with, but the Markdown files mean files that are sometimes 1/4th the file size.  When we are talking hundreds to thousands of files, this is significant savings.  So far, I’ve been skipping reviews if they have embedded images, but I already have those images saved elsewhere, so I may revisit that concept.

This also means finally sorting through some other “to sort” boxes.  For example, for a while, I was posting blog posts with Microsoft’s now discontinued “Windows Live Writer”.  The shitty part is, it used a proprietary format that even Word can’t open.  Fortunately, there is a open source alternative, “Open Live Writer”.  I don’t use it to post, but I can open those old Live Writer Files and convert them to useful Markdown Files.

One fun thing I did was export all of my Reddit Posts, and pull out anything over 500 characters as a “Journal Entry”.

Another source is old WordPress Exports. I have used my newfound l33t Pythonista Skills to build a sweet little script that takes a WordPress XML export, and parses through it for dates, titles, and content. Next, it cleans up the post content a bit (it’s not perfect sadly), and spits it all out to a series of files in the format I want.This script could easily be modified to work with other similar data exports like Reddit)

That code can be found over on Github. It’s probably buggy, but it works for the most part.

Which brings up sorting.  I have posted a few times about digital organization, and I’ve gotten the text down to a science as well.  A folder called “Journal” in my One Drive, which syncs to several PCs and my NAS.  Inside it’s sorted by year, inside each year are files in YYYY.MM.DD – TOPIC.md.  I’ve also incorporated this into my blogging workflow, and so partially written posts in the current year get X_ added to the front, so they all sort to the bottom, but I have an idea of when I had the idea.

This whole new system also allows me an easy way to just Journal occasionally.  One thing I’ve been trying to work on is that “not everything has to be a blog post”.  Sometimes it’s good to just, write, for myself, date it, and spit it out.

It’s healthy to get those thoughts out sometimes. For example, would you like to know how many times I’ve randomly bitched about the show Glee over the past 10-15 years?  Because it’s more than is probably healthy.

Anyway, this project is still a work in progress, but I’ve made a LOT of progress and I’m pretty happy with how it’s been going.

Organizing Digitally – Backups

Backups are the real key and benefit to digital media. It’s also best to have a multi layered plan for back ups. Specifically, i like the 3-2-1 plan that is often pushed. Three copies, Two different storage devices or media, One copy off site. I would like to add that it’s best if at least one of these is automatic. Ideally, you should have some sort of versioning, in case the backup becomes infected or locked by malware, but that tends to be more costly to do, and it’s kind of overkill unless your data is absolutely mission critical.

Old Backup Methods\

I wanted to touch on some old back up methods I’ve used over the year before going on to my current set up. Many of them were alright, but often had pitfalls. Though I am sure my current system has holes as well, it works well enough, for now.

The classic “oldest” would be CD-Rs. And an extension of this with DVD-Rs. I have not really relied on this method for something like 15-20 years but it’s one of the easiest methods of doing back ups of important data. Burn the data to disc, label the disc, store it away, A plus is that if you date the discs, you can also end up with redundant copies easily, in case one disk is damage or fails. Basic DVDs do have a shelf life, though which is a downside. There are long lasting archival discs available, and I’ve actually considered adding these into my Backup workflow again. I have since pulled all of my old archival DVDs and CDs forward to more modern solutions and sorted the data.

For a while I used a large capacity USB drive. It was something like 500GB, which was huge at the time. Eventually, unfortunately, this drive failed. I had also failed to have a second copy of the data, it was just an archive of data, so, while I managed to recover a lot of the data, I lost some family photos files.

For a while I used Amazon Photos as a backup. This worked fairly well, I get unlimited storage with Amazon Prime for photos. There were still several problems with this method. One, it was unlimited for Photos ONLY, which meant videos absolutely had to be sifted out since other files were limited to a measly 10GB. Eventually support for the automatic part that worked with my NAS was ended as well, so that pretty much killed that as a reliable backup. It also was flaky when deleting files. My wife had sorted out out photos and deleted fuzzy or duplicate images, and many returned and were re-synced from the cloud copies. To get around this I had to disable and disconnect the backup, purge out everything int he cloud, then let it re-sync entirely up after she finished sorting files. Not ideal.

Google photos has similar problems, coupled with Google’s new policy removing unlimited photo back ups. There also isn’t an automatic API based sync on my NAS for Google Photos.

One Drive

After shopping around on several different systems compatible with my NAS, I went ahead and chose One Drive. Specifically, Office 365 Family. I’ve considered subscribing since Office 365 was launched and based on quick rough calculations, for the same cost of something like Amazon EC2 or Glacier or Backblaze, or whatever (I can never keep all these names straight honestly), I decided I could get Office 365 Family instead, which effectively gives me 6TB of storage to work with. O365 Fmily gives 6 accounts 1 TB each. I also have 5 members in my family, all of which use Office to varying degrees (currently using an outdated copy I purchased for cheap through a work program). Except none of them are ever going to really use a lot of that 1TB, so I could easily create connections as needed through the NAS to sync backups to each account.

For now, most of the data sits in two accounts, mine, and a new account created solely for core backups. The fun part is, I could even manually push data up to one of the other accounts occasionally (say, yearly), as a slow backup if I wanted. The Core Backup has all of the Family Photos and Videos and a folder of important documents like taxes and bill statements etc pushed to it. My Onedrive has a copy of my personal document archive and personal photos archive.

The nice bonus is I can now access my documents more easily from anywhere. One Drive sharing also allows me to access the backup drive, using my main account. It also all syncs automatically, even though primary access to these files is all done over network shares. Plus it’s more reliable than Amazon Photos.

I also dumped Dropbox in favor of One Drive, with this new storage available. I had mostly been using Dropbox as a sync for “working files” between my laptop and desktop. Now I just use One Drive.

Flash Drives and Regular Drives

But hey, 3 copies right? More can be better though. I bought a couple of large capacity flash drives that I dump all of the family photos and important files to on an annual basis. These go in the fire safe.

Three Copies, Two (Three) different media types, One off Site.

I also have an external USB adapter for SATA Drives, and a pile of drives in storage. For general “data security”, I basically never throw out old hard drives. When I donate an old PC, I will always strip out the drive and dump it into storage somewhere. These are not always large drives, but they do usually still work. Hard Drives also have a longer shelf life than Flash Drives, so occasionally I will make a sort of “Deep Archive” copy of the data to a spare Hard Drive, that gets wrapped in a static bag and stored away.

The entire point of all this, is basically to avoid ever losing data again, like when my old USB drive crashed. It’s not 100% bulletproof, but it’s good enough that little damage could easily be done. If there were some sort of ransomware attack, I always have the drives backups, even if it got synced to One Drive. If the house burns down, there is always the cloud.

Organizing Digitally – Photos

On the surface, it seems a little goofy to separate Photos from Videos, but the reality is, both of these formats really deserve separate handling. For one, when backing up Photos, a lot of solutions will give you unlimited photo storage, but not unlimited video storage. Which makes automatic backup tricky.

Video is also rather massive compared to photos, from a file size perspective. A side effect of this is that having videos mixed with photos can dramatically increase load times when browsing photos later, since it takes more for the file system to chug through a video for thumbnails etc.

I have effectively two sorting systems for photos, depending on what the photos are. Also, unlike the videos, my wife does a lot of the actual sorting of Photos, at least the family photos, since she uses them for scrapbooks and such.

Family Photos

Similar to videos, I sort family photos by year, in one large blob. The difference is, I also sort them into folders by “event”. For example, there might be a folder, “2019.12.25 – Christmas at Home”. The extra details are helpful, because there might also be, “2019.12.23 – Christmas at Josh’s Parents” and “2019.12.26 – Christmas at Tina’s Parents”. (NOTE: I use the actual names of our parents). One minor mistake I made early on when I was doing all the sorting, was labeling them things like “Christmas at My Parents”. I changed all of those to be my parent’s names. I might also use “Christmas at Home with Josh’s Family”.

These folder names allow for easy sorting by date, and it allows a quick, at a glance description of who might be in the photos. I have tried several different Photo Organizing software solutions, and frankly, nothing beats just using the straight file system folders. The nice thing is a lot of software solutions will use the Folders as a way or sorting, so using these folders means the photos can easily and quickly be imported.

Each year also may have some more generic folders. These are catch all folders such as “2018 Cat Photos” or “2012 Kids School Artwork”. There might also be folders like “2014 Misc”, which is where less eventful photos might go. This would be things like, 1-2 lone photos at a local fair event, or single photos of weather or something at the house. They don’t deserve an entire folder, but they are still in the correct year.

Other Photos

I labeled this as “other photos” but it’s mostly just my photos. I take a lot of random photos of my toys and electronics projects, and random crap that is mostly unimportant. I keep these photos separate from the Family Photos, mostly because it’s just clutter my wife doesn’t care about, but because they are just different in their core nature.

These are sorted instead by type. For example, I might have folders for “Toys” then inside, “Transformers”, “Marvel”, “Imports”, “LEGO”, etc. Within those folders, I often will break it down further by lines, or individual figures, since I (used to) take little galleries for use in reviews. Other folders are broken down the same way, photos of projects, photos of electronics, photos of random scenery, sorted down and categorized.

We also use a similar set up for eBay photos, though I don’t really take any of those. I have a shared folder JUST for eBay photos, so my wife and daughters can keep everything sorted and together for the work they do selling on eBay and other online store fronts.

Not Photos

I do something similar with images that aren’t photos as well, though I don’t later back any of these up. Because I am a digital pack rat, I save a ton of random memes and images from the internet. I have a monthly reminder to clean my phone off. These files all dump onto my laptop in a folder named for the year and month, then these images are manually sorted down based on what they are. I’ve honestly gotten better about not just saving piles or random images lately.

Regular Consolidation

Speaking of the Monthly reminders, this system works best if it’s kept up regularly. I have a monthly reminder to offload my phone files, but this only works if I actually DO the sorting, which I make a point of doing.

We have also started regularly dumping everyone’s phones once a month. Though not necessarily removing the files, that’s up to everyone individually, everyone in my house is an adult at this point. But we still consolidate photos as needed for events or activities, since it’s not uncommon for say, my daughter to be taking photos at Christmas, that my wife may want to use in a scrapbook.

Old Photos

I have not gotten as far as I’d like in this project, but I have also started work scanning and archiving older printed photographs. It’s nice to have these digitized since it means they can be archived and backed up and even reproduced for scrapbook albums or whatever. My wife has made scrapbooks for each of our three kids, generally for each year (sometimes two) and so she often uses multiple copies of the same photograph.

Consolidation

The other good part of having everything together is it makes it way easier to keep backed up. I plan to do an entire separate post on the overall backup process, but having things consolidated, makes it way easier to manage and ensure everything is being captured and saved.