All About Back-ups!

Here's an overview of my process(es) for back-ups.  This isn't an instruction manual, just a walk-through that includes some fairly technical bits that will get you ready to take the plunge.  This guide emphasizes one specific program for back-ups, but the principles described apply to any back-up method if you want to explore other options.  No matter what approach you take, you definitely need more baskets for your eggs, especially when baskets are susceptible to fire, theft, and foxes.

External Hard Drive Back-ups
You need two external hard drives.  Ideally, you need them to be twice the size of your computer's hard drive or more unless you don't work with media (i.e., video or music) very much.  The extra capacity will give you flexibility to grow and to continue using them beyond the life of your current computer.  Additionally, you might use them for other purposes beyond back-ups such as:
  • Archiving files you want to save but don't want to waste space on your computer.
  • Transferring movies or other large files between friends.
  • Backing up a second (e.g., work, friend's) computer.

External hard drives are usually a little more than $100.  As of this writing, a 1TB is about $100 and a 4TB is about $150.  In another six months a 4TB will be $130 and they won't make 1TB drives anymore, so the entire investment will be under $300 in any case.  I currently have a 1TB and a 2TB (the latter bought after I dropped the 1TB while trying to carry it along with a toddler and library books).  They've lasted for years because I only plug them in about once a month.  I planning to replace them with a pair of 4TB drives simply because I collect a lot of digital junk (e.g., complete discographies, CBR comic books, etc.), but that would be excessive for most folks.

There are a lot of ways you can back up things, but each either has a learning curve and/or has flaws.  For example, you can back up everything manually (i.e., drag entire folders over).  That's very easy, but it's the most flawed approach because it's an indiscriminate transfer.  You're going to get everything backed up the first time you do it, but the second time you try, you're going to run into two main problems as well as some minor ones (I break these down more specifically in the Switches section below):
1) You either need to recopy everything or you aren't going to update any files that have be modified.
2) Your back-up will contain files that have since been deleted from the original, and these will only accumulate over time unless you engage in a tedious process of manually weeding.

I use XXCopy for all my back-ups.  It's completely free.  You probably haven't heard of it, but if the name sounds familiar, that's mainly because it's a play on the XCopy command from Windows.  However, the program is completely unrelated except in terms of some of its functions.  Like I said, there's a learning curve, but what I wrote below will get you more than halfway over it.

Basically, it's a command-line program, but I simply write these commands into a series of batch files.  (Batch files are just text files that run commands.  They just have the .bat extension that tells the computer they contain a command rather than text for humans.  If you had a computer in the early '90s, you probably remember the autoexec.bat and maybe even edited it.  It's that simple.)  By saving it as a batch file, I never have to type the command again; I just click the file, and the command is executed.

Each of my batch files deal with just one big folder (and its subfolders) at a time so that I don't have to do the entire hard drive at once.  You don't have to do it that way, of course, but there are several advantages such as letting you skip over a folder while you clean it up before you back it up or not making the hard drive run continuously for several hours if you have a lot to back up and want to break things into more than one session.  Almost all of them look the same other than the folder names, basically falling into two groups that address the above problems:

Group #1: Copy without updating/overwriting anything (i.e., just add new stuff since last time).
Group #2: Copy but update the modified files (i.e., replace old files with the newer version).

Additionally, I have things configured so the program deletes any extra files or subdirectories on the back-up drive that are no longer present on the original.  For example, say you no longer like a particular band and delete their music from your mp3 collection.  Those songs got backed up last time, but now they're gone from the original hard drive.  This ensures that they're scrubbed from the back-up drive too, otherwise these unwanted files would accumulate over the years.  (An alternative scenario might be simply that you moved the original files, so the back-up drive needs them gone from the original location since they'll be copied to the new folder when you run the back-up.)

Most of my folders fall under the Group #1 heading: I just need them backed up, not updated.  They are files that are never modified once they are created or downloaded.  This describes all your movies, photos, mp3s, and any other media that won't change.  (Note: I may edit them, but if so, I'm going to save them under a new name to indicate that it's an edit.)

A typical command might look like:
xxcopy "c:\my movies" "e:\my movies" /BB /S /Z

  • The quotes are because there's a space in the folder's name, and that confuses DOS because it doesn't recognize where the names end and the next item begins.  Quotation marks package it all to the syntax is comprehensible.  If the name didn't have a space, it would be fine to do without quotes, such as: xxcopy c:\mp3-jazz e:\mp3-jazz /BB /S /Z
  • I have the external drive listed as "E:" but that's not necessarily the case.  I have a work-around I won't detail here, but I can still do everything just fine if my external drive happens to identify itself as, say, J: or K: or L: and so on.
  • I gave the destination folder the same name as the source.  You don't have to, but it sure makes life simple.
  • The things at the end of the line are the "switches," which are covered below.

Group #2 includes folders where you are modifying files.  In other words, the file name stays the same, but you're going in and editing or adding to it periodically.  This includes anything that involves writing such as text documents (e.g., short stories, school assignments, journals, address book, etc.) or spreadsheets (e.g., taxes, bills) or any code (e.g., web pages, programming code, etc.).

A typical command might look like:
xxcopy "c:\my documents" "e:\my documents" /D /S /Z

Note the different switches (/BB vs. /D) from one group to the other.  See below for what they do.

These are modifiers (or modulators) of the command.  In a way they're qualifiers or adjectives that specify conditions for what/how to copy and/or delete.  As you can see in the examples above, it may take a string of switches to get XXCopy to do everything you want in one pass.  Fortunately my examples are a shortcut: I've probably already figured out the switches you need.

Here are the switches I use for my back-ups (as in the examples above):
/BB  Backs up brand new files only (does not overwrite existing ones).
/D Copies newer files and brand new files.
/S  Copies directories and subdirectories except empty ones.
/Z  Deletes extra files or subdirectories in destination.

To take it backwards, you can think of it as four scenarios and which switch addresses each to best mirror the contents of your hard drive:
  • The file exists on hard drive, but does not exist on the back-up drive = needs to be copied over = /BB
  • A newer version of the file exists on hard drive, but the old version exists on the back-up drive = needs to be overwritten to new version = /D
  • The file has been deleted from hard drive, but still exists on the back-up drive = needs to be deleted from back-up drive = /Z
  • The file exists on the hard drive, but already exists on the back-up drive = no action needed, but do not overwrite = both /BB and /D include this by default without prompting to ask about wasteful overwrites.

Additionally, we are dealing with not just a folder full of files, but with sub-folders and the respective files contained in each, so the /S switch makes certain they are included in all the above conditions as well.

Note that there are many, many more switches than just these.  I found that the above suit my needs perfectly, but you may be more particular, depending on your requirements for a given set of files.

But why two hard drives?
The hard drives are kept in two places: on-site and off-site.  Neither are ever connected to the computer until they're needed to run a back up or to restore from.

On-site = It's right there in your home, just not connected to your computer.  Should your hard drive fail (and it will, eventually), you can plug it in and restore your computer right away.

Off-site = It's away from the home.  Because maybe it isn't your computer that fails.  It could be the house burned down or was robbed.  This hard drive was somewhere else, so your family photos, dissertation, and your band's demo are safe.

For me, "off-site" means one drive stays in a bank safety deposit box.  I don't have to worry about someone looking through it, it getting damaged, etc.  I rotate these out every couple months, otherwise the off-site back-up would increasingly grow "stale" with old data.  At some point a back-upis too old to be of much use because your spreadsheets are missing the last two months, you don't have the things you downloaded for the latest project, etc.

Frozen Back-ups
In addition to the external hard drives, I also burn DVD back-ups of a few things.  Mainly I focus on just My Docs and web pages (i.e., things I created) and my pictures and movies (i.e., things that cannot be replaced).

DVDs may sound archaic and overly redundant (read: paranoid) in an era of flash drives and clouds, but they address two problems to which hard drive back-ups are susceptible.

  • Error propagation - The problem with updating files on your back-up (as opposed to saving all versions of each) is that you can potentially back up corrupt files without knowing it.  And there are a lot of ways you can end up with corrupt files: a physical error on your hard drive, a virus/malware that rewrites files (e.g., inserts spam links into html code), etc.  However, a DVD is written once, then kept away from the computer, so it's safe from read/write errors as well as anything running in the background that might modify it.  Problem solved.
  • Unnoticed deletions - What if you accidentally deleted a file you created?  If you notice it in time, then you can look it up on the back-up.  But what if you run the back-up before you notice the deletion?  Now it's gone from the back-up too.  That's a problem.  But DVDs are forever (well, for our purposes).  Problem solved.

Sure, a DVD can get scratched, but if you keep them safely out of reach, they're likely to remain untouched for decades, and these are your Plan D in any case, the two external hard drives serving as secondary and tertiary sources while you use your hard drive daily.

My schedule
When's the best time to do things?  That depends on how much you hard drive changes.  For example, if you take loads of pictures daily, maybe you need to do it weekly?  I don't know.  It's a cost/benefits analysis you'll have to make, but I think monthly is a reasonable schedule for me.  If you do it more often, you probably haven't accumulated much more work than that, but if you do it less, then you might have lost quite a bit.  Immediately after the birth of my kids I backed things up more than once a week simply because I didn't want to lose those images/video, but after those were safe, I went back to a more routine schedule.

I also clean up hard drive right before each back-up (i.e., put the temp files where they belong, get the files properly named, etc.).  I always do this on the 1st of the month, so it's easy to remember, and it forces me to file away things that I needed to finish up with anyway.  This all coincides with other related activities like cleaning off the phones and cameras.  All the pictures and movies move off other devices, then they get properly renamed, all before I back up the hard drive.

Other events include:
  • Every other month: Rotate out the on-site vs. off-site hard drives
  • Annually: Burn a dvd of My Docs and web pages
  • Every 700MB of pics: Burn dvds of pics.  When I accumulate a DVD's worth of pictures, that's when I burn a new DVD.  There's no fixed time.  I just make a note where I left off, and about once a year or so I have to burn it, so I check just about annually, though that period will depend on your use.

First step: Get organized
It's going to be a messy back-up if you don't have the contents of your hard drive organized.  You have to do what works for you, but some things I do that help me include:
  • Keep things in or near the root.  In other works, have a real "My Documents" folder.  Don't use the Windows defaults like \Users\Alex\blah\blah\blah\.  The same applies to pictures and other media.  Sure, you can back them up, but it's more complicated to track something three or four levels down, whereas I can get a good overview of all my belongings just looking at the folder listing under C:\.
  • Create "temp" folders.  Maybe you just need one, but I have three.  This is where I put new downloads or unzip files.  I also have a temp folder specifically for incoming pictures and videos from the phones and cameras so I can delete the unwanted/bad shots and rename everything before I move it where it belongs.  These folders never get backed up, so the don't catch anything that I may not save or have in the right place.
  • Make a "Stuff" folder.  This is a good catch-all for the misc. things you save.  I've been a digital pack-rat since long before Pinterest came online.  For example, "stuff" has the following sub-folders:
  • art - Collection of classical and modern art.
  • books - An ebook archive.  (I like to have searchable versions of things I've read.)
  • craft-projects
  • fonts
  • halloween - Photo references (screen caps, etc.) for costumes and decorations.
  • house - Ideas for projects plus a lot of wiring diagrams.
  • manuals - PDF files of manuals for tools, appliances, electronics, and so on.  (Easier to find and search than paper!)
  • media - Stuff from movies.
  • posters - A collection of movie posters.
  • software - Installation files.  I have sub-folders for this too: drivers, games, graphics, music, and web, just to keep it from turning into a massive list of .exe files.
  • Note: I also have a "guitar" folder, but that had so many subs that I pulled it into the root.
  • Create archive folders.  When a particular chapter of your life is complete, put the associated files into a folder that can be skipped when you do your back-ups.  For example, course-work/assignments that will never be revisited, your term papers and thesis, old tax records (you can pull them out if you're ever audited, but you won't need to add to them), etc.  Similarly, I keep my photos grouped by year.  When I run a back-up, it skips over everything but the folder for the current calendar year.  At the end of the year, I edit the batch file to read "2015" instead of "2014," and so on.

Postscript: What NOT to back up
I've already mentioned by example many things you should back up.  A good heading might be "things that cannot be replaced," but it might also include "things that could be replaced that would just be a big headache to download again."  For me, all of the above falls into three categories:
1. Personal - Files pertaining to finances, photos, scans, home movies, etc.
2. Creations  - Any art (written or visual) you've made.
3. Mass-media - Music, movies, and programs you've downloaded and want to save.

Pretty much everything that isn't something you made or is meaningful to you = Don't bother backing it up.  Specifically:
  • Don't back up program files, just the installation file (See my "Stuff" folder above). 
  • Don't back up operating systems.  If your hard drive ever fails, you'll have to reinstall, so just go with the installation disc(s).
  • Don't back up things you consume.  If you download tv shows and watch them next week, you don't need to waste the space they'll take up.  They'll be gone from the hard drive days later, but they'll stay on the back-up drive for a month, long after you had any use for them.  The same applies to podcasts and other media.
  • Don't back up temp folders.  Like I said above, I put a lot of things in temp folders: recent downloads, files I need to rename, files I need to convert, etc.  But then I process them and put them where they belong.  It doesn't make sense to back up things you 1) may not keep or 2) aren't in their final form yet.  I usually go ahead and process everything in my pre-back-up "clean-up" so everything new gets caught.

Similarly, as mentioned above, you can skip over folders that are rarely (if ever) updated such as the "archived" folders or folders associated with last calendar year and earlier.

The Beginning
This is just an overview.  I'm emphasizing principles, not really the nuts-and-bolts of what this will look like on your computer.  However, most of it is easy to apply, and I even have my collection of batch files that are easy enough to adapt to your use.  So here's your "to do" list:
  • Clean up your hard drive.
  • Get a couple external hard drives.
  • Do the first back-up manually.
  • When you feel like you have everything backed up, that's the template for all future back-ups, so you (or I) can create the batch files required.
  • Run the back-ups every month thereafter.
  • Sleep easier.

Copyright 2015 Ale[x]plorer.

Back to the index