Keeping up with my Photo Library, and Aperture 3 (part 4)

Backup, Redundancy and Disaster Recovery.

Because many of my photos are of my family and kids growing up, they are irreplaceable.  So far I have been lucky that I have not had any catastrophic failures, but, from my work life I know the risk well.  I will never be able to retake the photo of my kid's first step. So...

First, I backup on all active-media.  This means hard drives.  I don't use DVD for backup for two reasons.  The first is that I can't keep up with it.  The second is that they tend to deteriorate.   I know I won't bother to read every DVD periodically to make sure it is still good which defeats the purpose.

I choose a multiple hard drive solution using the reasoning that if I regularly access them I can tell when one starts to go bad or has died and then can replace it.  (I'll talk a little about RAID at the end)

My Storage:

• Laptop drive (250 GB)
• Backup drive (200 GB file sync, 300 GB time machine)
• Library drive (1TB portable USB drive)
• Library Backup drive (1 TB desktop USB drive)
• Offsite Library Backup drive (1 TB desktop USB drive)
• I know I have triple backup this way, but since I have the drives I do this.  You could easily use the Library Backup drive as offsite.

Key Workflow Elements:

• I keep one month's worth of photos on my laptop drive.  After which they are synced to the Library drive & Backup Library drive.

For each import or change:

• Current Month:
• Sync & run TimeMachine to Backup drive after every import and initial sort.
• CF cards are not formatted until this is complete.
• Copies exist on two drives: Laptop drive & Backup drive
• Sync & run TimeMachine to Backup drive after every edit.
• Previous Months
• On any edit or change:
• Sync from Library Drive to Backup Library drive
• Run TimeMachine to Backup drive

Monthly

• Move images from previous month on Laptop drive to Library drive
• Make sure Aperture library is in sync with the new master reference location.
• Sync from Library drive to Backup Library drive
• Run TimeMachine to Backup drive.
• Sync from Library to Offsite drive.

At Various Steps in the Workflow
Out of camera:

• CF cards.

Initial import, sort, and organization:

• CF Cards
• Laptop hard drive

After initial organization and editing completion:

• Laptop hard drive
• USB backup drive (both file sync & time machine)

Upon future edit (current month):

• Laptop hard drive
• USB backup drive (both file sync & time machine)

Upon future edit (previous months)

• Library hard drive
• Backup Library hard drive

Monthly

• Library drive
• Backup Library drive
• Offsite Backup Library drive

Sharing (as I feel like it)

• iPad
• Flickr 

It is worth noting that with this process I could loose up to a month's images in case of a fire or other location disaster.  I'm ok with that level of risk since I tend to post full size of some images to flickr fairly quickly after they are taken so it wouldn't be a total loss.

I use a sync utility (copyTwoFolders) to manage my synchronization.   I use the mode to repeat adds, changes, and deletes from the source to the target.   I use time machine to manage the local disk backup which includes the aperture Library file.  Since I use referenced masters I don't worry too much about it.

About RAID:

I think I would love a RAID solution to automatically create redundancy.   I haven't so far because of three issues:  

The first is that I don't want to spend the money.  I know it isn't that much more, but it just hasn't hit the price-point yet.

The second is that RAID isn't portable.   I want my 1TB portable drive so I can have access to my Library from my laptop anywhere.  I've tried networked over WiFi, but it is too slow and doesn't travel well.   Therefor I'd still need my main portable Library drive and a Library Backup drive.

The third is that RAID can't be geographically separated. 

So when there is a cost effective, portable (USB powered) 1TB RAID 0 mirror hard drive with user swappable drives; I'll get that.  Though I'm not holding my breath.  Its like the one pound, f/2.8, 17-400mm, lens that is less than 6" long, for under $1k;  not likely yet.  If I worked from a desk and wasn't as concerned about portability there are lots of good RAID solutions.

Up next: Editing and Sharing, finally the last part.

Keeping up with my Photo Library, and Aperture 3 (part 3)

Workflow and organization.

I keep all of my photos on disk in a folder structure that starts with the year with sub-folders per month. I use Nikon Transfer 2 to rename and transfer into the month folders sub-folders named by date and sequence number. E.g, 2011>2011-02>2011-02-03_01

Using this format makes it easy to avoid name collisions and is a good basic organization.

Once Nikon Transfer is complete I open Aperture 3 and import the folder to a project. I set Aperture to manage the masters in their current location (reference library). I have faces turned off because I don't use the feature and it takes disk space. I've limited the preview size to within 1200x1200. Again this manages disk space.

Aperture 3 uses generated preview jpgs to preview and to share. I typically share that size anyway. When I upload to flickr i use full size as a third off-site backup.

In aperture, I use a multiple-pass organization technique. I go through each image and mark either -1 to get rid of the image, 3 stars to keep and occasionally a 1 meaning probably delete. I then go back through the 3 stars and mark as 4 stars any I like enough to share and 5 star for the best album quality images. I'll also demote some to two stars or lower for eventual deletes.

I then delete all remaining -1 and 1 stars. I advise being aggressive in your selection. I have kept too many similar or low quality images and need clean them out.

Next for each 4 and 5 star images I locate in finder, open and edit in Capture NX2. I save the NEF then save as jpg the processed version. I then import the new jpg into the aperture project. If you open to Capture NX2 from within Aperture it sends over a TIFF version and you loose the value of raw.

I then run my backup sync and time machine for the local drive.

Now I re-format the memory card.

Monthly I move the working folder to the main library on my USB drive. You can do this via aperture or in finder than re-associate in aperture.

Up next, Backup, Redundancy and Disaster Recovery.

Keeping up with my Photo Library, and Aperture 3 (part 2)

What I want from my tools and my process.

Being an enthusiast I want a robust process with consumer tools that offers full capability and full protection.

My most demanding requirement is portability. I want to work on, view, find, share, and update my photos from anywhere. I don't work at a desk, I work around the house and on travel.

Secondly I want my entire library easily accessible. Not necessarily on the road but definitely around the house. When someone asks if I have images of something or I want to find a specific picture it needs to be relatively quick.

Third, I want full redundancy and disaster protection. These aren't commercially valuable pictures, they are my irreplaceable record of my kids and family.

Fourth, I prefer to edit with Capture NX2 directly on my raw files.

So, I identified the following requirements of my tools and process:

• Must be able to work off of my laptop.
• Organization software must work well with USB storage.
• Organization software must be able to tolerate editing with Capture NX2 on raw files.
• Must be able to sort and view my entire library even if my USB storage is disconnected.
• (or at least don't loose the data.)
• Be able to add more images to my iPad even when the USB drive is not attached.
• Have always two copies of my images on working active media (no DVD).
• Have my entire library in two physically different locations on active media ( in case of flood or fire)
• Be able to quickly organize and sort new images.
• Minimize usage of the local disk.

Next, my organization and workflow.

Keeping up with my Photo Library, and Aperture 3 (part 1)

I have been taking a lot of photos lately, though mostly of family so I haven't posted much here. I have however been fighting a disk space battle on my computer and was in need of a new process and more organization.

Cracks in my current process have been showing up for a while and came to a head about a month ago.

Previously, I would keep as many images on my local drive as I could and offload the rest to a USB drive on the network.  I was using Picasa for organization and Capture NX2 for editing.  This all worked well but last year I found I was only able to keep a few months worth of images on my local drive and access to the older files was painful.  Picasa is not meant for USB or networked drives and wifi networks are not meant for large file organizing.

While trying to figure out the most cost-effective and flexible setup I found lots of good information if you worked on a dedicated computer with dedicated storage within a single product suite (Adobe, say).

What I didn't find was specific info that was above consumer grade but not photo studio level.

So I decided to write a short series of posts on what I want, what I found, and what I do in hopes that others in a similar situation can benefit from my trials or at least decide that this is a crazy way to do things. Hopefully it helps either way.  Here begins the series of posts on the subject.

• My setup (this post)
• What I want out of my process and tools
• Workflow and organization
• Backup, Redundancy and Disaster Recovery
• Editing and Sharing

My Current Setup:

Computer: Macbook Pro 13" w/ 250 GB drive. Portable, powerful, up to 8 hours of battery. Good for home good for travel.

Storage:
Minimum Required

• 1 TB portable USB drive for full image masters library
• 250 GB local drive for Aperture 3 library file and current month working images.
• 1 TB desktop USB drive for image masters library backup.

Nice to Have

• 500 GB portable USB drive with: 200 GB Files partition for syncing local photos and 300 GB TimeMachine partition for local drive backup (excludes synced photos).
• 1 TB desktop USB drive for offsite backup (just because I already have it. Could use the other desktop 1TB drive for offsite).

Sharing: Flickr and iPad.

Software:

• Nikon Transfer2 for transferring images from card to computer.
• Aperture 3 for organization and sometimes editing.
• Capture NX2 for most editing.
• Sync2Folders for backing up images
• TimeMachine for local drive backup.

Useful numbers:

• About 60,000 images   (yes this is too much, I need better editing)
• Nearly 600 GB
• Aperture Library 24.5 GB on local drive. 
•  (Referenced Library with 1200x1200(ish) preview size limit and Faces turned off.)

Please read the following parts of this series where I explain why I'm setup this way.  It is a work in progress but is covering most of the basics.

P.S.  Here is a pic for no related reason.