Keeping up with my Photo Library, and Aperture 3 (part 4)

Backup, Redundancy and Disaster Recovery.

Because many of my photos are of my family and kids growing up, they are irreplaceable.  So far I have been lucky that I have not had any catastrophic failures, but, from my work life I know the risk well.  I will never be able to retake the photo of my kid's first step. So...

First, I backup on all active-media.  This means hard drives.  I don't use DVD for backup for two reasons.  The first is that I can't keep up with it.  The second is that they tend to deteriorate.   I know I won't bother to read every DVD periodically to make sure it is still good which defeats the purpose.

I choose a multiple hard drive solution using the reasoning that if I regularly access them I can tell when one starts to go bad or has died and then can replace it.  (I'll talk a little about RAID at the end)

My Storage:

• Laptop drive (250 GB)
• Backup drive (200 GB file sync, 300 GB time machine)
• Library drive (1TB portable USB drive)
• Library Backup drive (1 TB desktop USB drive)
• Offsite Library Backup drive (1 TB desktop USB drive)
• I know I have triple backup this way, but since I have the drives I do this.  You could easily use the Library Backup drive as offsite.

Key Workflow Elements:

• I keep one month's worth of photos on my laptop drive.  After which they are synced to the Library drive & Backup Library drive.

For each import or change:

• Current Month:
• Sync & run TimeMachine to Backup drive after every import and initial sort.
• CF cards are not formatted until this is complete.
• Copies exist on two drives: Laptop drive & Backup drive
• Sync & run TimeMachine to Backup drive after every edit.
• Previous Months
• On any edit or change:
• Sync from Library Drive to Backup Library drive
• Run TimeMachine to Backup drive

Monthly

• Move images from previous month on Laptop drive to Library drive
• Make sure Aperture library is in sync with the new master reference location.
• Sync from Library drive to Backup Library drive
• Run TimeMachine to Backup drive.
• Sync from Library to Offsite drive.

At Various Steps in the Workflow
Out of camera:

• CF cards.

Initial import, sort, and organization:

• CF Cards
• Laptop hard drive

After initial organization and editing completion:

• Laptop hard drive
• USB backup drive (both file sync & time machine)

Upon future edit (current month):

• Laptop hard drive
• USB backup drive (both file sync & time machine)

Upon future edit (previous months)

• Library hard drive
• Backup Library hard drive

Monthly

• Library drive
• Backup Library drive
• Offsite Backup Library drive

Sharing (as I feel like it)

• iPad
• Flickr 

It is worth noting that with this process I could loose up to a month's images in case of a fire or other location disaster.  I'm ok with that level of risk since I tend to post full size of some images to flickr fairly quickly after they are taken so it wouldn't be a total loss.

I use a sync utility (copyTwoFolders) to manage my synchronization.   I use the mode to repeat adds, changes, and deletes from the source to the target.   I use time machine to manage the local disk backup which includes the aperture Library file.  Since I use referenced masters I don't worry too much about it.

About RAID:

I think I would love a RAID solution to automatically create redundancy.   I haven't so far because of three issues:  

The first is that I don't want to spend the money.  I know it isn't that much more, but it just hasn't hit the price-point yet.

The second is that RAID isn't portable.   I want my 1TB portable drive so I can have access to my Library from my laptop anywhere.  I've tried networked over WiFi, but it is too slow and doesn't travel well.   Therefor I'd still need my main portable Library drive and a Library Backup drive.

The third is that RAID can't be geographically separated. 

So when there is a cost effective, portable (USB powered) 1TB RAID 0 mirror hard drive with user swappable drives; I'll get that.  Though I'm not holding my breath.  Its like the one pound, f/2.8, 17-400mm, lens that is less than 6" long, for under $1k;  not likely yet.  If I worked from a desk and wasn't as concerned about portability there are lots of good RAID solutions.

Up next: Editing and Sharing, finally the last part.

Keeping up with my Photo Library, and Aperture 3 (part 3)

Workflow and organization.

I keep all of my photos on disk in a folder structure that starts with the year with sub-folders per month. I use Nikon Transfer 2 to rename and transfer into the month folders sub-folders named by date and sequence number. E.g, 2011>2011-02>2011-02-03_01

Using this format makes it easy to avoid name collisions and is a good basic organization.

Once Nikon Transfer is complete I open Aperture 3 and import the folder to a project. I set Aperture to manage the masters in their current location (reference library). I have faces turned off because I don't use the feature and it takes disk space. I've limited the preview size to within 1200x1200. Again this manages disk space.

Aperture 3 uses generated preview jpgs to preview and to share. I typically share that size anyway. When I upload to flickr i use full size as a third off-site backup.

In aperture, I use a multiple-pass organization technique. I go through each image and mark either -1 to get rid of the image, 3 stars to keep and occasionally a 1 meaning probably delete. I then go back through the 3 stars and mark as 4 stars any I like enough to share and 5 star for the best album quality images. I'll also demote some to two stars or lower for eventual deletes.

I then delete all remaining -1 and 1 stars. I advise being aggressive in your selection. I have kept too many similar or low quality images and need clean them out.

Next for each 4 and 5 star images I locate in finder, open and edit in Capture NX2. I save the NEF then save as jpg the processed version. I then import the new jpg into the aperture project. If you open to Capture NX2 from within Aperture it sends over a TIFF version and you loose the value of raw.

I then run my backup sync and time machine for the local drive.

Now I re-format the memory card.

Monthly I move the working folder to the main library on my USB drive. You can do this via aperture or in finder than re-associate in aperture.

Up next, Backup, Redundancy and Disaster Recovery.

Keeping up with my Photo Library, and Aperture 3 (part 2)

What I want from my tools and my process.

Being an enthusiast I want a robust process with consumer tools that offers full capability and full protection.

My most demanding requirement is portability. I want to work on, view, find, share, and update my photos from anywhere. I don't work at a desk, I work around the house and on travel.

Secondly I want my entire library easily accessible. Not necessarily on the road but definitely around the house. When someone asks if I have images of something or I want to find a specific picture it needs to be relatively quick.

Third, I want full redundancy and disaster protection. These aren't commercially valuable pictures, they are my irreplaceable record of my kids and family.

Fourth, I prefer to edit with Capture NX2 directly on my raw files.

So, I identified the following requirements of my tools and process:

• Must be able to work off of my laptop.
• Organization software must work well with USB storage.
• Organization software must be able to tolerate editing with Capture NX2 on raw files.
• Must be able to sort and view my entire library even if my USB storage is disconnected.
• (or at least don't loose the data.)
• Be able to add more images to my iPad even when the USB drive is not attached.
• Have always two copies of my images on working active media (no DVD).
• Have my entire library in two physically different locations on active media ( in case of flood or fire)
• Be able to quickly organize and sort new images.
• Minimize usage of the local disk.

Next, my organization and workflow.

Keeping up with my Photo Library, and Aperture 3 (part 1)

I have been taking a lot of photos lately, though mostly of family so I haven't posted much here. I have however been fighting a disk space battle on my computer and was in need of a new process and more organization.

Cracks in my current process have been showing up for a while and came to a head about a month ago.

Previously, I would keep as many images on my local drive as I could and offload the rest to a USB drive on the network.  I was using Picasa for organization and Capture NX2 for editing.  This all worked well but last year I found I was only able to keep a few months worth of images on my local drive and access to the older files was painful.  Picasa is not meant for USB or networked drives and wifi networks are not meant for large file organizing.

While trying to figure out the most cost-effective and flexible setup I found lots of good information if you worked on a dedicated computer with dedicated storage within a single product suite (Adobe, say).

What I didn't find was specific info that was above consumer grade but not photo studio level.

So I decided to write a short series of posts on what I want, what I found, and what I do in hopes that others in a similar situation can benefit from my trials or at least decide that this is a crazy way to do things. Hopefully it helps either way.  Here begins the series of posts on the subject.

• My setup (this post)
• What I want out of my process and tools
• Workflow and organization
• Backup, Redundancy and Disaster Recovery
• Editing and Sharing

My Current Setup:

Computer: Macbook Pro 13" w/ 250 GB drive. Portable, powerful, up to 8 hours of battery. Good for home good for travel.

Storage:
Minimum Required

• 1 TB portable USB drive for full image masters library
• 250 GB local drive for Aperture 3 library file and current month working images.
• 1 TB desktop USB drive for image masters library backup.

Nice to Have

• 500 GB portable USB drive with: 200 GB Files partition for syncing local photos and 300 GB TimeMachine partition for local drive backup (excludes synced photos).
• 1 TB desktop USB drive for offsite backup (just because I already have it. Could use the other desktop 1TB drive for offsite).

Sharing: Flickr and iPad.

Software:

• Nikon Transfer2 for transferring images from card to computer.
• Aperture 3 for organization and sometimes editing.
• Capture NX2 for most editing.
• Sync2Folders for backing up images
• TimeMachine for local drive backup.

Useful numbers:

• About 60,000 images   (yes this is too much, I need better editing)
• Nearly 600 GB
• Aperture Library 24.5 GB on local drive. 
•  (Referenced Library with 1200x1200(ish) preview size limit and Faces turned off.)

Please read the following parts of this series where I explain why I'm setup this way.  It is a work in progress but is covering most of the basics.

P.S.  Here is a pic for no related reason.

Conowingo Bald Eagles

Conowingo, originally uploaded by Michael J. Mahoney.

A slightly distant but interesting photo. The eagles get very active when the damn turns on the turbines. When we arrived it was very foggy, no good for photos. I was a bit worried because my intel was saying the turbines were typically being run from 6 to 9am.

Fortunately this time the turbines didn't start until around 8:30am giving plenty of time for the fog to clear.

The pattern seemed to be one eagle would catch a fish, other eagles would try to steal it, 50 photographers with better gear than me documents it.

Fun trip.

Nobody every suspects the butterfly

Nobody every suspects the butterfly, originally uploaded by Michael J. Mahoney.

Went into the butterfly exhibit with the boy. Man it was warm in there. The butterflies liked it.

Conimicut Light

Conimicut Light, originally uploaded by Michael J. Mahoney.

This is the Conimicut Light as seen from my Aunt's deck. Given the conditions of the day, I thought the old sepia look brought out the character a bit more. This was built in 1883 and is still an active navigation aid.

Curious

Curious, originally uploaded by Michael J. Mahoney.

This little guy was kind of interested in what I was doing. I was surprised he hung out as long as he did.

Dragonfly

Dragonfly
Originally uploaded by Michael J. Mahoney

I was out one morning looking for a nice place to take some photos. I was looking for a spot to do some bird photography, but this one didn't pan out as I would have liked. On the upside, there were quite a few dragonflies out and this one was patient.

Dawn View of Jefferson

Dawn View of Jefferson
Originally uploaded by Michael J. Mahoney

Saturday morning I went down to the tidal basin to take pictures during the Cherry Blossom peak. Since I already had some nice cherry blossom landscapes from last year I focused on other types of images this year.

I was able to get there pre-dawn. It was a bit hazy with a few clouds that really lit the sky a nice purple. In about an hour, this bench and everywhere around it will be completely occupied by the crowds. For now, a nice serene, purple, scene.

Lift off

Lift off
Originally uploaded by Michael J. Mahoney

For some reason I like vultures. They play an important role in the cycle of life and the environment and have an interesting clumsiness. Just incase you think I'm odd, look into Tibetan sky burial and remember the vultures in The Jungle Book.

Even still, I was walking back up the path from Great Falls main overlook and he was in the tree over my head and spooked me a bit. I was able to catch him getting ready to take off. (Love the new lens.)

Very blue

Very blue
Originally uploaded by Michael J. Mahoney

This Cassowary, an endangered Australian bird, was particularly interesting. Apparently, this is the only bird with armor (the helmet) which protects his head from rain-forrest brush. Quite an amazing bird (and very thankfully flightless).

Hog heaven

Hog heaven
Originally uploaded by Michael J. Mahoney

Sometimes its about the simple pleasures in life, a little food, a little slop, and a nice red house.

Classic Ford Pickup

Classic Ford Pickup
Originally uploaded by Michael J. Mahoney

On a recent outing to Frying Pan Farm park they had the old work horse red ford pickup out. Nice red on blue.

Passing Farm - Train Ride

Passing Farm - Train Ride
Originally uploaded by Michael J. Mahoney

A little while back we took the boys to Strasburg Rail. They loved it. "Trains!, Trains!, Trains!" was exclaimed while pointing with both fingers. During a little more calm time on the train ride I followed this farm and cloud formation as it lined up for a nice look.

Warm light, cold ice

Warm light, cold ice
Originally uploaded by Michael J. Mahoney

After the first main snow this weekend I was able to get out for a quick walk. By walk, I mean shuffling through knee deep snow. The geese were pretty cold standing on the ice. On the way back to my warm house I took this nice warm light with the large icicle.

Hi Cow

Hi Cow
Originally uploaded by Michael J. Mahoney

We went to Frying Pan Park to see the animals. Brenden really likes the cows. He spent a good amount of time saying "Hi Cow, Hi Cow, Hi Cow, Hi Cow, Hi Cow..." You get the idea. This little guy was the most friendly and interested.

Morning after the blizzard

Morning after the blizzard
Originally uploaded by Michael J. Mahoney

I took this the morning after our "blizzard." Though I'm glad I shovled my driveway once it didn't seem like I had by this time.

Telegrams - Herndon

Telegrams - Herndon
Originally uploaded by Michael J. Mahoney

I stopped by the Herndon Museum yesterday. They have an interesting collection of W&OD railroad and Town of Herndon artifacts. They have a nice collection of photos from as old as the late 1800s.

Strasburg Railroad

Strasburg Railroad
Originally uploaded by Michael J. Mahoney

It has been quite a while since I posted anything. That is what happens with the new born. It isn't that I haven't been taking photos; its is just that most of them are of the kids and the ones that are not sit on my hard drive waiting to be processed.

This one is of the stairs and hitch between two of the dining cards at Strasburg Railroad. We saw Santa there too.

Commute

Commute
Originally uploaded by Michael J. Mahoney

I pulled off the road to take this picture just before sunrise on the way to work. I was able to get to work a bit early and with daylight savings time ending soon I will have to get up an hour earlier to get this shot again.

Agressive

Agressive
Originally uploaded by Michael J. Mahoney

It has been a while since I've been able to post a new picture. New baby and all, time is short. No, this is not Ryen. This is a particularly aggressive goose who wouldn't mind eating out of your hands and getting a finger or two if you aren't careful. Interesting angle using the camera to keep him from getting too friendly.

A rose for Ryen, and mommy

A Rose for Ryen
Originally uploaded by Michael J. Mahoney

One of the flowers that was sent to Mommy and Ryen.

Old Town Alexandria

Old Town Alexandria
Originally uploaded by Michael J. Mahoney

Had lunch and took a quick walk around Old Town Alexandria with a friend this weekend. I liked the Red door and the sailboat on an otherwise largely brick road.

Construction

Construction Equipment
Originally uploaded by Michael J. Mahoney

After the sun sets, the construction equipment gets a bit spooky.

Great Falls at Dusk

Great Falls at Dusk
Originally uploaded by Michael J. Mahoney

We stopped by great falls recently to see if we could catch some nice colors in the setting sun. Unfortunately the best clouds were to the left, but we ended up with something reasonably nice anyway.

Summer Moon

Summer Moon
Originally uploaded by Michael J. Mahoney

While walking around the pond with the family the sun was getting low and really lighting up the moon. I liked the blue sky and clouds with moon shining brightly through it. Nothing like an unusually cool summer evening.

Pig at Colonial Farm

Pig at Colonial Farm
Originally uploaded by Michael J. Mahoney

We went to a picnic at Turkey Run. We took a walk up to the Claude Moore Colonial Farm to show Brenden the animals. He enjoyed seeing turkeys, geese, and the pigs. This is one of the pigs who came over to say hello.

Another look at the zoo

Eyeing the Food
Originally uploaded by Michael J. Mahoney

This one was eye-balling the food on the other side of the gate. His trunk was just about long enough to get at it.

Wild Daisy

Wild Daisy
Originally uploaded by Michael J. Mahoney

I don't do a lot of macro work, but it was this or another shot of the great blue heron. This wild daisy had a beetle visitor on Saturday morning. It is kind of amazing what goes on around us that we are usually unaware of. Now only if the darn ants would be as inconspicuous....

I'm ready for my closeup

I'm ready for my closeup
Originally uploaded by Michael J. Mahoney

This Kori Bustard was more than happy to pose for me on our trip to the National Zoo.

Boat Billed Heron

Boat Billed Heron
Originally uploaded by Michael J. Mahoney

This was the first time I saw this type of heron. Quite unusual compared to the egrets and great blues I'm used to.

Aging

Aging
Originally uploaded by Michael J. Mahoney

I'm not looking forward to the day when my eyesight starts to go. I'm pretty sure it will as both my parents started with great vision and worked their way into a variety of types of glasses. No one and nothing, it seems, is immune to the aging process.

My Neighbor

Great Blue Heron local
Originally uploaded by Michael J. Mahoney

This great blue heron is my neighbor. He's lived here for quite a while. He was out fishing as I took Candy for a walk. He hung around long enough for me to get a nice shot even with the dog sitting next to me. Meanwhile a pair of great blue heron's were flying back and forth and another set of three flew overhead. I never would have figured that this is heron country.

Egret Chicks

Egret Chicks
Originally uploaded by Michael J. Mahoney

More from the Alligator Farm. These two great egret chicks were enjoying the morning light waiting for their breakfast. They were very active and interesting to watch.

Where Babies Come From

Wood Stork Babies
Originally uploaded by Michael J. Mahoney

Apparently these wood storks bring babies to Mommies and Daddies. Well.... Anyway here is a wood stork mommy and her babies. Happy Mother's Day

SR-71 Blackbird

SR-71 Blackbird
Originally uploaded by Michael J. Mahoney

I've always liked this plane. Apparently it can fly so fast that the heat from the friction of the air causes the engines to expand by 12 feet. Really a remarkable plane. We went to the Udvar-Hazy center with my parents, wife, and son. He liked watching the airplanes fly in.

Intensity

Intensity
Originally uploaded by Michael J. Mahoney

This portrait was taken at the St. Augustine Alligator Farm. It is a private zoo that was referred to me by Scott Bourne. I like his blog PhotoFocus. The tip was much appreciated and I came away with a whole lot of keepers. This Snowy Egret was preening himself. You can see some of the down stuck on his bill.

Click to my Flickr stream for more.

Egrets Nesting

Egrets Nesting
Originally uploaded by Michael J. Mahoney

This is one of many shots taken at the St. Agustine Alligator farm. It really is amazing the number of birds and how close they are. Thanks to Scott Bourne for the tip.

Spring Tulips

Spring Tulips
Originally uploaded by Michael J. Mahoney

Happy Easter.

The Old Cherry Tree

The Old Cherry Tree
Originally uploaded by Michael J. Mahoney

Another perspective from my day out at the Cherry Blossom Festival. I liked this old tree. Seems like it has been through a lot.

Cherry Winds

Cherry Winds
Originally uploaded by Michael J. Mahoney

I went to the Tidal Basin in DC Saturday with Sue and Raj. It was a great day beautiful weather with a bit of wind and great cherry blossoms. This is one of the many pictures I'm sorting through. If you haven't been to DC during the peak it is worth getting there early and braving the crowds.

Osprey on the way to the Cherry Blossoms

Osprey with Fish in Tree
Originally uploaded by Michael J. Mahoney

You may have guessed that I like bird photography. Well today I was out on my way to take some photos of the Cherry Blossoms and still managed to get a bird picture. We were walking from the car toward the tidal basin and saw this Osprey fly into the tree right in front of us holding a fish. I didn't have the bird lens to catch the flight but the light was good and he hung around.

Nice colors at sunset

Sunset
Originally uploaded by Michael J. Mahoney

I haven't had a lot of time to take pictures besides the boy. This is a shot of the neighborhood at sunset. I liked the sky's colors and the street lamp in the foreground.

My Neighbor

Red Tail Hawk
Originally uploaded by Michael J. Mahoney

This red tail hawk lives just down the street from me. Some days he's more patient with me. Some not.

Bald Eagle

Female Bald Eagle Fly By
Originally uploaded by Michael J. Mahoney

For quite a while I've wanted to get a good picture of a bald eagle. I finally have a good start. I went down to Mason Neck State Park and saw a couple of eagles flying around. Unfortunately they were a bit too far away for a good picture. I then went up to the GW Parkway to check out the new eagles nest and was surprised by all the activity. The observers think they recently laid eggs. This is the female as she flew right by us maybe 20 feet in the air heading back to her nest.

Mason Neck Refuge

Mason Neck Refuge
Originally uploaded by Michael J. Mahoney

I spent the morning out at Mason Neck Refuge and State park. It is a nice park with quite a bit of wild life. I was able to see quite a few birds even though it is between seasons. This scene was taken overlooking the water this morning.

View of the Potomac River

Patomac River
Originally uploaded by Michael J. Mahoney

This view of the Potomac was taken on a short trip to a local park. It is on the Virginia Birding Trail. I was skeptical at first, but there were lots of wood peckers some blue jays and cardinals. There were a couple of hawks. One came screaming in above me. Unfortunately he was moving too fast to capture. On one trail there was one in a tree I didn't see until he flew off. Anyway, nice view of the river.

Janelia Farm on a Snowy Day

Janelia Farm on a Snowy Day
Originally uploaded by Michael J. Mahoney

I have driven by this place many times and never found the opportunity to take a picture. Today I had to run to the grocery store so I stretched my errands into a small photo trip.

Valentine's Rose

Valentine's Rose
Originally uploaded by Michael J. Mahoney

This was one of the roses I gave my wife for Valentine's day. They lasted quite some time. I helped make sure they outlasted the chocolates I gave her too.

Canadian Geese

Canadian Geese
Originally uploaded by Michael J. Mahoney

I was out walking the dog and we saw our local canadian geese out the pond. The owner of the house to the left shooed them away. I guess he is tired of the evidence of their presence.

So many toys

So many toys
Originally uploaded by Michael J. Mahoney

It was Brenden's 1st Birthday on Thursday. He has so many toys now but loves them all. Even with all those toys books are still his favorite. He chose the pencils and books at his chut-dol. He had a great extended birthday celebration lasting almost two weeks. It's good to be the prince.

Brenden's Orchid

Brenden's Orchid
Originally uploaded by Michael J. Mahoney

This orchid was given to Brenden by his Grandma when he was born. It is blooming again and is now beating daddy's orchid in flowers.