Image Management Part 3

This is part 3 of a 3 part series on image management for travelling photography hobbyists. The first covered preparing the storage media for travelling and ensuring reliable and fast operation of storage media and the camera. The second focused on culling, editing, and backing up the images while on the road. The third and final post will focus on how to safeguard your images once you get them home.

Getting your images home is not very challenging, but keeping your images safe and accessible while at home can be quite a challenge. It’s so easy to become complacent and assume things will be alright. When you least expect it, and when you need the images the most, they will suddenly no longer be accessible or the files will become corrupted.

Everyone needs to assess their priorities for what’s important in how they store and manage their images at home. My priorities are:

1. Accessibility
2. Storage
3. Redundancy
4. Backup
5. Easy restoration

Many of you may wonder what the difference is between redundancy and backup. They are actually very different, and often, they get confused with each other and that’s where most home storage setups fail when you need them the most.

Let’s start with the easy stuff, accessibility and storage. There are several ways to store your images while keeping them accessible. Everything from large capacity external drives to network attached storage (NAS) and direct attached storage (DAS) devices.

In this post, I’ll share my personal take on home storage. My way is not the only way, there are several routes you can take. I’ll share the pros and cons of the approach I use as well as the other options available. There’s no way to cover it all, so if you don’t like the approaches I’ve laid out, you can do research online where there’s a wealth of information on backup solutions and ideas.

Accessibility

The ideal storage setup for accessibility is an internal drive on the main computer. This is typically where most people start, but storing your images on your main drive can quickly turn into a problem. Internal drives are usually limited in capacity and don’t typically have any redundancy. Internal drives, at least within desktop computers, are usually difficult to access when you are travelling or need to access a particular file remotely.

For good accessibility within your home network, as well as remotely, I highly recommend a NAS setup. There are two brands that I personally prefer, with Synology being the first and QNAP a close second. Each company has their niche and their pros and cons. Synology has an amazing app ecosystem and their operating system called DSM, is very easy to setup and use. I had my first Synology NAS up and running in less than an hour.

QNAP however has some compelling features. They give you the ability to not only use the device as a NAS, but some models also let you use it as a DAS. The benefits of a DAS are speed, both in improved latency and transfer speeds. By connecting the device directly to your computer, you can get transfer speeds much faster than what you’d normally get from a NAS on a 1GB Ethernet connection. Some QNAP NAS devices have Thunderbolt 3 connectivity which is a dream for hardcore photo and video editors. The two downsides to the QNAP are complexity in setup and running, and the prices tend to be higher than Synology.

I’m currently using the Synology DS918+ and love the device. It does everything I need at this time and has been remarkably trouble and maintenance free. It stays on for months at a time without needing any attention. The only times its uptime counter gets reset is when a new version of DSM gets released and the box has to restart. Because I’m more familiar with Synology, I’ll share my experience with their products here, but I encourage you to also watch youtube videos on QNAP devices before making your decision. 

There have been several times where I’ve wanted to access files stored on my NAS while I’ve been away from home. Synology makes this very easy with apps on the device as well as apps available for your mobile phone. Using the DS File app, I can access any file on my NAS while I’m on the road; it’s all very easy to setup and very convenient to use. One thing to keep in mind is that most home internet packages have high download speeds, but limited upload speeds; if you access files a lot remotely, you may want to bump up the upload speed as the files will “upload” from your NAS to the internet and then download to you at the remote location.

With a good NAS, you can access your files from any computer within your network, from any smart TV or smart device within your network, and remotely via a web interface or through dedicated mobile device apps. It all ends up being very convenient and I can’t imagine it will get much better than this in regards to accessibility.

Storage

Now that I have a Fujifilm GFX50R which generates mind boggling high resolution images, I’ve had to really think about storage capacity because the RAW files are large (50MB+) and even the JPEGs files are pretty big too (25MB+). With a good NAS/DAS, you can not only have large capacity, but you can also increase the capacity as you need it. The Synology DS918+ I use has four drive bays. I started by installing two 6TB Western Digital Red drives in the box and that served me well for quite a while. These WD 6TB Red drives are just fantastic, super quiet and reliable.

As my storage needs increased, due to the photos, but also as I became more familiar with the capabilities of the NAS and started to use it for all my storage needs including movies and other large files, I eventually added another two 6TB Western Digital Red drives. Synology makes adding drives very easy with their Synology Hybrid Raid (SHR) setup. You don’t need to know the technical details behind it all, but essentially, their RAID setup lets you use drives of differing sizes and maximizes the available usable capacity across the drives. 

I have the four drives setup in a bit of an odd arrangement, with three drives in the NAS in an SHR2 setup. The fourth drive is purely for internal backup, which I’ll go into a bit later. I ended up using the fourth drive as a backup after learning the hard way the importance of the phrase “RAID is not a backup”. With my setup, the Synology provides 10TB of storage capacity, with live redundancy such that one drive can fail and the other two will chug along fine with my data.

Redundancy

I’ve already touched on this tangentially by introducing RAID above. Redundancy builds on the desire for accessibility; if we have backups of our data, but if it takes days or weeks to rebuild the data, we lose the accessibility. In addition, if the source drive and the backup drives both fail, we will lose all our beautiful images and precious data. All NAS/DAS devices with two or more drive bays allow you to have live redundancy using RAID. There are several good articles online explaining RAID. For most hobby photographers, you don’t need to know too much because the modern devices make it so easy to setup and determine what’s best for you.

In summary, you will likely end up choosing RAID 1 for simplicity, redundancy and performance. I’ve noted this above, but I really want to stress this point again and again: RAID is not a backup. It feels so much like a backup because the device is essentially storing your data in two places, but it is not a backup, and here’s an example of what happened to me which demonstrates the perils of treating a RAID like a backup. After a long trip and lots of image taking on the road, I wanted to save the images on my NAS so that I could edit them from any of my computers, ensure they’re stored on a high capacity medium, and are properly backed up. 

There were hundreds of images to transfer and some video footage as well. To cut down on the transfer time through the network, I thought I’d be clever and use the handy USB 3.0 connection on the NAS itself to copy the files from a fast USB SD Card Reader. The Synology DSM software is very powerful and sometimes isn’t as clear on the implications of the choices given. When copying from a USB source, there is “basic option” which is a required selection, between “multi version”, “mirroring”, or “incremental”. Without thinking about it too much, I selected “mirroring”.

When I clicked OK, what I saw before my eyes put me in a state of disbelief and surreal panic. The Synology DSM software was deleting all of my image folders on the NAS along with all of my travel photos and trying its best to replicate or “Mirror” the folder structure on the SD card I was copying from! I immediately stopped the process, but some damage had already been done. The mirroring of the folder structures had deleted several once-in-a-lifetime trip images.

This is where the phrase, “RAID is not a backup” really hit home, hard. Even though I had the images stored on two disks on the NAS in the RAID, it didn’t matter because both instances were deleted at the same time when the USB import was initiated. That second copy is only for redundancy, it doesn’t help when legitimate instructions are given to delete them.

With hundreds of images now deleted, I had to turn to my backups to restore the images. This is a good lead into the next part of our home storage strategy, backup.

Backup

If you don’t have time or the inclination to read this in its entirety, there are three things I really want you to take away from this section:

1. RAID is not a backup
2. Onsite and offsite backup is required
3. Your restore process is as important as your backup process

With thousands of images gone in a mater of seconds, I turned to my backups to try and restore things back to normalcy. I had been keeping onsite backups via an external USB drive, but the last backup I had made was nearly a month prior to the mess I had created, so that option was off the table. 

I then turned to my cloud backup which is setup with an excellent company called BackBlaze; the service I used is called B2 and it connects directly and seamlessly through the Synology software. Every night at midnight, the Synology NAS uploads any new or changed files to BackBlaze via an automated process.

When I signed up for BackBlaze, I had it setup so that it keeps 30 days of file versions, so for example, if I edited an image five times over the months, it will keep the sidecar files for the last 30 days in case I want to go back and find a version I preferred more. You can determine what works best for you based on the balance between keeping old versions versus the cost of storage you’re willing to accept.

In the restoration process, BackBlaze was amazing. They provided several options including downloading the files, sending them on a USB drive, or sending them on a larger capacity hard disk drive. Since my lost files were just under 256GB, I tried the download method and also paid the US$99 for the USB drive to be sent. 

I let the download run for two days and to my surprise, all the files downloaded and I was able to restore them back onto the NAS. BackBlaze is such an excellent company that when I let them know I no longer needed the USB drive delivered, they promptly accepted the return of the USB drive and refunded the US$99 charge. I’m now a fan forever; at a time when they could have taken advantage of my desperation, they instead chose to be a considerate and kind company.  

Easy Restoration

As mentioned above, the restoration process is very important to getting your data back. The only way to really know if your end-to-end redundancy, backup and restore process works is to test it. You must test your restoration process to be confident you’ll be able to get your data back. To test the end-to-end process, I recommend doing the following:

1. Load the same set of files onto your NAS into four directories, named:
   • 1-day
   • 7-days
   • 30-days
   • 90-days
2. Allow your backup processes to complete, onsite and offsite
3. After one day, delete all four folders
4. Test your restore capability by attempting to:
   • Restore the 1-day folder right away
   • Restore the 7-days folder after a week
   • Restore the 30-days folder after a month
   • Restore the 90-days folder after a three months

Doing the restore test in this way will catch the subtleties such as the impact some settings like versioning have on your restore capabilities. For example, when I accidentally deleted the folders and files with the out-of-control USB import, had I not caught it right away, without versioning turned on in the cloud solution, the NAS/B2 synchronization would have eventually deleted those files from the cloud backup too, leaving only the older offsite backup as a viable solution.

Conclusion

To effectively manage files, an end-to-end approach needs to be taken with some thought put into your priorities. A NAS/DAS solution provides incredible flexibility and freedom to access, backup and restore your files. However, offsite backup is extremely important too and should not be overlooked just because you have an onsite NAS/DAS.

Whenever I mention NAS or DAS solutions to my non-technical friends, I see fear in their eyes of a highly complex device to setup. However, once they’ve invested a few hours to get it up and running, they always come back saying how much they love the device. Their files are now safe and accessible, and I often find they make use of the NAS/DAS beyond just file storage, by tapping into their vast APP ecosystems.

I hope you’ve found this three-part series useful. It took me a number of years to refine my image management routine, so I hope this series saves you some time and that you can spend that time enjoying taking photos instead.