This is part 3 of a 3 part series on image management for travelling photography hobbyists. The first covered preparing the storage media for travelling and ensuring reliable and fast operation of storage media and the camera. The second focused on culling, editing, and backing up the images while on the road. The third and final post will focus on how to safeguard your images once you get them home.
Getting your images home is not very challenging, but keeping your images safe and accessible while at home can be quite a challenge. It’s so easy to become complacent and assume things will be alright. When you least expect it, and when you need the images the most, they will suddenly no longer be accessible or the files will become corrupted.
Everyone needs to assess their priorities for what’s important in how they store and manage their images at home. My priorities are:
- Easy restoration
Many of you may wonder what the difference is between redundancy and backup. They are actually very different, and often, they get confused with each other and that’s where most home storage setups fail when you need them the most.
Let’s start with the easy stuff, accessibility and storage. There are several ways to store your images while keeping them accessible. Everything from large capacity external drives to network attached storage (NAS) and direct attached storage (DAS) devices.
In this post, I’ll share my personal take on home storage. My way is not the only way, there are several routes you can take. I’ll share the pros and cons of the approach I use as well as the other options available. There’s no way to cover it all, so if you don’t like the approaches I’ve laid out, you can do research online where there’s a wealth of information on backup solutions and ideas.
The ideal storage setup for accessibility is an internal drive on the main computer. This is typically where most people start, but storing your images on your main drive can quickly turn into a problem. Internal drives are usually limited in capacity and don’t typically have any redundancy. Internal drives, at least within desktop computers, are usually difficult to access when you are travelling or need to access a particular file remotely.
For good accessibility within your home network, as well as remotely, I highly recommend a NAS setup. There are two brands that I personally prefer, with Synology being the first and QNAP a close second. Each company has their niche and their pros and cons. Synology has an amazing app ecosystem and their operating system called DSM, is very easy to setup and use. I had my first Synology NAS up and running in less than an hour.
QNAP however has some compelling features. They give you the ability to not only use the device as a NAS, but some models also let you use it as a DAS. The benefits of a DAS are speed, both in improved latency and transfer speeds. By connecting the device directly to your computer, you can get transfer speeds much faster than what you’d normally get from a NAS on a 1GB Ethernet connection. Some QNAP NAS devices have Thunderbolt 3 connectivity which is a dream for hardcore photo and video editors. The two downsides to the QNAP are complexity in setup and running, and the prices tend to be higher than Synology.
I’m currently using the Synology DS918+ and love the device. It does everything I need at this time and has been remarkably trouble and maintenance free. It stays on for months at a time without needing any attention. The only times its uptime counter gets reset is when a new version of DSM gets released and the box has to restart. Because I’m more familiar with Synology, I’ll share my experience with their products here, but I encourage you to also watch youtube videos on QNAP devices before making your decision.
There have been several times where I’ve wanted to access files stored on my NAS while I’ve been away from home. Synology makes this very easy with apps on the device as well as apps available for your mobile phone. Using the DS File app, I can access any file on my NAS while I’m on the road; it’s all very easy to setup and very convenient to use. One thing to keep in mind is that most home internet packages have high download speeds, but limited upload speeds; if you access files a lot remotely, you may want to bump up the upload speed as the files will “upload” from your NAS to the internet and then download to you at the remote location.
With a good NAS, you can access your files from any computer within your network, from any smart TV or smart device within your network, and remotely via a web interface or through dedicated mobile device apps. It all ends up being very convenient and I can’t imagine it will get much better than this in regards to accessibility.
Now that I have a Fujifilm GFX50R which generates mind boggling high resolution images, I’ve had to really think about storage capacity because the RAW files are large (50MB+) and even the JPEGs files are pretty big too (25MB+). With a good NAS/DAS, you can not only have large capacity, but you can also increase the capacity as you need it. The Synology DS918+ I use has four drive bays. I started by installing two 6TB Western Digital Red drives in the box and that served me well for quite a while. These WD 6TB Red drives are just fantastic, super quiet and reliable.
As my storage needs increased, due to the photos, but also as I became more familiar with the capabilities of the NAS and started to use it for all my storage needs including movies and other large files, I eventually added another two 6TB Western Digital Red drives. Synology makes adding drives very easy with their Synology Hybrid Raid (SHR) setup. You don’t need to know the technical details behind it all, but essentially, their RAID setup lets you use drives of differing sizes and maximizes the available usable capacity across the drives.
I have the four drives setup in a bit of an odd arrangement, with three drives in the NAS in an SHR2 setup. The fourth drive is purely for internal backup, which I’ll go into a bit later. I ended up using the fourth drive as a backup after learning the hard way the importance of the phrase “RAID is not a backup”. With my setup, the Synology provides 10TB of storage capacity, with live redundancy such that one drive can fail and the other two will chug along fine with my data.
I’ve already touched on this tangentially by introducing RAID above. Redundancy builds on the desire for accessibility; if we have backups of our data, but if it takes days or weeks to rebuild the data, we lose the accessibility. In addition, if the source drive and the backup drives both fail, we will lose all our beautiful images and precious data. All NAS/DAS devices with two or more drive bays allow you to have live redundancy using RAID. There are several good articles online explaining RAID. For most hobby photographers, you don’t need to know too much because the modern devices make it so easy to setup and determine what’s best for you.
In summary, you will likely end up choosing RAID 1 for simplicity, redundancy and performance. I’ve noted this above, but I really want to stress this point again and again: RAID is not a backup. It feels so much like a backup because the device is essentially storing your data in two places, but it is not a backup, and here’s an example of what happened to me which demonstrates the perils of treating a RAID like a backup. After a long trip and lots of image taking on the road, I wanted to save the images on my NAS so that I could edit them from any of my computers, ensure they’re stored on a high capacity medium, and are properly backed up.
There were hundreds of images to transfer and some video footage as well. To cut down on the transfer time through the network, I thought I’d be clever and use the handy USB 3.0 connection on the NAS itself to copy the files from a fast USB SD Card Reader. The Synology DSM software is very powerful and sometimes isn’t as clear on the implications of the choices given. When copying from a USB source, there is “basic option” which is a required selection, between “multi version”, “mirroring”, or “incremental”. Without thinking about it too much, I selected “mirroring”.
When I clicked OK, what I saw before my eyes put me in a state of disbelief and surreal panic. The Synology DSM software was deleting all of my image folders on the NAS along with all of my travel photos and trying its best to replicate or “Mirror” the folder structure on the SD card I was copying from! I immediately stopped the process, but some damage had already been done. The mirroring of the folder structures had deleted several once-in-a-lifetime trip images.
This is where the phrase, “RAID is not a backup” really hit home, hard. Even though I had the images stored on two disks on the NAS in the RAID, it didn’t matter because both instances were deleted at the same time when the USB import was initiated. That second copy is only for redundancy, it doesn’t help when legitimate instructions are given to delete them.
With hundreds of images now deleted, I had to turn to my backups to restore the images. This is a good lead into the next part of our home storage strategy, backup.
If you don’t have time or the inclination to read this in its entirety, there are three things I really want you to take away from this section:
- RAID is not a backup
- Onsite and offsite backup is required
- Your restore process is as important as your backup process
With thousands of images gone in a mater of seconds, I turned to my backups to try and restore things back to normalcy. I had been keeping onsite backups via an external USB drive, but the last backup I had made was nearly a month prior to the mess I had created, so that option was off the table.
I then turned to my cloud backup which is setup with an excellent company called BackBlaze; the service I used is called B2 and it connects directly and seamlessly through the Synology software. Every night at midnight, the Synology NAS uploads any new or changed files to BackBlaze via an automated process.
When I signed up for BackBlaze, I had it setup so that it keeps 30 days of file versions, so for example, if I edited an image five times over the months, it will keep the sidecar files for the last 30 days in case I want to go back and find a version I preferred more. You can determine what works best for you based on the balance between keeping old versions versus the cost of storage you’re willing to accept.
In the restoration process, BackBlaze was amazing. They provided several options including downloading the files, sending them on a USB drive, or sending them on a larger capacity hard disk drive. Since my lost files were just under 256GB, I tried the download method and also paid the US$99 for the USB drive to be sent.
I let the download run for two days and to my surprise, all the files downloaded and I was able to restore them back onto the NAS. BackBlaze is such an excellent company that when I let them know I no longer needed the USB drive delivered, they promptly accepted the return of the USB drive and refunded the US$99 charge. I’m now a fan forever; at a time when they could have taken advantage of my desperation, they instead chose to be a considerate and kind company.
As mentioned above, the restoration process is very important to getting your data back. The only way to really know if your end-to-end redundancy, backup and restore process works is to test it. You must test your restoration process to be confident you’ll be able to get your data back. To test the end-to-end process, I recommend doing the following:
- Load the same set of files onto your NAS into four directories, named:
- Allow your backup processes to complete, onsite and offsite
- After one day, delete all four folders
- Test your restore capability by attempting to:
- Restore the 1-day folder right away
- Restore the 7-days folder after a week
- Restore the 30-days folder after a month
- Restore the 90-days folder after a three months
Doing the restore test in this way will catch the subtleties such as the impact some settings like versioning have on your restore capabilities. For example, when I accidentally deleted the folders and files with the out-of-control USB import, had I not caught it right away, without versioning turned on in the cloud solution, the NAS/B2 synchronization would have eventually deleted those files from the cloud backup too, leaving only the older offsite backup as a viable solution.
To effectively manage files, an end-to-end approach needs to be taken with some thought put into your priorities. A NAS/DAS solution provides incredible flexibility and freedom to access, backup and restore your files. However, offsite backup is extremely important too and should not be overlooked just because you have an onsite NAS/DAS.
Whenever I mention NAS or DAS solutions to my non-technical friends, I see fear in their eyes of a highly complex device to setup. However, once they’ve invested a few hours to get it up and running, they always come back saying how much they love the device. Their files are now safe and accessible, and I often find they make use of the NAS/DAS beyond just file storage, by tapping into their vast APP ecosystems.
I hope you’ve found this three-part series useful. It took me a number of years to refine my image management routine, so I hope this series saves you some time and that you can spend that time enjoying taking photos instead.
If you’ve found the above information useful, and are thinking of purchasing any of the gear mentioned, please consider using the links provided. As an Amazon Associate I earn from qualifying purchases.
4 thoughts on “Image Management Part 3”
Would you mind sharing what the folder structure on your NAS is like? I’m starting basic with a single SSD backing up to Backblaze (not B2) with all JPG and RAW in a simple Year > Month folder structure with a separate top level directory storing edits and albums. Great site!
Hi Patrick, because I use Capture One, my directory structure is pretty simple. My Synology NAS has a folder for Images > Capture One > Session Folder – Month-Year. For example, I went to Koh Samui in 02/2020 before the pandemic hit, so that session would be placed in Images > Capture One > Koh Samui – 02-2020. I then backup everything in the Capture One folder to Backblaze.
Within the Session Folder, I keep only the RAW files in the Capture folder. I also make sure that I cull images in the Capture folder since there’s no point backing up blurry or badly composed images. I also ensure that the Trash folder is emptied before moving it to the NAS. Finally, I always ensure that the Selects are exported to 90% JPEG and left in the Output folder.
If you’re a keen indexer, you’ll have noticed that this structure, while simple, doesn’t make it easy to find things. This is where I leverage modern technology like Apple Photos or Google Photos where AI technology makes it easy to find things. I take the JPEG images from the Output folder and import them into Apple Photos. This does two things, it lets me leverage the power of Apple AI to find images (you can search for dog and it will use pattern recognition to find all images of dogs for example), and provides for a central repository of all my finished images.
My workflow is not perfect and requires some steps and discipline, but it has worked for me as my session count has grown to over 200. I can easily find images from previous trips by searching in Apple Photos, getting the filename and then searching for that filename on the NAS, if I want to re-edit that RAW file as an example. If I were more disciplined, I would tag images, but I find Apple’s AI sufficient to find images when I need them.
Hope the above helps. Let me know if you need more info.
Concur completely with your Synology recommendation. I adopted a NAS solution after nearly losing all my images in the post-Katrina floodwaters. Long story, but I managed to retain my images, but knew I needed a better solution.
So, about 11 years ago I bought an Infrant ReadyNAS NV+. It was a bit fiddly, but with the help of a very proactive staff and the support forum I got it where I needed it. Unfortunately about this time, Netgear bought Infrant, and the rot set in. It wasn’t at first, but a gradual reduction in the support. My ReadyNAS had a semi-custom internal power supply that had the infamous bad Chinese capacitors. In their defense, Infrant nee NetGear replaced it twice, but the third time was no charm. And due to the semi-custom design, finding replacements was something of a challenge. I managed to find a lightly used one amazingly from an individual about 5 miles from me, again I dodged the bullet on losing my files.
One the ongoing issues I was having with the ReadyNAS was it would randomly hang needing a cold reboot. By this point the Infrant forums, that had been so helpful in the past, had been replaced by a new significantly less helpful forum system and the old info was no longer available.
So, I bit the bullet and bought a DS918+. I installed the 1gig drives from the ReadyNAS, and in the set up process the firmware informed me that one of the drives had a number of bad sectors. The firmware could block those off, but in hindsight I realized this is what was hanging the ReadyNAS, but there were no tools for managing the problem.
Setting up the Synology was simple, but there is a different philosophy regarding volumes and such from the ReadyNAS and took a bit of thought to configure in a way that worked with my previous setups. After doing that, I started exploring all the apps for the Synology and was simply amazed. The daily backup app is seamless and extremely stable, the Cloud sync is awesome as I automatically back up my photos to my Amazon Prime account real time. Simply awesome. I also use Resilio sync to automatically backup all the computers in the house, real time. simply amazing.
And the final plum was setting up Emby as a media server to distribute media files on the NAS all around the house.
So two enthusiastic thumbs up for the Synology NAS.
Hi Christopher, thanks for writing in and sharing your positive experience with the Synology DS918 NAS. It’s always good to hear from a readers’ of what has worked for them and what hasn’t. With SSD NAS drives starting to come down in price, I’m looking forward to going that route in the next year or so. I’m just trying to decide whether to stick the DS918 or go for the NAS boxes designed specifically for the SSD drives. I’m actually quite happy with the current setup which saturates my 1gbps Ethernet connection. Storing images on the NAS is totally doable with the cache setup on the NAS.