Update May 6, 2013: This article will give you a good foundation to start with. However, since the original writing of this post in 2011, I’ve updated my workflow to fit FCPX. Read about it at http://bit.ly/ronsfcpxworkflow.
I currently have a 100% tapeless workflow. Except for a few straggling projects here and there, everything I shoot is on the Canon DSLRs (I have projects shot on the 5D Mark II, the 7D and the T2i). Perhaps the biggest challenge in the tapeless workflow is back-up and archiving. What to do with all that digital media. Here’s how I handle it.
First, let me say that I am by no means an expert at this. Frankly, I’m still learning and who knows, next month I may have a completely different strategy. But for now, this will do. If you have a strategy you’d like to share, please, please do so in the comments.
When Shooting Away from Home
While I’m on a gig shooting a project away from my home office in the evenings (i.e. on a business trip), I’ll dump my cards onto two portable Seagate Go Drives (each about 500 GB). In general, it’s important to have at least two copies of the media in existence at all times. Forever. Then, when I get back home, I implement the regular strategy.
Terms
- Drobo – a RAID-like storage resource that uses proprietary technology.
- Archive Drive – a large capacity hard drive used to archive media and project files. Ideally kept off-site. If not, kept in a fireproof safe onsite.
- Work Drive – the primary hard drive on which I do the main editing of a project.
- Transcoded Files – If you have an older NLE, DSLR video files need to be transcoded (i.e. converted) to a format that your NLE can easily edit. (Note: most newer NLEs have the ability to handle native files, so this may not be as much an issue).
- Offline files – refers to a stage in the editing process where the files are transcoded to a lower quality format (usually ProRes Proxy) for speed and saving drive space.
- Online files – once a project is complete, if the final video is intended for anything on an HDTV or larger, I’ll create an “0nline” edit using a higher quality format like ProRes LT or 422. I simply re-connect all the media in Final Cut Pro to the new, higher quality transcoded files.
- Project & Media Files – this includes the FCP file, music, photos, contracts, etc.
- Self-Contained QuickTime Files – once a project is complete, I’ll save a full resolution QuickTime version of the file. “Self-contained” means that there are no render and media files referenced. Self-contained files take up much more space than media referenced files.
During Project
- Original raw footage to Drobo and also backed up to Archive drive.
- Create a project folder on the Work Drive that contains transcoded files (offline and/or online), media files, DVD files (if applicable), and other project related files. All files kept within their own sub-folders in the main project folder. I create one project folder per project. If a client has multiple projects within a year, I create a master client folder to hold all their individual project folders. The only files not included in the project folders are the original, unaltered raw footage files.
- Copy each project folder to the Archive drive.
After Completion
- Save self-contained QuickTime files to the project folders in their own respective sub-folder. Then update corresponding project folder on the Archive drive.
- Move project folders with the self-contained QuickTime files on Drobo to a folder marked “Archive.” As space is needed, add larger drives to Drobo or copy archived project folders to a second Archive drive then remove from Drobo.
- Keep project folders and their accompanying files on Work Drive for ONE YEAR (or until space is needed, whichever comes first).
- Also save Final Cut Pro project files and non-media project files (e.g. contracts, correspondence, etc.) online via Backblaze or Dropbox.com (Drop Box allows you to sync files on your computer online and with other computers. You get up to 2 GB free. Use this link to sign up and you’ll get an additional 250 mb free.)
As I mentioned before, this is an ever-evolving process. This is what I’m currently trying out based on a lot of research. As technology advances, and internet bandwidth speeds up, I look forward to the day when I can have an online back-up strategy for all of my video media the way my wife does for her photos.
Other Great Archive Strategy Resources
I tried to emulate my strategy as close to mega commercial photog/filmmaker super star Chase Jarvis. Check out his excellent blog post (pictures and video included).
Also check out dpBestflow.org’s post on the topic.
Care to share how you do it?
Thanks Ron. I don’t really have much of one to share YET. I do have a few external hard drives set-up and I use one for archiving BUT that ends up to be the only copy. I am working to save up and purchase some sort of solution like a Drobo.
Couple questions for you or anyone who wants to answer:
1) I was close to considering a Drobo then saw many mixed reviewed of drives suddenly unmounting and becoming unaccessible so I held off. Since my workflow would be very similar to yours Ron, do you have any “warnings” or caveats with the Drobo?
2) I have often wondered if there is a reason people don’t mention using their own website domain for archiving client-specific project files. I know the EULA of many hosting companies prohibit the use of the server space as a “backup solution”. But if it were permissible wouldn’t that work the same as a “Dropbox” type option? I have looked for answers online but can’t find anything specific as to what the downfall would be of doing this. Online disk space is so cheap these days, seems like an affordable way to go.
Thanks as always Ron!
Dan, regarding Drobo I too have heard mixed reviews about. On one photography board someone said that since Drobo has proprietary software, that if you lose two or more drives, you’re SOL. But on that same board, someone else negated that. I’ve invited Drobo to comment.
I once lost a drive, but Drobo did what it was supposed to do. Archived the footage on the 3 good drives so I could replace it. Took about 2 days for that process. My contact at the time suggested I use Western Digital SATA drives (the bad one was a Seagate SATA).
I use the “regular” 4-drive Drobo. The only slightly annoying issue is that when the drives are at rest for a while, it’ll take a few seconds for them to whirl back up when I access it. And sometimes it’ll do this when I’m not accessing it at all, but when I attempt to save a file from my browser (regardless of whether I’ve tried to save it to Drobo). As soon as I go to save, if the Drobo has been “rested”, it’ll start to whirl back up and my “Save as…” screen won’t appear until that’s done.
But, all that said, I’ve had this baby since early 2008 I think. That’s three years and still going strong. I love having the immediate access and the special back up security features built in. I wouldn’t recommend it if it wasn’t worth it. And who knows what the current versions are like. Mine is 3 years old.
As far as saving files to your own domain, if they’re not that big, sure, why not. Many website services have FTP space that you can use like that. It’s all about redundancy.
Hope this helps.
Thanks for that Ron, makes sense. This is a great post I hope others share their workflow. – D
I have a second generation, 4 drive Drobo for personal use at home and at work, we use a 5 drive DroboS.
The DroboS is incredibly quiet and I edit HD footage off of it regularly with no issues. The 4 drive Drobo on the other hand is still a great device, just not as quiet and can slow down the editing process a bit if editing HD.
Also, with a new firmware update currently available for some of the drives already, you can use the new 3 TB drives in your Drobo for added space.
Hope that helps!
Thanks for the comment Michael. My Drobo is indeed a little noisy. Not as bad or loud as the my old G5 Tower. Just a quiet hum that drives my wife batty when she has to work on my computer. 🙂 I’ll look into upgrading soon.
Hey Ron…for the most part a solid plan without going crazy. One thing that may bite you in the butt someday is a file getting corrupted in the process of copying to your external drives. I was at a recent workshop and Ray Roman and Konrad Cytowski mentioned a few incidents of their back up files being corrupted on the archive drives after being copied from the CF cards. They said on all their important and longer files they will do a quick preview of the file on the archive drivce to verify it is not corrupted. It isn’t very common but can happen. That means it’s possible you could then copy that archived file to another archive and have 2 bad copies. Last week I stumbled across some software that verifies the integrity of your file with checksums as it copies it to your backup. Problem is for the life of me I can’t remember the name of the software or where I found it. I googled around trying to uncover something that looked familiar and can’t find it anywhere. If I do stumble on it I will let you know. It may be worth doing a few minutes of research. Your projects are important enough that corrupted data could be costly. It rarely happens but you don’t want it to happen with your data. Just something to think about.
Thanks for the heads up Philip. So are you saying that in order to avoid this corruption, you have to use some kind of special software to copy from the cards to the drives? Please post if you do find the answer.
Yea Ron it is possible but very rare a file can be corrupted in the copy process. The Teracopy app isn’t what I saw the other day and it may be a little better than just copying. I checked the Teracopy app out and it appears it will fix corrupted copies if they are stopped mid-copy but I’m not 100% sure it compares the originals to the copies and verifies the data integrity. If it does then it would be a good solution or if it isn’t available on Mac one that is similar for Mac. Just think..if you are backing up to a redundant RAID or Drobo and in the process the file gets corrupted somehow then both copies on your RAID are also bad. If the transfer could cause a corruption then it makes sense to always make your backup copies directly from you CF cards instead of drive to drive. The odds of that file going corrupt on each transfer from card to drive are pretty slim. Most of the corruptions are just a file header or footer got messed up. There are often utilities to fix that problems as well.
The odds of the problem biting you are pretty small and it’s usually just a single file but it’s there. A little research may be a good idea to see if there is a good solution. Now if my old man memory would pretend it was young again I may remember the app I saw.
Your idea of copying from card to drive for each drive is a good one. I think Philip Bloom mentions a reader he uses that allows him to copy to two different destinations. I’ll need to check into that.
I’ve been using TeraCopy (Windows). It can integrate into contextual menus, and jumps in auto-magically when you copy from one drive to another. The task takes longer, of course, but it may be worth it.
I have no connection to the developer; just thought I’d add my $.02
I can’t wait for the day when third party service providers will offer the ability to manage storage for us. Online transfer will be ideal but until then, why not do a deal where we pay so much a month or year, or per drive, to have an outside storage facility store and maintain our drives. We could send them ours and they’d hook them up every so often to make sure they don’t fail. They could also dump all footage/files onto a ginormous raid system that would serve as our backups. One of the pains I’ve always had in this business when changing offices is trying to figure out where I’ll keep the closet full of hard drives. Also, it sucks when a drive fails and I realize that I didn’t have a backup copy. Backup drives are necessary but that adds twice the headache in the archiving and storage process. I’d gladly outsource that problem!.
Great idea Kris. I’ve thought of the same thing. Calling all entrepreneurs! 🙂
Hello Ron & Dan,
Hello from Spain. Ron’s Data Workflow looks good. Here is how I do it:
1. Raw files go into a RAID 1 Hard Drive.
2. Convert files to Work Hard Drive in RAID 0 for speed. (Here is where all the work is done)
3.Bounce final project into Drobo and a back up copy off site.
My set up is kind of expensive, but I make sure my workflow has full redundancy and is fast.
For more details feel free to contact me! Love to participate more often on these kind of posts.
Thank you, Maceo
Looks like a great plan Maceo. Thanks for sharing. Looking forward to your participation in future posts.
Hmmm. Sounds like you could really benefit timewise from some of our applications. ShotPut Pro http://tinyurl.com/ybtev2o and ProxyMill http://tinyurl.com/2byafxa
ShotPutPro makes sure your copies match the originals (verifies the copies), while doing so as efficiently as possible. So there’s no worries about corrupted files.
ProxyMill transcodes to self-contained files of any resolution you wish–and can do it while ShotPutPro is offloading the cards. This saves a tremendous amount of time.
Sorry to sound like a sales pitch, but thought you should know.
Can’t remember if you software is what I saw recently but it is exactly what I was referring to. The odds of a file getting corrupted on transfer are very small but it only takes one bad file that is critical to ruin your day or your company.
Ron you should take a look at Dan’s software. Will give you a little extra piece of mind on your archives.
Ron,
Great post and nice to see some good conversation on workflow. I figured I’d chime in to clear up a few misperceptions about our BeyondRAID technology, drive failure, and recoverability of data.
First off, ALL RAID technologies are proprietary. You cannot take drives from RAID vendor X and place them in RAID vendor Y. Even if those arrays happen to have the exact same RAID controller, you could still run into issues with different firmware levels, etc.
While BeyondRAID is unique to Drobos, we treat the drives in a Drobo as a “disk pack” that can be read in any like Drobo array or, in some instances, a different Drobo array. Not only is this really nice for offline archiving and upgrading to a bigger Drobo, but it also guarantees portability to another Drobo device should something happen to your Drobo. The entire list of supported disk pack migrations can be found here:
http://bit.ly/diYOd5
Regarding drive failure, this too is not unique to Drobo. For single drive protection mode, Drobo can protect your data from one drive failure regardless of which drive fails. This would be true for any RAID array that is running RAID 1, RAID 5, or similar. Should a second drive fail before the array is able to rebuild, the data would be lost, regardless of the vendor. There are recovery services that can help in these situations and we partner with Drive Savers to help recover data in the rare cases where this happens.
FYI – One thing Drobo does to further help protect data is self-healing. All Drobo arrays offer what we call “virtual hot spare” technology. In the event of a drive failure, Drobo will automatically put the data back into a protected state if it has enough free space on the remaining healthy drives. This greatly reduces the risk of data loss due to additional drive failures since it happens automatically and instantly without manual intervention.
If you are paranoid about your data (aren’t we all?), all Drobo 5 and 8 drive arrays offer the option of checking a box for dual drive protection which allow for up to 2 simultaneous drive failures without data loss. As we move from 2TB to 3TB and larger drives, rebuild times will inevitably grow. Having added protection against the probability of a second drive failure during rebuild is cheap insurance.
At the end of the day, using RAID to protect your data is a positive step. We feel the benefits that are unique to BeyondRAID (auto-configure, self-healing, mix and match different drives, instant and perpetual expansion) deliver the best possible storage experience for all users.
This is especially true for creative pros such as yourselves that would rather be shooting or editing video than trying to become a RAID storage expert.
Best,
Jim Sherhart
Drobo
Thanks for the feedback and inoput Jim. Very useful. I’m going to share your comment link with others I know who’ve asked.