Managing Filmmaking Data Like a Pro | On Set & at Home

As cinematographers and filmmakers we own tens of thousands of dollars of camera equipment, but for most of us, if our house was burning down and we could only save 1 item, I think we all know that we’d save our data and our footage above anything else, as without a doubt, the most valuable thing we own is our work and our files. And this is only made more extreme for ongoing projects that haven’t yet been delivered to the client, and when we’re on set. Our data is our livelihood, and if we lose those harddrives, or we make a mistake on set and format the wrong card the effects on our films, clients, and business would haunt us for years.

As filmmakers, we're constantly generating a ton of data, with the RED I’m generating terabytes of data on every shoot, from raw footage, drone footage, B and C camera files, second shooter footage and images, to sound recordings, scripts, project files, and more. Keeping all this organized and secure on set, and at home is no easy feat, but it's a skill every filmmaker needs to master, as messing this up can be catastrophic for your production and reputation. 

When you're on set, the chaos can be overwhelming, and we often find ourselves up at 11pm after a long day shooting, trying to copy and back up the days files from 4-7 different cameras, such that we can get our gear set up for sunrise the next day and try and squeeze a few hours of sleep in beforehand. We’re often fairly tired, and all we want is to get to bed and it becomes quite tempting to try and rush this process. And this is where mistakes happen, efficient data management is the key to a successful shoot and it’s critical that you don’t rush this process, or mess it up.

Of course on larger productions there will be a Data Wrangler on set, this is someone who is Assigned to oversee data transfer, ensure backups are made, and keep everything organized. But for smaller productions where this isn’t possible that responsibility falls to you.

So Here's how to do it like a pro:

The first and easiest tip is to Label Everything. Label Your memory cards, hard drives, and even your cables should be clearly labelled. This small step can save you from a lot of headaches later on, and it will help your tired mind at the end of a long day keep structure and avoid errors. I always separate the memory cards by not copied, and copied, with clearly distinguished piles such that the memory card goes in on end, and once copied, enters a separate ‘OUT’ pile where it can be formatted and erased.

It's always super helpful to separate the memory cards by camera, and tackle one camera at a time. This helps keep harddrives organised, and easily allows you to check and verify that you’ve copied the files off each camera, as on multi camera shoots, it can quickly become quite difficult to identify which cards on which cameras have been backed up.

The second tip is an obvious one. Backup, Backup, Backup: Always have multiple copies of your data. On location I always have a minimum of 2 separate harddrives with identical copies across both drives, and I always separate my harddrives and put them in different bags or pelican cases while travelling, and where possible keep at least one of them on my person. You never know what’s going to happen, and I have seen so many harddrives fail in the field. It’s a terrifying thought, but if you always have a back up or two and your footage and production will always be safe.

The third suggestion here, is to avoid cheap harddrives, and purchase a few SSD’s. I learnt this the hard way, and while SSD’s are so much more expensive, on location they save so so much time. To give you an idea, on a standard 4 to 5 camera shoot, where we have our underwater camera, topside camera, drone, the photographers underwater camera, and possibly the photographers topside camera, all of which use 1 to 2 SD cards a day, we can have up to 7 to 8 memory cards to copy each night. With productions typically wrapping late, and data wrangling happening well past bed time, it’s critical that these cards copy quickly. Using standard external harddrives, each memory card usually takes between 15 to 30minutes to copy over. With 7 memory cards, each of which requiring at least 2 copies, a primary and a back up, this process of copying and backing up data on these slower external harddrives can take over 5 hours. This is not practical, and it will keep you up at night. I’ve been here and I’ve done this, and take it from, it’s not worth it.

Pick up a pair of SSD’s and this entire process will take around 70% less time, meaning you can actually get your kit ready for the next sunrise much faster, and get to bed on time.

I should mention that there are methods to Offload data Safely. These more professional Use dedicated software for offloading data and they also verify the integrity of your files once copied. This is slightly more advanced, and if there is a data wrangler on set it’s likey that this is how they are transferring files. But to be honest, provided youre making 2 to 3 copies of each card before formatting it, you should be fine.

So now that we’ve got data management on set sorted, how should you manage data at home?

This is something i struggled with for years, and probably like a lot of you I amassed quite the collection of external harddrives. I reached a point that for a number of projects in post production I found myself drawing data from multiple different external harddrives at one time, and I found myself buying adaptors such that I could be pulling data from 4 to 5 external harddrives at once, as my files were spread out across multiple different drives. If you have ever been here you know how much of a mess working like this can be. And consequently, just how easy it is for those external harddrives to fail, as they’re not really built to be used like this.

This is where something like A NAS, or network attached storage device comes in handy. While this can be used in a number of different ways, like accessing files remotely anywhere on earth and synchronising project files up online between multiple different teams, I don’t use any of that, and actually have never had this connected to the internet. For me, this is one massive harddrive, that keeps all my footage and files in one place, and most importantly, ensures the files integrity by running regular checks, and by running a RAID. What this means is that even if 2 of the harddrives inside this system immediately fail, which is very unlikely to happen, all of my footage is completely secure due to the fail tolerance RAID system built into this device.  

Now I’m going to butcher this explanation, but essentially this system is constantly moving and copying all my data across all of these harddrives in parity, such that there is always a back up file on the drive, so that in the event the system integrity checks identify a corrupt file, this file can be immediately rebuilt without me even knowing about it, and as I mentioned in the event that a drive fails, all the data is safely secure elsewhere and can be rebuilt.

It's also a computer with a GPU and CPU, that you can install SSD cache drives in, and that runs off a much faster data connection, ethernet or thunderbolt, providing much much faster editing speeds than standard harddrives.

I told you id butcher that explanation. But I don’t really care about the IT specifics, for me, this is a failsafe storage system, that secures my data, with extremely fast run times, and 108TB of storage, that allows me to store all of my data in one place.

It’s a storage solution built for filmmakers and video editors, and while it hurt buying this, it has been worth every dollar.  

Managing filmmaking data can be a daunting task, but it's one of the most important aspects of our craft. Whether you're on set or working from home, staying organized and following these data management practices will save you time, stress, and potentially even your project. Going cheap on data management, is not worth it.