Phil was living the dream, working on a Hollywood feature film. The film's budget was in the $35M range, putting it toward the low end for feature films. This movie in particular would utilize a lot of green screen, and they planned to film entirely digital. The director had (correctly) decided that JPEG-style compression that was common to most tape formats was not acceptable, and that they'd need something that could handle raw, uncompressed high-definition video. They were making a movie, after all, not some podunk town's weather report!

Because of the sheer volume of raw data, they needed a professional-grade Digital Field Recorder (DFR). We're talking eight-disk RAID-5s, video capture cards in the tens-of-thousands-of-dollars range, tons of memory, ruggedized cases, etc. Phil's production company knew this, and since they knew of only one vendor at the time that provided hardware specific to their needs, they began their negotiation. Phil saw a golden opportunity here.

The Golden Opportunity

Having worked on films before, Phil knew many of the constraints they'd be working under. They needed an insurable setup that the completion guarantors would accept, easy set-up/tear-down for constant moving, and quick and simple operaion. Phil was ready to hit it out of the park. He'd done some research on rough pricing and could tell right away that his solution wouldn't be cheap, but it'd certainly be cheaper. It'd be a powerhouse comprising big rack cases, Supermicro motherboards, Adaptec RAID controllers, several SCSI drives, the whole enchilada. All of which would cost about half of a rental of the vendor's equipment. All told, his solution would've cost around $27,000.

He presented his proposal to the producers. "We definitely like the cost," the executive producer told him, "but we just can't risk it. I'm sure you've done your homework, but these guys are professionals. We're at $5,000/hour for set time, and we just can't afford any downtime with your rig."

Phil was told that he'd have to familiarize himself with the vendor's solution, as it'd be arriving on Monday for prep week. Phil was disappointed, but not defeated.

The Professional Solution

The first day of prep week came and went with no DFR. Phil was a bit worried, since he really wanted to learn the set-up and train others on it. And then Tuesday came and went with no DFR. Then Wednesday, then Thursday, and then Friday. Each day, the vendor promised that the equipment was "on it's way."

Finally, on the first week of production, a crate arrived labeled "Digital Field Recorder." It had most of the parts, but the remaining hardware trickled in via smaller shipments for the first half of the week. They couldn't record a single frame until the last component arrived on Wednesday. So far, two full days of set time were lost.

After unboxing everything, it was clear that they'd received mostly consumer-grade hardware. The DFR itself was a Red Hat server that seemed to have little more than Red Hat's bundled software. When booting up, it ran all kinds of unnecessary software like NTP, CUPS, SAMBA, and others that had no use in embedded hardware on a film set. The DFR was managed by a Compaq laptop running Windows XP Home Edition via ethernet, running a Cygwin shell, over an SSH connection to the recorder, then running an X app on the XP side over SSH.

Also troubling was the fact that the hardware would be moved around frequently, which meant it'd have to be shut down and rebooted often. Because of all the unnecessary software running on the Red Hat box, it took a minimum of three minutes from plugging in to being ready to record. It wouldn't take long for that time to add up. The silver lining was that it did have an impressive RAID system — a six-disk setup.

Except those six disks were on RAID-0 — meaning the disks basically worked as a hard drive with the total combined capacity of the disks, with data striped. Ergo, a drive fails, there's zero redundancy, and you're SOL to the tune of $200,000 in data.

The setup recorded scenes as a series of still images named by scenes and takes, with filenames like sc_1_tk_1_tcf_1012003. If for whatever reason a scene was rejected (say, you've recorded scenes 1, 2, and 3, but deleted 2), it would re-use scene 2 for the next recorded scene. When scene 2 was done, it'd pick the next scene, which would probably overwrite the existing scene 3. Phil never had the guts to try it out on his own.

The Compaq laptop served as a single point of failure, and the whole control structure, top-to-bottom was completely myopic. When he raised objections to the producers, even trying to explain it in the simplest possible terms, it was ignored. "This is a professional product," the producer countered, "this is what professionals use."

Going in Production

By the third day of production, everyone was starting to feel the pain of the DFR. And unfortunately for Phil, he was the DFR point man.

"Why the hell does this take you so long," the director shouted at Phil, "we're always waiting on you and your stupid recorder!" There was really no point in trying to explain the unnecessary boot-up process, especially with all the cast and crew standing around, watching him work.

Late that day, one of the single points of failure (the laptop) did exactly what single points of failure tend to do — it failed. Some part of the rickety setup of the XP laptop, the Cygwin shell, the SSH connection, running an X app wasn't working consistently, so the laptop had to be replaced. This led to even more downtime, for which Phil naturally took the blame.

Things just continued to go badly, and Phil's life just became more and more miserable. Fortunately, production only lasted five weeks and, after all was said and done, he was only blamed for a few hundred thousand dollars in budget overruns. But hey, that's how it goes when you're living the dream in show business!

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!