The problem of moving a large file from point A, to point B, is one that occurs regularly with distributed production teams. Most productions resort to sending hard drives or DVDs via courier. I just last week had to send eight packages of DVDs to points around the world. What a pain.

I’ve been looking into adding a “set it and forget it” asset transfer to ShotRunner and it seems like a piece-wise torrent prototocol is the ticket. Still doing research, but looking for a protocol that would keep the chunk size small, that could be throttled back to consume some fraction of the available bandwidth, that is secure, and that would be undetectable by ISPs.

I want to build a facility that lets you say: “I need these plates to be in Zimbabwe by next Tuesday” and the system computes how to make it happen, and does the transfer.

As my buddy Martin Weber points out, torrents are very robust as the files are broken down into manageable chunks, and every chunk has its own checksum to verify whether the transfer was OK. If a chunk gets corrupted during the transfer, the client notices it immediately and the chuck is transferred again. Partial or corrupted chunks are discarded.

The peer-to-peer aspect of a torrent protocol won’t help us much as most assets simply need to arrive in one location. But dailies for feedback and approval, ah ha! That’s another story.

Incidentally, in my tests I’ve found that SFTP is a completely different beast from FTP and, aside from being secure, seems to be far, far more reliable than FTP. My upload speeds are abysmal, 56KB/sec, but I’ve been able to routinely upload 1-2GB files over SFTP.