Software: Recording audio and video in sync
It turns out that writing software to record audio and video in sync is not as easy as you might expect. I had to do this for a recent project, and below are some of the issues I encountered, with solutions. I was working on Mac OS X (Snow Leopard) using openFrameworks. I used the openFrameworks movieGrabberExample as a starting point, along with ofxQTVideoSaver to store the video. There are some useful posts on the forums here.
Well, onto the issues, starting with simplest first:
Don’t assume constant frame duration - It turns out that a lot of the code out there assumes that frames will be captured from the camera at a constant rate. However, many (most?) cameras under default settings will automatically adjust frame rate based on lighting conditions. You can handle this by simply updating a timestamp each time you store a frame, to keep track of the duration of each.
Check for audio buffer overruns - The code I found online used libsndfile to write audio data to disk every time, the audio data callback was called. Writing data to the disk takes forever, and in this case caused the audio buffer to overflow, losing data and throwing the audio and video out of sync. This can be addressed by storing the audio data in a large buffer and periodically writing this buffer out to disc.
Audio and video start at different times - Even if you start recording audio and video at the same point in your code, it is possible that one of these two systems takes a few hundred milliseconds longer than the other to start sending data. In my case, this offset was constant, so I simply measured it and threw away that much data off of the video stream at the beginning, so that the two streams were synced. I’d appreciate hearing about a more elegant solution if someone has one.
Rounding error on frame durations - Quicktime uses an integer value to indicate the length of a frame in a Quicktime movie. I believe the default units for this measurement are 1/600ths of a second. However, ofxQTVideoSaver uses a float value to indicate the fraction of a second. Those float values are rounded to the ints that Quicktime uses, and that rounding error can add up over time, throwing your audio and video out of sync. Strangely, I found this bug because video that I recorded in the morning or afternoon was in sync, but video I recorded at night was out of sync. It turned out that the longer exposure times at night happened to cause a more severe rounding error than those during the day. As a simple fix, I adapted the addFrame function on ofxQTVideoSaver so that it took a Quicktime TimeValue instead of a float, and that fixed the problem. A fix that maintains backwards compatibility would be to have addFrame take a float, convert it to a TimeValue, and measure the error in that conversion, so that it could be added to the next float before the next conversion.
Here is ofxQTVideoSaver.cpp … the only thing that is changed is the addFrame function.