Fixing the video/audio de-sync problem when re-encoding a .MKV/.MP4/.RM/.RMVB/.WMV file

Summary:
With AVISynth installed, create a .avs file, and in the first line enter:
DirectShowSource("My Documents\Video1.mkv", fps=23.976, convertfps=true)
And feed that .avs file into the encoder

If I wrote a guidebook, the chapters would probably be in the reversed order; meaning you'd have to read from the back.

Yes, again, this guide should've been a follow-up to another guide, maybe AVISynth usage or 3GP Converter customization guide. Because these are the only situations where this trick can be applied, and to apply this trick skillfully you'll need knowledge on the above 2, too. (add maybe AVISynthesizer)

Ok, so basically, you downloaded an episode of a recent Anime, used an all-in-one encoder to encode it into AVI or MP4 for your MP4(MP5) player or PSP or DivX player or watever. Then when you played-back the video, the sound is either too fast or too slow after some time. You play it with your computer. Same problem. You check the original file, no problem. You get frustrated and blame the encoder. You ask for help in the forums but get no reply.

Actually, it isn't the audio that went out of sync. It's the video that was either too fast or too slow at certain points. (strictly speaking, de-syncing means the audio and video are not together, so it's both that's de-synced to reach other) The cause of this, we blame variable frame rate (VFR) encoding.

Basically, a hybrid (frame rate) video is one that has different frame rates at different parts of the video. Usually it doesn't occur in movies or broadcast signals, but it happens sometimes in NTSC shows. To be precise, the most affected genre is animation, that makes use of NTSC film (23.976) most of the time and NTSC video (29.97) in higher action scenes e.g. opening. PAL videos aren't affected by this because there's only one legitimate official frame rate of 25fps.

When you simply dump the file into an encoder, what happens is that the encoder gets the frame rate of the source video at the start, and assigns that frame rate to the encoded video, fixed. Why so? Because you need additional input if you want to encode a VFR video, mainly which parts of the clip are what frame rate. It won't get the variable frame rate right even if the encoder reads the file throught DirectShow (in fact, most "cheapo" encoders read the file through DirectShow, and sometimes with the help of AVISynth), although it will play back fine using a player (as in the original video, not the encoded one).

Also, if you're encoding into an AVI file, then there's no way to get VFR (except using the 119.88fps hack, which you wouldn't want to use). The common container file formats that support VFR are Mastroka (.MKV), .MP4, RealMedia (.RM/.RMVB), and Windows Media (.WMV). .AVI files can be hacked to pseudo-VFR by changing the frame rate to 119.88, usually used with WMV9 VCM (VFW version of the WMV9 codec), but this combination is rarely used for good reasons. Other combinations do not make sense since AVI also has no proper support for B-frames and hence H.264 anyway i.e. to get H.264/AVC you have to use MP4 or MKV.

The solution of this, is frame rate conversion. Basically, we need to tell the encoder to encode at a specific frame rate, then convert some parts of the video to match that frame rate. You can do it in VirtualDub, and if I'm not wrong, in TMPGEnc Encoder Xpress (aka Tsunami Encoder Xpress) though not entirely successful, but for this guide I'll use AVISynth for the highest compatibility.

Firstly, you need to have AVISynth prepared (skip this if you already know AVISynth)
1. Get version 2.5 or later here: http://sourceforge.net/project/showfiles.php?group_id=57023
2. Create a .avs file. To do that, create a new .txt file using Notepad and when saving, select save as type - all files and type .avs at the end. Double-click the .avs file to edit later on.
3. Type in the first line - DirectShowSource("full path and name of file"), e.g. DirectShowSource("C:\My Documents\Video1.mkv")

That's about all you need to do to play the video. I used DirectShowSource file reader instead of the (default?) AVISource for 2 important reasons:
- DirectShowSource is THE reason why AVISynth is so useful and popular - it allows the input of ANY file type as long as you can play it with Windows Media Player. This gives it high compatability, and this is used in VirtualDubMod and Video9 and many other programs for them to read all the otherwise not readable formats.
- Only by using DirectShowSource you can get subtitles and other streams (other than the first video and audio streams) to show. This is less important than the first reason, because seldom will you find subtitles and AVI files and they'd be hacked, and people tend to use MKV for multiple streams, which you'll have to use DirectShowSource to open anyway.

So now you have the .avs file to open the video, now add some additional stuff to that line:

The same line after, in example format:
DirectShowSource("My Documents\Video1.mkv", fps=23.976, convertfps=true)

What it does is self-explanatory, converts the entire clip's frame rate to 23.976.

Why 23.976?

Because most of the time you'll be dealing with Anime for this as mentioned above. And in Anime, most of the time the 29.97 part would only be the opening, which you'd skip. The rest of the video would be 23.976. It's disastrous to frame rate conversion, so we'll leave the more important parts untouched.

Furthermore, it's better to convert from 29.97 to 23.976 than the other way round. Because with 29.97 -> 23.976, you're removing 1 frame out of 5 to give 4 different frames, but for 23.976 -> 29.97 you're adding 1 frame to 4 to give 4 different frames and 1 clone, the clone will be very obvious and results in more stuttering. Note that this is different from 3:2 pulldown (and removal), because again, 3:2 pulldown (and removal) assumes the entire clip to have the same frame rate. Applying it only causes more problems. Speaking of which, you're more likely going to get back pseudo-23.976 if you do the latter, than get back pseudo-29.97 if you do the former, by applying 3:2 pulldown (or removal). It's not important info at all and is just trivia, but you may want to use logic to find out why so.