As of this HandBrake svn commit and its corresponding Nightly Build, HandBrake is no longer overriding x264′s default of b-pyramid being on. Previously HandBrake would internally turn off b-pyramid by initializing it to 0 as it caused issues with the version of QuickTime that existed at that time. QuickTime has progressed and can handle b-pyramid just fine … except for the modified 7.0 version found in the AppleTV (much like weightp addressed prevously). At any rate using b-pyramid on the current AppleTV firmware will not only screw up your encodes … but will in fact freeze the AppleTV which only a restart can correct. So the way to fix this is by adding:
b-pyramid=none
to all of your custom presets advanced panel *if* you are using any HandBrake nightly after svn 3424 (NOTE: this does not affect the last public release of HandBrake 0.9.4 however in fact there is no reason not to add it if using 0.9.4 in the interest of “future proofing”). This also does not affect the AppleTV built in presets as that is also added in the commit. So only critical for your custom presets if using a version after 3424.
The advance custom string discussed in HandBrake AppleTV Hi Profile Setting: Part 2 has been changed accordingly to: