Dynamic Aspect Ratio Detection With FFmpeg

Christopher Hranj -

This week I've been working on a challenging new feature for one of GIPHY's GIF creation tools. The goal is to detect every aspect ratio change of a long-form video, which is useful for videos such as smart-phone videos that contain a combination of vertical and horizontal video. Here's an example of such a video:

Notice at ~1:23 mark, the video changes from vertical to horizontal.

For a seasoned FFmpeg user or a command-line guru, this would probably be an easy task. Unfortunately, I am neither of those.

After a ton of Google/StackOverflow hunting and documentation scanning, I was finally able to cobble together a solution.

Getting Started

This post will assume a UNIX/Linux development environment. The only tools required are youtube-dl and FFmpeg:

  • Install instructions for youtube-dl are available here.
  • Install instructions for FFmpeg are available here.

The following command can be used to download the example video from above:

yt-dl -f "bv*[ext=mp4]+ba[ext=m4a]/b[ext=mp4] / bv*+ba/b" "https://www.youtube.com/watch?v=06kdC28AGKg" -o video.mp4

This command will download the best quality version available of the video and save it as video.mp4.

The Solution

A great deal of trial/error and refactoring led to the following solution:

ffmpeg -nostats -i video.mp4 -vf cropdetect=reset=1 -f null - 2>&1 | sed -ne 's/\[[^][]*\] //p' | awk '$11!=old {print $0}{old=$11}'

There's a lot going on in this (relatively) short command. Here's a breakdown of each piece:

  • ffmpeg -nostats -i video.mp4 -vf cropdetect=reset=1 -f null - 2>&1
    • -nostats is required to prevent FFmpeg's progress bar from disrupting the expected output
    • -i video.mp4 simply specifies the name of the video file to run the crop detection on
    • -vf cropdetect=reset=1 tells FFmpeg to use the cropdetect filtergraph with the reset=1 parameter so that the filter will reset to detect the current optimal crop area after every frame
    • -f null - is the null muxer which is used to prevent an output file from being generated
    • 2>&1 is used to redirect stderr to stdout since FFmpeg's output is sent to stderr in this case
  • sed -ne 's/\[[^][]*\] //p'
    • sed is being used here to parse out only the necessary info. the ouput being piped to this command initially looks as such:
      [Parsed_cropdetect_0 @ 0x158f078d0] x1:437 x2:842 y1:0 y2:719 w:400 h:720 x:440 y:0 pts:2002 t:0.066733 crop=400:720:440:0
    • the -n flag is used to suppress each line of output by default
    • the -e flag followed by s/\[[^][]*\] // is a substitution. this will pattern match everything within the brackets (and the trailing space) and replace it with an empty string
    • the p at the end of the substitution is the print flag which will print the line only if the substituion succeeds
  • awk '$11!=old {print $0}{old=$11}'
    • awk will assign the 11th field of the line, which is the crop= field in this case, to a temporary variable, and compare it to the previous line
    • if the crop= field does not match the previous line, it will execute {print $0} which prints out the entire line
    • it's important to note that awk field numbers are indexed starting at 1, and field 0 refers to the entire line

The output produced by this command should look something like this:

x1:437 x2:842 y1:0 y2:719 w:400 h:720 x:440 y:0 pts:2002 t:0.066733 crop=400:720:440:0
x1:0 x2:1279 y1:0 y2:719 w:1280 h:720 x:0 y:0 pts:2499497 t:83.316567 crop=1280:720:0:0

The {print $0} argument in the awk command can be updated to print specific fields if desired. For example, the timestamp field is located at field 10, so `awk '$11!=old {print $10}{old=$11}' can be used to print only the timestamps of each aspect ratio change:

$ ffmpeg -nostats -i video.mp4 -vf cropdetect=reset=1 -f null - 2>&1 | sed -ne 's/\[[^][]*\] //gp' | awk '$11!=old {print $10}{old=$11}'
t:0.066733
t:83.316567

Wrapping Up

This post has covered how to detect every change in a video's aspect ratio. Although this may be a pretty specific use case, this code will hopefully end up being useful to somebody, somewhere.

Special thanks to hackNY's Chris Wiggins and Devon Peticolas for showing me some command line intricacies.

If you have any questions or feedback please reach out to me @brodan_. Thanks for reading!