在常规的视频图像编辑处理场景中，画中画是必不可少的一个功能，而 FFmpeg 的 overlay 滤镜通过进行参数设置就可以轻松地达到画中画效果，下面就详细讲解一下如何使用 overlay 滤镜实现画中画效果。
在做画中画效果实现之前，首先需要了解基本的需求，那么问题来了，画中画就涉及到主画面和子画面，会遇到如下几个问题：
- 子画面显示在主画面的什么位置？
- 子画面从主画面的哪个时间点开始显示？
- 子画面从子画面的哪个时间点开始显示？
- 子画面是按照时间段显示还是一直显示？
- 如果子画面或者主画面不等长怎么办？
解决以上问题之前，首先了解一下overlay的一些基本参数：
overlay AVOptions:
x <string> ..FV..... set the x expression (default "0")
y <string> ..FV..... set the y expression (default "0")
eof_action <int> ..FV..... Action to take when encountering EOF from secondary input (from 0 to 2) (default repeat)
repeat ..FV..... Repeat the previous frame.
endall ..FV..... End both streams.
pass ..FV..... Pass through the main input.
eval <int> ..FV..... specify when to evaluate expressions (from 0 to 1) (default frame)
init ..FV..... eval expressions once during initialization
frame ..FV..... eval expressions per-frame
shortest <boolean> ..FV..... force termination when the shortest input terminates (default false)
format <int> ..FV..... set output format (from 0 to 5) (default yuv420)
yuv420 ..FV.....
yuv422 ..FV.....
yuv444 ..FV.....
rgb ..FV.....
gbrp ..FV.....
auto ..FV.....
repeatlast <boolean> ..FV..... repeat overlay of the last overlay frame (default true)
alpha <int> ..FV..... alpha format (from 0 to 1) (default straight)
straight ..FV.....
premultiplied ..FV.....
framesync AVOptions:
eof_action <int> ..FV..... Action to take when encountering EOF from secondary input (from 0 to 2) (default repeat)
repeat ..FV..... Repeat the previous frame.
endall ..FV..... End both streams.
pass ..FV..... Pass through the main input.
shortest <boolean> ..FV..... force termination when the shortest input terminates (default false)
repeatlast <boolean> ..FV..... extend last frame of secondary streams beyond EOF (default true)
This filter has support for timeline through the 'enable' option.解决第 1 个问题，主要设置 x, y 这两个参数就可以达到效果；
解决第 2 个问题，看最后一行，这个滤镜是支持 timeline 的，通过 enable 配合可以设置可见时间区间；
解决第 4 个问题，与第 2 个问题解决方法相同；
第 5 个问题，如果两个视频流时长不等长，可以通过 framesync 参数来处理，具体的说明均在上面，如果英语不好，可以长按英文，然后复制说明内容，贴到翻译软件中翻译一下。
那么，第 3 个问题怎么办呢？第 3 个问题单单用 overlay 滤镜解决不了，想一下场景：
主画面播放 3 秒后，子画面从子画面的第 0 秒开始显示出来，显示 3 秒钟后不继续显示
这种场景，子画面的 start time 与主画面的 start time 之间存在了偏移，那么视频偏移可以通过 setpts 滤镜设置：
Filter setpts
Set PTS for the output video frame.
Inputs:
#0: default (video)
Outputs:
#0: default (video)
setpts AVOptions:
expr <string> ..FVA.... Expression determining the frame timestamp (default "PTS")
pts 设置比较简单，通过 expr 也就是咱们现在这个公众号以往的分享内容中可以找到。
overlay 配合 pts 即可解决第三个问题。
下面举个例子，争取把三个问题都串起来。
命令行部分
先创建两个素材,为了查看方便，都创建的是带时间显示的视频。
1. 创建主画面视频素材
StevenLiu:ffmpeg StevenLiu$ ffmpeg -f lavfi -i testsrc2=s=800x480 -vcodec libx264 -t 10 input1.mp4
ffmpeg version N-94165-g6e988b75df Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
configuration: --enable-libass --enable-opengl --enable-libx264 --enable-libmp3lame --enable-gpl --enable-nonfree --prefix=/usr/local --enable-libtesseract --enable-libspeex --enable-libfreetype --enable-libfontconfig --enable-libfdk-aac --enable-videotoolbox --enable-libxml2 --enable-librsvg --enable-libvmaf --enable-version3 --disable-stripping --disable-optimizations
libavutil 56. 30.100 / 56. 30.100
libavcodec 58. 53.101 / 58. 53.101
libavformat 58. 28.101 / 58. 28.101
libavdevice 58. 7.100 / 58. 7.100
libavfilter 7. 56.100 / 7. 56.100
libswscale 5. 4.101 / 5. 4.101
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100
Input #0, lavfi, from 'testsrc2=s=800x480':
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 800x480 [SAR 1:1 DAR 5:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x7ff3f2801200] using SAR=1/1
[libx264 @ 0x7ff3f2801200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 @ 0x7ff3f2801200] profile High, level 3.0
[libx264 @ 0x7ff3f2801200] 264 - core 133 r2334M a3ac64b - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'input1.mp4':
Metadata:
encoder : Lavf58.28.101
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 800x480 [SAR 1:1 DAR 5:3], q=-1--1, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder : Lavc58.53.101 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame= 250 fps= 92 q=-1.0 Lsize= 1297kB time=00:00:09.88 bitrate=1075.4kbits/s speed=3.63x
video:1294kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.224270%
[libx264 @ 0x7ff3f2801200] frame I:1 Avg QP:24.72 size: 11441
[libx264 @ 0x7ff3f2801200] frame P:180 Avg QP:27.72 size: 5503
[libx264 @ 0x7ff3f2801200] frame B:69 Avg QP:30.78 size: 4673
[libx264 @ 0x7ff3f2801200] consecutive B-frames: 50.0% 36.8% 8.4% 4.8%
[libx264 @ 0x7ff3f2801200] mb I I16..4: 72.5% 13.7% 13.9%
[libx264 @ 0x7ff3f2801200] mb P I16..4: 3.9% 0.4% 0.3% P16..4: 8.0% 4.9% 3.6% 0.0% 0.0% skip:78.9%
[libx264 @ 0x7ff3f2801200] mb B I16..4: 0.6% 0.1% 0.2% B16..8: 13.2% 3.7% 0.9% direct: 2.2% skip:79.0% L0:47.6% L1:44.4% BI: 8.0%
[libx264 @ 0x7ff3f2801200] 8x8 transform intra:8.5% inter:18.7%
[libx264 @ 0x7ff3f2801200] coded y,uvDC,uvAC intra: 6.6% 12.4% 11.2% inter: 5.3% 9.2% 7.9%
[libx264 @ 0x7ff3f2801200] i16 v,h,dc,p: 82% 17% 2% 0%
[libx264 @ 0x7ff3f2801200] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 4% 81% 0% 0% 0% 0% 0% 1%
[libx264 @ 0x7ff3f2801200] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 19% 37% 5% 2% 2% 3% 3% 4%
[libx264 @ 0x7ff3f2801200] i8c dc,h,v,p: 63% 17% 20% 0%
[libx264 @ 0x7ff3f2801200] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7ff3f2801200] ref P L0: 50.7% 7.4% 24.6% 17.2%
[libx264 @ 0x7ff3f2801200] ref B L0: 69.9% 27.6% 2.5%
[libx264 @ 0x7ff3f2801200] ref B L1: 95.4% 4.6%
[libx264 @ 0x7ff3f2801200] kb/s:1059.602. 创建子画面素材
StevenLiu:ffmpeg StevenLiu$ ffmpeg -f lavfi -i testsrc2=s=320x240 -vcodec libx264 -t 5 input2.mp4
ffmpeg version N-94165-g6e988b75df Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
configuration: --enable-libass --enable-opengl --enable-libx264 --enable-libmp3lame --enable-gpl --enable-nonfree --prefix=/usr/local --enable-libtesseract --enable-libspeex --enable-libfreetype --enable-libfontconfig --enable-libfdk-aac --enable-videotoolbox --enable-libxml2 --enable-librsvg --enable-libvmaf --enable-version3 --disable-stripping --disable-optimizations
libavutil 56. 30.100 / 56. 30.100
libavcodec 58. 53.101 / 58. 53.101
libavformat 58. 28.101 / 58. 28.101
libavdevice 58. 7.100 / 58. 7.100
libavfilter 7. 56.100 / 7. 56.100
libswscale 5. 4.101 / 5. 4.101
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100
Input #0, lavfi, from 'testsrc2=s=320x240':
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x7f8faa805200] using SAR=1/1
[libx264 @ 0x7f8faa805200] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 @ 0x7f8faa805200] profile High, level 1.3
[libx264 @ 0x7f8faa805200] 264 - core 133 r2334M a3ac64b - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'input2.mp4':
Metadata:
encoder : Lavf58.28.101
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 320x240 [SAR 1:1 DAR 4:3], q=-1--1, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder : Lavc58.53.101 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame= 125 fps=0.0 q=-1.0 Lsize= 173kB time=00:00:04.88 bitrate= 289.9kbits/s speed=12.4x
video:171kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.129911%
[libx264 @ 0x7f8faa805200] frame I:1 Avg QP:25.53 size: 4375
[libx264 @ 0x7f8faa805200] frame P:84 Avg QP:27.86 size: 1519
[libx264 @ 0x7f8faa805200] frame B:40 Avg QP:32.16 size: 1056
[libx264 @ 0x7f8faa805200] consecutive B-frames: 47.2% 24.0% 19.2% 9.6%
[libx264 @ 0x7f8faa805200] mb I I16..4: 66.7% 7.3% 26.0%
[libx264 @ 0x7f8faa805200] mb P I16..4: 4.6% 0.4% 0.5% P16..4: 13.5% 8.8% 6.7% 0.0% 0.0% skip:65.5%
[libx264 @ 0x7f8faa805200] mb B I16..4: 0.8% 0.1% 0.1% B16..8: 19.7% 6.5% 1.8% direct: 2.6% skip:68.6% L0:45.5% L1:49.1% BI: 5.4%
[libx264 @ 0x7f8faa805200] 8x8 transform intra:6.7% inter:34.2%
[libx264 @ 0x7f8faa805200] coded y,uvDC,uvAC intra: 7.5% 23.3% 18.9% inter: 5.6% 15.1% 12.2%
[libx264 @ 0x7f8faa805200] i16 v,h,dc,p: 75% 21% 4% 0%
[libx264 @ 0x7f8faa805200] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 14% 8% 68% 6% 0% 0% 2% 1% 1%
[libx264 @ 0x7f8faa805200] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 42% 20% 30% 1% 1% 1% 2% 1% 1%
[libx264 @ 0x7f8faa805200] i8c dc,h,v,p: 45% 21% 34% 1%
[libx264 @ 0x7f8faa805200] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7f8faa805200] ref P L0: 61.3% 7.5% 18.9% 12.2%
[libx264 @ 0x7f8faa805200] ref B L0: 77.5% 18.8% 3.7%
[libx264 @ 0x7f8faa805200] ref B L1: 94.8% 5.2%
[libx264 @ 0x7f8faa805200] kb/s:278.713. 开始画中画操作
StevenLiu:ffmpeg StevenLiu$ ffmpeg -i input1.mp4 -filter_complex "movie=input2.mp4[in2];[in2]setpts=PTS+2/TB[out2];[0:v][out2]overlay=x=20:y=120:enable='between(t,2,4)':shortest=1" output.mp4
ffmpeg version N-94165-g6e988b75df Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
configuration: --enable-libass --enable-opengl --enable-libx264 --enable-libmp3lame --enable-gpl --enable-nonfree --prefix=/usr/local --enable-libtesseract --enable-libspeex --enable-libfreetype --enable-libfontconfig --enable-libfdk-aac --enable-videotoolbox --enable-libxml2 --enable-librsvg --enable-libvmaf --enable-version3 --disable-stripping --disable-optimizations
libavutil 56. 30.100 / 56. 30.100
libavcodec 58. 53.101 / 58. 53.101
libavformat 58. 28.101 / 58. 28.101
libavdevice 58. 7.100 / 58. 7.100
libavfilter 7. 56.100 / 7. 56.100
libswscale 5. 4.101 / 5. 4.101
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input1.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.28.101
Duration: 00:00:10.00, start: 0.000000, bitrate: 1062 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 800x480 [SAR 1:1 DAR 5:3], 1060 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
Metadata:
handler_name : VideoHandler
Stream mapping:
Stream #0:0 (h264) -> overlay:main
overlay -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[libx264 @ 0x7fab3a827000] using SAR=1/1
[libx264 @ 0x7fab3a827000] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX AVX2 FMA3 LZCNT BMI2
[libx264 @ 0x7fab3a827000] profile High, level 3.0
[libx264 @ 0x7fab3a827000] 264 - core 133 r2334M a3ac64b - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'output.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.28.101
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 800x480 [SAR 1:1 DAR 5:3], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
Metadata:
encoder : Lavc58.53.101 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
[Parsed_movie_0 @ 0x7fab3a4072c0] EOF timestamp not reliablebitrate= 873.9kbits/s speed=1.87x
frame= 250 fps= 52 q=-1.0 Lsize= 1311kB time=00:00:09.88 bitrate=1087.2kbits/s speed=2.05x
video:1308kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.226632%
[libx264 @ 0x7fab3a827000] frame I:1 Avg QP:22.64 size: 11415
[libx264 @ 0x7fab3a827000] frame P:173 Avg QP:26.25 size: 5606
[libx264 @ 0x7fab3a827000] frame B:76 Avg QP:30.55 size: 4705
[libx264 @ 0x7fab3a827000] consecutive B-frames: 44.4% 43.2% 6.0% 6.4%
[libx264 @ 0x7fab3a827000] mb I I16..4: 64.3% 22.3% 13.4%
[libx264 @ 0x7fab3a827000] mb P I16..4: 3.2% 0.6% 0.4% P16..4: 9.4% 5.0% 3.4% 0.0% 0.0% skip:77.9%
[libx264 @ 0x7fab3a827000] mb B I16..4: 0.4% 0.1% 0.2% B16..8: 14.2% 3.8% 1.0% direct: 2.3% skip:78.1% L0:49.1% L1:44.1% BI: 6.8%
[libx264 @ 0x7fab3a827000] 8x8 transform intra:16.0% inter:20.7%
[libx264 @ 0x7fab3a827000] coded y,uvDC,uvAC intra: 8.9% 17.4% 15.5% inter: 5.4% 10.0% 8.5%
[libx264 @ 0x7fab3a827000] i16 v,h,dc,p: 84% 14% 2% 0%
[libx264 @ 0x7fab3a827000] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 14% 13% 68% 0% 1% 1% 1% 0% 1%
[libx264 @ 0x7fab3a827000] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 19% 37% 5% 2% 2% 3% 3% 4%
[libx264 @ 0x7fab3a827000] i8c dc,h,v,p: 65% 15% 20% 0%
[libx264 @ 0x7fab3a827000] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fab3a827000] ref P L0: 52.4% 7.5% 23.5% 16.6%
[libx264 @ 0x7fab3a827000] ref B L0: 73.4% 25.2% 1.5%
[libx264 @ 0x7fab3a827000] ref B L1: 97.1% 2.9%
[libx264 @ 0x7fab3a827000] kb/s:1071.14
StevenLiu:ffmpeg StevenLiu$1我们解析一下命令行参数：
- 输入的主文件是 input1.mp4
- 输入的子文件是 input2.mp4
- 将子画面的所有的时间戳都加上 2 秒钟的偏移
- 将子画面显示在主画面的 x 坐标为 20，y 坐标为 120 的位置
- 子画面在主画面显示时间段是第 2 秒到第 4 秒之间
- 以主画面和子画面中时长最短的那个视频流长度为结束点
API 调用实现
为了让觉得用命令行 Low B 确喜欢调用 API 的大佬们开开心心抄代码，我把代码部分贴上来：
StevenLiu:ffmpeg StevenLiu$ git diff
diff --git a/doc/examples/transcoding.c b/doc/examples/transcoding.c
index e48837cbd2..cf46a61492 100644
--- a/doc/examples/transcoding.c
+++ b/doc/examples/transcoding.c
@@ -391,7 +391,7 @@ static int init_filters(void)
if (ifmt_ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO)
- filter_spec = "null"; /* passthrough (dummy) filter for video */
+ filter_spec = "movie=input2.mp4[in2];[in2]setpts=PTS+2/TB[out2];[in][out2]overlay=x=20:y=120:enable='between(t,2,4)':shortest=1"; /* passthrough (dummy) filter for video */
else
filter_spec = "anull"; /* passthrough (dummy) filter for audio */
ret = init_filter(&filter_ctx[i], stream_ctx[i].dec_ctx,
StevenLiu:ffmpeg StevenLiu$下面看一下生成出来的视频的效果：
如果不设置 PTS 的话，效果如下：
今天就介绍这些。
作者：悟空；公众号：流媒体技术
版权声明：本文内容转自互联网，本文观点仅代表作者本人。本站仅提供信息存储空间服务，所有权归原作者所有。如发现本站有涉嫌抄袭侵权/违法违规的内容， 请发送邮件至1393616908@qq.com 举报，一经查实，本站将立刻删除。
