今天有个为 FFmpeg 贡献高质量 patch 的朋友问到了一个问题，如何编写一个 FATE 测试用例，当时简单地回复了操作步骤，那么接下来我将会在这里详细介绍一下如何编写 FFmpeg 的自动化测试用例。
一. 测试用例结果参考数据说明
首先看一下测试用例参考数据内容大概的展现形态：
bogon:ffmpeg_up liuqi$ cat tests/ref/fate/hls-segment-size
#tb 0: 1/44100
#media_type 0: audio
#codec_id 0: pcm_s16le
#sample_rate 0: 44100
#channel_layout 0: 4
#channel_layout_name 0: mono
0, 0, 0, 1152, 2304, 0x907cb7fa
0, 1152, 1152, 1152, 2304, 0xb8dc7525
0, 2304, 2304, 1152, 2304, 0x3e7d6905
0, 3456, 3456, 1152, 2304, 0xef47877b
0, 4608, 4608, 1152, 2304, 0xfe916b7e
0, 5760, 5760, 1152, 2304, 0xe3d08cde
0, 6912, 6912, 1152, 2304, 0xff7f86cf
0, 8064, 8064, 1152, 2304, 0x843e6f95
0, 9216, 9216, 1152, 2304, 0x81577c26
0, 10368, 10368, 1152, 2304, 0x04a085d5
0, 11520, 11520, 1152, 2304, 0x1c5a76f5
0, 12672, 12672, 1152, 2304, 0x4ee78623
0, 13824, 13824, 1152, 2304, 0x8ec861dc
0, 14976, 14976, 1152, 2304, 0x0ca179d8
0, 16128, 16128, 1152, 2304, 0xc6da750f
0, 17280, 17280, 1152, 2304, 0xf6bf79b5
0, 18432, 18432, 1152, 2304, 0x97b88a43
0, 19584, 19584, 1152, 2304, 0xf13c7b9c
0, 20736, 20736, 1152, 2304, 0xdfba83af
0, 21888, 21888, 1152, 2304, 0xc9467d4b
0, 23040, 23040, 1152, 2304, 0xbbb58e2b
0, 24192, 24192, 1152, 2304, 0x3a1078ea
0, 25344, 25344, 1152, 2304, 0xe9587a5c
此后省略好几百行
0, 880128, 880128, 1152, 2304, 0x3cb18530
0, 881280, 881280, 1152, 2304, 0x5a0c5e7b
bogon:ffmpeg_up liuqi$从内容上看，这是一个音频数据的测试文件，timebase是1/44100，采样率是44100，媒体类型是音频，音频格式是pcm_s16le，音频通道布局是mono。然后下面就是每一帧音频数据的输出信息，一共 6 列，分别代表，
- 当前 packet 的流所在的索引就是 packet->stream_index
- 当前 packet 的 dts
- 当前 packet 的 pts
- 当前 packet 的 duration
- 当前 packet 的数据包大小
- 当前数据包的 crc 值
二. 实际操作
那么这一堆数据是怎么生成的呢？下面从FATE中找到一个样例看一下：
tests/data/hls_segment_size.m3u8: TAG = GEN
tests/data/hls_segment_size.m3u8: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data
$(M)$(TARGET_EXEC) $(TARGET_PATH)/$< \
-f lavfi -i "aevalsrc=cos(2*PI*t)*sin(2*PI*(440+4*t)*t):d=20" -f hls -hls_segment_size 300000 -map 0 \
-hls_list_size 0 -codec:a mp2fixed -hls_segment_filename $(TARGET_PATH)/tests/data/hls_segment_size_%d.ts \
$(TARGET_PATH)/tests/data/hls_segment_size.m3u8 2>/dev/null
FATE_AFILTER-$(call ALLYES, HLS_DEMUXER MPEGTS_MUXER MPEGTS_DEMUXER AEVALSRC_FILTER LAVFI_INDEV MP2FIXED_ENCODER) += fate-hls-segment-size
fate-hls-segment-size: tests/data/hls_segment_size.m3u8
fate-hls-segment-size: CMD = framecrc -flags +bitexact -i $(TARGET_PATH)/tests/data/hls_segment_size.m3u8 -vf setpts=N*23从例子中可以看到，首先是生成一个m3u8列表，也就是这一段：
tests/data/hls_segment_size.m3u8: TAG = GEN
tests/data/hls_segment_size.m3u8: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data
$(M)$(TARGET_EXEC) $(TARGET_PATH)/$< \
-f lavfi -i "aevalsrc=cos(2*PI*t)*sin(2*PI*(440+4*t)*t):d=20" -f hls -hls_segment_size 300000 -map 0 \
-hls_list_size 0 -codec:a mp2fixed -hls_segment_filename $(TARGET_PATH)/tests/data/hls_segment_size_%d.ts \
$(TARGET_PATH)/tests/data/hls_segment_size.m3u8这一段最终执行时也就是这样的：
/Users/liuqi/project/ffmpeg_up/dash/ffmpeg \
-f lavfi -i "aevalsrc=cos(2*PI*t)*sin(2*PI*(440+4*t)*t):d=20" -f hls -hls_segment_size 300000 -map 0 \
-hls_list_size 0 -codec:a mp2fixed -hls_segment_filename /Users/liuqi/project/ffmpeg_up/dash/tests/data/hls_segment_size_%d.ts \
/Users/liuqi/project/ffmpeg_up/dash/tests/data/hls_segment_size.m3u8 2>/dev/null然后是下面那一段生成framecrc数据的操作
FATE_AFILTER-$(call ALLYES, HLS_DEMUXER MPEGTS_MUXER MPEGTS_DEMUXER AEVALSRC_FILTER LAVFI_INDEV MP2FIXED_ENCODER) += fate-hls-segment-size
fate-hls-segment-size: tests/data/hls_segment_size.m3u8
fate-hls-segment-size: CMD = framecrc -flags +bitexact -i $(TARGET_PATH)/tests/data/hls_segment_size.m3u8 -vf setpts=N*23内容也比较好理解，首先是执行一句tests/data/hls_segment_size.m3u8，在make fate-hls-segment-size的时候首先会执行tests/data/hls_segment_size.m3u8，然后会执行这一句
CMD = framecrc -flags +bitexact -i $(TARGET_PATH)/tests/data/hls_segment_size.m3u8 -vf setpts=N*这里的framecrc是一个shell封装函数，这个函数是封装在tests/fate-run.sh文件中。
124 ffmpeg(){
125 dec_opts="-hwaccel $hwaccel -threads $threads -thread_type $thread_type"
126 ffmpeg_args="-nostdin -nostats -cpuflags $cpuflags"
127 for arg in $@; do
128 [ x${arg} = x-i ] && ffmpeg_args="${ffmpeg_args} ${dec_opts}"
129 ffmpeg_args="${ffmpeg_args} ${arg}"
130 done
131 run ffmpeg${PROGSUF}${EXECSUF} ${ffmpeg_args}
132 }
133
134 framecrc(){
135 ffmpeg "$@" -bitexact -f framecrc -
136 }
137
138 ffmetadata(){
139 ffmpeg "$@" -bitexact -f ffmetadata -
140 }
141
142 framemd5(){
143 ffmpeg "$@" -bitexact -f framemd5 -
144 }
145
146 crc(){
147 ffmpeg "$@" -f crc -
148 }
149
150 md5pipe(){
151 ffmpeg "$@" md5:
152 }
153
154 md5(){
155 encfile="${outdir}/${test}.out"
156 cleanfiles="$cleanfiles $encfile"
157 ffmpeg "$@" $encfile
158 do_md5sum $encfile | awk '{print $1}'
159 }最后拼出来的fate执行的效果大概是这样的：
/Users/liuqi/project/ffmpeg_up/dash/ffmpeg -nostdin -nostats -cpuflags all -flags +bitexact -h然后将最后一个字符’-‘替换成自己的文件名即可，那么这里可以替换成tests/ref/fate/hls-segment-size，也就基本上大功告成了。
然后剩下的FATE相关信息，可以参考FFmpeg的官方文档：
FFmpeg Automated Testing Environment
Table of Contents
1 Introduction
2 Using FATE from your FFmpeg source directory
3 Submitting the results to the FFmpeg result aggregation server
4 Uploading new samples to the fate suite
5 FATE makefile targets and variables
5.1 Makefile targets
5.2 Makefile variables
5.3 Examples
1 Introduction
FATE is an extended regression suite on the client-side and a means for results aggregation and presentation on the server-side.
The first part of this document explains how you can use FATE from your FFmpeg source directory to test your ffmpeg binary. The second part describes how you can run FATE to submit the results to FFmpeg’s FATE server.
In any way you can have a look at the publicly viewable FATE results by visiting this website:
http://fate.ffmpeg.org/
This is especially recommended for all people contributing source code to FFmpeg, as it can be seen if some test on some platform broke with their recent contribution. This usually happens on the platforms the developers could not test on.
The second part of this document describes how you can run FATE to submit your results to FFmpeg’s FATE server. If you want to submit your results be sure to check that your combination of CPU, OS and compiler is not already listed on the above mentioned website.
In the third part you can find a comprehensive listing of FATE makefile targets and variables.
2 Using FATE from your FFmpeg source directory
If you want to run FATE on your machine you need to have the samples in place. You can get the samples via the build target fate-rsync. Use this command from the top-level source directory:
make fate-rsync SAMPLES=fate-suite/
make fate SAMPLES=fate-suite/
The above commands set the samples location by passing a makefile variable via command line. It is also possible to set the samples location at source configuration time by invoking configure with --samples=<path to the samples directory>. Afterwards you can invoke the makefile targets without setting the SAMPLES makefile variable. This is illustrated by the following commands:
./configure --samples=fate-suite/
make fate-rsync
make fate
Yet another way to tell FATE about the location of the sample directory is by making sure the environment variable FATE_SAMPLES contains the path to your samples directory. This can be achieved by e.g. putting that variable in your shell profile or by setting it in your interactive session.
FATE_SAMPLES=fate-suite/ make fate
Do not put a ’~’ character in the samples path to indicate a home directory. Because of shell nuances, this will cause FATE to fail.
To use a custom wrapper to run the test, pass --target-exec to configure or set the TARGET_EXEC Make variable.
3 Submitting the results to the FFmpeg result aggregation server
To submit your results to the server you should run fate through the shell script tests/fate.sh from the FFmpeg sources. This script needs to be invoked with a configuration file as its first argument.
tests/fate.sh /path/to/fate_config
A configuration file template with comments describing the individual configuration variables can be found at doc/fate_config.sh.template.
The mentioned configuration template is also available here:
slot= # some unique identifier
repo=git://source.ffmpeg.org/ffmpeg.git # the source repository
#branch=release/2.6 # the branch to test
samples= # path to samples directory
workdir= # directory in which to do all the work
#fate_recv="ssh -T fate@fate.ffmpeg.org" # command to submit report
comment= # optional description
build_only= # set to "yes" for a compile-only instance that skips tests
ignore_tests=
# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
as=
cc=
ld=
target_os=
sysroot=
target_exec=
target_path=
target_samples=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf= # extra configure options not covered above
#make= # name of GNU make if not 'make'
makeopts= # extra options passed to 'make'
#makeopts_fate= # extra options passed to 'make' when running tests,
# defaulting to makeopts above if this is not set
#tar= # command to create a tar archive from its arguments on stdout,
# defaults to 'tar c'
Create a configuration that suits your needs, based on the configuration template. The slot configuration variable can be any string that is not yet used, but it is suggested that you name it adhering to the following pattern ‘arch-os-compiler-compiler version’. The configuration file itself will be sourced in a shell script, therefore all shell features may be used. This enables you to setup the environment as you need it for your build.
For your first test runs the fate_recv variable should be empty or commented out. This will run everything as normal except that it will omit the submission of the results to the server. The following files should be present in $workdir as specified in the configuration file:
configure.log
compile.log
test.log
report
version
When you have everything working properly you can create an SSH key pair and send the public key to the FATE server administrator who can be contacted at the email address fate-admin@ffmpeg.org.
Configure your SSH client to use public key authentication with that key when connecting to the FATE server. Also do not forget to check the identity of the server and to accept its host key. This can usually be achieved by running your SSH client manually and killing it after you accepted the key. The FATE server’s fingerprint is:
‘RSA’
d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51
‘ECDSA’
76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86
If you have problems connecting to the FATE server, it may help to try out the ssh command with one or more -v options. You should get detailed output concerning your SSH configuration and the authentication process.
The only thing left is to automate the execution of the fate.sh script and the synchronisation of the samples directory.
4 Uploading new samples to the fate suite
If you need a sample uploaded send a mail to samples-request.
This is for developers who have an account on the fate suite server. If you upload new samples, please make sure they are as small as possible, space on each client, network bandwidth and so on benefit from smaller test cases. Also keep in mind older checkouts use existing sample files, that means in practice generally do not replace, remove or overwrite files as it likely would break older checkouts or releases. Also all needed samples for a commit should be uploaded, ideally 24 hours, before the push. If you need an account for frequently uploading samples or you wish to help others by doing that send a mail to ffmpeg-devel.
#First update your local samples copy:
rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite
#Then do a dry run checking what would be uploaded:
rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite
#Upload the files:
rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite
5 FATE makefile targets and variables
5.1 Makefile targets
fate-rsync
Download/synchronize sample files to the configured samples directory.
fate-list
Will list all fate/regression test targets.
fate
Run the FATE test suite (requires the fate-suite dataset).
5.2 Makefile variables
V
Verbosity level, can be set to 0, 1 or 2.
0: show just the test arguments
1: show just the command used in the test
2: show everything
SAMPLES
Specify or override the path to the FATE samples at make time, it has a meaning only while running the regression tests.
THREADS
Specify how many threads to use while running regression tests, it is quite useful to detect thread-related regressions.
THREAD_TYPE
Specify which threading strategy test, either ‘slice’ or ‘frame’, by default ‘slice+frame’
CPUFLAGS
Specify CPU flags.
TARGET_EXEC
Specify or override the wrapper used to run the tests. The TARGET_EXEC option provides a way to run FATE wrapped in valgrind, qemu-user or wine or on remote targets through ssh.
GEN
Set to ‘1’ to generate the missing or mismatched references.
HWACCEL
Specify which hardware acceleration to use while running regression tests, by default ‘none’ is used.
KEEP
Set to ‘1’ to keep temp files generated by fate test(s) when test is successful. Default is ‘0’, which removes these files. Files are always kept when a test fails.
5.3 Examples
make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate三 结语
其实要看过程的话，make 自己的tag就可以了，例如这样：
bogon:dash liuqi$ make fate-hls-segment-size V=1
/Users/liuqi/project/ffmpeg_up/dash/ffmpeg \
-f lavfi -i "aevalsrc=cos(2*PI*t)*sin(2*PI*(440+4*t)*t):d=20" -f hls -hls_segment_size 300000 -map 0 \
-hls_list_size 0 -codec:a mp2fixed -hls_segment_filename /Users/liuqi/project/ffmpeg_up/dash/tests/data/hls_segment_size_%d.ts \
/Users/liuqi/project/ffmpeg_up/dash/tests/data/hls_segment_size.m3u8 2>/dev/null
TEST hls-segment-size
src/tests/fate-run.sh fate-hls-segment-size "" "" "/Users/liuqi/project/ffmpeg_up/dash" 'framecrc -flags +bitexact -i /Users/liuqi/project/ffmpeg_up/dash/tests/data/hls_segment_size.m3u8 -vf setpts=N*23' '' '' '' '1' '' '' '' '' '' '' '' '' '' ''
/Users/liuqi/project/ffmpeg_up/dash/ffmpeg -nostdin -nostats -cpuflags all -flags +bitexact -hwaccel none -threads 1 -thread_type frame+slice -i /Users/liuqi/project/ffmpeg_up/dash/tests/data/hls_segment_size.m3u8 -vf setpts=N*23 -bitexact -f framecrc -
bogon:dash liuqi$读这篇内容需要一些基础：
- Makefile 基本原理和用法
- bash 脚本基本用法
作者：悟空 公众号：流媒体技术
版权声明：本文内容转自互联网，本文观点仅代表作者本人。本站仅提供信息存储空间服务，所有权归原作者所有。如发现本站有涉嫌抄袭侵权/违法违规的内容， 请发送邮件至1393616908@qq.com 举报，一经查实，本站将立刻删除。
