本地使用ClusterFuzz

本地使用ClusterFuzz

获取代码

代码语言：javascript代码运行次数：0运行复制

git clone 
cd clusterfuzz
git pull

处于稳定性考虑，建议使用最新的发行版本，而不是master分支

代码语言：javascript代码运行次数：0运行复制

git checkout tags/vX.Y.Z

通过git tag -l或者在github上面可以看到发行版本

代码语言：javascript代码运行次数：0运行复制

$ git tag -l
1.2.0
1.2.1
reproduce-tool-stable
v1.0.0
v1.0.1
v1.1.0
v1..0
v1.4.0
v1.5.0
v1.5.1
v1.6.0
v1.6.1
v1.7.0
v1.7.1
v1.8.0
v1.9.0
v2.0.0
v2.0.1

我用v2.0.1

代码语言：javascript代码运行次数：0运行复制

git checkout tags/v2.0.1`

依赖

首先谷歌建议使用python .7，python2已经不支持

代码语言：javascript代码运行次数：0运行复制

$ python butler.py
Traceback (most recent call last):
  File "butler.py", line , in <module>
    ()
  File "src/local/butler/guard.py", line 41, in check
    check_virtualenv()
  File "src/local/butler/guard.py", line 0, in check_virtualenv
    'You are not in a virtual env environment. Please install it with'
Exception: You are not in a virtual env environment. Please install it with `./local/install_deps.bash` or load it with `pipenv shell`. Then, you can re-run this command.

安装python .7：

代码语言：javascript代码运行次数：0运行复制

apt-get install -y apt-transport-https software-properties-common build-essential git curl \
      libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite-dev \
      libgdbm-dev libdb5.-dev libbz2-dev libexpat1-dev liblzma-dev libffi-dev uuid-dev
	  
curl -sS .7.7/Python-.7. | tar -C /tmp -xzv && \
      cd /tmp/Python-.7.7 && \
      ./configure --enable-optimizati && make altinstall && \
      rm -rf /tmp/Python-.7.7

安装golang

代码语言：javascript代码运行次数：0运行复制

sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt update
sudo apt install golang-go

修改./local/install_deps_linux.bash中的 bower install 为bower install --allow-root（因为bower install的时候默认不允许root用户）

最后执行./local/install_deps.bash

代码语言：javascript代码运行次数：0运行复制

./local/install_deps.bash

环境搭建

运行python.7 -m pipenv shell即可进入clusterfuzzer需要的环境

我们运行python butler.py，就看到所有功能了

代码语言：javascript代码运行次数：0运行复制

usage: butler.py [-h]
                 {bootstrap,py_unittest,js_unittest,format,lint,package,deploy,run_server,run,run_bot,remote,clean_indexes,create_config,integration_tests,reproduce}
                 ...

Butler is here to help you with command-line tasks.

positional arguments:
  {bootstrap,py_unittest,js_unittest,format,lint,package,deploy,run_server,run,run_bot,remote,clean_indexes,create_config,integration_tests,reproduce}
    bootstrap           Install all required dependencies for running an
                        appengine, a bot,and a mapreduce locally.
    py_unittest         Run Python unit tests.
    js_unittest         Run Javascript unit tests.
    format              Format changed code in current branch.
    lint                Lint changed code in current branch.
    package             Package clusterfuzz with a staging revision
    deploy              Deploy to Appengine
    run_server          Run the local Clusterfuzz server.
    run                 Run a one-off script against a datastore (e.g.
                        migration).
    run_bot             Run a local clusterfuzz bot.
    remote              Run command-line tasks on a remote bot.
    clean_indexes       Clean up undefined indexes (in index.yaml).
    create_config       Create a new deployment config.
    integration_tests   Run end-to-end integration tests.
    reproduce           Reproduce a crash or error from a test case.

optional arguments:
  -h, --help            show this help message and exit

初始化

第一次运行，初始化数据，试了下跟python butler.py bootstrap执行的功能一样。。。

代码语言：javascript代码运行次数：0运行复制

python butler.py run_server --bootstrap

假如不是第一次就直接

代码语言：javascript代码运行次数：0运行复制

python butler.py run_server

如果依赖早已安装好，使用下面的就可以跳过依赖的再次安装

代码语言：javascript代码运行次数：0运行复制

python butler.py run_server --skip-install-deps

最后终于起来了，访问9000端口即可

代码语言：javascript代码运行次数：0运行复制

$ python butler.py run_server --skip-install-deps
Running: pkill -KILL -f "dev_appserver.py"
| Return code is non-zero (-9).
Running: pkill -KILL -f "CloudDatastore.jar"
| Return code is non-zero (-9).
Running: pkill -KILL -f "pubsub-emulator"
| Return code is non-zero (-9).
Running: pkill -KILL -f "run_bot"
| Return code is non-zero (-9).
Created symlink: source: /root/clusterfuzz/configs/test, target /root/clusterfuzz/src/appengine/config.
Created symlink: source: /root/clusterfuzz/src/protos, target /root/clusterfuzz/src/appengine/protos.
Created symlink: source: /root/clusterfuzz/src/python, target /root/clusterfuzz/src/appengine/python.
Running: python polymer_bundler.py (cwd='local')
| App Engine templates are up to date.
Created symlink: source: /root/clusterfuzz/local/storage/local_gcs, target /root/clusterfuzz/src/appengine/local_gcs.
Running: gunicorn -b :9000 main:app (cwd='src/appengine')
| [2020-05-18 22:51:55 +0800] [185] [IFO] Starting gunicorn 20.0.4
| [2020-05-18 22:51:55 +0800] [185] [IFO] Listening at: :9000 (185)
| [2020-05-18 22:51:55 +0800] [185] [IFO] Using worker: sync
| [2020-05-18 22:51:55 +0800] [1841] [IFO] Booting worker with pid: 1841

访问9000端口如下：

之后启动botpython butler.py run_bot --name my-bot /path/to/my-bot，比如下面

代码语言：javascript代码运行次数：0运行复制

python butler.py run_bot --name my-fuzzing-bot `pwd`/my-fuzzing-bot

有时候空闲运行时间过长就退出了

代码语言：javascript代码运行次数：0运行复制

[1]    1446 killed     python butler.py run_bot --name my-fuzzing-bot `pwd`/my-fuzzing-bot

我们执行

代码语言：javascript代码运行次数：0运行复制

python butler.py run_bot `pwd`/my-fuzzing-bot

代码语言：javascript代码运行次数：0运行复制

$ python butler.py run_bot `pwd`/my-fuzzing-bot
Created symlink: source: /root/clusterfuzz/configs/test, target /root/clusterfuzz/src/appengine/config.
Bot directory already exists. Re-using...
| /root/clusterfuzz/my-fuzzing-bot/clusterfuzz/src/python/crash_analysis/stack_parsing/stack_analyzer.py:84: FutureWarning: Possible nested set at position 4
|   r'\s*[[][^]]*[:]([^](]*).*[]].*Check failed[:]\s*(.*)')
| /root/clusterfuzz/my-fuzzing-bot/clusterfuzz/src/python/crash_analysis/stack_parsing/stack_analyzer.py:202: FutureWarning: Possible nested set at position 
|   r'.*[[][^]]*[:]([^](]*).*[]].*Security CHECK failed[:]\s*(.*)\.\s*')
| /root/clusterfuzz/my-fuzzing-bot/clusterfuzz/src/python/crash_analysis/stack_parsing/stack_analyzer.py:204: FutureWarning: Possible nested set at position 
|   r'.*[[][^]]*[:]([^](]*).*[]].*Security DCHECK failed[:]\s*(.*)\.\s*')

在网页上也可以看bot的状态

可以看到bot应该是将clusterfuzz复制了一份

查看bot的log

代码语言：javascript代码运行次数：0运行复制

cd /path/to/my-bot/clusterfuzz/bot/logs
tail -f bot.log

可以看到先由于没有fuzzing任务所以显示Failed to get any fuzzing tasks

代码语言：javascript代码运行次数：0运行复制

$ tail bot.log
2020-05-19 11:16:49,456 - run_bot - IFO - Using local source, skipping source code update.
2020-05-19 11:16:49,457 - run_bot - IFO - Running platform initialization scripts.
2020-05-19 11:16:49,99 - run_bot - IFO - Completed running platform initialization scripts.
2020-05-19 11:16:50,74 - run_bot - ERROR - Failed to get any fuzzing tasks. This should not happen.
oneType: one
2020-05-19 11:21:50,478 - run_bot - IFO - Using local source, skipping source code update.
2020-05-19 11:21:50,478 - run_bot - IFO - Running platform initialization scripts.
2020-05-19 11:21:51,009 - run_bot - IFO - Completed running platform initialization scripts.
2020-05-19 11:21:51,194 - run_bot - ERROR - Failed to get any fuzzing tasks. This should not happen.
oneType: one

实际fuzz实例

ClusterFuzz支持覆盖率指导的模糊测试（libfuzzer和afl）和黑盒测试

接下来我们以心脏出血漏洞Heartbleed为例

使用这个平台，我们首先的编译出自己的fuzzer或者二进制程序

文档已经提供了针对OpenSSL的构建libfuzzer的代码，这样我们就得到了openssl-fuzzer-build.zip

代码语言：javascript代码运行次数：0运行复制

# Download and unpack a vulnerable version of OpenSSL:
curl -O .0.1/openssl-1.0.gz
tar xf openssl-1.0.gz

# Build OpenSSL with ASan and fuzzer instrumentation:
cd openssl-1.0.1f/
./config

# $CC must be pointing to clang binary, see the "compiler section" link above.
make CC="$CC -g -fsanitize=address,fuzzer-no-link"
cd ..

# Download the fuzz target and its data dependencies:
curl -O 
curl -O .key
curl -O .pem

# Build OpenSSL fuzz target for ClusterFuzz ($CXX points to clang++ binary):
$CXX -g handshake-fuzzer -fsanitize=address,fuzzer openssl-1.0.1f/libssl.a \
  openssl-1.0.1f/libcrypto.a -std=c++17 -Iopenssl-1.0.1f/include/ -lstdc++fs   \
  -ldl -lstdc++ -o handshake-fuzzer

zip openssl-fuzzer-build.zip handshake-fuzzer server.key server.pem

之后回到平台（9000端口那个），来到Jobs，看到ADD EW JOB表单

依次填写：

代码语言：javascript代码运行次数：0运行复制

“libfuzzer_asan_linux_openssl” for the “ame”.
“LIUX” for the “Platform”.
“libfuzzer” and “engine_asan” for the “Templates”.
CORPUS_PRUE = True for the “Environment String”.

其中CORPUS_PRUE = True是开启语料库修剪

最后将我们openssl-fuzzer-build.zip选择文件，再add即可

下面这个是之前忘记装go，导致模板没有导入

代码语言：javascript代码运行次数：0运行复制

但是报错，Invalid template name

不过好像确实我这个搭建完一个template也没有啊，不填template又上传不了

不过看页面确实没有template，我在源码中到了设置template的代码`src/local/butler/scripts/setup.py`

可能是初始化失败了，导致没有template，所以我手动添加了libfuzzer和engine_asan

![](.png)

![](.png)

到后面是上传压缩包失败，抓包好像是请求是发给localhost的gcs的，但是服务器在远程，那当然失败了

所以在服务器那边装个图形界面，终于ko了

还有一个解决方案，就是修改代码，让gcs监听0.0.0.0，请求的url也相应修改

上传成功后，就可以来到Fuzzer页面，选择libfuzzer那行的edit，将我们新建的jobs勾选上，之后提交即可

可以看到libfuzzer这里也有updated

查看bot这里，可以看到my-bot2先拿到了任务

从log也可以看到

代码语言：javascript代码运行次数：0运行复制

$ tail ./my-bot2/clusterfuzz/bot/logs/bot.log
2020-05-26 11:16:22,89 - run_bot - IFO - Completed running platform initialization scripts.
2020-05-26 11:16:2,509 - run_bot - IFO - Executing command 'fuzz libFuzzer libfuzzer_asan_linux_openssl'
2020-05-26 11:16:28,551 - run_bot - IFO - Setting up fuzzer and data bundles.
2020-05-26 11:16:29,779 - run_bot - IFO - Retrieving custom binary build r1.
2020-05-26 11:17:47,56 - run_bot - IFO - Unpacked /.
2020-05-26 11:17:47,578 - run_bot - IFO - Picked fuzz target handshake-fuzzer for fuzzing.
2020-05-26 11:17:47,579 - run_bot - IFO - Retrieved custom binary build r1.
2020-05-26 11:17:47,579 - run_bot - IFO - Setup application path.
2020-05-26 11:17:48,461 - run_bot - IFO - Checking for bad build.
2020-05-26 11:17:49,769 - run_bot - IFO - Recorded use of fuzz target libFuzzer_handshake-fuzzer.

但是报错了,ERROR - libFuzzer: engine encountered an error (target=handshake-fuzzer).

代码语言：javascript代码运行次数：0运行复制

$ tail bot.log
2020-05-26 11:6:19,70 - run_bot - IFO - Strategy pool was generated according to default parameters. Chosen strategies: value_profile, corpus_mutati_ml_rnn, corpus_subset
2020-05-26 11:6:19,58 - run_bot - IFO - Corpus is empty. Skip generation.
2020-05-26 11:6:19,822 - run_bot - ERROR - libFuzzer: engine encountered an error (target=handshake-fuzzer).
oneType: one
2020-05-26 11:6:19,824 - run_bot - IFO - Skipped corpus merge since no new units added by fuzzing.
2020-05-26 11:6:19,824 - run_bot - IFO - Extracting and analyzing recommended dictionary for handshake-fuzzer.
2020-05-26 11:6:19,825 - run_bot - IFO - o recommended dictionary in output from handshake-fuzzer.
2020-05-26 11:6:19,825 - run_bot - IFO - Used strategies.
2020-05-26 11:6:20,287 - run_bot - IFO - Uploaded file to logs bucket.
2020-05-26 11:6:20,288 - run_bot - IFO - Uploaded file to logs bucket.

后来又分配给第一个bot了

一直没结果，我就换了一个简单的程序，命名项目为libfuzzer_asan_my_project

代码语言：javascript代码运行次数：0运行复制

#include <stdint.h>
#include <stdio.h>
extern "C" int LLVMFuzzerTestOneInput(ct uint8_t *Data, size_t Size) {
  uint8_t tmp = Data[10];
  return 0;  // on-zero return values are reserved for future use.
}

提交后，不一会就有minimize任务了

这个简单程序的结果就出来了

点进去可以查看更多信息

参考

/

本文参与 腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。 原始发表：2020-05-18，如有侵权请联系 cloudcommunity@tencent 删除前往查看localroot程序服务器python