Evaluating Directed Fuzzers: Are We Heading in the Right Direction?
Directed fuzzing recently has gained significant attention due to its ability to reconstruct proof-of-concept (PoC) test cases for target code such as buggy lines or functions. Surprisingly, however, there has been no in-depth study on the way to properly evaluate directed fuzzers despite much progress in the field. In this paper, we present a first systematic study on the evaluation of directed fuzzers. In particular, we analyze common pitfalls in evaluating directed fuzzers and empirically confirm that different choices in each step of the evaluation process can significantly impact the results. For example, we find that the choice of the crash triage method can substantially affect the performance of a directed fuzzer, while the majority of the papers we studied do not fully disclose their crash triage scripts. We argue that disclosing the whole valuation process is essential for reproducing research and facilitating future work in the field of directed fuzzing. In addition, our study reveals that several common evaluation practices in the current directed fuzzing literature can mislead the overall assessments. Thus, we identify such mistakes in previous papers and propose guidelines for evaluating directed fuzzers.
Thu 18 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
16:00 - 18:00 | FuzzingDemonstrations / Journal First / Ideas, Visions and Reflections / Research Papers / Industry Papers at Sapoti Chair(s): Maxime Lamothe Polytechnique Montreal, Montreal, Canada | ||
16:00 18mTalk | Dodrio: Parallelizing Taint Analysis Based Fuzzing via Redundancy-Free Scheduling Industry Papers Jie Liang , Mingzhe Wang Tsinghua University, Chijin Zhou Tsinghua University, Zhiyong Wu Tsinghua University, China, Jianzhong Liu ShanghaiTech University, Yu Jiang Tsinghua University | ||
16:18 18mTalk | Evolutionary Generative Fuzzing for Differential Testing of the Kotlin Compiler Industry Papers Călin Georgescu Delft University of Technology, Mitchell Olsthoorn Delft University of Technology, Pouria Derakhshanfar JetBrains Research, Marat Akhin JetBrains Research, Annibale Panichella Delft University of Technology | ||
16:36 18mTalk | Evaluating Directed Fuzzers: Are We Heading in the Right Direction? Research Papers Tae Eun Kim KAIST, Jaeseung Choi Sogang University, Seongjae Im KAIST, Kihong Heo KAIST, Sang Kil Cha KAIST Pre-print Media Attached | ||
16:54 9mTalk | When Fuzzing Meets LLMs: Challenges and Opportunities Ideas, Visions and Reflections Yu Jiang Tsinghua University, Jie Liang , Fuchen Ma Tsinghua University, Yuanliang Chen Tsinghua University, Chijin Zhou Tsinghua University, Yuheng Shen Tsinghua University, Zhiyong Wu Tsinghua University, China, Jingzhou Fu Tsinghua University, Mingzhe Wang Tsinghua University, Shanshan Li National University of Defense Technology, Quan Zhang Tsinghua University Pre-print | ||
17:03 9mTalk | Look Ma, No Input Samples! Mining Input Grammars from Code with Symbolic Parsing Ideas, Visions and Reflections Leon Bettscheider CISPA Helmholtz Center for Information Security, Andreas Zeller CISPA Helmholtz Center for Information Security Link to publication DOI | ||
17:12 9mTalk | VinJ: An Automated Tool for Large-Scale Software Vulnerability Data Generation Demonstrations Yu Nong Washington State University, Haoran Yang Washington State University, Feng Chen University of Texas at Dallas, Haipeng Cai Washington State University DOI Pre-print Media Attached | ||
17:30 18mTalk | The Human Side of Fuzzing: Challenges Faced by Developers During Fuzzing Activities Journal First Olivier Nourry Kyushu University, Yutaro Kashiwa Nara Institute of Science and Technology, Bin Lin Radboud University, Gabriele Bavota Software Institute @ Università della Svizzera Italiana, Michele Lanza Software Institute - USI, Lugano, Yasutaka Kamei Kyushu University | ||