This program is tentative and subject to change.

Fri 19 Jul 2024 11:54 - 12:12 at Baobá 2 - Testing 4

Abstractive summarization (AS) systems, which aim to generate a text for summarizing crucial information of the original document, have been widely adopted in recent years. Unfortunately, factually unreliable summaries may still occur, leading to unexpected misunderstanding and distortion of information. This calls for methods that can properly evaluate the quality of AS systems. Yet, the existing reference-based evaluation approach for AS relies on reference summaries as well as the automatic evaluation metrics (e.g., ROUGE). Therefore, the reference-based evaluation approach is highly restricted by the availability and quality of reference summaries as well as the capability of existing automatic evaluation metrics. In this study, we propose MTAS, a novel metamorphic testing based approach for evaluating AS in a reference-free way. Our two major contributions are (i) five metamorphic relations towards AS, which involve semantic-preserving and focus-preserving transformations at the document level, and (ii) a summary consistency evaluation metric SCY, which measures the alignment between a pair of summaries by incorporating both the semantic and factual consistency. Our experimental results show that the proposed metric SCY has a significantly higher correlation with human judgment as compared to a set of existing metrics. It is also demonstrated that MTAS can break the dependence on reference summaries, and it successfully reports a large number of summary inconsistencies, revealing various actual ummarization issues on state-of-the-art AS systems.

This program is tentative and subject to change.

Fri 19 Jul

Displayed time zone: Brasilia, Distrito Federal, Brazil change

11:00 - 12:30
11:00
18m
Talk
Partial Solution Based Constraint Solving Cache in Symbolic Execution
Research Papers
Ziqi Shuai School of Computer, National University of Defense Technology, China, Zhenbang Chen School of Computer, National University of Defense Technology, China, Kelin Ma School of Computer, National University of Defense Technology, China, Kunlin Liu School of Computer, National University of Defense Technology, China, Yufeng Zhang Hunan University, Jun Sun School of Information Systems, Singapore Management University, Singapore, Ji Wang School of Computer, National University of Defense Technology, China
Pre-print
11:18
18m
Talk
Java JIT Testing with Template Extraction
Research Papers
Zhiqiang Zang The University of Texas at Austin, Fu-Yao Yu The University of Texas at Austin, Aditya Thimmaiah The University of Texas at Austin, August Shi The University of Texas at Austin, Milos Gligoric The University of Texas at Austin
11:36
18m
Talk
Natural Symbolic Execution-based Testing for Big Data Analytics
Research Papers
Yaoxuan Wu UCLA, Ahmad Humayun Virginia Tech, Muhammad Ali Gulzar Virginia Tech, Miryung Kim UCLA and Amazon Web Services
11:54
18m
Talk
MTAS: A Reference-Free Approach for Evaluating Abstractive Summarization Systems
Research Papers
Xiaoyan Zhu Zhejiang Sci-Tech University, Mingyue Jiang Zhejiang Sci-Tech University, Xiao-Yi Zhang University of Science and Technology Beijing, Liming Nie Nanyang Technological University, Zuohua Ding Zhejiang Sci-Tech University
12:12
18m
Talk
Property-based Testing for Validating User Privacy-Related Functionalities in Social Media Apps
Industry Papers
Jingling Sun University of Electronic Science and Technology of China, Ting Su East China Normal University, Jun Sun School of Information Systems, Singapore Management University, Singapore, Jianwen Li East China Normal University, China, Mengfei Wang ByteDance, Geguang Pu East China Normal University, China