EyeTrans: Merging Human and Machine Attention for Neural Code Summarization
Neural code summarization leverages deep learning models to automatically generate brief natural language summaries of code snippets. The development of Transformer models has led to extensive use of attention during model design. While existing work has primarily and almost exclusively focused on static properties of source code and related structural representations like the Abstract Syntax Tree (AST), few studies have considered human attention — that is, where programmers focus while examining and comprehending code. In this paper, we develop a method for incorporating human attention into machine attention to enhance neural code summarization. To facilitate this incorporation and vindicate this hypothesis, we introduce EyeTrans, which consists of three steps: (1) we conduct an extensive eye-tracking human study to collect and pre-analyze data for model training, (2) we devise a data-centric approach to integrate human attention with machine attention in the Transformer architecture, and (3) we conduct comprehensive experiments on two code summarization tasks to demonstrate the effectiveness of incorporating human attention into Transformers. Integrating human attention leads to an improvement of up to 29.91% in Functional Summarization and up to 6.39% in General Code Summarization performance, demonstrating the substantial benefits of this combination. We further explore performance in terms of robustness and efficiency by creating challenging summarization scenarios in which EyeTrans exhibits interesting properties. We also visualize the attention map to depict the simplifying effect of machine attention in the Transformer by incorporating human attention. This work has the potential to propel AI research in software engineering by introducing more human-centered approaches and data.
Fri 19 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
15:30 - 16:00 | |||
15:30 30mPoster | Predicting Failures of Autoscaling Distributed Applications Posters Giovanni Denaro University of Milano - Bicocca, Noura El Moussa USI Università della Svizzera Italiana & SIT Schaffhausen Institute of Technology, Rahim Heydarov USI Università della Svizzera Italiana, Francesco Lomio SIT Schaffhausen Institute of Technology, Mauro Pezze USI Università della Svizzera Italiana & SIT Schaffhausen Institute of Technology, Ketai Qiu USI Università della Svizzera Italiana | ||
15:30 30mPoster | On the Contents and Utility of IoT Cybersecurity Guidelines Posters Jesse Chen University of Arizona, Dharun Anandayuvaraj Purdue University, James C. Davis Purdue University, Sazzadur Rahaman University of Arizona | ||
15:30 30mPoster | Demystifying Invariant Effectiveness for Securing Smart Contracts Posters Zhiyang Chen University of Toronto, Ye Liu Nanyang Technological University, Sidi Mohamed Beillahi University of Toronto, Yi Li Nanyang Technological University, Fan Long University of Toronto | ||
15:30 30mPoster | Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillation Posters | ||
15:30 30mPoster | Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We? Posters Kaixuan Li East China Normal University, Yue Xue Metatrust Labs, Sen Chen Tianjin University, Han Liu East China Normal University, Kairan Sun Nanyang Technological University, Ming Hu Singapore Management University, Haijun Wang Xi'an Jiaotong University, Yang Liu Nanyang Technological University, Yixiang Chen East China Normal University | ||
15:30 30mPoster | Predicting Code Comprehension: A Novel Approach to Align Human Gaze with Code Using Deep Neural Networks Posters Tarek Alakmeh University of Zurich, David Reich University of Potsdam, Lena Jäger University of Zurich, Thomas Fritz University of Zurich | ||
15:30 30mPoster | Decomposing Software Verification Using Distributed Summary Synthesis Posters DOI Pre-print | ||
15:30 30mPoster | EyeTrans: Merging Human and Machine Attention for Neural Code Summarization Posters Yifan Zhang Vanderbilt University, Jiliang Li Vanderbilt University, Zachary Karas Vanderbilt University, Aakash Bansal University of Notre Dame, Toby Jia-Jun Li University of Notre Dame, Collin McMillan University of Notre Dame, Kevin Leach Vanderbilt University, Yu Huang Vanderbilt University | ||
15:30 30mPoster | Mining Action Rules for Defect Reduction Planning Posters Khouloud Oueslati Polytechnique Montréal, Canada, Gabriel Laberge Polytechnique Montréal, Canada, Maxime Lamothe Polytechnique Montreal, Foutse Khomh Polytechnique Montréal | ||
15:30 30mPoster | How does Simulation-based Testing for Self-driving Cars match Human Perception? Posters Christian Birchler Zurich University of Applied Sciences & University of Bern, Tanzil Kombarabettu Mohammed University of Zurich, Pooja Rani University of Zurich, Teodora Nechita Zurich University of Applied Sciences, Timo Kehrer University of Bern, Sebastiano Panichella Zurich University of Applied Sciences | ||
15:30 30mPoster | Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice Posters Ranim Khojah Chalmers | University of Gothenburg, Mazen Mohamad Chalmers | RISE - Research Institutes of Sweden, Philipp Leitner Chalmers | University of Gothenburg, Francisco Gomes de Oliveira Neto Chalmers | University of Gothenburg |
This room is conjoined with the Foyer to provide additional space for the coffee break, and hold poster presentations throughout the event.