Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example
Because of the naturalness of software, developers often repeat the same code changes within a project or across different projects. We call these “code change patterns” (CPATs). Automating CPATs is crucial to expedite the software development process. While current Transformation by Example (TBE) techniques can automate CPATs, they are limited by the quality and quantity of the provided input examples. Thus, they miss transforming code variations that do not have the exact syntax, data-, or control-flow of the provided input examples, despite being semantically similar. Large Language Models (LLMs) are pre-trained on extensive datasets of source code. If we can harness LLMs’ creativity to produce semantically equivalent, yet previously unseen variants of the original CPAT, we can significantly increase the effectiveness of TBE systems.
In this paper, we discover best practices for harnessing LLMs to generate code variants that meet three criteria: correctness (semantic equivalence to the original CPAT), usefulness (absence of hallucinations), and applicability (aligning with the primary intent of the original CPAT). We instantiate these practices into our tool PyCraft, which synergistically combines static code analysis, dynamic analysis, and LLM capabilities. By employing chain-of-thought reasoning, PyCraft generates both variations of input examples and comprehensive test cases that can identify correct variations with an F-measure of 96.6%. Our algorithm uses a fixed-point iteration to create relevant variations and expands the original input examples by a factor of 44x. Using these richly generated examples, we inferred transformation rules and then automated these changes, resulting in an increase of up to 39x, with an average increase of 14x in target codes compared to a previous state-of-the-art tool that relies solely on static analysis. We submitted patches generated by PyCraft to a range of projects, notably esteemed ones like microsoft/DeepSpeed and IBM/inFairness. Their developers accepted and merged 83% the 86 CPAT instances submitted through 44 pull requests. This confirms the usefulness of these changes.
Thu 18 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
10:30 - 11:00 | |||
10:30 30mPoster | DyPyBench: A Benchmark of Executable Python Software Posters Islem BOUZENIA University of Stuttgart, Bajaj Piyush Krishan University of Stuttgart, Michael Pradel University of Stuttgart | ||
10:30 30mPoster | Shadows in the Interface: A Comprehensive Study on Dark Patterns Posters Liming Nie Nanyang Technological University, Yangyang Zhao Zhejiang Sci-Tech University, Chenglin Li Zhejiang Sci-Tech University, Xuqiong Luo Changsha University of Science and Technology, Yang Liu Nanyang Technological University | ||
10:30 30mPoster | Do Large Language Models Recognize Python Identifier Swaps in their Generated Code? Posters DOI Pre-print Media Attached File Attached | ||
10:30 30mPoster | Understanding Developers' Discussions and Perceptions on Non-Functional Requirements: The Case of the Spring Ecosystem Posters Anderson Oliveira Pontifical Catholic University of Rio de Janeiro (PUC-Rio), João Lucas Correia Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Wesley Assunção North Carolina State University, Juliana Alves Pereira Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rafael de Mello Federal University of Rio de Janeiro (UFRJ), Daniel Coutinho Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Caio Barbosa Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Paulo Vítor C. F. Libório Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Alessandro Garcia Pontifical Catholic University of Rio de Janeiro (PUC-Rio) | ||
10:30 30mPoster | ProveNFix: Temporal Property guided Program Repair Posters Yahui Song National University of Singapore, Xiang Gao Beihang University, Wenhua Li National University of Singapore, Wei-Ngan Chin National University of Singapore, Abhik Roychoudhury National University of Singapore | ||
10:30 30mPoster | PBE-based Selective Abstraction and Refinement for Efficient Property Falsification of Embedded Software Posters | ||
10:30 30mPoster | A Transferability Study of Interpolation-Based Hardware Model Checking to Software Verification Posters DOI Media Attached | ||
10:30 30mPoster | Evaluating and Improving ChatGPT for Unit Test Generation Posters Zhiqiang Yuan Fudan University, Mingwei Liu Fudan University, Shiji Ding Fudan University, Kaixin Wang Fudan University, Yixuan Chen Yale University, Xin Peng Fudan University, Yiling Lou Fudan University | ||
10:30 30mPoster | Testing AI Systems Leveraging Graph Perturbation Posters Zhaorui Yang University of California, Riverside, Haichao Zhu Tencent America, Qian Zhang University of California, Riverside | ||
10:30 30mPoster | Predictive Program Slicing via Execution Knowledge-Guided Dynamic Dependence Learning Posters Aashish Yadavally University of Texas at Dallas, Yi Li University of Texas at Dallas, Tien N. Nguyen University of Texas at Dallas | ||
10:30 30mPoster | Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example Posters Malinda Dilhara University of Colorado Boulder, Abhiram Bellur University of Colorado Boulder, Timofey Bryksin JetBrains Research, Danny Dig University of Colorado Boulder, JetBrains Research | ||
10:30 30mPoster | A Deep Dive into Large Language Models for Bug Fixing Posters Soneya Binta Hossain University of Virginia, Nan Jiang Purdue University, Qiang Zhou Amazon Web Services, Xiaopeng LI Amazon Web Services, Wen-Hao Chiang Amazon Web Services, Yingjun Lyu Amazon Web Services, Hoan Nguyen Amazon Web Services, Omer Tripp Amazon Web Services | ||
10:30 30mPoster | A Quantitative and Qualitative Evaluation of LLM-based Explainable Fault Localization Posters Sungmin Kang Korea Advanced Institute of Science and Technology, Gabin An Korea Advanced Institute of Science and Technology, Shin Yoo Korea Advanced Institute of Science and Technology | ||
10:30 30mPoster | IRCoCo: Immediate Rewards-Guided Deep Reinforcement Learning for Code Completion Posters Bolun Li Shandong Normal University, Zhihong Sun Shandong Normal University, Tao Huang Shandong Normal University, Hongyu Zhang Chongqing University, Yao Wan Huazhong University of Science and Technology, Chen Lyu Shandong Normal University, Ge Li Peking University, Zhi Jin Peking University | ||
This room is conjoined with the Foyer to provide additional space for the coffee break, and hold poster presentations throughout the event.