Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user’s personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.
Fri 19 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
15:30 - 16:00 | |||
15:30 30mPoster | Predicting Failures of Autoscaling Distributed Applications Posters Giovanni Denaro University of Milano - Bicocca, Noura El Moussa USI Università della Svizzera Italiana & SIT Schaffhausen Institute of Technology, Rahim Heydarov USI Università della Svizzera Italiana, Francesco Lomio SIT Schaffhausen Institute of Technology, Mauro Pezze USI Università della Svizzera Italiana & SIT Schaffhausen Institute of Technology, Ketai Qiu USI Università della Svizzera Italiana | ||
15:30 30mPoster | On the Contents and Utility of IoT Cybersecurity Guidelines Posters Jesse Chen University of Arizona, Dharun Anandayuvaraj Purdue University, James C. Davis Purdue University, Sazzadur Rahaman University of Arizona | ||
15:30 30mPoster | Demystifying Invariant Effectiveness for Securing Smart Contracts Posters Zhiyang Chen University of Toronto, Ye Liu Nanyang Technological University, Sidi Mohamed Beillahi University of Toronto, Yi Li Nanyang Technological University, Fan Long University of Toronto | ||
15:30 30mPoster | Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillation Posters | ||
15:30 30mPoster | Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We? Posters Kaixuan Li East China Normal University, Yue Xue Metatrust Labs, Sen Chen Tianjin University, Han Liu East China Normal University, Kairan Sun Nanyang Technological University, Ming Hu Singapore Management University, Haijun Wang Xi'an Jiaotong University, Yang Liu Nanyang Technological University, Yixiang Chen East China Normal University | ||
15:30 30mPoster | Predicting Code Comprehension: A Novel Approach to Align Human Gaze with Code Using Deep Neural Networks Posters Tarek Alakmeh University of Zurich, David Reich University of Potsdam, Lena Jäger University of Zurich, Thomas Fritz University of Zurich | ||
15:30 30mPoster | Decomposing Software Verification Using Distributed Summary Synthesis Posters DOI Pre-print | ||
15:30 30mPoster | EyeTrans: Merging Human and Machine Attention for Neural Code Summarization Posters Yifan Zhang Vanderbilt University, Jiliang Li Vanderbilt University, Zachary Karas Vanderbilt University, Aakash Bansal University of Notre Dame, Toby Jia-Jun Li University of Notre Dame, Collin McMillan University of Notre Dame, Kevin Leach Vanderbilt University, Yu Huang Vanderbilt University | ||
15:30 30mPoster | Mining Action Rules for Defect Reduction Planning Posters Khouloud Oueslati Polytechnique Montréal, Canada, Gabriel Laberge Polytechnique Montréal, Canada, Maxime Lamothe Polytechnique Montreal, Foutse Khomh Polytechnique Montréal | ||
15:30 30mPoster | How does Simulation-based Testing for Self-driving Cars match Human Perception? Posters Christian Birchler Zurich University of Applied Sciences & University of Bern, Tanzil Kombarabettu Mohammed University of Zurich, Pooja Rani University of Zurich, Teodora Nechita Zurich University of Applied Sciences, Timo Kehrer University of Bern, Sebastiano Panichella Zurich University of Applied Sciences | ||
15:30 30mPoster | Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice Posters Ranim Khojah Chalmers | University of Gothenburg, Mazen Mohamad Chalmers | RISE - Research Institutes of Sweden, Philipp Leitner Chalmers | University of Gothenburg, Francisco Gomes de Oliveira Neto Chalmers | University of Gothenburg |
This room is conjoined with the Foyer to provide additional space for the coffee break, and hold poster presentations throughout the event.