Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user’s personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.
Fri 19 JulDisplayed time zone: Brasilia, Distrito Federal, Brazil change
11:00 - 12:30 | Human Aspects 3Research Papers / Industry Papers at Mandacaru Chair(s): Eduardo Santana de Almeida Federal University of Bahia | ||
11:00 18mTalk | Exploring Hybrid Work Realities: A Case Study with Software Professionals From Underrepresented Groups Industry Papers Ronnie de Souza Santos University of Calgary, Cleyton Magalhaes Universidade Federal Rural de Pernambuco, Robson T. de Souza Santos UNINASSAU, Jorge Correia-Neto Universidade Federal Rural de Pernambuco | ||
11:18 18mTalk | Rocks Coding, Not Development–A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks Research Papers Wei Wang Beijing University of Posts and Telecommunications, Huilong Ning Beijing University of Posts and Telecommunications, Gaowei Zhang Beijing University of Posts and Telecommunications, Libo Liu School of Computing and Information Systems, University of Melbourne, Yi Wang Beijing University of Posts and Telecommunications DOI Pre-print | ||
11:36 18mTalk | Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice Research Papers Ranim Khojah Chalmers | University of Gothenburg, Mazen Mohamad Chalmers | RISE - Research Institutes of Sweden, Philipp Leitner Chalmers | University of Gothenburg, Francisco Gomes de Oliveira Neto Chalmers | University of Gothenburg Pre-print | ||
11:54 18mTalk | How to Gain Commit Rights in Modern Top Open Source Communities? Research Papers Xin Tan Beihang University, Yan Gong Beihang University, Geyu Huang Beihang University, Haohua Wu Beihang University, Li Zhang Beihang University DOI Pre-print | ||
12:12 18mTalk | How does Simulation-based Testing for Self-driving Cars match Human Perception? Research Papers Christian Birchler Zurich University of Applied Sciences & University of Bern, Tanzil Kombarabettu Mohammed University of Zurich, Pooja Rani University of Zurich, Teodora Nechita Zurich University of Applied Sciences, Timo Kehrer University of Bern, Sebastiano Panichella Zurich University of Applied Sciences |