Fri 19 Jul 2024 11:36 - 11:54 at Mandacaru - Human Aspects 3 Chair(s): Eduardo Santana de Almeida

Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that, rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how (i) purpose of the interaction, (ii) internal factors (e.g., the user’s personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.

Fri 19 Jul

Displayed time zone: Brasilia, Distrito Federal, Brazil change

11:00 - 12:30
Human Aspects 3Research Papers / Industry Papers at Mandacaru
Chair(s): Eduardo Santana de Almeida Federal University of Bahia
11:00
18m
Talk
Exploring Hybrid Work Realities: A Case Study with Software Professionals From Underrepresented Groups
Industry Papers
Ronnie de Souza Santos University of Calgary, Cleyton Magalhaes Universidade Federal Rural de Pernambuco, Robson T. de Souza Santos UNINASSAU, Jorge Correia-Neto Universidade Federal Rural de Pernambuco
11:18
18m
Talk
Rocks Coding, Not Development–A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks
Research Papers
Wei Wang Beijing University of Posts and Telecommunications, Huilong Ning Beijing University of Posts and Telecommunications, Gaowei Zhang Beijing University of Posts and Telecommunications, Libo Liu School of Computing and Information Systems, University of Melbourne, Yi Wang Beijing University of Posts and Telecommunications
DOI Pre-print
11:36
18m
Talk
Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Research Papers
Ranim Khojah Chalmers | University of Gothenburg, Mazen Mohamad Chalmers | RISE - Research Institutes of Sweden, Philipp Leitner Chalmers | University of Gothenburg, Francisco Gomes de Oliveira Neto Chalmers | University of Gothenburg
Pre-print
11:54
18m
Talk
How to Gain Commit Rights in Modern Top Open Source Communities?
Research Papers
Xin Tan Beihang University, Yan Gong Beihang University, Geyu Huang Beihang University, Haohua Wu Beihang University, Li Zhang Beihang University
DOI Pre-print
12:12
18m
Talk
How does Simulation-based Testing for Self-driving Cars match Human Perception?
Research Papers
Christian Birchler Zurich University of Applied Sciences & University of Bern, Tanzil Kombarabettu Mohammed University of Zurich, Pooja Rani University of Zurich, Teodora Nechita Zurich University of Applied Sciences, Timo Kehrer University of Bern, Sebastiano Panichella Zurich University of Applied Sciences