Patient-reported outcome (PRO) refers to “any report of the status of a patient's health condition that comes directly from the patient without interpretation of the patient's response by a clinician or anyone else.” [1]. It objectively reflects the patient's subjective perception of therapeutic interventions and finds widespread application in various clinical scenarios such as quality-of-life monitoring [2] and clinical care [3]. In the early stages of research, PRO information was collected through paper-based forms. However, this method faced challenges related to data consistency and real-time interaction [4]. With the proliferation of personal computers, smartphones, and other electronic interactive media, the electronic Patient-reported Outcome (ePRO) system, utilizing electronic forms for interaction with patients on mobile platforms, effectively overcame previous obstacles [5], [6].

Nevertheless, the overuse of electronic forms has led to survey fatigue [7] and the unidirectional input of information lacks sufficient feedback to maintain patient compliance with ePRO systems. Additionally, some PRO items involving privacy concerns may elicit resistance or confusion from patients, further reducing compliance in the absence of clarification. For instance, in the Breast Cancer Body Image and Quality of Life Questionnaire (BIBCQ) [8], the item “I am satisfied with the shape of my buttocks” may be offensive to patients and raise questions about its relevance to breast cancer.

In recent years, the emergence of low-code development platforms for chatbots, such as Dialogueflow, has made it possible to build rule-based chatbot ePRO system. By manually defining rules and knowledge bases, chatbots can assist patients in obtaining additional information during the question-and-answer process, providing encouragement and feedback. Some studies suggest that chatbots can decrease experimenter demand effects [9], encourage self-disclosure [10] and increase response quality with open answers [11]. Therefore, rule-based chatbot ePRO systems [12], [13] have become a new focus in the field of research. However, rule-based chatbots require the manual construction of a sufficiently large rule system and knowledge base tailored to specific objectives to perform intelligent question-and-answer processes. The upfront human cost further impedes the widespread adoption of this method.

The emergence of Large Language Models (LLMs) has introduced new generative solutions for chatbot-based ePRO systems. The substantial increase in network scale and corpus knowledge enables LLMs to accurately comprehend complex natural language texts and provide responses on medical datasets that approximate those of professional doctors [14], [15]. To collect ePRO form information, LLMs proactively engage in conversations with patients, facilitating seamless data collection during the dialogue. Importantly, this process eliminates the need to construct intricate rules and knowledge bases tailored to specific clinical scenarios, significantly reducing the upfront development costs of chatbots.

To our knowledge, there are currently no published articles on the development of ePRO systems based on LLMs. Related research includes the work of Jing Wei et al. [16] in 2023, where they attempted to use prompt engineering based on GPT-3 to collect user self-reported information. The study primarily analyzed the impact of different prompts on the language model's generation of questions and summarized potential errors that may arise when generating self-report questions with GPT-3, exploring the design of prompt engineering. However, Jing Wei et al.'s work was limited to 4–5 self-report questions, lacked experimentation with large language models, and did not consider the interaction between the dialogue system and a database. Therefore, there is a lack of explicit guidance for constructing LLM-based ePRO systems for longer PRO forms (>10 items).

Therefore, there is currently a lack of exploration and feasibility validation of developing ePRO systems based on LLM. Thus, this study utilized prompt engineering to drive ChatGPT (GPT-3.5-turbo) [17] combined with offline knowledge distillation algorithms to construct the first ePRO system based on LLM (Chat-ePRO), and designed a pilot study to preliminarily apply Chat-ePRO at the Sir Run Run Shaw Hospital. We proposed and attempted to validate two hypotheses:

Hypothesis 1 (H1): The algorithms of the Chat-ePRO system can drive LLM to complete ePRO data collection tasks.

Hypothesis 2 (H2): Compared to ePRO systems based on electronic forms and rule-based chatbot, Chat-ePRO can improve patient compliance with the system in clinical applications.