記住我
Cigarette smoking accounts for 8 million premature deaths and 25% of all cancer deaths annually [,]. Despite advancements in government policies, antismoking campaigns, and shifting societal norms, existing smoking cessation interventions continue to have limited treatment engagement and cessation rates [-]. While this is a problem for the general population of people who smoke, the issue is particularly pronounced in communities considered marginalized, synonymous with groups considered vulnerable or disadvantaged, which are segments of society facing systemic disadvantages and barriers in accessing resources and opportunities. Populations considered marginalized, marked by factors such as racial or ethnic minority status, sexual or gender identity differences, low education and income levels, higher unemployment rates, or an increased prevalence of mental illness, encounter discrimination, social exclusion, and limited influence in decision-making processes.
Challenges in treatment engagement and cessation efficacy across all communities of people who smoke are compounded by a scarcity of trained clinicians and significant barriers, including cost and lack of insurance, hindering access to existing clinician-delivered interventions [-]. Given that 1.3 billion people in the world smoke cigarettes, with 70% of them wanting to quit, it would be impractical to have enough trained clinicians to help people quit smoking [,]. Indeed, only 5% of cessation attempts are aided by a health professional []. Consequently, there is an enormous need for high-impact, cost-effective, population-level interventions for smoking cessation.
A well-documented finding from research on clinician-delivered treatments has emphasized the significance of therapeutic conversations as powerful drivers of patient engagement [-]. Therapeutic conversations, which form a social-emotional bond with the user, have predicted treatment engagement and, subsequently, health outcomes across various treatments and settings [,,]. A new technology provides an opportunity to leverage engaging therapeutic conversations. Advances in machine learning, large language models (LLMs), and cloud computing are now making it possible to create and widely disseminate conversational chatbots for behavior change coaching.
Unlike the chatbots used in customer service contexts, conversational chatbots for behavior change coaching are designed to form long-term social-emotional connections with users, even as they are made aware that chatbots are merely computer software that use language to communicate with users [,]. Conversational chatbots for coaching are designed to be supportive and empathic, offering reflective listening, personalized responses, and timely advice aligned with the user’s individual needs [,]. In the context of cessation, conversational chatbots can enhance engagement through an informal therapeutic conversational style tailored to users’ unique barriers to quitting smoking []. Conversational chatbots only require a text response to operate, making them ideal for all individuals who smoke, including those with low technology literacy []. Overall, conversational chatbots offer a cost-effective communication platform, accessible at any time, and have the potential for high population-level reach, making them a valuable tool in smoking cessation interventions.
To date, research on conversational chatbots for smoking cessation is scarce. Existing literature revealed a limited number of empirical studies, often exhibiting low methodological quality []. There is a notable paucity of randomized controlled trials (RCTs) focusing on conversational chatbots for smoking cessation, and while promising results have emerged, they have been limited by low quit rates []. Several conversational chatbots for smoking cessation in the public domain include Florence (World Health Organization) [], Bella (Solutions4Health) [], and Alex AI (Alex Therapeutics) []. However, we are not aware of publications on their efficacy, with only the Florence app having reported user’s receptivity results []. Critical to creating useful and engaging conversational chatbots is following a user-centered design development process []. Similar to most chatbots, the development of the chatbots listed above lacks context for how they were designed and any user-centered design that involved conducting a needs assessment or including user feedback during the development process [,]. The few studies that have provided development details only describe early design phases, such as coding 30 quit coaching calls for prototype development, without empirical efficacy data [,,]. In sum, the literature on chatbots for smoking cessation offers only partial accounts on how they were developed or report on initial stages of development.
ObjectiveTo address these gaps, this paper describes the comprehensive 4-year, 11-step user-centered design development process for a novel quit smoking conversational chatbot named “QuitBot.” This single report aims to summarize the entirety of the QuitBot development process.
The 4 years of formative research for developing QuitBot followed an 11-step process, consistent with a user-centered design framework () [,].
The steps were as follows: (1) specifying a conceptual model to guide the QuitBot intervention targets; (2) conducting content analysis of existing smoking cessation interventions to guide the QuitBot coaching conversations; (3) conducting a needs assessment to determine what an adult seeking help in quitting smoking would need from a cessation chatbot; (4) developing the QuitBot persona, or personality of the chatbot, to shape the user’s experience of and bond with the QuitBot chatbot; (5) prototyping QuitBot’s basic content and persona; (6) developing the full functionality of the QuitBot; (7) programming the QuitBot; (8) conducting a diary study for user feedback on their interactions with QuitBot and its design and content; (9) conducting a pilot RCT to test QuitBot for smoking cessation; (10) reviewing results of the pilot RCT; and (11) adding a free-form question and answer (QnA) function, based on user feedback from pilot RCT results. The process of adding the QnA function itself involved a three-step process: (1) generating QnA pairs, (2) fine-tuning LLMs on the QnA pairs, and (3) evaluating the LLM model outputs.
Figure 1. Overview of QuitBot’s formative research process. Step 1: Specifying the Conceptual Model Guiding QuitBot for Smoking CessationThe conceptual model guiding the development of QuitBot for smoking cessation () focuses on impacting user engagement through 4 therapeutic alliance processes. The four processes are as follows: (1) bond with QuitBot, (2) agreement on smoking cessation goal, (3) agreement on tasks for achieving smoking cessation goal, and (4) perception that QuitBot understands user’s current needs [].
These working alliance processes have predicted smoking cessation [] and quit attempts [] and have mediated the impact of human therapist–delivered smoking cessation interventions []. QuitBot uses various strategies to establish a therapeutic alliance, including expressing empathy for the user’s struggles [,], engaging in social dialogue [,], using metarelational communication (ie, discuss the relationship) [], and expressing happiness while interacting with the user []. Language constructs such as inclusive pronouns [], politeness strategies [], and the use of greetings and farewells rituals [] contribute to the creation of this alliance as well. Compared to a technology that did not use these verbal behaviors, a conversational chatbot for physical activity increased these therapeutic alliance processes, which in turn was predictive of higher engagement with the chatbot [].
Agreement on smoking cessation goal starts by collaboratively setting a quit date (eg, “Have you thought about a specific day you would like to quit? Generally, I recommend about 14 days away.”). QuitBot enhances perceived understanding by promptly addressing the user’s immediate needs (eg, “You say you are tempted by friends who smoke. Here’s a tip that might help...”). In addition, self-disclosure [] is used to foster perceived understanding, generating various positive outcomes, especially when the listener responds with support and validation []. A chatbot that used self-disclosure increased the user’s perception that the chatbot understood their needs, which in turn predicted more positive mood [].
Figure 2. Conceptual model of QuitBot for smoking cessation. Step 2: Conducting Content Analysis to Guide QuitBotThe content analysis aimed to establish a natural flow of coaching conversations for QuitBot, aligned with US Clinical Practice Guidelines for smoking cessation []. In the initial phase of the content analysis, we interviewed a panel of experts, including 3 smoking cessation counselors, a smoking cessation master trainer, and a tobacco cessation scientist from our team. This panel consisted of 4 women and 1 man, with 20% (1/5) from racial and ethnic minority backgrounds. Among them, 40% (2/5) held a PhD in clinical psychology, while 60% (3/5) had master’s degrees in counseling or social work. Collectively, they had a wealth of experience ranging from 3 to 20 years, with an average of 8 (SD 4.6) years, in developing and delivering smoking cessation interventions. Deductive coding of these interviews and expert consensus iteratively lead to the formulation of common themes, domain-specific responses, and anticipated user interactions that QuitBot should address. We identified common conversation topics about smoking cessation, including triggers to smoke (ie, physical, emotional, and situational triggers), motivations to quit, and barriers to quitting. Interviews also highlighted the importance of QuitBot’s persona to be sensitive and empathetic to the user and to express that their concerns are being heard.
Guided by this expert consensus on conversation topics, the second phase was to extract the content and flow of smoking interventions as they naturally occur in actual interactions between cessation counselors and patients. To achieve this, we conducted semantic analysis of verbatim manually transcribed intervention conversation transcripts obtained from our telephone counseling intervention trial, randomly selected among those who did and did not quit smoking (R01 DA038411) []. A total of 159 call transcripts (equating to 63 h and 23 min) from 117 unique participants were randomly selected, constituting a 7.8% (159/2038) sample from each of the 5 sessions (with an average session duration of 22.9 min) of an efficacious behavioral intervention for smoking cessation with a 25% thirty-day point prevalence abstinence (PPA) rate at the 12-month follow-up []. These sessions covered various topics, including motivations to quit, triggers to smoke, barriers to quitting, setting a quit date, developing a quit plan, education and proper use of Food and Drug Administration (FDA)–approved medications for quitting smoking, coping skills for dealing with urges, enlisting social support, and strategies for avoiding external cues to smoke. Participants had a mean age of 47.4 (SD 12.7) years, with 43.6% (51/117) male participants and 21.4% (25/117) from racial and ethnic minority backgrounds.
Transcripts underwent deductive coding using a predefined codebook to identify common conversation topics related to smoking cessation, such as triggers to smoke (ie, physical, emotional, and situational triggers), motivations to quit, and barriers to quitting. Using a supervised machine learning approach, these topics formed the basis of QuitBot’s entity extraction, wherein elements of the unstructured transcript text were coded into predefined categories. Subsequently, we determined intent classifications, which involved discerning the meaning of the user’s text. Finally, we mapped out the natural conversational flow of both the chatbot and the range of verbal responses and comments that users might provide in response to the chatbot. The entity extraction, intent classifications, and conversational mapping were conducted using the LUIS conversational artificial intelligence (AI) program [].
Step 3: Needs Assessment of Adults Seeking Help in Smoking CessationAssessing the needs of adults seeking help in smoking cessation interventions shapes what the user should be able to do with a chatbot. To assess user needs, we first analyzed the results of the content analysis phase. Subsequently, we conducted interviews with 5 adults who had participated in our human-delivered smoking cessation interventions within the past year (including 2 who quit and 3 who did not quit) []. Participants had a mean age of 46.1 (SD 10.4) years, with 40% (2/5) female participants and 40% (2/5) from racial and ethnic minority backgrounds. The interviews queried participants about their personal background and smoking history, expectations for a smoking cessation program, experiences with a human cessation coach, perceptions regarding setting, keeping and changing quit dates, coping skills for urges to smoke, and attitudes toward and expectations of what a chatbot could do for helping them quit smoking. Semistructured interviews were conducted in person at the lead author’s user experience (UX) Research HABIT laboratory. The deductive thematic analysis organized the user’s responses by grouping them into themes, reviewing the themes, and then interpreting them [-]. The themes of the key user needs identified were (1) a coach who can make a personal connection, (2) on-demand help with urges, and (3) skills for preparation to quit and preventing relapse.
Step 4: Developing the QuitBot “Persona”The user’s bond with the chatbot is impacted by its persona []. On the basis of interviews with smoking cessation coaches and our master trainer, we created the persona to foster a strong bond with users. Presented to the user as a computer program (eg, “I’m a bot designed to help you live smoke free”), elements of the QuitBot persona included expressions of empathy [,], social dialogue [,], metarelational communication (ie, discuss the relationship) [], and expressing happiness to see the user []. In addition, specific language constructs, including inclusive pronouns [], politeness strategies [], and greetings and farewells rituals, were integrated to enhance the UX and promote a respectful dialogue []. Finally, we established 11 core values for the persona, serve as guiding principles for QuitBot’s behavior throughout conversations.
Step 5: Prototype Testing of QuitBot’s Basic Content and PersonaThe prototyping testing of QuitBot’s basic content and persona aimed to assess users’ initial responses to basic smoking cessation conversations between the user and the persona. Stimuli were built using botmock [,] to develop the chat dialogue, which was then integrated into Facebook Messenger (FM; Meta Platforms, Inc) using Chatfuel []. Participants had a guided initial chat conversation introducing the chatbot and program goals, querying about triggers for smoking, and setting a quit date. Subsequently, they interacted with QuitBot for a second conversation, focusing on tracking triggers to smoke. For both conversations, a UX researcher frequently paused to prompt participants to think-aloud their experiences with QuitBot. Real-time interactions between the user and QuitBot were facilitated by a UX researcher using the Chatfuel program []. To evaluate this process, 75-minute individual interviews were conducted with 8 adults interested in quitting smoking. Four were chosen because they thought a chatbot could be helpful for quitting smoking, while the remaining 4 were selected because they were unsure or skeptical that a chatbot would be helpful for quitting smoking. The mean age of the participants was 42 (SD 11.1) years, with 38% (3/8) male participants, 50% (4/8) female participants, and 12% (1/8) transgender participants. In addition, 38% (3/8) of the participants had high school education or less and 25% (2/8) reported being from racial or ethnic minority backgrounds.
Semistructured InterviewsSemistructured interviews were conducted in person at the lead author’s UX Research HABIT laboratory. A deductive thematic analysis method was used to organize user responses into themes, review those themes, and then interpret them [-]. Despite initial skepticism from half of participants (4/8, 50%) regarding the usefulness of interacting with a digital coach, the results showed a notable shift in the interest in QuitBot by the end of the interview: 100% (8/8) reported that a chatbot such as QuitBot would be valuable for helping someone quit smoking, with 88% (7/8) expressing willingness to try this chatbot for quitting. In addition, all participants (8/8, 100%) found QuitBot easy to use, noting its conversational tone as “encouraging,” “polite,” and “reassuring.” They deemed the length and speed of onboarding conversations appropriate and felt comfortable providing conversational responses. Participants expressed surprise at the “humanness” of QuitBot’s avatar, noting its informal, reassuring, accessible, and easy-to-talk-to demeanor.
When discussing whether the avatar should have a gender or a name, there was consensus among participants in favor of a female persona, with the name “Ellen” deemed appropriate (interestingly, one of the initial participants spontaneously suggested “How about a woman named ‘Ellen’?”). Later participants concurred with this choice when asked by the UX researcher.
Participants also expressed a desire for more actionable suggestions and to open and close each conversation with a specific plan of action. In response, we added a plan outlining what to anticipate, letting them know that the avatar would initiate a chat the following day and introduce a new quitting smoking skill in the subsequent conversation. Some participants wanted additional time to decide on a quit date, prompting us to include a dialogue indicating that they postponed setting a quit date until they felt ready. In addition, participants suggested visualizing their progress in quitting smoking, such as through a graph. In response, we added a progress chart displaying the number of cigarettes smoked over time. Overall, participants described feeling “captivated” by the content and expressed eagerness to learn more.
Step 6: Developing Full FunctionalityBuilding upon the prototype as the foundation, we applied insights from the earlier steps to develop a full program consisting of 42 days of 2- to 3-minute focused conversations. These conversations were distributed over several phases of treatment: a prequit phase (14 days of content), quit day (1 day of content), and postquit phase (27 days of content). There are also conversations for those not ready to quit smoking by day 14 (6 days of content) and conversations for those who have relapsed (3 days of content). The content, described in and , follows US Clinical Practice Guidelines for cessation interventions []. The program content was presented as a continuous conversation, built on user input from prior conversations. This ensures a personalized and adaptive approach based on the user’s stated motivations to quit, triggers to smoke, and number of cigarettes smoked. QuitBot was proactive and provided daily prompts to start a structured text conversation with Ellen at the user’s preferred time, such as “Hi Alex, are you free to chat?” Users also had the flexibility to reach out to Ellen at any time for on-demand help with urges, inspiration, mood, and slips.
Textbox 1. QuitBot and SmokefreeTXT (SFT): phases and corresponding content.Phase (number of days) and content of both SFT and QuitBot
Prequit day (14 days)Triggers to smoke, motivations for quitting, setting a quit date, Food and Drug Administration–approved medication information, skills to be aware of and cope with urges, and cessation progress trackingQuit day (1 day)Encouragement and smoking status check-inPostquit (28 days)Withdrawal symptoms education, slips and relapse prevention, managing mood, managing cravings, and cessation progress trackingNot ready or quit date >14 daysReviews motivations for quitting and cessation progress trackingAnytime helpSkills to cope with urges, mood, and slipsTextbox 2. Content communication.How content is communicated:SmokefreeTXT (SFT) and QuitBot
SFT: sends texts of the content, answers to daily check-ins (eg, number of cigarettes smoked today), get 1-2 text responses, answers to entering anytime help keywords (eg, “CRAVE”), and get 1 text responseQuitBot: digital coach sends the user a greeting to start a 2- to 3-minute conversation, presents content in a dialogue with the user via engagement features described in (eg, tailored responses and empathy), and answers to entering anytime help keywords (eg, “CRAVE”) initiate a dialogue
Figure 3. Representative functionalities of QuitBot include (A) determining triggers, (B) maintaining motivation, and (C) providing anytime urge help. Step 7: Programming QuitBotWe initially sought a development architecture with the flexibility to interact with QuitBot on any major consumer communication channel (eg, as a stand-alone app, FM, and Slack). Such flexibility adapts to current consumer trends in communication technology use, making QuitBot available for use on the channels with current high population-level reach. To determine which communication channel would be used for interacting with QuitBot, we conducted a web-based survey of 100 US adults who smoke, asking them which of these channels they would prefer for a chatbot: stand-alone app, WhatsApp (Meta Platforms, Inc), FM (Meta Platforms, Inc), Skype (Skype, Inc), or Slack (Slack Technologies, LLC). The majority of respondents (74/100, 74%) preferred FM, citing its familiarity, ubiquity, and ease of use. FM is an instant messaging service for online chats. At the time of the study (ie, 2019), there were >133 million FM users in the United States (1.3 billion globally) and FM hosts >300,000 chatbots, with 27% of them for health care (eg, exercise) [-]. Following these findings, we hosted QuitBot on FM.
Therefore, we custom built an architecture using the Microsoft Bot Framework that uses Microsoft Azure for the cloud computing and Microsoft Language Understanding (LUIS) platform for the natural language understanding of the QuitBot guide Ellen. The preference for natural language understanding over an if-then decision-based conversation flow was made to ensure a more natural and open-ended interaction, allowing a broad range of responses and better conveying that the user is being heard and understood. QuitBot’s LUIS allows it to understand common text shorthand. Users can respond freely or simply select from a menu of responses. If QuitBot does not understand a free response, it will say so and ask the participant to rephrase the response. QuitBot was written in the programming language of Node.js [].
Step 8: Conducting a Diary StudyWe conducted a diary study to obtain ongoing feedback on users’ interactions with QuitBot, its design, and content. In user-centered design research, a diary study of 2 weeks with 6 to 12 participants is recommended to obtain this initial feedback [,]. Accordingly, we conducted a single-arm 14-day diary study of the program with 9 adults who were smoking at least daily (all smoked ≥30 cigarettes/d), were interested in quitting smoking, and recruited from around the United States via Facebook advertisements. Four were chosen because they were skeptical about chatbots being able to help someone quit smoking, while the remaining 5 were neutral about them. Participant demographics were as follows: mean age 40.4 (SD 13.4) years, 11% (1/9) from racial and ethnic minority backgrounds, 44% (4/9) female, and 67% (6/9) had less than a bachelor’s degree.
All 9 participants completed the following: (1) the 60-minute video-based orientation focusing on how to use QuitBot and complete the daily diary entries; (2) 14 evening diary entries (15 min each) about their daily interactions with QuitBot, its design, and content; (3) on day 7, a midpoint 15-minute video call with a member of our user research team to review their impressions to date; and (4) a 60-minute video call exit interview with a member of our user research team. A PhD-level UX researcher with >20 years of experience conducted the interviews. (Example questions from the exit interview are as follows: “Which parts of the app did you find the most helpful? Why?”) Semistructured interviews were conducted in person at the lead author’s UX Research HABIT laboratory. The deductive thematic analysis organized the user’s responses by grouping them into themes, reviewing the themes, and then interpreting them [-].
The results showed that, although the focus was on usability, by day 14, three participants quit smoking and the remaining 6 participants reduced to 3 or 4 cigarettes per day. Ratings for usefulness (“Overall, how useful was the QuitBot app for helping you quit smoking?”), satisfaction (“Overall, how satisfied were you with the QuitBot?”), and likelihood of recommending QuitBot (“To what extent would you recommend QuitBot to someone who would like to quit smoking?”) were all high: 4.33, 4.67, and 4.88, respectively, on a 0 (not all) to 5 (extremely) scale. All 9 users felt highly supported by Ellen and liked her persona. They liked the skills training for coping with smoking urges and lapses. Their feedback yielded minor content edits and fixes of technical bugs. Representative functionalities of QuitBot are shown in .
Diversity and Inclusion in UX DesignThe diversity of race, gender, age, and educational characteristics of users who participated in our UX design studies influenced the design in many ways, including Ellen’s persona design (eg, men and women both preferred a female persona), Ellen’s stories of people who have quit smoking (eg, they were diverse in age, gender, race, and education), use of language (eg, fifth-grade reading level, informal, and respectful), and user interface (eg, larger response buttons and larger font size []).
Step 9: Conducting a 3-Arm Pilot RCTThe favorable feedback from the diary study led us to conduct a 3-arm parallel pilot RCT comparing QuitBot (n=200) to the SmokefreeTXT (SFT; n=149) intervention and to a QuitBot delayed access control group (n=55). Following expert recommendations for pilot RCT design [,], the feasibility outcomes were the study’s primary focus to inform the further development of QuitBot and design of a future full-scale trial of QuitBot. As this pilot RCT was the first time QuitBot was tested and no prior RCTs had been reported on any quit smoking chatbot, estimated effect sizes were unknown. Instead, the sample sizes were based on comparable sample sizes from prior pilot studies we had conducted in our laboratory [,]. Participants were recruited nationwide and were randomized to the intervention arm using randomly permuted blocks of size 2, 4, and 6, stratified by biological sex (male vs female), heaviness of smoking index score (≤4 vs >4), and percent confidence in being smoke-free in 12 months (≤70% vs >70%). The study was double-blinded, with both interventions called “QuitBot.”
Ethical ConsiderationsAll study procedures were approved by the Fred Hutch Cancer Center Institutional Review Board (8659/RG1001766). The clinical trial protocol was approved by the Fred Hutch Scientific Review Committee (FHIRB008659), and the trial was registered on ClinicalTrials.gov (NCT03585231). There were no deviations to the registered protocol. All study participants provided informed consent, and data were deidentified for privacy and confidentiality.
Eligibility Criteria for the Pilot RCTThe inclusion criteria were as follows: (1) age ≥18 years; (2) having smoked at least 1 cigarette a day for at least the past 12 months; (3) wanting to quit cigarette smoking within the next 14 days; (4) if concurrently using any other nicotine or tobacco products, wanting to quit using them within the next 14 days; (5) being interested in learning skills to quit smoking; (6) being willing to be randomly assigned to either condition; (7) residing in the United States; (8) having daily access to their own smartphone; (9) having both SMS text messaging and FM on their smartphone (criteria 8 and 9 were required to receive each interventions’ content); (10) being willing and able to read in English; and (11) not using other smoking cessation interventions. Individuals deemed ineligible to participate were directed to the smokefree.gov website and the 800-QUIT-NOW number for access to their state’s quitline resources.
SFT Comparison ConditionFor the past 20 years, mobile phone–delivered SMS text messaging interventions have been a prominent technology for delivering smoking cessation interventions [-]. Each year, SMS text messaging smoking cessation interventions are reaching >300,000 US adults who smoke and 6 million adults who smoke worldwide [,]. SFT’s 42-day program was developed by the National Cancer Institute (NCI). SFT is the most widely accessible SMS text messaging program in the United States. SFT is nonproprietary and free to the public, thereby providing maximal accessibility and replicability. Daily messages are sent about the importance of quitting smoking, setting a quit date, preparing to quit, quitting, and maintaining abstinence. Daily messages check in about quit status. Three keywords can be proactively sent by users to receive help anytime: “CRAVE” (on how to cope with urges), “MOOD” (on how to cope with moods triggering smoking), and “SLIP” (on how to cope with lapses). Participants do not need to respond to or otherwise engage with SFT messages to complete the SFT program. Refer to for sample messages.
NCI’s SFT contractor (ICF International []) provided us with the full content of SFT so that we could internally host a secured private version for research. In both SFT and QuitBot, participants receive 2 prompts per day (3 on the quit day). Comparisons between QuitBot and SFT are shown in and .
Figure 4. Sample SmokefreeTXT text message. QuitBot Delayed Access Comparison ConditionTo explore the unique impact of QuitBot on smoking cessation, considering that some participants might quit smoking without intervention, we introduced a delayed access comparison condition. In this condition, 55 participants received delayed access to QuitBot after completing the 3-month follow-up survey. The delayed access comparison condition served the ethical purpose of providing participants access to a treatment (as opposed to no treatment at all).
Outcome MeasuresOutcome data were collected through an encrypted web-based survey. Participants not completing the web-based survey were sequentially offered the survey via phone, mailed survey, and postcard. The primary feasibility outcomes were (1) sufficient accrual of the planned number of study participants, (2) balanced demographic and smoking characteristics at baseline between study arms, and (3) retention of the primary 30-day PPA smoking outcome at the 3-month follow-up. Intervention engagement was assessed based on comparing the active treatment study arms on the number of times and number of days participants interacted with their assigned intervention. All interactions with the participants’ assigned interventions were objectively logged using an internally hosted secure server. The primary smoking cessation outcome was 30-day PPA, based on compete-case analysis, and 7-day complete-case PPA was secondary.
Statistical Analysis for the Pilot RCTThe feasibility of the pilot RCT was assessed based on sufficient accrual, balanced randomization, and adequate follow-up data retention rates that did not differ between arms. Baseline characteristics were compared between the 3 study arms using ANOVA for continuous variables and Fisher exact tests for categorical variables and were summarized with the “arsenal” package in R (version 4.2.3; R Foundation for Statistical Computing) [,]. We used generalized linear models to assess differences between study arms in the number of days participants used their intervention.
We used negative binomial models, implemented with the R package “MASS” [], to compare treatment arms on total number interactions because the data were heavily right-skewed. Logistic regression models were used to test the effect of the treatment arm on binary smoking cessation outcomes. On the basis of evidence from meta-analyses of SMS text messaging trials [], all outcome models were adjusted for the 3 factors used in stratified randomization: biological sex (male vs female), heaviness of smoking index score (≤4 vs >4), and percent confidence in being smoke-free in 12 months (≤70% vs >70%). Wald tests for pairwise comparisons of each outcome between study arms were adjusted for multiple comparisons with the Holm procedure []. Statistical tests were considered significant at α<.05. Deductive thematic analysis organized participants’ comments about QuitBot by grouping them into themes, reviewing the themes, and then interpreting them [-].
On the basis of our successful methods for national recruitment [], we developed and tailored Facebook advertisements with ongoing monitoring and adjustment of recruitment yield. These efforts resulted in screening 2954 participants, with 1380 eligible, 583 consenting, and 418 randomized between September 2018 and June 2019. After the completion of study participation, 14 participants were found to be cases of fraud, duplicate participants, or in the same household as another participant, leading to a total of 404 participants included in analyses.
RandomizationThe 3 stratification conditions were balanced at baseline on all measured characteristics (all P values >.05). As shown in , participants were on average 36 years old, 70% (283/404) were women, 28.9% (116/401) reported being from racial or ethnic minority backgrounds, 52.7% (213/404) were unemployed, 83.9% (339/404) had no college degree, 71.5% (289/404) smoked more than one-half pack daily, and 59.9% (242/404) had high cigarette dependence (Fagerström Test for Cigarette Dependence scores of ≥6). The characteristics of this FM sample are very similar to those of other digital health intervention trials [,,].
Table 1. Baseline participant characteristics by study arm.CharacteristicTotal (n=404)SmokefreeTXT (n=149)Delayed (n=55)QuitBot (n=200)P valueBaseline characteristicaFTCD: Fagerström Test for Cigarette Dependence.
The 3-Month Follow-Up Rates Were HighTo maximize outcome data completion, we followed our team’s successful protocol []: 4 sequential survey modalities (first web, followed by phone, mail, and postcard). As agreed in the informed consent, participants received US $25 for submitting their responses and received an additional US $10 bonus for completing the web survey within 24 hours. The achieved outcome survey completion rate of 96% provided confidence in the follow-up survey methods. The data retention did not differ between study arms (P=.54). Given the limitations of the pilot budget, cessation data were self-reported.
Engagement and Cessation Results Were Promising for QuitBotThe number of times participants interacted with their assigned intervention was 1.3 times greater in QuitBot as compared to SFT (incidence rate ratio 1.33, 95% CI 1.04-1.70; P=.02; ). Participants used their assigned intervention 11 days longer in the QuitBot arm than in the SFT arm (point estimate 11.5, 95% CI 4.9-18.1; P=.001). QuitBot’s intervention completion results are substantial when considering that each day’s content involved a 2- to 3-minute conversation. (By contrast, SFT participants did not need to respond to or otherwise engage at all with their messages to complete their program; daily SFT text messages were sent automatically.) Participant engagement was limited by QuitBot’s inability to answer participants’ open-ended questions (see the Representative QuitBot Comments section). Therefore, cessation results are reported for all participants and for participants who completed their assigned intervention.
For all participants, the 30-day PPA rates at 3-month follow-up were 31.1% (59/190) for QuitBot versus 34.7% (50/144) for SFT (QuitBot vs SFT: odds ratio [OR] 0.81, 95% CI 0.50-1.29; P=.36; ) versus 7% (4/54) for delayed treatment (QuitBot vs delayed: OR 5.97, 95% CI 2.04-17.45; P=.002). For those who completed their assigned intervention (ie, viewed all 42 days of planned content), the 30-day, complete-case, PPA rates at 3-month follow-up were 63% (39/62) for QuitBot versus 38.5% (45/117) for SFT (QuitBot vs SFT: OR 2.58, 95% CI 1.34-4.99; P=.005). The pattern of results was highly similar for the outcome of 7-day, complete-case, PPA rates at 3-month follow-up, albeit with higher abstinence rates in each study arm.
Table 2. Comparison of QuitBot and SmokefreeTXT (SFT) interventions on 3-month engagement outcomes.Study engagement outcomeSFT (n=149), mean (SD; median)QuitBot (n=200), mean (SD; median)QuitBot vs SFTaIRR: incidence rate ratio.
bPE: point estimate.
cNot applicable.
Table 3. Comparison of QuitBot and SmokefreeTXT (SFT) interventions and delayed intervention on 3-month cessation outcomes.Study outcomeSFT (n=149), n (%)Delayeda (n=55), n (%)QuitBot (n=200), n (%)QuitBot vs SFTQuitBot vs delayedaaThree-month delay in receiving QuitBot.
bOR: odds ratio.
cNot applicable.
Representative QuitBot CommentsComments from QuitBot arm trial participants reflected a strong overall bond with the chatbot’s persona:
I loved Ellen. She was always there when I needed her.Conversely, participants were frustrated by QuitBot’s inability to respond to their specific questions about quitting smoking:
I could not ask questions and get real answers back.Our main conclusions were as follows: (1) the intervention demonstrated potential for rigorous testing based on sufficient accrual, balanced randomization, and high retention rates; (2) overall, there was a strong engagement with QuitBot; and (3) promising quit rates were observed, particularly among participants who completed the content of their assigned intervention. The effectiveness of QuitBot was evident, as quit rates in the delayed condition group were significantly lower (59/190, 31.1% vs 4/54, 7%; P=.002), indicating a net percentage point increase in smoking cessation of 24%. Therefore, it is highly unlikely that effects of QuitBot were merely due to the passage of time or baseline motivation to quit smoking (ie, few participants quit without offering intervention).
Challenges were also identified, potentially impacting participant engagement and quit rates. Specifically, QuitBot’s inability to respond to participants’ own questions about quitting smoking led to a significant level of frustration. While the participant can answer questions asked by the QuitBot (eg, “Tell me what is triggering your urge.”), the reverse was not possible: participants could not ask QuitBot their own questions. A QuitBot feature that allows participants to ask free-form questions would be needed to address this limitation.
Technical Limitations of the FM PlatformWhile FM was the preferred communication platform from our survey results, Facebook introduced changes that would limit participants’ engagement with QuitBot as well as our own access to user data: (1) Facebook made policy changes that revoked access permissions to proactively outreach (eg, to invite participant to check in or start a conversation), effectively removing our ability to proactively contact users (restricting that ability to news-related apps only); and (2) Facebook made platform changes that restricted our ability to access demographic information of users, inhibiting data collection. Facebook’s changes raised concerns about the feasibility of QuitBot’s conversational functionality and data collection. This critical limitation could be addressed by transitioning to a stand-alone smartphone app communication platform, enhancing accessibility and control for both participants and the development and research teams.
Step 11: Building a Main Function Enabling Users to Pose Free-Form Questions About SmokingOverviewThe goal of this specific QuitBot refinement was to build a main function of QuitBot that would enable users to pose free-form questions about quitting cigarette smoking and for the QuitBot to respond with accurate, concise, professional, and nonrepetitive answers. This was an iterative 3-step process, which is detailed in .
Table 4. Steps, sources, and results of QuitBot’s question and answer (QnA) iterative development process.StepSource (year)Results1. Generate QnA pairsAlexander Street therapy transcripts (2020)
留言 (0)