The secret shopper methodology can be broadly applied to many healthcare settings. One particularly potent use of secret shopper methodologies is as a mechanism to assess the delivery of healthcare. For example, family medicine, with its emphasis on preventative healthcare—which at its core has the aims to provide equitable care to all—represents one ideal opportunity to assess and improve healthcare delivery, elevate the patient experience, and reduce disparities in access. The methodology outlined here can be used to ascertain areas of potential improvement within the healthcare cascade [17, 18].

It is challenging to objectively measure patients’ ability to access healthcare. When surveyed about appointment availability, physicians’ offices tend to overestimate their capacity to accept hypothetical new patients [19]. Patient and physician surveys often fail to accurately capture biases—particularly those related to sensitive or stigmatized topics—which may impact patients’ experiences with healthcare providers and, consequently, their access to appropriate care [19]. Furthermore, physicians’ offices may not be forthcoming about their health insurance policies, especially when discussing their ability or willingness to promptly accept Medicaid patients [19]. Each of these elements is integral to providing good care in family medicine.

The secret shopper methodology serves as an objective measure for studies seeking to evaluate patient access to care [20]. This method is particularly useful in overcoming the “Hawthorne effect” in situations similar to the one described above, in which providers’ behaviors or policies might change if they were aware that they were being observed [21]. For example, an audit of orthopedic surgery practices to evaluate patient access to knee and primary and revision arthroplasty found that orthopedic surgeons’ offices responded differently to a faxed survey about Medicaid acceptance policies than they did to a simulated Medicaid patient calling and attempting to schedule an appointment [22]. A study examining delays to emergent surgical care revealed discrepancies in emergency department referrals based on insurance status [23]. These results suggest that practice policies and staff behavior may not be accurately captured in a survey. Therefore, the results of these studies can be used to enhance patient access to care and the quality of care they receive.

Insurance-based discrepancies in access to care impact the healthcare system in a variety of ways. Understanding shifts in the burden of healthcare service utilization can reveal gaps between policy and practice. Accordingly, the United States Department of Health and Human Services recommends using secret shoppers to measure healthcare access [24]. A 1994 study by the Medicaid Access Study Group employed this methodology and found that Medicaid patients were less able to access timely medical appointments, a finding that contributed to the explanation for that population’s increased use of emergency departments for nonurgent issues [25]. Other secret shopper studies have revealed insurance-related disparities in access to primary [8, 18, 26,27,28], follow-up [11], orthopedic [29], dermatologic [30], pediatric [31, 32], newborn [33], reproductive [5], and psychiatric [34] care, among others [35]. In addition to highlighting challenges associated with insurance, secret shopper studies can assess administrative staff’s knowledge of details related to coverage, such as the likelihood of receiving a surprise bill or the availability of alternative payment options [36, 37].

Furthermore, simulated patients can obtain more detailed information about the intricacies of patients’ experiences within a complex and evolving healthcare system. The last decade has seen substantial increases in vertical and horizontal consolidation of health organizations and medical practices [38, 39]. These consolidations have affected all sectors, but particularly family medicine. To cite a few downstream effects: increased costs, increased travel time for patients, and no improvement in quality of care [40]. These mergers are often associated with higher prices and spending, and budgeting decisions inevitably affect reimbursement, clinical decision making, local competition, referrals, and patient experiences [41,42,43]. These nuances impact access to care but might not be apparent on practice websites or in healthcare policy. For example, in addition to determining appointment availability based on insurance type, secret shopper studies can probe for information about how patients are treated by providers or staff based on their insurance. Secret shoppers may also identify additional barriers to receiving care, such as referral requirements, long wait times before obtaining appointments, and the requirement to send records and obtain testing results prior to the visit, which might not be expected of patients with private insurance [22].

However, it is important to also address the challenges of deploying these studies. Secret shopper studies need to be done systematically or can be subject to bias. It can be difficult to achieve large enough numbers to power studies, so careful considerations need to be taken to begin with a broad targeted cohort. There is also a subjective nature of data acquisition that is challenging to control. Iterative processes should be undertaken, when possible, as the role of discrimination vs. human emotions needs to be teased apart. Additionally, once providers learn of the secret shoppers, this can perhaps engender distrust on the part of the healthcare providers being assessed, which may damage any future relations. Therefore, it is important that researchers of this methodology, when publishing and sharing their results, do not identify specific entities for privacy reasons given that these entities never gave consent to participate.

These findings are especially useful for evaluating access to care in a period following Medicaid expansion and the COVID-19 pandemic in the United States, when policymakers, providers, and patients alike seek to understand whether health insurance coverage alleviates differences in healthcare access to the underinsured. Secret shopper studies are well-positioned to expose these disparities and provide insight into how patients navigate the healthcare system [44].

Case studyExamining access to urgent care based on insurance type

This study sought to examine insurance-related disparities in access to care at urgent care centers in the United States. We designed a secret shopper study to assess whether callers posing as simulated patients with either private insurance or Medicaid were able to successfully schedule an appointment. Below, we analyze our process in accordance with the best practices outlined.

1. Seek IRB exemption

Our study design received an institutional review board exemption from our institution’s IRB.

2. Define the primary outcome variable

Our primary outcome variable was acceptance of Medicaid.

3. Change only onevariable at a time

Our callers called once asking about a patient paying with private (Blue Cross) insurance and once about paying with Medicaid. All other details remained the same.

4. Inquire about a real scenario

Our callers explained that they were inquiring on behalf of their father, who had been previously diagnosed with an incarcerated inguinal hernia and was currently experiencing symptoms of strangulation. They stated his insurance type and asked if they could bring him into the urgent care center.

5. Don’t pre-screen offices to ensure that they accept Medicaid

We used a random number generator to select 25 urgent care centers to call per state, and did not pre-screen any of them to verify that they accepted Medicaid.

6. Include questions that probe for nuances

Our call script included a variety of follow-up questions after we presented the initial scenario: “What kind of provider would he be seeing (doctor, nurse, PA)?” “What would the wait time be? Do you have a self-pay policy for uninsured patients?” “Are there any discounts?” “Do you know the maximum price you would have to pay?” “What’s the closest Emergency Department to your office?” These questions allowed us to capture a more accurate understanding of a patient’s experience attempting to access care.

7. Pilot and refine the call script (and prioritize plausibility and efficiency)

Prior to beginning the data collection, we piloted the script by contacting several offices. We adjusted our scripts accordingly to streamline the conversation and ask more relevant questions.

8. Consider the health policy implications of data reported

We knew that the urgent care industry had experienced astronomical growth in recent years, but we wanted to determine the degree to which urgent care is accessible and how it impacts the care of surgical conditions. Collecting and reporting this data has important health policy implications.

9. Verify that the practice type is categorized correctly

When we called, we specified that we were seeking an urgent care center and excluded the center from our results if the office said they did not meet our criteria.

10. Record details that are relevant to the research objective; consider including additional demographic information and analytical tools to describe a practice within a state context.

We recorded data describing both individual centers and entire states. Our categories included: urgent care center classification (independent, private practice, primary care, health network, academic); accreditation status; zip code median income; driving distance from an academic medical center; state Medicaid expansion status and reimbursement level for a new patient visit; and state population size. We analyzed our data using JMP Pro. We also identified urgent care centers within a 5-mile driving radius of an academic medical center using the ArcGIS Mapping System.

We presented a univariate and multivariate analysis of center-specific and community-level urgent care center characteristics. We concluded with relevant policy findings about predictors of Medicaid acceptance, suggested that urgent care centers may be referring Medicaid patients to safety-net hospitals. We urged future investigators to delve deeper into understanding the impact of urgent care centers on referral rates and patterns, healthcare disparities, and delays to receiving appropriate care.

11. Train callers in delivering standardized responses and periodically monitor their calls

We trained callers and provided feedback on several practice calls before they began calling sites independently.

12. Block phone numbers

When possible, we blocked our numbers. Notably, several urgent care practice offices did not accept calls from a blocked number, so we made sure to call from a different phone for the second call.

13. Space out calls

We called practices at least > 14 days apart and, whenever possible, from a different phone number and on a different weekday and time.

14. Avoid centralized phone numbers of operating systems

When we encountered a centralized phone system, we attempted to look online for direct extensions that would lead us to individual centers instead. If an operator realized that we had called multiple times, we explained that we were trying to get a better sense of regional healthcare options for the patient (the simulated caller’s father).

15. Be on the lookout for aberrances or unusual activity

Those making the phone calls must make each call with an eye open for anomalous or peculiar results. We have commonly been surprised by some of the answers given to us by the operators, and at times have uncovered findings that led us to alter our study design or propose a new study to probe a different question.