We chose the top five keywords directly related to PCOS from Google Trends on February 28, 2022 to conduct subsequently our searches in the separate search engines. The keywords included [1] polycystic ovary syndrome symptoms, [2] polycystic ovary syndrome treatment, [3] polycystic ovary syndrome causes, [4] polycystic ovary syndrome pregnancy, and [5] polycystic ovary syndrome test [26]. One investigator performed the keyword search. We used Google®, Bing®, and Yahoo® search engines for separate searches not limited to any specific geographical region to search for webpages using the abovementioned keywords describing PCOS information in English. Google Chrome version 99 (99.0. 4844.88) was used for all the searches.

We included webpages with information describing information about PCOS intended for patients or the lay public (newspaper websites, government and academic institutions, health center or hospital websites, or non-profit institutions), that had text with more than 30 sentences or 100 words long, and had identifiable domains (e.g., “.com”, .edu, “.gov”, “.info”, “.net”, “.biz”) to allow for their categorization. Webpages were categorized as commercial (.com), scientific resources (.edu and .gov), private foundations/advocacy (.health, .info, and .net), and non-profit organizations (.org) [27]. We excluded webpages that required a subscription, were videos, scientific journal articles, were intended for health care professionals, or were inaccessible.

One investigator (HV) copied and pasted the text of unique, non-duplicate webpages from each search from March to May 2022. Another investigator (SMP) checked the eligibility. Any disagreement about the categorization of the webpages was discussed until consensus was reached without the involvement of a third author. Two investigators rated the quality, clarity, and accuracy of the PCOS information with the DISCERN and EQIP tools in a 10% random sample of the webpages (5 from each search engine).

Inter-rater reliability was high for DISCERN items (kappa range 0.826 to 1.00). We resolved through consensus discussion the differences in our interpretation of item 9 “description of how each treatment works”, which had the lowest kappa in any single category of the DISCERN items (0.83, 95% confidence interval [CI] 0.77–0.89), before the full data extraction by HV. Inter-rater reliability was high for EQIP items (kappa range 0.83 to 1.00). We resolved through consensus discussion the differences in our interpretation of item 26 “use of generic names”, which had the lowest kappa in any single category of the EQIP items (0.83, 95% confidence interval (CI) 0.77–0.88) before the full data extraction by HV.

Inter-rater reliability was also high for accuracy of symptoms with a kappa range 0.891 (95% CI 0.84–0.94) to 1.00. We resolved through consensus discussion the differences in our interpretation before the full data extraction by HV.

We extracted the top 10 webpages from each of the three databases from which we performed five separate searches using the keywords chosen from Google Trends [26]. The web history log and cookies were deleted between each keyword and webpage search to avoid the influence of previous searches. The text and images were copied and pasted into Google Word documents marked with the date of extraction. We assessed the quality of the PCOS information using the DISCERN tool and the clarity with the EQIP tool. Due to the presence of figures and other images needed to rate the webpages that might have otherwise become altered when pasted to the Word document, we scored the webpages in live format directly, given that the date of the latest review or update was prior to the initial text extraction date. When this was not possible, a copy of the extracted text was used. We stipulated a maximum two-clicks per webpage to access the information included in the assessment.

The 16-item DISCERN tool utilizes four criteria to assess the authorship, attribution, currency of information, and ownership of a publication (website owner and conflict of interest of health information in written form. The tool contains Likert scores of 1 (no), 2, 3 (partially), 4, and 5 (yes) for items 1–15 to judge the presence of [1] clear aims, [2] information that addresses the stated aims, [3] relevant or realistic treatment information, [4] references to the sources used as evidence, [5] dates of the main sources of information, [6] a bias assessment, [7] suggestions for further reading or additional sources of information, [8] acknowledgement of gaps in knowledge, [9] the effectiveness of each treatment, [10] benefits of each treatment, [11] a description of the risks of each treatment, [12] descriptions of disease progression in the absence of treatment, [13] adverse events and the impact on the overall quality of life, [14] description of treatment choice options, [15] suggestions to discuss the health information on the website with family or health practitioners. To facilitate rating the webpages, we separated the intermediate ratings to 2 (somewhat low), 3 (moderate), 4 (somewhat high), while the lowest and highest ratings remained 1 (low or not available) and 5 (high). Item 16 is a summary score that addresses overall quality, denoted as 1 (low), 2, 3 (moderate), 4, and 5 (high). The minimum DISCERN score is 16, while the maximum score is 80. The quality of the information was classified according to the median score as “excellent” (63 to 80), “good” (51 to 62), “fair” (39 to 50), “poor” (28 to 38), or “very poor” (≤ 27).

We used the modified 36-item EQIP tool to assess the clarity of the PCOS information. There are 18 items related to content, 6 items for the identification of information, and 12 items regarding the structure of the information. For each item, we recorded yes or no responses to indicate the presence or absence of information and used not applicable (N/A) if the item in question was not relevant for a particular webpage. We excluded Item 27 from the EQIP because this item describes the “use of short sentences (< 15 words on average)” which was already automatically assessed in a separate readability analysis that we performed (described in the next section). Therefore, the maximum total EQIP score was 35 in the present study. Webpages with an EQIP score greater than 22.0, which corresponds to the 75th percentile, were deemed as high-scoring webpages. Low-scoring webpages were those with an EQIP score less than or equal to 22.0 [21].

Readability was analyzed using the online readability calculator at readable.io by directly pasting the webpage text from and including the title to the last sentence into the readability calculator [1]. We reported the Flesch-Kincaid Reading Ease scored from 0 to 100, where lower scores indicate difficult to understand text and higher scores indicate easier reading. Lower grade levels correspond to easier readability with the Flesch-Kincaid grade (FKG) level (ranges from grade 0 to 18 [college graduate] the Gunning-Fog (GF) score (ranges from grade 0 to 20 [college graduate]. The Coleman-Liau index (CLI) and automated readability index (ARI), ranging from 5 to 22 (college graduate), the New Dale-Chall Readability (NDCR) [ranges from grade level 4 to college graduate], and simple measure of gobbledygook (SMOG) [ranges from grade level 3 to college graduate] scores correspond to the years of education needed to understand written material. We additionally collected word count, syllables per word, words with more than two syllables, words per sentence, and sentence count.

We assessed the accurate and inaccurate statements of the symptoms of PCOS on the webpages using systematic reviews published from 2019 to 2022 [2, 23,24,25]. We based the accuracy of the PCOS information on the proportion of total accurate statements on the symptoms in the webpage text compared to the total number of statements about symptoms. Symptoms not accepted as accurate symptoms according to the systematic reviews included enlarged clitoris, headache, sleep apnea, sleep problems, pelvic pain, eating disorders, sexual dysfunction, oily skin, deeper voice, decreased breast size, mood changes, insomnia, fatigue, increased appetite, hypertension, swollen belly, endometrial hyperplasia, hidradenitis suppurativa, fatty liver, recurrent miscarriage, hyperkeratosis, inappropriate male features, and behavioral changes together with urinary and fecal incontinence. We assessed the accuracy of symptoms by counting the number of accurate listed symptoms in the website text and counting the total number of statements or words describing symptoms on a webpage. With the goal of producing one accuracy score for each website, the total number of accurate symptom descriptions was divided by the total number of statements or words describing symptoms. Scores were based on the proportion of accurate data ranging from 1 (lowest) to 5 (highest) [28, 29].

A score of 0 was assigned when the webpage did not list any symptoms. A score of 1 represented less than 25% agreement with evidence-based information, a score of 2 represented 26-50% agreement with evidence-based information, a score of 3 represented 51-75% agreement with evidence-based information, a score of 4 represented 76-99% agreement, and a score of 5 denoted 100% agreement with evidence-based sources on PCOS symptoms [28].

The DISCERN, EQIP, readability, and accuracy scores were treated as continuous variables. The EQIP responses were treated as dichotomous (rated as a 1 for yes or 0 for no) categorical variables, where each item scored as 1 contributed to the total score for each webpage. We used the Kolmogorov-Smirnoff test to determine the distribution of the numerical data and used non-parametric tests for non-normally distributed numerical variables. We reported descriptive data as n (%), median (Md), and interquartile range (IQR). Our analysis of webpage quality, clarity, and accuracy involved the assessment of the interrater reliability with Cohen’s kappa for agreement along with 95% confidence intervals (CI). The Kruskal-Wallis test was used to determine whether the DISCERN, EQIP, readability, and accuracy scores differed between search engine or webpage category. Dunn-Bonferroni post-hoc analysis was used for the Kruskal-Wallis test to determine in which category differences existed. The statistical significance was set at P < 0.05 for all the comparisons. MedCalc version 9.1.2 (MedCalc software bv) and IBM SPSS Statistics for Windows, versions 22.0 (IBM Corp., Armonk, N.Y., USA) were used for all analyses.