This study is part of a larger project, the COSMO-SPAIN Project, aimed to inform COVID-19 outbreak response measures, including policies, interventions and communications. The results have been shared with the Spanish health authorities and are publicly available (https://portalcne.isciii.es/cosmo-spain/).

A cross-sectional, observational study was conducted on the Spanish general population from 27 July to 3 August 2020 by an online survey agency. The sample, composed of 1033 participants, was extracted from a panel of approximately 982,000 participants. The sampling was stratified matching the Spanish general population in terms of age, gender, and area of residence. Inclusion criteria were being 18 years or older and living in Spain.

Data were collected after the first wave of the pandemic and after the end of the general lockdown (21st June 2020). At the time of the study the epidemiological situation in Spain had improved, and the accumulated incidence was 37.9 cases by 100,000 inhabitants [17]. The use of face masks was mandatory for people aged 6 years or older in enclosed spaces and outdoors. The government opened all internal borders among autonomous communities as well as international travel restrictions with other European Union countries and the United Kingdom. Other restrictions related to mass gatherings and closure of public spaces were handled by each autonomous community independently.

Participants were asked about their gender, age, education (incomplete primary; primary education; secondary education; high school education, and university), area of residence (2000 to 50.000; 50.000 to 400.000; > 400.000 inhabitants) and employment situation.

CHL-Q items included in the COSMO-WHO survey tool [18], originally in English, were translated by professional translators and adapted by the COSMO-SPAIN team.

The original HLS-EU-Q, which assess generic HL, consists in 47 items, but shorter versions have been developed and tested in several languages and contexts [19,20,21,22,23,24,25,26,27]. In Spain, a short version was found to be an adequate and valid tool to measure the level of generic HL in the population [19].

The CHL-Q assesses people's knowledge, motivation and competencies to access, understand, evaluate, and apply information about COVID-19 in order to make informed decisions to prevent the disease. Includes a general question “How easy or difficult is it for you to…?” followed by 9 specific tasks, for which participants rate their perceived difficulty on a four-category Likert scale very difficult (1), difficult (2), easy (3) and very easy (4).

In accordance with the original questionnaire, the CHL-Q index was standardized from 0 to 50 [index = (mean – 1) × (50/3)], using the mean of all items for each respondent.

We addressed COVID-19 infection by asking participants “To your knowledge, are you, or have you been, infected with COVID-19?”, with yes/no response options. In order to assess information-seeking behaviour, we asked the respondents about how often they used 8 common sources of information (TV news, TV and radio magazines, government press briefings, newspapers, social networks, internet, government website and WHO reports) to stay informed about the coronavirus, answered in a scale from 1 (never) to 5 (very often).

COVID-19 knowledge was measured by the degree of agreement with 12 correct and incorrect statements about COVID-19 (Supplementary file 1). The items were developed according to health authorities’ guidelines at the time of the survey, and included symptoms, transmissibility and face mask use (e.g. people who do not have fever can be contagious; the coronavirus is spread by droplets when coughing/talking; face masks must cover mouth and nose). The total number of correct answers composed the COVID-19 knowledge score (0–12).

To assess confusion related to COVID-19 information we asked the participants: “Have you encountered information on the novel coronavirus where you found it hard to decide whether it was right or wrong?” (yes/no).

Adherence to recommended preventive behaviours was assessed by 8 items regarding basic protective measures recommended at that time by the health authorities, “During the last 7 days, which of the following measures have you taken to prevent infection from COVID-19?” (yes/no). The total number of preventive measures taken by each participant composed the preventive adherence score, ranging from 0 to 8.

CHL-Q index data was normally distributed, and thus, parametric statistics were applied. Continuous variables were expressed as central tendency measures (means, medians), measure of dispersion (standard deviation—SD, range), and the categorical variables were expressed by frequencies and percentages. Bivariate descriptive analyses were performed on the continuous variables by Spearman's correlation. To test the association between variables, for independent samples, ANOVA tests were used.

The psychometric properties of the CHL-Q were explored using Rasch analysis and classical test theory (CTT): data quality and acceptability, construct (structural and hypotheses testing) validity, and reliability (internal consistency) [28].

For data quality and acceptability, the mean, median, SD, range of observed vs. theoretical values, skewness (criterion: − 1 to + 1), floor and ceiling effects (≤ 15%) of the CHL-Q items and index were calculated [29]. Internal consistency of the CHL-Q was examined by computing by Cronbach’s α coefficient (≥ 0.70), item-total corrected correlations (r ≥ 0.40), inter-item correlations and the item homogeneity index (> 0.30) [30].

The corrected item-total correlation (≥ 0.20), for structural validity, and the inter-correlation of CHL-Q, for internal validity, using Spearman’s rank correlation coefficients were calculated.

Discriminative or known-groups validity were explored by calculating the differences in CHL-Q scores in the sample grouped by age, sex, education level, employment status, infection status, adherence to preventive measures and confusion related to COVID-19 information. Lower CHL-Q index scores were expected for older [5], those with lower education level [31], feel confused about COVID-19 information [32] and had been infected [33, 34].

Convergent validity was calculated using the Spearman’s rank correlation coefficient between the CHL-Q index and age, information seeking frequency, COVID-19 knowledge score and number of preventive measures taken. Our hypothesis was a low correlation of the CHL-Q index with age [5] and a moderate one with knowledge about COVID-19 [13], number of preventive measures [13, 14] and frequency of information seeking [35].

The Rasch model purports that the probability of a given response to an item is a function of the person’s capability (or level of health literacy) and item difficulty (or degree of the construct measured by the item), expressed in logits in an interval-level scale [36]. Fit to the Rasch model is obtained when there is a non-significant interaction chi-square difference between the observed data and the Rasch model [37]. Since a large sample size might lead to signalling small model deviations as significant, and to unnecessary model modifications, a random sample of 300 was used. This sample size provides stable calibrations independently of the targeting [38]. Item and person fit residuals are expected to follow a normal distribution with mean of 1 and SD of 0, and fall within the -2.5 to 2.5 interval. Reliability is assessed through the Personal Separation Index (PSI), interpreted similarity to Cronbach’s alpha. Low correlations (< 0.30 of the average correlation) between item residuals indicate item local independency, i.e., that the response to one item do not lead to the response to another item. Unidimensionality was ascertained by a principal component analysis of the residuals and comparison through t-tests; the lower bond of the associated binomial 95% confidence interval should be ≤ 0.05. Item thresholds, or the point of equal response probability between two adjacent response categories, are expected to be ordered. For Differential Item Functioning (DIF), analyses of variance were performed by the following groups of persons: age (lower than the median 46 vs. 46 + years), sex, and education level (low: up to 14 years old; medium: secondary or professional training; high: university).

Analysis was performed using IBM SPSS Statistics version 22 (IBM, Armonk, NY). Rash analysis was performed iteratively, using RUMM2030 software.