Cultural consensus theory for two-dimensional location judgments

In many domains in the social sciences and particularly in psychology, participants provide responses to questions for which correct answers are not known or not defined. For instance, researchers may elicit probability judgments of future events (Anders, Oravecz, & Batchelder, 2014) or ask whether one agrees or disagrees with a set of statements about a certain topic such as beliefs about the contagiousness of AIDS (Trotter et al., 1999). Cultural consensus theory (CCT, Romney, Weller, & Batchelder, 1986) is a method for aggregating the responses from several informants to estimate the shared knowledge of a group. Essentially, the model infers the latent cultural consensus of a group while considering variance both in the competence of informants and in the difficulty of items. This is achieved by assuming that the experts in a domain are those informants who provide “correct answers” in the sense that their responses consistently reflect the shared cultural beliefs.

The fact that normatively correct answers are unknown complicates the aggregation of informants’ responses because it is not clear which of the informants are most competent in the sense that they provide judgments close to the unknown cultural truth. As a remedy, CCT allows researchers to infer the latent cultural truth as well as the competence of each informant simultaneously. The main principle of CCT is that informants with more cultural knowledge are likely to show similar answer patterns across the set of questions asked because their judgments consistently reflect the shared cultural truth (Romney et al., 1986). Based on the correlation of the observed answer patterns, the method jointly estimates the cultural truth at the group level and the informants’ competence at the individual level. This requires that multiple informants provide judgments to a set of items from the same knowledge domain (Weller, 2007).

CCT was first developed in anthropological research for questionnaires about cultural topics with a dichotomous response format (Romney et al., 1986) and has also been described as “test theory without an answer key” (Batchelder & Romney, 1988). For instance, one of the first applications investigated the intracultural variability of beliefs about whether illnesses are contagious (Romney et al., 1986). The method has since been applied in various contexts such as aggregating eyewitness reports (Waubert de Puiseau et al., 2012, Waubert de Puiseau et al., 2017), obtaining forecasts for various events (Anders et al., 2014, Merkle et al., 2020), or estimating social networks where individuals provide information about social relations among different people (Batchelder, 2009, Batchelder et al., 1997).

The original version of CCT was applicable only to dichotomous data and assumed that all informants belong to a single shared cultural truth. As it may be possible that not all informants share a common consensus, Anders and Batchelder (2012) extended CCT to multiple cultural truths (see also Aßfalg & Klauer, 2020). Essentially, such extended models assume that informants belong to separate latent classes which differ with respect to the assumed cultural truth. For instance, medical professionals and lay people may differ with respect to medical beliefs resulting in two cultural truths which are latent if group membership is unknown.

CCT has also been extended to response formats other than binary answers. Extensions have been developed for continuous data (Anders et al., 2014, Batchelder and Anders, 2012), ordinal responses (Anders & Batchelder, 2015), and mixed response formats (Aßfalg, 2018) in order to aggregate ratings about the grammatical acceptability of English phrases or to measure shared beliefs about the importance of various health behaviors. Statistical inference for such extended CCT models has often relied on hierarchical Bayesian modeling in which parameter estimates are obtained via Markov chain Monte Carlo (MCMC) sampling (Anders and Batchelder, 2012, Anders et al., 2014, Aßfalg and Klauer, 2020). Overall, these extensions have enabled researchers to adapt the CCT approach to various types of data while assuming a certain structure of cultural truths underlying informants’ answers.

CCT is also applicable to scenarios in which correct answers are not known during the time of data collection but may become available later. Such forecasting applications are especially interesting because the performance of different aggregation methods including CCT can be directly compared against each other once the correct answers become available. If factually correct answers are available, it is also possible to check whether the expertise estimates of CCT correlate with the accuracy scores of individuals. In judgment and decision making, it is well known that the aggregation of independent individual judgments (e.g., by computing an unweighted average) results in highly accurate group estimates, a phenomenon referred to as wisdom of crowds (Hueffer et al., 2013, Larrick and Soll, 2006, Steyvers et al., 2009, Surowiecki, 2005). This high level of accuracy across various tasks and contexts is surprising given that all judgments are weighted equally without considering informants’ competence (or incompetence) regarding the domain of interest.

The accuracy of wisdom of crowds can be improved by weighing individual judgments by the expertise of informants. For instance, Budescu and Chen (2015) relied on prior judgments of participants to estimate the competence of each individual relative to the crowd. Using the estimated expertise as weights improved the accuracy of the aggregate estimates. CCT also weighs judgments by expertise, but it does not rely on informants’ performance on previous items. Instead, CCT relies on a statistical model to simultaneously estimate individuals’ expertise while using these estimates as weights for the aggregation of judgments. With respect to forecasting, Merkle et al. (2020) showed that such a CCT-inspired aggregation mechanism does indeed outperform unweighted averaging. Similarly, the accuracy of aggregated eyewitness testimonies increases when using a CCT model since it infers the witnesses’ competence (Waubert de Puiseau et al., 2017). Overall, CCT is thus a useful tool for the aggregation of judgments when the ground truth becomes available only at a later time.

CCT has been adapted to several types of response formats and applications, but an extension for two- or higher-dimensional judgments has not been developed yet. One important type of two-dimensional continuous data are geographical judgments which arise whenever individuals locate sites on a map (e.g., Friedman, Kerkman, and Brown, 2002, Mayer and Heck, 2022). In such scenarios, the actual locations are usually unknown to the informants, either because of a lack of precise knowledge (e.g., for cities) or because there is no factually “correct” location (e.g., when asking for preferences or beliefs). Extending CCT to location judgments provides a principled method of inferring the common consensus of a group about the location of the sites of interest.

A CCT model for two-dimensional location judgments can be beneficial in various contexts and tasks, both in psychology and beyond. First, CCT is a useful tool for aggregating subjective judgments even when the actual locations of sites can in principle be known. For instance, Friedman and colleagues (Friedman, Brown, and Mcgaffey, 2002, Friedman, Kerkman, and Brown, 2002, Friedman et al., 2005, Friedman et al., 2012) examined the role of individuals’ place of residence on their geographical knowledge and representation. Several studies showed that there are considerable differences in location judgments for individuals living in Canada, Mexico, and the United States when it comes to locating cities in all three countries. However, participants in these studies only provided one-dimensional judgments of the latitude of cities as this facilitated the statistical analysis. An extended CCT model for two-dimensional continuous data would allow researchers to collect and aggregate location judgments with respect to both latitude and longitude. Moreover, a CCT model could be used to explain variance in judgments by informants’ expertise, or to compare model-based location estimates between different manifest groups or cultures. Note that location judgments of cities have also been used to compare the performance of different approaches of judgment aggregation in online collaborative projects (Mayer & Heck, 2022).

Second, an extension of CCT for two-dimensional continuous data is especially useful for the aggregation of individual location judgments when the factually correct or optimal locations are unknown. For instance, Surowiecki (2005) describes how a lost submarine was found by aggregating the judgments of experts on its most likely location. Similar applications are in principle possible when selecting optimal locations for park-and-ride facilities (Faghri, Lang, Hamad, & Henck, 2002), suitable areas for ecotourism (Mahdavi, Niknejad, & Karami, 2015), or uncovering ancient archaeological sites (Casana, 2014) and natural resources (e.g., water harvesting sites, Al-shabeeb, 2016). However, a statistical aggregation of location judgments based on CCT is only applicable in scenarios where several informants provide location judgments for multiple sites. If these requirements are met, CCT is ideally suited to infer the shared, common consensus about the unknown locations.

In the following, we develop a new CCT model for two-dimensional location judgments based on Anders et al.’s (2014) CCT model for one-dimensional continuous responses. We check the validity and performance of the proposed CCT model and its Bayesian implementation in JAGS (Plummer, 2003) by investigating parameter convergence and recovery in a Monte Carlo simulation. Moreover, we use simulations to examine under which conditions the weighting of judgments by individuals’ competence improves the accuracy of location estimates at the group level. Empirically, we apply the new model to reanalyze location judgments of European cities on maps (Mayer & Heck, 2022) and compare the accuracy of the aggregate location estimates to those obtained with unweighted averaging. Thereby, our work contributes to prior research showing that wisdom of crowds can be improved by weighing judgments by expertise (Budescu and Chen, 2015, Merkle et al., 2020). Overall, the results of our simulation studies and the empirical reanalysis show that the weighting of individual location judgments by informants’ competence improves estimation accuracy of CCT compared to weighting all judgments equally.

The proposed extension of CCT for two-dimensional continuous responses is specifically tailored to geographical data where informants provide location judgments. Of course, two-dimensional continuous data are also collected in other tasks and contexts in psychology. For instance, participants may have to rate items with respect to two features such as the valence and arousal of images (Funke and Reips, 2012, Reips and Funke, 2008) or facial images for their attractiveness and trustworthiness (Oosterhof & Todorov, 2008). The proposed CCT model may not be directly applicable to such data because of certain assumptions that are specific to location judgments (e.g., assumptions about the dimensionality of informants’ competence or the correlation of errors across dimensions). In the Discussion, we elaborate on how the model can be adapted to multidimensional, continuous judgments in other tasks and contexts besides location judgments.

留言 (0)

沒有登入
gif