The Measurement Pitfalls of Research Designs in Cultural Studies of Religions

Cross-cultural comparability and generalizability are the problems that come up in religious studies and need to be solved for scientific progress (Karandashev, 2021a; Karandashev et al., 2022; Fischer, 2022). When studying behavioral and social phenomena in various populations and religious contexts, culture matters. In this regard, the lead article by Ronald Fischer (2022) in the recent issue of the journal Religion, Brain & Behavior is particularly useful. The author of that article shares personal reflections on the study that their team reported on during their exploratory journey. Here is a summary of one of the two points covered in his commentary “Cultural lessons missed and learned about religion and culture.” It is about “how important cultural context is for thinking about, and researching, religion, morality, and evolution.”

The Typical Mistakes and Their Effective Solutions to Studying Religions from a Cross-Cultural Perspective

The study’s goal was to investigate the universality and evolutionary perspective of religious concepts. Researchers considered cultural dynamics throughout the process, including the specification of key variables, variable operationalization, measurement context, and result interpretation. Researchers summarized the new efficient translation methods (Harkness et al., 2003). They proposed the updated checklists for use by cultural researchers (Hambleton & Zenisky, 2010; Harkness et al., 2003, 2010; Hernández et al., 2020).

In this lead article, Ronald Fischer (2022) addressed two groups of methodological issues:

The first one is the problems of cross-cultural universality of the concepts under study, their conceptual equivalency, the selection of major variables, and their conceptual descriptions and operationalization. These questions are summarized in another article.

The second one is the problems such as cultural contexts of measurement, technical procedures of measurement, cultural biases in measurements, measurement invariance across cultural samples, and culturally sensitive interpretation of results. These questions are summarized in this article.

Confounding Cultural Variables in the Studies of Religions

In complex cross-cultural research, the design itself may create confounding factors. Who is a local co-religionist as opposed to a remote one in a religious context? Religions frequently make fine distinctions in group membership. In the cultural context of Candomblé religion, this includes questions about

  • who went through the initial initiation (“bori”) with you,
  • who is a member of the same “terreiro,” house of worship, typically organized around extended family ties),
  • who has the same sitting “orixá.”

Without knowledge of these regionally relevant group distinctions, the research design of a cultural study lacks these essential local details.

In addition, classic cross-cultural research has demonstrated that both familiarity and theoretically irrelevant features can influence

  • behavioral and cognitive responses (Serpell, 1979),
  • social expectation or experimenter effects that can be difficult to identify or avoid (Smith et al., 2013).

The Cultural Biases in Religious Studies

Typically referred to as technique biases, these difficulties involve

  • how tests are conducted,
  • by whom, and in what (implicit or explicit) context.

Humans are sociable experts. They try to predict what others want from them. These attempts may lead to an array of behavioral adaptations with the intentions

  • to make favorable impressions,
  • form alliances, or
  • gain tiny advantages over local competitors or
  • trade favors with outside visitors.

Depending on how the participants interpret the testing circumstances, these motivations can reverse the expected behavioral responses.

This is another challenge for cultural research. Individuals in small-scale societies converse and make assumptions as to why someone may or may not have received the money. The questions arise

“Does the payout matrix align with the implicit group lineages that participants construct while participating in the experiment?

Does the knowledge of pay-outs affect the next participant’s strategy of playing? ” (Fischer, 2022, p. 214)

In environments with greater interdependence, individuals are likely to respond depending on who has already been tested or how many individuals remain to be evaluated (Yamagishi et al., 2008). These different techniques’ biases provide considerable obstacles for evaluating the outcomes of money distribution and frequently necessitate ingenious and observant researchers conversant with local cultures and standards.

The Pitfalls of Priming Research Designs in Cultural and Religious Studies The research with priming tasks poses other questions. The procedure of priming requires locally salient categories regardless of the question of replicability concerns with priming. This brings scientists back to the principles of functional and structural equivalence, which we talked about above.

“What is a moralistic god vs. a local god?”

(Fischer, 2022, p. 214).

The Christian “God,” which is not part of the Candomblé religion, and Ogum, a particular orixá linked with ironwork and war, are very different planes of existence. Therefore, a contrast between those two may not convey what the researchers intended.

For Candomblé believers, the Christian “God” is familiar. It is simple to identify and acknowledge this deity’s significance in the larger community. However, it is not necessarily an entity with personal meaning for a Candomblé devotee. In the same vein, depending on the context, Ogum may be appropriate for particular goals or for particular individuals.

What is an adequate and comparable indication of the idea of interest within the local cultural context? Questions like this are very important in the context of structural equivalence, specifically the issue of conceptual domain representation.

The Importance of Local Context in Cultural Research In conclusion, Ronald Fischer (2022) encourages cultural researchers to pay more attention to the local cultural context of their studies. He suggests learning the lessons from researchers of previous generations who made progress through these challenging paths.