When Iain Chalmers, Andy Oxman and others from the Informed Health Choices team (www.informedhealthchoices.org) convened a cross-field forum to develop a generic framework to support people in making choices informed by critical understanding of the relevant available evidence (that is, ‘informed choices’), they did not know how such an interdisciplinary group would work. A generic framework to support these informed choices was subsequently produced and published elsewhere (Aronson et al., 2019). This companion paper describes how we collaborated, the underlying discussions, and our experiences of working together across disciplines and fields (see Box 1 for definitions). The project had an inherently experimental feel from the outset. When we began this task, for instance, we were not sure whether we would all agree on the nature of the problem we were working to solve, whether we could identify enough shared items for inclusion in our framework, or whether we could produce something that was applicable across the full range of potential uses that our fields represented.
|Discipline||A branch of knowledge as studied in higher education|
|Field||A sphere of activity, or a domain, that can incorporate more than one academic discipline|
|Informed choice||A choice that is based on critical understanding of the relevant available evidence|
There is a growing body of research on the value of breaking down traditional boundaries in academia (see, for example, Heinze et al., 2009). This is combined with increased awareness of the need to work in transdisciplinary and cross-field ways to prepare our students for the future, and to generate useful knowledge for decision making (Sharples et al., 2017). Several organisations, including the Global Research Council (2016), the Wellcome Trust (2017) and the Research Council of Norway (2019) have encouraged interdisciplinary innovation, but there is little understanding of how such innovation can occur. To gain a sense of the challenges that interdisciplinary work imposes, one must first understand that each discipline trains its followers to engage with ideas and with peers, using frameworks that often differ fundamentally from discipline to discipline. For example, psychology very broadly focuses on individuals and their rights, responsibilities, norms and practices, while sociology prioritises the collective, and focuses on a shared community-level identity and centres of control. Educational research operates generally under an assumption that each learner, classroom and school is unique. A clinician, on the other hand, seeks to individualise the treatment of a patient with some condition, based on the average results of research in a population. These examples demonstrate the widely varying paradigms within which different academic disciplines operate, and provide some idea of why cross-disciplinary work is more unusual and more challenging than one might expect.
Evidence-based approaches focus on informing real-world choices. Helping individuals, communities, citizens and policymakers tackle important problems often requires solutions that span more than one field (Wilkins and Cooper, 2019). Improving evidence-informed decision making for a young child’s health, for example, requires input from primary schools, public health practitioners, clinicians and families. Addressing issues such as high youth unemployment requires evidence on economic growth, higher education and learning, business development, entrepreneurship and innovation. The more complex the problem, the greater the need to break down traditional disciplinary boundaries.
The SARS-CoV-2 coronavirus disease (COVID-19) pandemic has highlighted the need for cross-field collaboration. For example, addressing the accompanying ‘infodemic’ (WHO, 2020a) requires more than epidemiology. Meanwhile, the recommendation to randomise school closures to improve certainty about their effects on both health and non-health outcomes (Fretheim et al., 2020) might be strengthened with collaborations involving colleagues from education and economics. There has been criticism that some national pandemic response teams are limited by their failure to include, for example, economists or behavioural or social scientists, while others have been challenged to be more transparent in their processes (see, for example, Clark, 2020). Meanwhile, the COVID-19 Evidence Network to support Decision-making (COVID-END) is bringing together several fields to help decision makers find the best available evidence, and to help researchers coordinate their efforts, rather than duplicating them (www.mcmasterforum.org/networks/covid-end).
Perhaps encouragingly, a large number of disciplines within broad fields, such as health, education and development (see Table 1), have committed to a shared goal of ensuring that decisions are informed by the best available evidence, and have established their own norms and practices for supporting evidence-informed decision making. However, apart from a few exceptions, such as in health promotion and other allied health professions, these parallel initiatives seldom work together. This is despite a recognition of the potential for cross-field activities. Academic discussion and debate across evidence-based fields has largely taken place in silos, with limited cross-field discussion or fertilisation of ideas. For example, we know that parallel discussions have taken place in field-specific evidence-related conferences, such as the Cochrane Colloquia and the Collaboration for Environmental Evidence conferences, with little or no awareness that other experts in a different field are focusing on solving the same problems – the development of a multitude of methods for the synthesis of qualitative research is just one example (see Barnett-Page and Thomas, 2009). It was the combination of the shared commitment to the use of evidence across fields, and the recognition that the fields rarely work together, that motivated this cross-field initiative.
|Agriculture||Centre for Evidence-Based Agriculture, Harper Adams University||UK|
|Economics||Institute of Health and Society, Newcastle University||UK|
|Education/schools||Education Endowment Foundation|
Graduate School of Education and Statistics Department, University of Pennsylvania
|Engineering||School of Engineering and Applied Science, Aston University||UK|
|Environment||Collaboration for Environmental Evidence|
Centre for Evidence-Based Conservation, Bangor University
|International development||International Initiative for Impact Evaluation (3ie)|
Africa Centre for Evidence, University of Johannesburg
|Health care, including a number of specialisms (in particular: public health, general internal medicine, mental health)||Centre for Informed Health Choices, Norwegian Institute of Public Health|
Faculty of Health Sciences, Oslo Metropolitan University
Regional Centre for Child and Adolescent Mental Health and Child Welfare (Eastern and Southern Norway)
Centre for Evidence-Based Medicine, University of Oxford
Faculty of Medicine, University of Manchester
|Journalism and media/fact checking||Africa Check||Africa|
|Management and business||Centre for Evidence-Based Management (CEBMa)||Global|
|Museums/public engagement||Museum of Natural History, University of Oxford||UK|
|Nutrition||Centre for Evidence-Based Health Care, Stellenbosch University||South Africa|
|Planetary health||Centre for Climate Change and Planetary Health, London School of Hygiene and Tropical Medicine||UK|
|Policing/crime reduction||What Works Centre for Crime Reduction, College of Policing||UK|
|Psychology||Department of Experimental Psychology, University of Oxford||UK|
|Social welfare/social work||Campbell Collaboration|
School of Social Work, Saint Louis University
|Speech and language therapy||School of Health Sciences, University of Central Lancashire||UK|
|Veterinary medicine||Centre for Evidence-based Veterinary Medicine||UK|
Interdisciplinarity and informed choices
The Informed Health Choices (IHC) project team has developed key concepts for thinking critically about health-care claims, research and choices (Chalmers et al., 2018). In part because it is hard to change adults’ behaviours, they have focused largely on working with young people (Semakula et al., 2020). This reflects a general trend in this kind of work towards initiatives among schoolchildren or undergraduates (Oxman et al., 2021). The challenge, however, is not one that applies only in the health-care field, nor only to young people. Claims are made in the public arena every day about topics such as nutrition, ways of guaranteeing success in examinations, and how to make money quickly. Unfortunately, people often fail to question such claims, that is, to ask for or question supporting evidence. In recognition of the commonality of the problem of ensuring that decisions are informed by the best available evidence, and in the belief that solutions to our common problems are better sought together, the authors volunteered to join a cross-field group which convened in late 2018. Our aim was to develop a generic framework for assessing claims and supporting informed choices. The group included 24 researchers from a wide range of disciplines (see Table 1). What we had in common was that we all work to support the use of evidence in decision making in our respective fields.
This paper shares lessons of interest to researchers and others working in cross-disciplinary collaborations. Our aim in sharing our observations is to engage further with what we have learned and identify areas for improvement to enable greater trans-disciplinary and cross-field work in the future.
We developed the shared framework in three phases: before, during and after a two-day meeting in Oxford (UK) in December 2018. Before the meeting, the IHC framework was circulated (Chalmers et al., 2018). All were encouraged to bring reflections on this to the meeting. This included sharing preliminary elements of the Informed Choices framework with colleagues, and getting their inputs into which elements they saw as relevant to their discipline. After the meeting, we continued to refine and develop our thinking by editing the emergent framework and preparing a report on our work (Aronson et al., 2019), and developing website content (https://thatsaclaim.org). Table 2 summarises aspects of our engagement.
Because of her personal interest in boundary-spanning initiatives, Ruth Stewart took notes about the process of working together across fields, which have directly contributed to this paper. In the subsequent months, she documented and shared key observations with the group by email, and invited others to reflect on these and help identify key lessons from the process of our cross-field working. Our shared reflections and contributions were distilled and grouped into themes, and are shared in this paper.
The resulting framework
Our cross-field work resulted in a framework that includes many concepts that are important to both laypeople and professionals aiming to make or support informed choices, within at least 14 fields. The concepts are in three overarching groups: (1) claims of effects that ‘should be supported by evidence from fair comparisons’; (2) comparisons that should be ‘fair comparisons, designed to minimise the risk of systematic errors (biases) and random errors (the play of chance)’; and (3) choices that ‘depend on judgements about the problem, the relevance (applicability or transferability) of evidence available and the balance of expected benefits, harm and costs’ (Aronson et al., 2019: 304). Each overarching group is unpacked within the cross-field framework reported in full and available online (https://thatsaclaim.org).
Our findings about the process of developing this framework reflect three broad themes:
the extent and nature of agreement over the problems we were trying to solve, and when diversity of experiences and views were voiced
the extent to which we identified common problems and prioritised them for inclusion in the framework we were developing, while also recognising differences
the breadth of applicability of the framework we developed.
In addition, we found several themes that cut across the elements of the Informed Choices framework that we developed. These cross-cutting themes are summarised in Table 3.
|Elements of the final Informed Choices framework||Themes of commonalities that arose from the cross-field discussions|
|Claims: Claims about effects should be supported by evidence from fair comparisons. Other claims are not necessarily wrong, but there is an insufficient basis for believing them.||Common observation that interventions get implemented in good faith based on a belief that something works, but belief is not enough to substantiate the claim, and resources can be wasted and harm caused by such claims.|
|Claims are not made in a vacuum, but are influenced by a wide range of other factors, and often become the accepted norm, even if contrary to the evidence base.|
|Small-scale studies are common across disciplines, and are part of the problem.|
|There is a need for examples of harms that can occur, and resources wasted, when you are not making decisions based on evidence, to make the point in a powerful way so that people will take notice.|
|Comparisons: Studies should make fair comparisons, designed to minimise the risk of systematic errors (biases) and random errors (the play of chance).||There is a need for common terms for describing study designs that make fair comparisons, and that are designed to minimise biases.|
|There is a lack of high-quality evidence on which to base decisions at all levels, from practice to policy.|
|Choices: What to do depends on judgements about the problem, the relevance (applicability or transferability) of the evidence available, and the balance of expected benefits, harms and costs.||Recognition of the trade-offs that decision makers make, and that these take place at several levels, balancing multiple benefits, harms and costs.|
|Cognitive bias seems to be a stumbling block.|
The extent and nature of agreement about the problems we were trying to solve
On the first day, we focused on understanding the work to date by those who had convened the meeting (the IHC project), and to develop shared understanding of the value of a cross-field framework. All participants acknowledged early on that our shared purpose centred on helping individuals and groups make well-informed decisions about interventions, primarily by improving the quality of communication of scientific information. We quickly recognised the value of working together, and the need to enable individuals to think critically about claims. At no point was the goal of developing a shared framework to support the use of evidence for informed choices challenged.
Despite the breadth of backgrounds and experiences within the group, several of us had taken part in cross-disciplinary work previously, and we self-selected to an extent in accepting the invitation to take part in this initiative. We were therefore each already convinced about the value of working across traditional boundaries. Discussions at the meeting were rich, varied and wide ranging. We agreed that across all fields, choices (when options are considered) and decisions (when one action is chosen) are wide ranging and differ from one another in degree, scale and nature. Discussions across fields about the need for a common framework helped us to think more broadly about the nature of the informed choices people make, and the kinds of claims with which they are faced. We learned from both our commonalities and our differences.
We also recognised the need to consider all levels of decision makers, from individuals making choices for themselves and their families, to health-care professionals and government policymakers making choices on behalf of individuals and whole populations. We recognised the extent to which public policy in its broadest sense (that is, policy that affects the public, whatever its source) is often shaped by a wide range of considerations. This included discussion of how policymakers understand public debate. We acknowledged that their decisions are shaped by perceived public support, official considerations, and a complex interaction between what the evidence suggests and how much policymakers are willing to risk political failure in following the evidence. We also perceived barriers for researchers/scientists in whether they inform policy, and how that occurs. We discussed the role that misinformation can play, and agreed on the importance of tackling it at all levels.
Discussions across fields helped us to see how the framework we were developing could be useful beyond health choices. For example, we discussed the strength of the available evidence and agreed that small-scale studies, which are common across disciplines, are part of the problem behind many misleading claims. We found examples from each of our fields, for example, a systematic review of management research that found a predominance of small studies with low internal validity (Barends et al., 2014).
The commonalities across fields not only related to experiences and ideas, but also to the degree of applicability of the framework across those fields. We all agreed that a mechanistic hypothesis, although important, is not on its own enough to justify implementation of an intervention until it has been tested. A key observation was that interventions, implemented in good faith based on a theory that something should work, can lead to wasted resources and even to harm. We did nevertheless observe that the development of theory, often using tools such as logic models, is a common precursor to any experiment, with differing levels of importance in different fields.
We observed that informed choices are not made in a power vacuum. The power and influence behind some ideas are forceful. For example, microloans to people living in poverty, with the intention of supporting them to generate income through small businesses, are usually provided at very high interest rates, and an overwhelming body of evidence suggests that this does more harm than good. Nevertheless, the desire to find solutions for poverty makes the intervention hugely attractive and very popular. Muhammad Yunus – considered by many as the founding father of microfinance – was awarded the Nobel Peace Prize in 2006. It is difficult to challenge accepted truths and change norms.
Other examples include the lack of global support for electric cars (Paine et al., 2006), the decades before the harms of smoking were accepted (Warner, 1991), and the limited action against the sugary drinks industry (Public Health England, 2019). Sometimes those with influence, whether big pharma, big donors or political parties, put their weight and their money behind an intervention, and it takes a lot to counter this. Most recently, we have observed persistence of the belief that hydroxychloroquine is an effective treatment for COVID-19, despite high-quality evidence to the contrary, following support for it from US President Donald Trump (Khuroo et al., 2020). A few public initiatives specifically challenge accepted truths, including Ask for Evidence (www.askforevidence.org). We agreed that all decision makers, including at national and international levels, have multiple factors to balance in decision making.
We found common ground when considering the important roles that the media (including social media) play in influencing the choices that people make, and explored how to counter the power of misinformation in all forms of media. Another shared challenge related to the power of commercial interests, and how well-resourced (but not necessarily evidence-based) organisations can use their financial weight to drive policies and interventions that are extremely difficult to challenge without considerable funding and effort.
The potential value of a generic framework of key concepts to inform choices was agreed, as reflected in a willingness to press on with understanding applications in different fields, and implications for the detailed sections of the framework (see more below).
How we identified and prioritised common problems for inclusion in the framework
When we considered adapting the original Informed Choices tool, which was developed for decisions in health care, into a framework for use across different fields, one of the main topics we discussed related to wording, the use of particular terminology and semantics, as opposed to differences in the concepts themselves. Colleagues in different disciplines use different jargon to characterise studies that focus on comparisons – randomised clinical trials, randomised social experiments, cluster-randomised trials, place-randomised trials. For example, in most disciplines, a randomised trial is referred to as an ‘experiment’, but in the domain of management, an experiment means ‘trying something out to see what happens’, and is a synonym for ‘pilot’. Nevertheless, it was understood that the phrases embody a common idea, that of fair comparisons in estimating outcomes. In another example, colleagues in each discipline alluded in varied ways to the tentativeness of evidence. We agreed that this concept of uncertainty is fundamental, and a reason for doing more than one study and for learning from mistakes.
Once we had aired, discussed and recognised the subtle differences in how we use words such as ‘trial’, ‘experiment’, ‘decision’, ‘policy’ and ‘harm’, we moved on to discussing the framework itself. This helped us to identify common factors that shape decisions, which in turn were included in the new cross-field framework.
We observed that across all fields, decision making involves trade-offs at several levels, balancing multiple benefits, harms and costs. We learned that in some fields it is routine to consider harms and costs (medicine, dentistry, veterinary medicine and international development, for example). In other fields, benefits, harms and costs might take longer to become apparent, and harms may not even be considered, for example, in education (Zhao, 2017). With some decisions it is necessary to balance benefits, harms and costs across multiple people, groups or populations of other species, for example, in environmental management. In some fields, such as planetary health, some people for whom this balance must be considered may not even have been born yet. Costs vary enormously across fields. They include both fiscal costs and opportunity costs, and they can be immediate or long term. In agriculture, for example, intensification through increased use of pesticides and/or fertilisers may meet increasing food production requirements, but can cause harm in the wider environment. In the longer term, there may also be cost implications, particularly to future generations, as damage to natural support services, such as soils or pollinators, can have economic consequences.
Opportunity costs vary according to context: what we have to give up in order to adopt a course of action can vary for each individual, and from group to group and place to place. In resource-poor settings, the opportunity cost of seemingly trivial actions can be enormous. Furthermore, we recognised that across fields, decisions can be made at many levels (personal, for family or friends, in a school or hospital, in a city, regionally, nationally or internationally), and the distribution of harms, benefits and costs varies with context. Who gains and who loses can sometimes be as important as maximising an outcome, if not more so. In evidence-based management, this is known as ‘stakeholder evidence’ (yet another term which means different things in different fields) – evidence that allows analysis of impact on different groups and introduces an ethical dimension. The need for fair comparisons to enable balancing of options cuts across fields and levels of decision being made.
Language commonly generates tension in all fields; for example, the contrast between warning language and neutral statements – ‘Beware false news’ versus ‘This is what an evidence-based claim looks like’. We agreed that the framework should contain statements that are as neutral as possible about the value of evidence, which could be adapted in different fields and for different uses. For example, the framework states: ‘Comparisons of interventions should be fair’ (Aronson et al., 2019: 304), rather than ‘Beware comparisons that are unfair’.
We developed the option for each field of enhancing the framework and sharing additional field-specific information. We agreed that claims are not made in a vacuum, and we discussed the importance of ethical issues. However, we decided not to include them in the framework, because they can be context specific. We agreed that irrespective of the field, there is a need for relevant examples of what happens when decision making is not evidence based, so that people will take notice. These have been included in field-specific sections of the project website (www.thatsaclaim.org).
The breadth of applicability of the framework
While we discussed how we might adapt the framework for different fields, there was never any suggestion that the framework would not be useful in any specific field. Rather, there was a lot of common ground across fields. Participants described cross-field work as a pleasure, reflecting a new way of working without disciplinary boundaries, engendering optimism. We felt reassured that we all speak the same broad language, and that the principles of evidence-informed decision making (and thus the framework that we developed) apply across all fields. To date, the project website includes field-specific enhancements for 8 of the 14 fields involved in the development of the framework (www.thatsaclaim.org).
Finally, there was a shared recognition that the framework is not an endpoint for any of our fields. Implementation of decisions is an essential aspect of informed choices, and is often lacking, whether we are discussing implementation of individual or policy choices. Policy choices in any field require implementation.
Cross-field, interdisciplinary approaches are possible, and the learning from them is hard to overestimate. With COVID-19 dominating many current policy debates, and the proliferation of related research requiring synthesis, there are calls for disciplinary breadth (Stewart et al., 2020; WHO, 2020b). The lessons from our work are highly relevant for anyone convening these forums. Agencies that aim to tackle other global priorities, from climate change to Black Lives Matter, all require cross-field engagement. We hope that in sharing our experiences, we can contribute a little to these engagements.
An increasing number of organisations and initiatives support the use of evidence in decision making, as well as seeking to address misinformation about claims. Fact-checking organisations, such as Africa Check, Ask for Evidence and iHealthFacts, are on the rise around the world. The potential to apply, reflect and collectively learn from cross-field engagements is considerable, and increasing in importance.
We know of many other cases in which agreement has been more difficult to achieve. In 2003, the International Campaign to Revitalise Academic Medicine (ICRAM), led by an interdisciplinary working party, accepted the challenge of reinventing academic medicine, but found the task too difficult (ICRAM, 2004). Some within the group suggested that they had not understood the problem (Clark, 2005). Relative to ICRAM, our focus was much more specific, and we began with the common ground of evidence synthesis in supporting decision making.
It has also been suggested that because disagreement on committees is rife, consensus in health care is reached only on ‘bland generalities that represent the lowest common denominator of debate and are embalmed as truths’ (Buetow et al., 1997: 269). However, we were not a ‘committee’ responsible for reaching a consensus, but had freedom to engage and to choose whether or not the resulting framework was useful for our own fields. We were also encouraged to adapt, and either broaden or narrow, the key concepts for our own fields. There was no pressure from a funder or other authority that required a consensus, or even an output.
We suspect that respect for one another also encouraged constructive dialogue. We consistently shared concrete examples from our work, and this paper reports just a few of those. We believe that voicing opinion alone would not have been sufficient to reach consensus, and that the examples enabled us to find common ground.
In conclusion, despite representing broad and varied interests, and belonging to traditionally siloed academic disciplines, we were able to reach agreement on a framework. Through the process of working together, we came to appreciate the breadth of applicability of the evidence-based approach. Readers who find the boundaries of traditional disciplines frustrating should be encouraged by our experience.