Background
Persons living with various forms of disabilities are part of every society: estimates are that approximately 15 per cent of the world’s population are living with disabilities (World Bank, 2021). Of these, about 80 per cent live in developing countries in the Global South, including 80 million people in Africa (Vesper, 2019). Although it is difficult to quantify the challenges faced by persons with disabilities (PWD) due to incomplete data, we know that children with disabilities account for a large proportion of those who do not complete primary education in developing countries (UNICEF, 2022).
Recently, there has been growing attention paid to the use of research evidence to inform policy and practice for all, not only increasing transparency and accountability in how public services are designed and delivered, but also increasing their potential for effectiveness, and reducing the likelihood of wasting scarce public funds on ineffective policies (Stewart, 2019; White, 2019). Evidence-informed policy therefore has the potential to improve the lives of disadvantaged minority groups, including PWD (McAuley et al., 2006; Pecora et al., 2018). An argument can be made that efforts to increase the use of evidence in policy and practice for those with disabilities will only be meaningful if PWD can be part of the process, from the scoping of issues that affect them, and the generation of research about them, to the identification and integration of evidence for policy and practice (Read and Maslin-Prothero, 2011; Walmsley, 2001).
Searchable online databases of research are one of the most common sources of evidence for policy (Halladay et al., 2015). To be inclusive of PWD and the issues they face, the online databases themselves, and the resources within them, therefore need to be accessible. This accessibility must include the ability of PWD to reach, obtain, understand and use the resources. Despite this, research evidence is still not fully accessible to some PWD (Cockburn et al., 2014). For example, a visually impaired person will not be able to understand a figure within a publication or manuscript that does not have a description using alternative text (alt text). Producing research that considers disability, but then publishing it in formats which are not compliant with disability standards limits the usefulness of the research, because it limits PWD from accessing and making use of the research findings.
Lack of understanding of the needs of PWD, and limited services to support them, can prevent social inclusion and lead to isolation (Vanderschuren and Nnene, 2021). To some extent, the internet of things can now offer PWD the assistance and support system required to achieve a good quality of life and social inclusion, but this assistance is only possible if they live in places where these devices, systems and connections are well supported (Domingo, 2012). The potential to be free of reliance on others in daily life, and the right to have equal opportunities in utilising all basic related products and essential services, is an important consideration for those with disabilities (Panagiotopoulou and Tsirintani, 2020). A range of technologies are available: for example, people with vision impairments can read websites using screen readers that read text aloud from a computer. The lived realities of a PWD are increasingly recognised as important within research (Aldersey and Assumani, 2015), and issues of equity are acknowledged as essential by those who advocate evidence-informed decision making (Rios et al., 2016). However, the rhetoric is only meaningful if the necessary changes are made to enable PWD to play an active role within the evidence ecosystem. For this meaningful engagement to happen, those with disabilities must be able to visit evidence websites, search online databases, download evidence, resources and content, and understand the content, including figures and graphs.
Accessibility, usability and inclusion are closely related aspects in creating a website that works for everyone, and there are guideline standards for web accessibility for all, such as the Web Content Accessibility Guidelines (WCAG), which is the international standard ISO/IEC 40500 (W3C Web Accessibility Initiative, 2019). Most designers, developers and editors do not pay attention to all requirements for accessibility for all, and, as a result, the inclusion of PWD is often not achieved (W3C Web Accessibility Initiative, 2019). Accessibility requires several components of a website to work together, including: (1) web content, including text, images and multimedia, as well as any markup code and scripts; (2) authoring tools, including the software or services that people use to produce web content, including code editors, document conversion tools, blogs and database scripts; and (3) the user agent, which is the software that people use to access web content, including desktop graphical browsers, voice browsers, mobile phone browsers, multimedia players, plug-ins and some assistive technologies (W3C Web Accessibility Initiative, 2019). By assessing these elements of a website, we can assess its accessibility. In 2019, the World Health Organization (WHO) launched an inclusive programme to promote Global Cooperation on Assistive Technology, and to encourage basic services for persons living with disabilities (Layton and Borg, 2019; WHO, 2018). These services include assistive products which help to maintain or improve an individual’s functioning and independence, thereby promoting their well-being (Layton and Borg, 2019). However, research suggests that only 10 per cent of people in need of assistive technology products worldwide have access to them (Boot et al., 2018). This gap is particularly evident in low- and middle-income countries, where such resources are scarce. There is a critical need for inclusive practices when designing accessible websites to ensure social participation and equal opportunities (Gamache et al., 2019). For example, a study by Yuh et al. (2020) conducted in Cameroon on people living with disabilities using assistive software revealed that public policies and political processes for the rights and needs of PWD have been developed, inspired by both the United Nations Convention on the Rights of Persons with Disabilities and the Sustainable Development Goals. Nevertheless, these policies are not implemented due to poor implementation strategies, lack of understanding, and poor participation of PWD in the political process (Yuh et al., 2020).
International laws on the protection of the rights of PWD guarantee their socio-economic integration, including access to information to make informed choices (UK Government, 2010). This further strengthens the case for inclusive practices when designing evidence websites for ensuring social participation and equal opportunities (Gamache et al., 2019; O’Young et al., 2019). The monitoring and evaluation of evidence databases is therefore important to assess accessibility for PWD, and to allow for improvements that will enable their full participation in the evidence ecosystem. The purpose of this article is to report on a process that can be used to monitor and evaluate evidence databases to encourage improvements in website accessibility.
Methodology
We set out to answer two research questions: (1) How accessible are online evidence databases for people with disabilities, as defined by the WCAG?; and (2) How accessible are resources on evidence databases for people with disabilities, as measured by the WCAG? In doing so, we aimed to demonstrate a process for monitoring and evaluating evidence databases to drive improvements for PWD.
Study design and sampling technique
A cross-section of five online databases of research evidence was selected for assessment through consensus by two of the authors using a purposive sampling technique. The databases were selected based on ease of access, global recognition, and common use by a specialist evidence centre (Effective Basic Services [eBASE] Africa). The names of the selected databases were anonymised, and they are referred to below as A, B, C, D and E. All five databases were among the most commonly promoted in the evidence-informed decision-making field.
Data collection
Data were collected using two online web accessibility evaluation tools by two of the authors working independently. These evaluation tools were selected based on the following criteria: they had to be open access, make use of the WCAG, and have global recognition. These criteria were recommended during a presentation at the Partnership for Inclusive Research and Learning (PIRL) Institute held in Bafoussam, Cameroon in 2019. The chosen tools were AChecker (www.achecker.com) and WebAccessibility (www.webaccessibility.com). The tools were user-friendly, and they provided an opportunity for a uniform resource locator (URL) link to be submitted and assessed, and for the provision of a stepwise compliance score, along with a list of actionable recommendations. The two tools provided somewhat different information about the accessibility of the sites they assessed.
Data analysis
Our evaluation was conducted using the two-level analogy of a dinner, where the invitation to the dinner is the first level of access, and the provision of a seat at the table is the second level. The invitation to dinner was equated with access to an online database, and it was assessed using AChecker and WebAccessibility. The seat at the table was equated with access to the resources within the online database, and it was assessed by applying a checklist based on WCAG criteria.
First-level evaluation: the invitation to dinner
First, we described the five online evidence databases using some basic descriptive criteria, including their geographical scope, content, language, number of visitors, audience and profile in the evidence field.
Access to online databases (the first-level assessment, or the invitation to dinner) was then assessed using both AChecker and WebAccessibility. AChecker allowed us to assess Hypertext Markup Language (HTML) for accessibility by inputting the location of the web page, uploading an HTML file, or pasting the complete HTML source code from the web page. A copied URL link of each online database’s home page was pasted on AChecker, and a check was run. AChecker reported on three criteria: known problems, likely problems and potential problems. Known problems are problems that have been identified with certainty. These are problems that present accessibility barriers that must be addressed, and modifications that must be made to the page to fix these problems. Likely problems are problems that have been identified as probable barriers, and which require a human to decide whether it is a problem and will need modification of the page to fix the problem. Last, potential problems are problems which AChecker cannot identify, and which require a human decision to assess, and then perhaps modify the page to fix the problem. In most cases of likely and potential problems, the human assessor needs to clarify whether the problems identified are present.
The second tool used during the first-level assessment was WebAccessibility. The process used was the same as for AChecker. This site uses two criteria: the percentage compliance with best practice, where results portray how much of the web page adheres to practices that follow WCAG recommendations; and the number of best practice instances where WCAG recommendations have been followed.
Once each website had been assessed using both tools, a comparative analysis of results was made. Our approach is mapped out in Figure 1. Following discussions, not all the outputs from the tools were included in our final analysis. An initial comparison was made of the results from both of the first-phase tools (AChecker and WebAccessibility), and, after a series of team discussions about what the results could really tell us, we settled on using percentage compliance with best practices as the cut-off point for access to the database site. The AChecker analysis outputs were therefore not included in our final judgement about the five websites. This decision was made because known/likely or potential problems did not portray explicitly what the problem was and how to fix it, nor did they provide a narrative to compare. Counting instances of best practice just results in a number. These numbers did not measure or compare resources, so we also dropped this indicator during our analysis.
Second-level evaluation: the invitation to the table
Access to resources or contents (the second level of evaluation, or the invitation to sit at the table) was assessed using WCAG criteria. From each of the five online evidence databases, five evidence-based resources published between 2015 and 2020 were downloaded. Resources or contents included manuscripts and publications of systematic reviews and other supplementary documents. This created a sample of 25 resources which were selected at random. The choice to include resources published between 2015 and 2020 was pragmatic, based on the study period. All resources, whether a systematic review, a research article or a report, were assessed using the same checklist based on the WCAG criteria.
Results from the 25 resources generated a raw data set using Microsoft Excel 2013, and they were analysed using the WCAG criteria. The data on the collected resources were analysed in relation to the 16 indicators in the WCAG; 8 categories were judged to be applicable, and 8 were not applicable. The applicable 8 were: alternative text, multimedia presentation, colour, saver-side image maps, table headings, complex tables, scripts and forms.
Results
Our initial description of the five online evidence databases is presented in Table 1. All five had a global scope, were in English, and had a high profile in the evidence field. Three focused on health content, one on education content, and one on both.
Characteristics of selected databases
| Database | Geographical scope | Content | Language | Number of visitors | Nature (online/paper/both) | Audience | Profile in the evidence field (high/moderate/low) |
|---|---|---|---|---|---|---|---|
| A | Global | Systematic reviews, health, education | English | Not available | Both | WHO, World Bank, development agencies, government | High |
| B | Global | Health | English | Not available | Both | WHO, World Bank, development agencies, government | High |
| C | Global | Health | English | Not available | Both | WHO, World Bank, development agencies, government | High |
| D | Global | Education | English | Not available | Both | WHO, World Bank, development agencies, government | High |
| E | Global | Impact evaluations, health, systematic review | English | Not available | Both | WHO, World Bank, development agencies, government | High |
The five home pages of the respective online evidence databases of research scored an average of 78.6% (range: 73% to 87%) compliance with best practice according to WebAccessiblity. While this result falls below the ideal of 100% compliance, it does indicate that all five databases were somewhat accessible for persons with disabilities. The 25 resources selected from the 5 online databases scored an average of 52.5% (range: 12.5% to 70%) using the eight WCAG indicators. These results are illustrated in Figure 2, which clearly shows the contrast between the online databases and the resources within them.
For all the databases, the resources they contained were less accessible than the databases themselves. Furthermore, in one case, the resources were extremely inaccessible, scoring only 12.5% on the WCAG scale. Our results show how resources failed completely on some indicators, such as colour, and were extremely poor when assessed for alt text and server-side maps, while they scored consistently highly on other indicators, in particular, multimedia presentation. Figure 3 represents compliance with accessibility across resources in each database. The databases are shown in different colours. The indicators by which they are assessed were: alt text, multimedia presentation, colour, server-side maps, table headings, complex tables, scripts and forms.
While Database B provided good access for those with disabilities, the resources within it were judged to have the lowest scores. Only Databases C and D recorded an adequate score for alt text. The other three sites scored zero, implying that no efforts were made to provide alt text.
Figure 4 summarises the general analyses of the data sets based on the eight applicable WCAG indicators. All the resources from the databases have an aggregate score of 12% for alt text, scored 100% for multimedia representation, and all scored 0% for colour.
All the databases scored 100% on multimedia presentation. This means that all the multimedia items on every resource were accessible to PWD. No database made any effort to check the colour contrast on their resources, as indicated by all of them scoring 0% on colour. For server-side image maps, Databases B and C scored up to 20%. For table headings, all databases except Database B made an effort to use the expected formatting for their headers to be accessible to PWD.
Our experience of applying these tools also allows us to conclude that, while the results generated from our preliminary AChecker analysis were not informative, both our analysis using the WebAccessibility tool and our analysis using the checklist that we developed from eight of the WCAG criteria were helpful in assessing the accessibility of these online evidence resources.
All in all, we conclude that while the online databases were fairly accessible, the resources within them varied widely in terms of accessibility, and tended not to meet the guidelines used in the assessment. Our findings suggest that PWD are invited to dinner, but not invited to the table. The implications of these results are discussed in more detail below.
Study limitations
We selected five online evidence databases from an unknown large number of online evidence databases available internationally. The purposive sampling technique used to identify the five databases means that our sample was not fully representative of all evidence databases. Hence, the findings from this study cannot be generalised to all evidence databases. Nevertheless, even finding that just five commonly used evidence databases fall short of international accessibility standards is cause for concern. The evaluation tools used for the analysis of the study were free versions of the applications, and therefore there were restrictions on certain features. Using the more comprehensive versions that require payment might have yielded different results. However, the results obtained with the free versions of the analysis tools still provided useful evaluative information to guide the improvement of online databases to enable accessibility of research evidence for all. Our process can also inform future evaluations that could be conducted by many evidence website developers and organisations. A more in-depth study using WCAG standards evaluation software with more features, as well as more comprehensive consideration of all online evidence databases, would generate more robust feedback to guide improvements. However, this recommendation does not limit the legitimacy of our findings. It merely suggests that we may only be scratching the surface of the issues regarding the accessibility of research evidence.
Discussion and conclusion
Due to the digital revolution, most resources from research evidence are widely available in digital forms. This increased availability highlights the importance of making digital resources accessible to all. Based on our online evaluation of the accessibility of online databases of research evidence to PWD, the data show how PWD have fairly good access to online databases, yet when they get further into the sites, they begin to experience challenges in accessing the resources contained within them. We can see an inconsistency in the consideration of accessibility for PWD. These results are a clear indication of how users are ‘invited to dinner, but not to the table’. Our findings of the challenges faced by PWD are in line with a related systematic review study (Bernard et al., 2016) which indicates that PWD faced challenges when accessing web content. Similarly, a study conducted by Adam and Kreps (2009) indicates that some websites were very complex for PWD, making access to databases very challenging. Access to, and consideration of, the best available evidence in decision making has great potential for improved livelihoods and equal opportunities for all (Garg et al., 2020; White, 2019). Limiting access to research evidence for those with disabilities therefore not only suggests that research is not in fact ‘for all’, but also limits the scope of evidence-informed decision making to serve the needs of PWD.
The findings from this study provide some guidance for editors, authors, policy makers and journals. Specifically, we have identified the following steps forward that could be taken.
Towards achieving inclusion and the United Nations Sustainable Development Goals: Given the current global agenda for disability inclusion through the Sustainable Development Goals 2015–30, it is essential to increase understanding of the situation of PWD, while further improving their accessibility to evidence-resource websites. Feedback evaluation on the level of participation and access of PWD to evidence websites is important for both the planning and monitoring of disability-inclusive policies and programmes (Foti et al., 2017). Our findings show that there is a need to improve web content accessibility for PWD. Addressing these needs, aimed at improving the quality of life of PWD, is one of the objectives of the both the Campbell Collaboration’s Disability Coordinating Group (Campbell Collaboration, 2023) and Cochrane’s Equity Working Group (Cochrane Methods Equity, 2023). All the envisaged long-term goals for inclusive development of PWD may fall short of reaching their targets, if resources are not equitable for them to make informed choices with the best available evidence (Welch et al., 2017).
Promoting equity through the use of WCAG: While we support the activities within several evidence collaborations, including the Campbell Collaboration through the Campbell Disability Coordinating Group, and Cochrane through the Cochrane Equity Working Group, and we stand with the equity extensions within the CONSORT and STROBE statements (Cochrane Methods Equity, 2023; Welch et al., 2017), our findings suggest that such promotion of equity and diversity may fall short of reaching its target, if attention to using the WCAG is not included. We therefore encourage advocates of evidence use to also become advocates for the use of the WCAG, increasing the potential for accessibility of evidence for more people. Each of us can lobby the editors and curators of research databases and journals, urging them to improve not only their own websites, but also to ensure that there is content accessibility, allowing for more inclusion in the evidence-based train. Journal editors and publishers can ensure that all authors comply with the WCAG when submitting manuscripts, perhaps through automated reviews within the online submission processes.
Alignment with our eBASE perspective: eBASE’s perspectives are geared towards promoting equity and inclusion for PWD and underserved populations, evidence generation and synthesis to inform policies, and both innovation and best practices within the evidence ecosystem. Data on web content for PWD is inadequate, which makes it difficult to identify challenges faced by PWD. Providing data on web accessibility, and promoting web content accessibility for PWD, is relevant to our work within the evidence ecosystem and, by definition, our effort towards leaving no one behind the evidence-based practice train. Adequate policies and guidelines on digital content to improve accessibility, equity and inclusiveness for PWD are strongly recommended.
Call to action for editors and authors: There is an opportunity for journal editors and database curators to improve access to resources by requiring all authors to comply with WCAG criteria as a condition of publication of research, and inclusion of such research in online databases of evidence. In addition, we recommend that editors provide adequate policies and guidelines on digital content to improve accessibility, equity and inclusiveness for PWD.
The novelty of this article is based on evidence synthesis through web content accessibility for PWD within five evidence databases. The results cannot be generalised for all databases, but findings are nonetheless useful to inform policies and practice for equity and inclusion, and guideline recommendations for web content. This article provides valuable insights into the digital content accessibility complexities faced by PWD, adding to the evidence base for best-practice guidance for websites and journal editors. The article highlights existing conceptual thinking around disability, and the importance of incorporating this thinking into discussions and practices for well-informed decision making. The eBASE Africa team and collaborators continue to work to provide support to increase access to evidence resources globally.