Why research is important for small and medium-sized enterprises within the educational technology sector
The educational technology (edtech) sector is a subsection of the global technology industry that is particularly focused on the design, development and scaling of digital products and services that address issues that relate to all aspects of human learning. This learning is lifelong in that it might concern young children’s language development, school-age children’s collaborative science skills, older children’s examination preparation, and all manner of workforce development within professional and vocational settings.
However, in many cases this community lacks knowledge, skills and experience to do the main task that will enable them to produce products and services that are both educationally and ethically sound. The task to make use of the existing research evidence in the learning sciences and associated research methods to enable them to validate the effectiveness and efficacy of their endeavours by designing and conducting their own studies is daunting for most. Access to such knowledge helps companies to demonstrate the impacts their products have on their users. Consequently, without such skills, they put themselves at risk of producing products that may not align with their intended learners’ needs and/or the needs of the educators who may be expected to adopt them.
UK Research and Innovation’s vision for public engagement with research is ‘to create a more prosperous, healthy and sustainable society. Creating opportunities for people to discuss, create and participate in research and innovation is an important way to achieve this’ (UKRI, 2019: 2). The EDUCATE project, which provides the context for this set of articles, was positioned firmly within this vision, as it sought to bring edtech enterprises and educational researchers closer together to work towards all the stated wider benefits. However, university institutional goals for such engagement activities are not always conducive to such initiatives, an issue highlighted by Burchell et al. (2017) in their review of the experiences and perspectives of UK researchers. Burchell and colleagues concluded that this was a ‘work in progress’ (ibid.: 199), and they suggest that targeting particular domains might be a promising approach. In the case of EDUCATE, the university made a substantial financial commitment to co-fund a large-scale project within the educational technology sector.
Furthermore, the third (electronics, telecommunications and computers) and fourth (internet of things, robotics, virtual reality and artificial intelligence) industrial revolutions are both underpinned by increasingly ubiquitous access to the internet via more portable, faster, higher-capacity devices. Consequently, around the world, there are thousands of entrepreneurs who are working to devise such products, and around 1,200 companies within the UK alone. These entrepreneurs have a wide range of prior experiences as academic researchers, educators in formal and informal settings, entrepreneurs from other sectors and, in some cases, as learners themselves.
The problem lies in the fact that those that develop edtech (edtech enterprises), those that research edtech (academia) and those that use edtech (users – teachers/learners/parents) operate in silos, and information regarding how to demonstrate impact, on what to demonstrate the impact, and for whom, is not accessible enough to create truly impactful educational technologies. Compounding the issue is that no theories exist to inform how to bring these three particular communities closer together. A construct, the golden triangle of evidence-informed edtech, was first put forward by Rose Luckin, Professor of Learner Centred Design at UCL, to fill this gap (Luckin, 2016; see Figure 1).
The academia–education axis of the golden triangle is well-served by different research and policy initiatives that aim to bridge academic research with practitioner communities (Coburn and Stein, 2010; see the Education Endowment Foundation in the UK, https://educationendowmentfoundation.org.uk/, and the What Works Clearinghouse in the USA, https://ies.ed.gov/ncee/wwc/). Furthermore, the established edtech industry has long-standing relationships with both academia and education. However, the prerequisite is always funding, which either comes from the businesses themselves or from traditional research funding routes, both of which exclude small to medium-sized businesses (SME, defined here as fewer than 250 employees or less than £5 million annual turnover).
The golden triangle of evidence-informed edtech implies that the more tightly connected the three communities are, the better the quality of evidence to support the ultimate common goal, which is to produce the best, most evidence-informed, impactful edtech enterprises that are underpinned by sustainable business models. Luckin devised the EDUCATE programme to address this gap for the edtech SME community.
The EDUCATE programme for edtech enterprises
The overall purpose of EDUCATE is to advance the efficacy of EdTech through an evidence-informed process by making the best research evidence and practice accessible for educators, researchers and technology developers. The project will enable developers and educators to integrate four sources of information: research evidence, local context, practitioner experience and judgement, and user values and preferences to further drive and inform EdTech design and implementations. (Cukurova et al., 2019: 495)
The main goal of EDUCATE is to enable companies to learn how to access and use existing research to inform how they can conduct their own research – both ethically and objectively – in ways that support them to grow and sustain their business. Furthermore, where any resulting research findings suggest that an initial educational concept is flawed, the enterprises should be open to ‘pivot’ to a more educationally sound concept. In the process, the companies are supported by the programme to overcome tensions that can arise when what appears to be a viable commercial edtech may not be sound from an educational or sociopsychological perspective.
During 2017–19, the EDUCATE programme team worked with 252 London-based companies in 12 cohorts of between 7 and 23 participating companies. Between one and three participants from each company accessed at least 12 hours of research training and mentoring over a six-month period. Additional business and product design workshops were also available. In brief, this research training and mentorship programme had the objectives to support the participants to:
understand the nature of evidence for educational technology: what it is; how to locate and interpret it; and why it is important?
learn how theories of change are useful within the design and development of educational technology, and how to operationalize these using a logic model approach?
devise and critique research questions as appropriate to their particular technology
develop a research proposal based on the revised research question(s)
gain a broad understanding of experimental and exploratory research methods and a more in-depth understanding of the methods appropriate to their proposed research.
A more detailed account of the programme design and the key activities undertaken by participants is provided in Cukurova et al. (2019).
Theoretical framing for the EDUCATE programme
Boundary objects (Star and Griesemer, 1989) grounded in activity theory play a key role in the EDUCATE programme. Star and Griesemer (ibid.) define these as the documents, concepts, artefacts and other items that reside at the boundary of intersecting communities. As Akkerman and Bakker (2011) note, learning at the boundaries cannot take place without human facilitation, and it is the quality of this communication and collaboration that is key to EDUCATE impacting positively on its participants by supporting them to achieve their respective goals. If each of the participating communities is perceived as a community of practice with shared working practices, the role of brokers is key to the transfer of information from one community of practice to another (Wenger, 1998). Hence the role of research mentors from the academic community to fulfil this brokering role became increasingly clear as the EDUCATE programme evolved. Through the iterations of the programme, the nature of this brokering role became more evident, as was the importance of productive relationships between the academics and the entrepreneurs.
EDUCATE has four main boundary objects:
the language of the programme
the UCL Lean platform – an online tool for research development and business planning
the logic model – the theory of change devised by each company for their edtech product or service
the research proposal – the ultimate goal of programme completion and the boundary object that earns participants an ‘Evidence-aware Edward’, the programme’s mark of research mindedness (a second ‘Evidence-applied Edward’ is earned once participants have conducted and reported their proposed research)
As already mentioned, a theoretical framing did not exist to guide the design of the EDUCATE programme, given the unique context of the project, although it has its foundations in activity theory. Consequently, to validate the golden triangle in practice, the research team followed a design-based research methodology to iterate and improve the programme’s resources and processes (Anderson and Shattuck 2012; Brown 1992). The goal of each cycle of research was to develop both theoretical and methodological knowledge. The overarching aim was for the EDUCATE team to get closer to a ‘best learning design’ to support the participating edtech SMEs to achieve a level of research mastery as evidenced by an appropriate research plan for their enterprises. Figure 2 shows the main theoretical and methodological focuses for each cycle.
Cycle 1 was unique in that it was the most exploratory round. The companies attended four two-hour taught sessions that included learning content at the level of a master’s in educational technology research methods course. This first cycle revealed a first set of challenges in relation to the programme language, or ontology, which was complex and firmly situated in the world of the researchers. The information was highly theoretical and ‘dense’, and edtech companies had issues translating the content to enable them to understand how it applied to them in the context of their edtech business. However, in this phase, the companies in this cohort also received the most attention from the academic mentors, as the mentors were also learning what was needed most in this process. Mentors often had to translate content from the research training programme into a language that was more relevant to the companies, and they spent time sourcing examples from the research literature that the enterprises could understand. Consequently, with the time constraints of the programme, there were inefficiencies on both sides, although the mutual learning that took place was a necessary part of the process to refine the programme further. Furthermore, in this cycle, the EDUCATE team sought to match each company with a subject matter expert from the academic research community in response to each company’s identified research focus, as revealed by the needs analysis process. However, as the programme scaled quickly, it became increasingly difficult and time-consuming to find suitable matches (it was a manual process), hence the programme was adapted to offer generic research support from a dedicated team of academic researchers.
Cycle 2 began with a review of the research training programme content and, founded on the experiences of the research mentors, a revised ontology became apparent, which was closer to the language of the edtech enterprises. The ontology emerged from programme interviews and mentoring sessions as the research team began to understand and capture the contexts, goals and challenges of the enterprises. This resulted in a more applied curriculum that included further case studies that brought the theoretical ideas to life, but also involved more practical activities in which participants were tasked to work on the boundary objects such as their logic models. The research mentoring team all attended and participated in the taught sessions to enable them to build relationships with their allocated companies, while also becoming increasingly knowledgeable about the research training content and its processes. Companies were paired with the most appropriate mentor based on the mentor’s past experience and expertise most relevant to the company’s research goals, so as not only to optimize experience but also to develop relationships such that the mentor acted as an extension of their teams. It was becoming evident that for the companies, working through the EDUCATE programme in teams would prove to be a necessary aspect of overcoming the challenges of the programme and successfully designing and implementing their research proposals such that it did not derail other aspects of their business (Moeini, 2020).
Cycle 3 concerned the further refinement of the EDUCATE programme to optimize the knowledge exchange processes within the available time. This included greater use of digital tools to support at-distance work – the research training materials and tasks had been digitized, and the research team was now proficient in supporting synchronous online video communication. Both the mentoring and the research training programme had been streamlined such that the programme was delivered towards an optimal efficiency. The final two cohorts engaged with the programme as three-day ‘bootcamps’ that combined the research training, practical tasks and research mentoring sessions in a condensed programme.
In summary, the lessons from this experience were three-fold:
The ontology must be accessible and useful for the companies, especially as the EDUCATE programme is a short-term intervention for the companies, who have limited time (and perhaps no need) to learn the new language of ‘educational research’.
The research training content needs to be ‘applied’ in its approach, whereby relevant theory is brought to life with real cases, and participants are given the opportunity to apply learnings and iterate their logic models and research proposals in situ with the support of mentors.
Research mentors are seen as extensions of company teams, and the conversations that take place become invaluable assets to the programme for two reasons. First, it ensures that participants develop correct understandings of research processes and negotiate an application of the research to business processes that is rigorous and sound. Without a consistent mentor relationship, it is unrealistic to expect edtech enterprises to deliver a sound logic model, let alone research proposal, within the six-month timescale. Second, the expertise that the research mentors developed through the hundreds of conversations with companies created an immense knowledge base for the project.
Research readiness in practice
Before companies were offered a place on the programme, the only criteria were that: the edtech product had an overarching educational goal for its users; there was a minimum viable product that could provide a context for research; and the company was able to assign time for one or two representatives to participate in the programme. This was because the programme was addressing a gap for educational technology SMEs that could otherwise not afford a research team and whose dynamic nature would make it more difficult to perform research activities than a larger, more established counterpart.
The work of Anissa Moeini (a PhD student conducting her research within the EDUCATE programme) has developed a new construct called the evidence-informed learning technology enterprise (ELTE), which acts as a framework for understanding the conditions that need to be in place within edtech enterprises to be successful on the EDUCATE programme (Moeini, 2020). ELTE proposes six sub-constructs (sense of purpose, research know-how, teamwork, leadership vision, learning culture and action orientation), developed through a process of co-design with the mentors on the EDUCATE programme in the first instance, and validated ‘in the wild’ (Hutchins, 1995) with EDUCATE companies and external participants. The ELTE framework can be used to gauge the success of the enterprise on the EDUCATE programme and, indeed, in general as a research-minded enterprise that has the conditions in place to successfully implement research activities in their enterprise.
The logic model
The central boundary object was indeed the logic model, a tool to support the companies to articulate how their product or service should be used to achieve the educational impacts for which it is being designed (W.K. Kellogg Foundation, 2004). Logic models have their roots in evaluation research, and serve to break down the processes and products of an intervention. For the EDUCATE companies, the creation of the logic model proved painful for many, as it required them to think hard and deeply about their edtech. Similarly, for the research team, it simultaneously served to reveal the most relevant form of research that might be possible for the company to devise, as it revealed the status of the edtech product development and the associated access to, and availability of, useful research data. For both communities, although the components of the logic model (inputs, activities, outputs, outcomes and impacts) are all widely understood words, their meanings were unique to each edtech.
Further research
The research data set created by the EDUCATE programme, which includes hundreds of logic models, research-industry conversations, research proposals and research reports is underpinning a number of further studies, including:
The evolution of the participants’ perspectives on the nature of educational research within the context of educational technology. This includes how participants developed a wider view of the different forms of evidence, and how to discern what types of evidence are needed for the different development stages of the innovation.
An analysis of the type of educational research (exploratory, experimental and so on) conducted by companies and the related methodologies (survey, interview, observation and so on) alongside the different development stages of the innovation.
Dynamic visualizations of the edtech ontology that has been developed through the project for use by community members to support and enhance academic–industry–practitioner collaborations.
What we have learned
EDUCATE began as an exploratory project that had not been done before. Consequently, there were no relevant theories to inform how best to design the programme. The experience and expertise of the team designing the training suggested that a descriptive theory of learning informed by activity theory would form a robust foundation, informed by the careful selection of boundary objects. However, the exact nature of the activity of the programme and its actors was something that the programme team played a key role in determining. Similarly, the choice of boundary objects, and the evolution of these through the programme, was fundamental to the evolution of the programme.
The role of the ontology was critical, as the research mentoring team learned to understand the more business- and product-focused language of ‘edtech’ and the companies simultaneously learned to understand the most relevant educational research terminology for them.
About the collection of articles
The collection of articles below highlights how the engagement of the SMEs with academics enabled them to further their business development, project development and research goals. Critical to the EDUCATE approach is that the academics did not conduct the research on behalf of, or in collaboration with, the SMEs. In all cases the academic’s role was to support the entrepreneur to make sense of the research training by providing feedback on their emerging logic models and research proposals. This required the academic to become familiar with both the current stage of design of the educational product/service and the prevailing size and scale of the business. With this knowledge, it was possible for the academic to support the proposed research design by aligning the research methodologies to capitalize on existing educational data, or suggest data collection methods that drew on the current human/financial resources that were available within the business.
This focus on developing the research capabilities of the business resulted in the locus of control remaining with the entrepreneurs, which is exemplified within the collection of articles. However, the academic researchers provided important guidance on the positioning of educational theory and the appropriateness of the research design for the resulting studies conducted by the companies. The article by De Ossorno Garcia and Doyle illuminates the nature of the mentor–mentee processes, as experienced by one of the academic–industry collaborations within EDUCATE.
The special collection includes two articles – one by Morrison, Blake, Embleton-Smith, Gosiewski and Zvesper, the other by Rogers and Weatherby – that have been co-authored by edtech entrepreneurs with their research mentors. Both articles report the outcomes of their research, and also reflect on the nature of the industry–academic relationship and its impacts on the research process.
This special section is preceded by an enlightening article by the EDUCATE programme director, Professor Rose Luckin. In this, Rose provides her personal rationale of the motivations for EDUCATE, the principles underpinning the programme design and its ambitions for the future.
The impacts of EDUCATE are still being researched and reported, and further articles will appear in future issues of Research for All. However, the emerging evidence suggests that it is supporting SMEs to build their conceptual understanding of educational research by enabling them to embed a ‘research mindset’ within their companies through the more systematic collection and reporting of research evidence. Furthermore, the programme is strengthening research capacity within the business ecosystem through its resulting collaborations and industry engagement events. Finally, the instrumental impacts of the programme are evident within UK national edtech policy (Department for Education, 2019).