Background
Social pedagogy is values-led. It seeks to create environments and practices that support well-being and inclusion as well as learning. Social pedagogy is a growing approach to service and staff development in the UK, particularly in social care (Kirkwood et al., 2019; Petrie, 2013; ThemPra, 2015a). This makes it important to evaluate and reflect on the contribution that programmes informed by social pedagogy make.
The need for cultures to be supportive of holistic learning within social care is clear. We recognise that there is no universally agreed definition of social care (Cameron et al., 2021) and we use the term to refer to organisations that provide non-medical care and support to people to live a ‘life on equal terms’ (Beresford, 2020, p. 2). This can include a diverse range of practical and emotional support to engage in activities, support with personal care, support for families and young people, and community development. In the UK, local authority social care services are required to demonstrate how evidence has informed their policy and practice (Lamont et al., 2020) and there are similar expectations in other economically developed countries (Head, 2010; Rycroft-Malone, 2008). These requirements assume that multiple forms of evidence can enhance practice, leading to better care and well-being. It is also hoped that evidence will elucidate the questions to be addressed in further research and practice development.
Furthermore, there is a clear need for new initiatives to enhance learning and development cultures within social care. The challenges of engaging social care staff in working with evidence are well documented (Boote et al., 2010; Choi et al., 2005; Ghate and Hood, 2019). Resource constraints are often cited (Lamont et al., 2020; Orme and Powell, 2007), but challenges exist in several domains. There is limited infrastructure both within (Seddon et al., 2021) and outside social care organisations to support evidence work (Arnold, 2006; Inoue et al., 2017) and inconsistent approaches to evidence governance (Boody and Oliver, 2010). There can be a gap between the foci of academic research and the evidence that practitioners want (Huxley, 2009). In addition, there are workforce shortages (Lamont et al., 2020) and a dearth of evidence skills in the workforce (Orme and Powell, 2007), issues that are compounded by the complexity of undertaking research in social care (Ghate and Hood, 2019; McLaughlin, 2012). Moreover, there are few career pathways for practitioners interested in evidence work (Gazeley et al., 2019).
Historically, initiatives to foster evidence work have focused predominantly on methods rather than processes and evidence has often been conceptualised as meaning only research, without consideration of other forms of knowledge. In contrast, the programme Developing Evidence Enriched Practice (DEEP – www.deepcymru.org) emphasises process rather than discrete methods. It also emphasises social justice values and the social construction of knowledge. Informed by social pedagogy principles and co-production, it promotes the use, synthesis and contextualisation of diverse forms of evidence to achieve a better life for people and communities and those working in social care. Evaluation (observing and reflecting for ongoing improvement) is a core component of social pedagogy practice and given the distinctiveness of DEEP in social care, there was an impetus to learn from what DEEP had contributed. Before presenting the evaluation, we will share more about the programme and the context in which it operated.
Consistent with the values and practices in social pedagogy as referenced in Cameron et al. (2021), DEEP can be characterised as a holistic and values-based approach to learning and development based on story and dialogue. DEEP emerged from a participatory action research project (Andrews et al., 2015), which showed the value of thinking about evidence in more complex ways, that is, evidence as inclusive of lived experience, practitioner wisdom and knowledge generated through research. The project also highlighted the practical dilemmas of creating the learning environments where diverse forms of knowledge can be drawn together, especially in organisations imbued with managerial values and practices. Creating these learning spaces requires respecting how people learn and an openness to critically thinking, talking and sharing across epistemic divides. DEEP draws on the principles of social pedagogy to support the use of evidence in an ongoing and dynamic educational process.
The initial Joseph Rowntree Foundation-funded DEEP project (Andrews et al., 2015) across six sites in Wales and Scotland started with a commitment to the collective exploration and application of evidence involving older people, unpaid carers, practitioners, researchers and managers. From the outset, it was clear that relationships and dialogue-learning would play a key part in the project’s success and connections were made with oracy experts in the University of Cambridge.
Building on the success of the initial DEEP project, connections were later made with social pedagogy through a serendipitous meeting between the principal investigator and Gabriel Eichsteller in ThemPra in 2016. At that time, the Welsh government had invested further funding to support the national roll-out of the DEEP approach – the challenge being to articulate what this ‘approach’ was. This was where the synergy between DEEP and social pedagogy was recognised. Both are approaches rather than methods and, as such, need support with articulation. The diamond model provided a helpful framework to help articulate the DEEP approach. This model was incorporated into the eight DEEP principles and the DEEP training curriculum, which is framed around a gardening metaphor:
prepare the ground (creating good learning environments through applying social pedagogy principles)
gather and sow the seeds (of diverse types of evidence)
grow the garden (using caring dialogic-learning methods).
The context in which DEEP operates
The DEEP programme operates at a national level within complex systems in Wales. Wales is a devolved UK nation where social care is delivered by a range of public, private and not-for-profit providers. Social pedagogy is not widely known or applied in Wales and yet there are strong Welsh government policy imperatives around relational mutuality, and collective learning and development as outlined in the Social Services and Well-being (Wales) Act 2014 (Welsh Government, 2014) and the Well-being of Future Generations (Wales) Act 2015 (Welsh Government, 2015). The Welsh government’s commitment to this can be seen in the extension of funding for DEEP from 2023 to 2025 to work in partnership with Social Care Wales (the national social care workforce regulation and service improvement agency in Wales) to develop a strategic approach to DEEP across Wales.
As DEEP embarked on its learning/evaluation journey, members of its steering group, which comprised practitioners, academics and interested citizens, identified enablers and challenges for the programme (see Table 1). They not only highlighted examples of positive social care practice in Wales, but also noted organisational practices that pull in a different direction to the values of social pedagogy. The evaluation period (2020–3) was also a time of significant strain on social care services in the UK due to the Covid-19 pandemic and the cost-of-living crisis contributing to workforce turnover, shortages and burnout (BBC, 2022; Senedd Research, 2021). As Cameron et al. (2021) have highlighted, although the policy context in Wales can be seen as supportive of social pedagogy approaches, there are systemic and institutional challenges in social care.
Contextual analysis for the DEEP programme.
| DEEP operating environment | |
|---|---|
| Enablers | Challenges |
| A clear DEEP programme vision | Short-term funding and decision-making |
| Legislative support for innovation | Innovative legislation not fully implemented |
| Examples of positive social care practice | The change process can be difficult |
Between 2020 and 2023 there were other UK initiatives in Wales supporting the generation and use of evidence in social care. These included the British Association of Social Work Practitioner Research Network and IMPACT – the UK centre for evidence implementation in adult social care, funded by the Economic and Social Research Council and the Health Foundation. DEEP was distinct from other initiatives in its combination of:
a foundation in the principles and practices of social pedagogy
a focus on dialogical tools (such as Exploratory Talk and Community of Enquiry)
an emphasis on relational and organisational learning and development processes.
DEEP was arguably also more inclusive of diverse social care stakeholders. A literature search undertaken by the authors found papers detailing capacity-building interventions in social care emanating from the US, Australia, Canada, New Zealand, Sweden and the UK (for examples, see Harper and Dickson, 2019; Lachini et al., 2016; Lunt et al., 2012), but none of these reached out to collectively and simultaneously include citizens, social care staff, providers, policymakers and researchers.
The DEEP programme
Programme overview
From 2015 DEEP was a work package in the Wales School for Social Care Research. Between 2020 and 2023 it was funded as a standalone programme by the Welsh government through Health and Care Research Wales (2022). It is currently funded by Social Care Wales. The 2020–3 programme had a full-time equivalent staff of 3.8.
The DEEP approach
Informed by social pedagogy, DEEP also draws on theoretical insights from community development, implementation science and research. DEEP aims to inform social care research and practice through multiple types of evidence. Consistent with social pedagogy, rather than constituting a precise method, the DEEP approach comprises five principle-based elements (see Table 2 and Andrews et al., 2020).
The five elements of the DEEP approach.
| Element | Explanation | Principles | Theoretical basis |
|---|---|---|---|
| Create enriched environments of care and learning | Safe, inclusive learning spaces where everyone feels welcomed, valued and able to contribute | Support well-being Start with what people know and find interesting Help people make sense of what they learn | The SENSES Framework (Nolan et al., 2006) Human-Centred Learning (Lowe and Plimmer, 2019) Social Pedagogy (Hatton, 2013) |
| Recognise and address structural obstacles | Address systemic issues and consider context when using evidence | Share leadership and decision-making Be mindful of context when using evidence | Applied phronesis (Flyvbjerg, 2001) Radical social work theory (Ferguson and Woodward, 2009) Complexity theory (Auspos and Cabaj, 2014; McMillan, 2008) Community development (Ife, 2018) |
| Value and use a range of evidence | Evidence from lived experience, practice, research and organisational knowledge | Share all sorts of knowledge | Knowledge democracy (Beresford, 2018; Hall and Tandon, 2017) Participatory democracy (Escobar, 2011) |
| Gather and present evidence in meaningful formats | Collect and share evidence in ways that engage the head and the heart | Use stories in learning | Storytelling and narrative-based learning (Bruner, 1991) Experience-based co-design (Bate and Robert, 2007) |
| Effectively talk and think together about evidence | Engage in dialogue to enhance learning and promote implementation | Be brave and challenge each other kindly | Social constructionism (Berger and Luckmann, 1966) Dialogue-learning and inter-think (Littleton and Mercer, 2013) |
A focus on strengths, assets and empowerment is integral in these elements. They seek to support well-being, foster meaningful relationships, create positive experiences and generate opportunities for holistic learning.
Programme aims and objectives
The 2020–3 programme aims were that the DEEP approach:
was clearly articulated and widely understood
was embedded in strategic initiatives in Wales
enhanced social care policy and practice
increased capacity in social care research
shared its evidence base.
Objectives were detailed in five work packages, as seen in Table 3.
Work packages and objectives.
| Work packages | Objectives |
|---|---|
| Evaluation | |
| Learning courses and DEEP catalyst role* | |
| Resources | |
| Social care priority areas | |
| Bridging research and practice |
* DEEP catalysts are people trained to use and promote the DEEP approach
Between 2020 and 2023 the programme team devised and delivered learning opportunities, held events on priority social care topics using methods such as Community of Enquiry and worked with individuals and organisations to help them use the DEEP approach in development projects. The programme team were active in supporting research and sat on several project steering groups as well as supporting research students to use aspects of the DEEP approach in their work. The team also engaged with the social care research infrastructure in Wales supporting capacity-building initiatives. For instance, the team contributed to the development of the Social Care Wales evidence offer. How DEEP operated was informed by recommendations in the research capacity building and the learning and development literature. The programme:
worked at different layers of the social care and research infrastructure (Huxley, 2009), working with colleagues who were policy leads, infrastructure staff, providers and practitioners
provided support in multiple ways (Donley and Moon, 2021; Harper and Dickson, 2019), including through experiential opportunities on learning courses, mentoring and facilitating peer support (Withington et al., 2020)
connected people (Rubin et al., 2016), focusing on establishing relationships (Fox and Hopkins, 2021), which included supporting research and practice development groups (RPDGs)
offered information and support that responded to ‘what mattered’ (Mignone et al., 2018) through learning courses and through working with individuals and organisations.
Evaluation method
The evaluation was designed with and supported by Matter of Focus. Matter of Focus is an evaluation approach based on contribution analysis (Matter of Focus, 2017). Contribution analysis is theory-based and seeks to explore whether an intervention has furthered noticeable outcomes and, if so, how (Mayne, 2012). Contribution analysis is a useful approach when interventions are multifaceted and work within complex systems (Matter of Focus, 2017). Contribution analysis is consistent with the values of social pedagogy, which emphasises learning and reflection rather than summative analyses that imply ‘endpoints’. The DEEP evaluation:
co-developed a theory of change with the programme steering group and Matter of Focus (see Table 4)
plotted a pathway illustrating how activities and outputs led to outcomes
evidenced the pathway
reviewed the evidence
synthesised the learning as the basis for further reflection.
DEEP theory of change.
| What we do | Who with | How they need to feel | What they learn and gain | What they do differently | What difference does this make? |
|---|---|---|---|---|---|
| Catalyse dialogue Support people to share their evidence, amplifying marginalised voices Pilot and adapt a range of story and dialogue-based methods Help people to use the DEEP approach | People who the DEEP programme can learn from People who want support to gather and/or use evidence Policymakers and national organisations in Wales | Safe, supported, valued and listened to Open to reflection and new ideas That engaging with the DEEP approach is a good use of time | A deeper valuing of different forms of evidence Skills and confidence in using the DEEP approach A greater understanding of their own context for using evidence | Use the DEEP approach in their role Champion the DEEP approach People and their experiences are put at the centre of decision-making | Practice, policy and research are informed by the DEEP approach The DEEP approach becomes widespread The DEEP approach has an accessible evidence base People in Wales experience better social care and improved well-being |
Swansea University College of Human and Health Sciences Research Ethics Committee (reference: 21072021) approved the data collection and analysis plan. All participants provided informed consent and were aged 18 or older. Evidence was collected from a range of stakeholders including citizens, practitioners, people working in regional roles, policy advisers and academics. Multiple forms of evidence were collected.
Exemplar cases
We captured the impacts achieved in exemplar engagements and identified facilitators and barriers to using the DEEP approach in practice. Impacts were captured through semi-structured online interviews. Each interview lasted approximately an hour and was recorded using the Microsoft Teams platform. Three exemplars were selected through team discussion:
a project conducted by a health board and a third-sector (charitable) organisation to develop prevention services (two interviews);
work in a local authority to develop policies to support unpaid carers (three interviews); and
a project in a charity to capture evidence of change (three interviews).
The key respondent in each exemplar was also invited to contribute evidence that illustrated the work being done and its impacts. In exemplars one and three, key respondents asked other staff members to provide short accounts of the positive effects that they had experienced, which were then shared anonymously with DEEP. In exemplar two, a poem that had been written as part of the work was shared.
The collated evidence was analysed using framework analysis (Ritchie and Spencer, 1994), which can accommodate different types of data and is appropriate when there is a pre-specified exploration lens, such as the DEEP theory of change (Srivastava and Thomson, 2009). The analysis involved five stages: familiarisation; applying the DEEP theory of change as the thematic framework; indexing; charting; and mapping and interpretation.
Questionnaire data
A range of questionnaires were employed. These were completed anonymously online or in hard copy and comprised Likert scale items and open-ended questions. The questionnaires asked about the respondent’s experience of engaging with DEEP and how they intended to use their learning. Following the DEEP learning course, a three-month follow-up questionnaire explored whether practice intentions had been implemented. This questionnaire also gathered feedback on the mentoring and peer support provided. We used descriptive statistics to investigate Likert-scale responses and content analysis to explore free text questions. Content analysis considers manifest meaning units and common content categories, and our analysis was informed by the approach detailed by Graneheim and Lundman (2004).
Documentary evidence
Documents included polices, guidance documents and standards, as well as programme outputs. Documents were often provided by the people we worked with. If a document was not in the public domain, we sought permission for citations.
Synthesis
OutNav (Matter of Focus, 2017) provided a platform for collating the data. Some sections of the DEEP theory of change were evidenced by a single data source (for example, questionnaire data) but most sections drew on multiple types of data. The synthesis considered data consistency and the extent of triangulation when determining the strength of evidence for each section of the theory of change.
Findings
The DEEP theory of change provides a structure for reporting the evaluation findings. Key learning and reflections are summarised in the discussion.
What DEEP did and with whom
Between April 2020 and December 2022, DEEP delivered 113 events about the DEEP approach and other topics that had arisen from the interests and needs of practitioners. Example events included a session on using the DEEP approach in parent advocacy work, a bitesize session on the DEEP principles and a workshop on participatory evaluation. Registrants included 1,131 practitioners, 670 managers, 29 policymakers, 71 researchers and 118 citizens.
During 2020–3, DEEP connected with 175 organisations. Geographically, these connections covered all regions of Wales and reached into England, Scotland, Northern Ireland and parts of Europe, as well as the USA and Indonesia. We engaged with 16 of the 22 local authorities in Wales. Key partners were national social care organisations in Wales, including the social care regulator and service inspectorate.
DEEP supported and mentored 53 DEEP catalysts, who attended a 10-week DEEP learning course. During 2020–3, it was challenging for social care staff to attend time-intensive learning opportunities and the attrition rate on this course was approximately 26 per cent. DEEP also supported at least 18 connections, where we linked together like-minded organisations and individuals, and two RPDGs.
How the people we worked with needed to feel
People needed to believe that engaging with the DEEP approach was worthwhile. Respondents who represented the range of stakeholders detailed above, indicated that DEEP events could be useful networking opportunities (mean rating 3.63 out of a maximum of 5). Comparing the number of returns with the number of registrants suggests the response rate to the events questionnaire was at least 19 per cent.
It was also beneficial if people engaged with an open mind. It is hard to evidence this, but we observed people being receptive to new ideas. For instance, an event co-host provided this unsolicited feedback: ‘in particular, your presentation into DEEP and the impact of stories has really made them think differently, which is exactly what we wanted.’
We investigated whether DEEP activities provided the elements for effective learning (security, belonging, continuity, purpose, achievement and significance) detailed in the Senses framework (Nolan et al., 2006). Mean ratings from events and the learning course are illustrated in Figure 1 and Figure 2.
What people learned and gained
The DEEP learning course covered the principles and methods of the DEEP approach. Attendees practised and gained experience in methods such as Most Significant Change (Davies and Dart, 2005) and Community of Enquiry (Muirhead, 2018), as well as understanding the DEEP approach’s underpinning principles, such as valuing all types of evidence. Bitesize sessions provided three-hour taster sessions in areas including the Senses framework, the DEEP principles and Community of Enquiry.
Bitesize sessions could improve confidence, as attested to in this unsolicited feedback: ‘This session gave me the ability and confidence to use exploratory talk to help understand the story more.’ After attending the learning course, some catalysts also said that they felt more confident using the DEEP approach:
I feel much more confident in some areas such that will help support the teaching and learning and the gathering of evidence. I feel the course has given me a better understanding of how research can be applied in practice. (Other role: assessor)
After engaging with DEEP, some people said that they could better appreciate the value of different forms of evidence. For example, a service manager reflected in exemplar one that they had learned the value of stories:
Yeah we, I think, we’ve always captured numbers so, this was perhaps going, you know, going down a different road in terms of evaluating a project … but I think we’ve seen it as a more meaningful way of capturing outcomes.
DEEP had also helped partners in exemplars one and two to develop a deeper understanding of their institutional context for using evidence. This example was provided by the project manager in exemplar one:
and then the conversations about, you know, that about how much wider those panels need to be, who we need to involve in them. Strategically it was something like a thing that started with health and social care but that has grown into a … this needs to be reported through to, you know, the health board internally. It needs to go locally up to the public services board cos that’s a wider membership … but they’ve got kind of similar engagement themes and health and social care themes, so taking it wider than just the people that were originally involved in those discussions.
What people did differently
Event attendees said they intended to change their practice informed by the DEEP approach. For example, respondents wrote about adopting the values of working in a caring way and using story telling. A frontline staff member said: ‘it has given me knowledge of how to work in a caring way with individuals who have dementia, and also information that I can begin to pass on to my staff team’ while another frontline staff member wrote: ‘I want to use story telling more in my work’.
Due to the discrete nature of DEEP events, we could not evidence whether these intentions were implemented. However, three months after the learning course, responses on the follow-up questionnaire indicated that catalysts were using aspects of the DEEP approach in their roles (11 per cent response rate). A catalyst who was a frontline staff member said, ‘I am currently completing a master’s course and I have used some of the techniques to describe how to explore the cultural climate within organisations.’ Another catalyst who identified themselves as being in an ‘other role’ commented that ‘my practice has changed as I now discuss the DEEP principles with more confidence to encourage others to think about how their actions impact on those they support’.
The DEEP approach encourages organisations to put citizens at the centre of decision-making. The exemplars provided evidence of this shift in decision-making processes. The project manager in exemplar one said, ‘Yeah and, you know, for the panels this time, you know, we’ve got community representatives as well … we’ve trained up service users and carers to be on there.’ While a member of the commissioning team interviewed in exemplar two commented: ‘I would believe that the principles of putting the individual at the centre of thinking has been pretty much adopted across the board.’ Furthermore, a researcher interviewed in exemplar three shared, ‘but the thinking and the sharing of ideas and that sort of approach is sort of been, Yeah, we’re trying to look at doing that a bit more.’
Event attendees also said that they intended to give more consideration to the views of people they work alongside. A frontline staff member intended to have ‘more discussions with residents and families’, and a manager wrote about ‘giving people a safe space to share their stories, which will assist in the development of services and giving those people participating a sense of purpose and the opportunity to develop services for those that follow them’. DEEP catalysts were able to apply this principle in their varied roles, with a researcher saying, ‘the course has changed my approach to teaching and supporting students, allowing students to use their experiences at the pace to design and plan learning’.
Event attendees gave a mean rating of 4.59 (out of 5) to the likelihood that they would share their learning with others. A lead carers officer interviewed in exemplar two commented: ‘I’ve referred some colleagues across Wales to … the DEEP process as a good way, a good way of working.’ Catalysts said they would champion the DEEP approach through embodying its principles. A catalyst who was a frontline staff member said: ‘I will lead by example I hope and offer what I have gained to others’, and a catalyst who worked as an assessor said: ‘rather than telling learners and colleagues about DEEP principles I will practice them.’
The difference made by these changes
DEEP achieved proximal impacts in policy, practice and research.
Policy
A DEEP team member helped create a commissioned set of recommendations for unpaid carer short break services in Wales (Carers Trust Wales, 2021). The Welsh government agreed to implement two of the four recommendations made and committed £9 million to the recommended short breaks fund.
Practice
DEEP informed three national guidelines in Wales. Guidance (co-written with partners) on personal outcomes focused recording (Social Care Wales, 2020) was well received:
Just a quick email to let you know that WG [Welsh Government] have included our ‘Friend not foe’ links into their qualitative evidence guidance and CIW [Care Inspectorate Wales] have agreed to endorse the resource … this is really down to this being a great piece of work that has been welcomed by the sector.
The exemplars provided evidence of more local practice change. The project manager interviewed in exemplar one talked about how ‘they’ve started at every carer’s group meeting … they read a Most Significant Change story … to remind people why they’re there’. Meanwhile, the commissioning team member interviewed in exemplar two said ‘that principle of engagement and, co-production, you know, we’ve recognised the benefits from that work we did with [DEEP] and we’re making sure that we continue to do it in other service areas’. In addition, the head of learning and impact interviewed in exemplar three shared that ‘we are at the moment using Most Significant Change … So, we’re currently collecting the stories … and we’ve been collecting stories from commissioners, staff, participants, family members and we’re still in the process.’
Improved citizen well-being
It was difficult to evidence the pathway from enhanced practice, policy and research to improved services and citizen well-being. However, we captured evidence that this could happen. A member of the commissioning team interviewed in exemplar two highlighted that working with the DEEP approach had contributed to a better specification for commissioning services for unpaid carers:
Exemplars one and two also provided insight into how practice change that resulted from using the DEEP approach could enhance citizen well-being. The lead carers officer interviewed in exemplar two said: ‘So, on the day there were carers that could be helped ... They’d identified that they needed help with a particular thing that was easily done on the day so there was an immediate result.’ In exemplar one, the community connector manager commented:Sustainability
Sustainability can encompass the programme itself continuing, the incorporation of learning into policy and everyday practice, a physical infrastructure, lasting resource and continuance of the benefits (Jones and Verity, 2021). By 2023 the DEEP approach was becoming embedded within strategic partner organisations, and aspects of the approach were included in two social care training and qualification syllabuses in Wales. Other foundations for sustainability are outlined in Table 5. The passage of time will evidence whether these foundations are sufficient to sustain the DEEP approach in Wales.
Foundations for sustaining the DEEP approach.
| Foundations of sustainability | Strategies | Outcomes |
|---|---|---|
| Incorporating the DEEP approach into policy and the ‘everyday’ | The DEEP approach included in training syllabuses DEEP catalyst role Contribution to policy and national guidance documents | Two training syllabuses reference the DEEP approach 53 DEEP catalysts working in Wales One policy and three guidance documents informed by the DEEP approach |
| Resources | DEEP learning course DEEP catalysts DEEP materials | Learning course (N: 4) and bitesize sessions (N: 37) provided with accompanying materials Catalysts share and model the DEEP approach |
| Continuing benefits | Inherent characteristics of the DEEP approach Creating critical mass | DEEP ‘teaches people to fish’ At least 18 like-minded individuals and organisations connected |
Discussion: learning and reflections
The collated evidence provides relatively strong evidence that the DEEP approach made proximal contributions. The programme increased people’s confidence in using the DEEP approach (though ongoing support was also important) and people identified a deepening in how they valued different types of evidence. Other capacity-building initiatives have reported similar findings, for example Mugabo et al. (2015) and Wenke and Mickan (2016). Others have expressed hope that approaches informed by social pedagogy would bring such impacts in social care in the UK. For instance, Hunter (2020, p. 8) proposed that social pedagogy could support the use of creative approaches with older adults by balancing ‘head, hands and hearts’. Increased confidence has also been reported as an outcome in other social pedagogy-informed initiatives (ThemPra, 2015b). There was weaker evidence that people gained a greater understanding about their context for using evidence. This might be an evaluation artefact as we did not ask about this on every questionnaire, but it was notable that DEEP catalysts did not identify this as an outcome.
The evaluation suggests that the DEEP programme motivated people to put their learning into practice, which could include promoting the approach to others and ensuring that citizens were central in decision-making. This outcome was undoubtedly enhanced by the synergy between the DEEP approach and social care policy in Wales. Unsolicited feedback provided evidence that some of these intentions had been actioned, but the low response rate on the questionnaire that was sent three months after the learning course limits our understanding of the practice changes made by DEEP catalysts.
Distal impacts were harder to evidence. There was strong evidence that the DEEP approach contributed to practice, policy and research as has been reported for other initiatives (for example, Harper and Dickson, 2019; Karlsson et al., 2008). We have evidence from the exemplars that the DEEP approach could support service improvement and enhance citizen well-being, but we cannot quantify how often this contribution occurred. Furthermore, although foundations for sustaining the DEEP approach were established, we cannot determine whether these foundations will be sufficient.
The evaluation context is likely to have moderated the programme impacts. The Health Protection (Coronavirus Restrictions) (Wales) Regulations 2020 prohibited face-to-face events during some of 2020–2023. Given the relational nature of the DEEP approach, it is possible that greater impact would have been captured if more face-to-face events had been held. Indeed, event attendees gave comparatively low ratings for networking opportunities.
Reflecting on the evaluation findings we conclude that the DEEP approach has put in place the foundations for successful engagement with different types of evidence. The DEEP programme created supportive learning environments, increased confidence and inspired people to take their learning into practice. A contributory factor to these impacts seemed to be the programme focus on relational and responsive processes rather than fixed and predetermined methods. The elements of DEEP and its principles seemed to resonate with many people working in social care. We observed that whenever social pedagogy principles were introduced through DEEP learning and development work, they were well received. Anecdotally they seemed to motivate and encourage people as they were an alternative to more methods-based learning approaches. The findings also highlighted that because DEEP is not a single method, like social pedagogy, it can be hard to communicate what it does. This suggests that incorporating more practice opportunities into our learning courses might further increase people’s confidence in using the approach.
Strengths and limitations of the evaluation
The theory of change that formed the basis for the evaluation was co-produced with the programme steering group. The evaluation covered a three-year period and incorporated multiple types of evidence. The contribution analysis approach recognised the multifaceted nature of the programme and the complex environment within which DEEP operated. The evaluation was not conducted independently, and to minimise bias:
the case exemplar data was collected and analysed by a team member not involved in the engagement
questionnaires were returned anonymously and some returns suggested areas for improvement, such as this suggestion from a frontline staff member following an event: ‘more participatory practice of the principles to help see it in action’
documentary evidence could be from independent sources
unsolicited feedback was included.
As DEEP events were often discrete, we have weaker evidence that the DEEP approach was put into practice, and unfortunately there was a low response rate to the learning course three-month follow-up questionnaire. In future evaluations it would be beneficial to monitor if and how intended practice changes are implemented. This will also contribute learning about what impacts are sustained in the longer term.
Implications for future work
The learning from the evaluation can be distilled into six criteria that inform how to build capacity in the gathering and use of evidence. Although speaking to the social care sector, we believe these lessons could be transferable across sectors. To build foundations for better services and well-being, capacity-building initiatives in evidence work must be:
relational: meaningfully engaging stakeholders who come from different world views
holistic: covering all aspects of evidence work
practice-focused: providing mentoring, experiential and tailored support
multifaceted: working at individual, organisational and strategic levels
co-produced: informed by intended beneficiaries and local needs
contextualised: grounded in the specific issues and circumstances that call for evidence.
Conclusion
DEEP is an approach informed by social pedagogy principles and practices that takes a holistic and relational approach to building capacity in evidence work. It operates across the social care sector in Wales, reaching out to citizens, practitioners, policymakers and infrastructure staff. It has distinctive features in the context of previously evaluated initiatives and concurrent UK programmes.
There is relatively strong evidence that the 2020–3 programme contributed to the acquisition of new knowledge and increased confidence in using the DEEP approach. Following our engagements, people intended to put the DEEP approach into practice and there was evidence that some practice change did result. These practice changes could improve social care services and enhance citizen well-being.
The evaluation findings inform future capacity-building initiatives. Programmes that promote capacity in gathering and using evidence should be relational, holistic, practice-focused, multifaceted, contextualised and be co-produced with intended beneficiaries.