Author name: Jamie Taylor

Featured in order of appearance (L-R): Dr Katie Shearn, Martins Iyekekpolor, Dr Sebastian Potthoff, Dr Rebecca Hunter, Professor Angela Bate We’re proud to mark one year since IDEAS-NET began its funding period – and what an extraordinary year it has been. From our official launch at The Future of Evaluation in Health and Social Care Symposium, to building a dedicated and talented team, and receiving our first National Evaluation, the past 12-months have been defined by momentum, collaboration, and ambition. Each milestone has strengthened our commitment to research excellence and set a clear direction for the impactful work ahead. Held at The Common Room in Newcastle upon Tyne, this event provided a valuable opportunity to explore our ways of working in greater depth and to take part in an interactive workshop delivered by our co-applicants, Peter van der Graaf and Sebastian Potthoff. The workshop offered practical insights into how we can maximise our impact through the NIHR School for Public Health Research’s six knowledge-sharing principles and the Implementation Stakeholder Engagement Model (I-STEM) Toolkit, developed by Sebastian alongside fellow co-applicant Professor Tracy Finch and colleagues. Click below to view images Featured in order of appearance (L-R): Dr Jo Wildman, Leanne Anderson-Clements, Dr Eduwin Pakpahan, Professor Peter Van der Graaf, Dr Sebastian Potthoff, Dr Katie Shearn, Professor Angela Bate As we look ahead, we remain focused on driving high-quality evaluation and expanding our networks. Find out more about the NIHR SPHR Knowledge Sharing Principles Discover the I-STEM Toolkit Stay Connected Linkedin Envelope

Author: Dr Julie Williams (IDEAS-NET Research Fellow) Contact: julie3.williams@northumbria.ac.uk How do research teams make sure they do meaningful and non-tokenistic PPIE? Here in the IDEAS team, as we start our Patient and Public Involvement and engagement (PPIE) work, we’ve been thinking a lot about how we involve people in our evaluation work equitably and ethically. So, it was good timing to attend training on developing and evaluating complex interventions hosted by the University of Glasgow, and two conferences, the UK Evaluation Society conference, also in Glasgow, and the European Implementation Event in Newcastle. The training was focused around the Framework for developing and evaluating complex interventions updated in 2021 which has engaging stakeholders as one of its core elements. The conferences had a different emphasis – UKES on evaluation and EIC on implementation, but PPIE was a key focus at both. The training highlighted the importance of involving all stakeholders including the public and patients and outlined some of the reasons why this was important which included understanding different perspectives on any one ‘problem’ and helping to understand what needs to be evaluated and how. These will be key to us at IDEAS as we undertake any evaluation. Both conferences included presentations and discussions on PPIE and were often focused on how to do it rather then why to do it. I want to focus on one presentation from UKES and one panel session from EIE to give some sense of different ways of approaching PPIE. https://www.youtube.com/watch?v=K2Oo0-kzX2Y Video: Measuring Humanity. Find our more about their REALITIES project. At the UKES many presentations and keynotes talked about PPIE. In one panel session titled ‘Participation or Pretence’ there was a robust debate about how to make PPIE inclusive and non-tokenistic. The panel ended with showing a video called Dear Human which was developed as part of the Measuring Humanity project which has the aim of measuring health and inequalities through connectivity and creativity, and was featured in a Lancet article. At the EIE conference they had a Public Involvement and Community Engagement (PICE) committee who oversaw all aspects of the conference to make sure it was inclusive-I’ve not seen this before at a conference and I hope that other conferences begin to do this too. I would love to know what was changed due to their work. Image: European Implementation Event 2025  This committee, that included two people with disabilities, also ran a workshop titled ‘Hard to reach? How hard are you trying?’ which was a really thought-provoking interactive session on what is needed for PPIE to be inclusive and why this is important. They used this cartoon and asked us to make sure they didn’t just get bums and feet, a striking way to illustrate the importance of inclusive PPIE. So, for all of us working in PPIE, how do we make sure people get faces, and not bums and feet? Disclaimer:For any individuals or organisations wishing to re-use or distribute materials from the IDEAS-NET website, please contact ideas@northumbria.ac.uk

Authored by: Sofie Jaspers*, Dr Stine Dandanell Garn*, and Charlotte Brøgger Bond** Contents Guiding Concepts Charlotte’s Reflections Stine’s Reflections Sofie’s Reflections Conclusions We kicked off 2025 in style by attending the Future of Evaluation Symposium at Northumbria University, Newcastle Upon Tyne, UK. As three early-career researchers connected through the Danish realist network, we boarded the plane with high hopes of soaking up new ideas for our upcoming research and networking efforts. Spoiler alert: it exceeded all expectations. The symposium proved to be a treasure trove of fresh perspectives on evaluation research, bringing together experts from a wide array of disciplines and methodologies. We were particularly inspired by the keynote presentations from esteemed researchers in the field, including Professors Ray Pawson (one of the founders of realist evaluation), Carl May (normalization process theory), Laurence Moore and Dr Kathryn Skivington (MRC framework), and Dr Emily Warren (realist trials). A particularly engaging aspect of the symposium was its format. Rather than traditional academic presentations, thought leaders posed bold, thought-provoking challenges, offering a “helicopter view” of the field’s future and sparking insightful discussions on the direction of evaluation research. Special guiding concepts introduced at the symposium that we will carry forward into our future work: The Black Elephant: This concept merges the ideas of a “black box” and “elephant in the room,” emphasizing how decision-makers often overlook or oversimplify the inherent complexities of programs and policies, even when these complexities are both evident and significant. The Iron Law: This concept refers to the tendency of large social and health interventions to have little or no significant net effect. When evaluating the overall impact of many programmes, they frequently show minimal or no long-term effects—at least not to the extent initially expected. However, this does not necessarily mean that all social programs are ineffective. Rather, it suggests that they are often designed, implemented, or evaluated in ways that fail to drive meaningful systemic change. Instead of relying on simplistic judgments of whether a program “works” or “does not work,” this concept urges a deeper analysis of why an intervention has led to certain outcomes and how we can better explain and improve its impact. Cumulative Knowledge Generation: This concept highlights the need to systematically gather and use knowledge so that lessons from one program can be transferred to others. Many complex interventions are designed without considering previous learnings from similar programs, leading to repeated mistakes and ineffective solutions. As such, rather than evaluating each program as an isolated unit, this principle encourages us to develop a deeper understanding of the key factors that make an intervention successful, allowing them to be applied across different projects and contexts. Do not harm: Interventions are naturally designed to create improvements. However, when developing programme theories and logic models, we often overlook potential harmful effects. The symposium highlighted the need to look beyond intended outcomes, consider, and assess potential unintended consequences and their underlying mechanisms, using tools like realist ripple effects mapping or dark logic models. In addition to the keynote sessions, the symposium featured engaging breakout sessions. We participated in one titled The Future of Evaluation: Where Have We Been and Where Are We Going?, organized by Professors Angela Bate and Sonia Dalkin. We actively contributed to a creative process of writing on post-it notes, helping to create a map of ideas that reflected the collective hopes, beliefs, challenges, and things to avoid in the future of evaluation. We also attended another breakout session, Mobilising Knowledge from Complex Intervention Evaluations into Policy and Practice: How to Deal with Lazy Academics and Stubborn Policymakers, presented by Dr. Peter Van der Graaf and Dr. Sebastian Potthof. They introduced their I-STEAM tool, demonstrating structured approaches to implementation and stakeholder engagement. We found it highly relevant and are excited to apply it in practice. At the National Research Centre for the Working Environment, Stine works with knowledge mobilization, and it was inspiring to see how other research groups structure this work. We each took valuable insights home for our upcoming projects:   Charlotte: As a PhD student conducting a realist evaluation of a stress management intervention, I set out to present a poster outlining my initial programme theory. My colleagues back home had warned me not to expect much engagement, especially at larger conferences where posters often don’t draw much attention. So, I was both surprised and a little nervous when I stepped into the intimate setting of the symposium. It was a bit overwhelming to stand face to face with leading researchers in the field and present my work. While I did receive some positive feedback, I was also gently challenged by a respected editor from a high-impact journal. The editor encouraged me to think more deeply about the audience for my research and offered valuable advice on how to further develop my work. It’s moments like these, when you’re pushed to think critically, that truly help you grow as a researcher. Stine: As a recent PhD graduate, having applied the MRC framework alongside realist principles to evaluate a peer support intervention for vulnerable diabetes patients—and with a strong interest in evaluation methods—the symposium felt tailor-made for me. More than just an academic event, it provided a rare and invaluable opportunity to connect in person with colleagues and collaborators. Notably, it marked the first face-to-face meeting for many of us in the Danish realist network. A standout moment was presenting and discussing my PhD research with researchers who have profoundly shaped my thinking—scholars I have cited, learned from, and been taught by. Sharing my proposal for an extended modified CMO configuration, designed to emphasize broader measurement of outcome patterns, was both insightful and humbling. I returned home with a suitcase full of new insights, strengthened relationships, exciting collaborations, and valuable tools and concepts I look forward to applying in my work.   Sofie: Over the past 10 years, I have developed a methods interest alongside my subject area of research in organizational health and safety, and this symposium was finally

Authors: Professor Angela Bate, Professor Sonia Dalkin & Dr Jason Scott Contents Provocation Plenaries Breakout Sessions & Poster Submissions Photography Summary Photography: Del Stevyn Provocation Plenaries  Over the course of three days in early January 14th-16th 2025, we were privileged to welcome esteemed speakers across a range of disciplines to share provocation plenaries on the past, present and future of evaluation in health and social care. We were joined by over 100 attendees from academic, health service and policy backgrounds, at all career stages, representing 9 countries. Our speakers did as asked and provoked the audience to critically reflect on all aspects of evaluation.  Copies of all the presentations are linked below. Be sure to cite them if you reference any material contained within them: Dr Kathryn Skivington & Professor Laurence Moore (University of Glasgow) – “Strength through diversity? A journey through the MRC/NIHR Framework.” Professor Ray Pawson (University of Leeds) – “Farewell to Evaluation.” Professor Dave Byrne (Durham University) – “Understanding causes and effects as system states: the implications for evaluation.” Dr Emily Warren (London School of Hygiene & Tropical Medicine) – “Are realist RCTs possible? Reflections from the INCLUSIVE trial.” Dr Rachel Meacock (University of Manchester) – “Reconciling the micro and macro impacts of health and care interventions.” Professor Carl May (London School of Hygiene & Tropical Medicine) & Professor Tracy Finch (Northumbria University) – “What are we evaluating when we do process evaluations?” Professor Mike Kelly (University of Cambridge) – “Why is Changing Health Inequalities So Difficult?” Photography: Jamie Taylor   Breakout Sessions & Poster Submissions Plenaries were complemented by a series of breakout sessions and posters presentations. Special shout out to our poster presentation award winners: Dr Andrew Fletcher, Dr Carsten Hinrichsen, Lauren Lawson and Dr Petra Makela for their exceptional contributions in advancing evaluation in health and social care, through methods innovation and impactful dissemination. Dr Meghan Kumar (Northumbria University) and Dr Ferdinand Mukumbang (University of Washington) – “Future of Evaluation in Global Health.” Professor Sonia Dalkin and Professor Angela Bate (Northumbria University) – “Future of Evaluation: Where have we been and where are we going?” Dr Manbinder Sidhu and Dr Sophie Spitters (University of Birmingham) – “Rethinking Rapid Evaluation: Challenges in Scoping and Decision-Making. Professor Darren Flynn (Northumbria University), Professor Joanne Gray (Northumbria University), Dr Alan Bagnall (Consultant Interventional Cardiologist) and I-Lin Hall (Strategic Head of Digital and Programmes – NENC) – “Managing big data sets and data linkage.” Dr Jason Scott (Northumbria University), Professor Brian Castellani (Durham University), Dr Efundem Agboraw (London School of Hygiene and Tropical Medicine), Dr Morgan Beeson (Newcastle University) – “Managing complexity in Evaluation.” Dr Peter Van der Graaf and Dr Sebastian Potthoff (Northumbria University) – “Mobilising knowledge from complex intervention evaluations into policy and practice: how to deal with lazy academics and stubborn policymakers.” Photography: Del Stevyn Photography   We wanted to take a moment to acknowledge the incredible work of Del Stevyn, our event photographer for The Future of Evaluation in Health and Social Care symposium. Del’s keen eye and thoughtful approach have resulted in a fantastic collection of images that capture the energy, engagement, and key moments of the event. As we share these images, we recognise how they vividly illustrate the connections made, the ideas exchanged, and the spirit of collaboration that defined the symposium. View the images here. Summary As promised, the symposium allowed attendees to: Hear from a range of speakers across a variety and disciplines and meet participants across these disciplines to break down those silos and communicate with one another; Spend time and space to think through current conceptual challenges in evaluation and gain different perspectives on live problems; Connect evaluators from academia and consultancy with service providers, local and national policy makers, and the voluntary, community, and social enterprise sector (VCSE). Disclaimer:For any individuals or organisations wishing to re-use or distribute materials from The Future of Evaluation in Health & Social Care symposium, please contact the IDEAS Team at ideas@northumbria.ac.uk.

Authors:  Professor Angela Bate, Professor Sonia Dalkin & Dr Jason Scott  Contents IDEAS More Than Minutes The IDEAS-NET Team (L-R), Dr Eduwin Pakpahan, Professor Tracy Finch, Dr Sebastian Potthoff, Professor Katie Haighton, Professor Angela Bate (Co-Director), Professor Sonia Dalkin (Co-Director), Professor Joanne Gray, Professor Tom Sanders, Dr Jason Scott (Deputy Director), Professor Darren Flynn, Jamie Taylor (Project Manager) Photography: Del Stevyn IDEAS The Future of Evaluation in Health and Social Care symposium was pivotal in celebrating the launch of the recently funded National Institute for Health Research (NIHR) National Evaluation Team (NET), IDEAS, located at Northumbria University. The event also offered the opportunity for IDEAS to come together with other NIHR evaluation teams to share experience and expertise. Thanks especially to the contributions from the BRACE (Birmingham, RAND and Cambridge) Rapid Evaluation Team and ROSE-NET (Manchester-Leeds Collaborative) National Evaluation Team. The IDEAS-NET Executive Team (L-R), Professor David Croisdale-Appleby (Chair of our National Advisory Group), Professor Angela Bate (Co-Director), Professor Sonia Dalkin (Co-Director) & Dr Jason Scott (Deputy Director) Photography: Del Stevyn More than Minutes  Now that we have had the time to recover from an intense and stimulating 3-days, we thought it would be opportune to share some of the initial lessons and reflections from the symposium, drawing on the visual minutes that were sketched by More Than Minutes which brilliantly capture key moments from the Symposium.  Whilst we credit some of our speakers, we acknowledge that many themes recurred throughout the plenary presentations, breakouts and delegate feedback.  Visual minutes from the symposium, sketched by ‘More Than Minutes’, pg.1/2. Visual minutes from the symposium, sketched by ‘More Than Minutes’, pg.2/2. Embrace complexity in evaluation. Portrayed by the black elephant standing in the pool of complexity (pg. 1 bottom left of the visual minutes), the image signifies the concept that both evaluation and policy-makers often seek to ignore or downplay complexity even when the complexities are anticipated, widely reported, and significant. Mike Kelly emphasised the importance of this to policy making it clear that we need to move away from seeking ‘easy answers’ and simple solutions to the inherently complex problems in health and social care evaluation and simple solutions to.   Recognise the limits of silos and hierarchies (disciplinary, evidence, and methods) and engage in collaborative, inter-disciplinary working to determine not only ‘if’ but also understand ‘how’ interventions or policies work (as represented by the ‘black box’ at the top of pg. 1 in the visual minutes) in order to develop ‘practical but contingent advice that is useful in the real world’. The visual minutes highlight moving away from a hierarchy of evidence, and instead towards a toolkit of evaluation approaches, each fit for purpose when used to answer the appropriate research question (pg. 1 top left of the visual minutes).   Don’t scrimp on the scope a term coined by our colleagues at BRACE (illustrated by the telescope pg. 2 of the visual minutes) to emphasise the importance of allocating sufficient time and resources to understanding and defining: what it is being evaluated, the range of perspectives and stakeholders involved, the evaluation questions that need to be addressed, and the evaluability of these questions based on the availability of historical and prospective evidence and data (among other things).   Learn from the past to guide the present and future avoid institutional amnesia (represented pg. 1 bottom left of the visual minutes), by documenting and synthesising previous evidence to learn from past mistakes, contextualise past decisions (recognising that these don’t occur in a political vacuum), and move toward a model of cumulative learning (illustrated as Ray Pawson waving farewell to the accreditation (of programmes) train, and thus moving away from pass or fail verdicts and towards cumulative learning, pg. 1 middle of the visual minutes).   Seek alternative explanations and be iterative in our designs to allow further exploration. Rachel Meacock used the example of community healthcare (pg. 2 top left of the visual minutes), to illustrate how different data and levels of data analysis can tell different stories, and illustrated the importance of theorising possible causal contextual and mechanistic explanations of how and why this might be to make sense of the results and inform future data collection; highlighting the need to theorise and to move between the empirical and the abstract to do so (pg. 1 bottom left of the visual minutes).  Take risks in commissioning and conducting evaluations including those borne by universities (employ researchers on secure contracts), funders and funding committees (fund larger interdisciplinary teams, longitudinal studies, and novel and live evaluations, employing stop/go criteria and staggered funding), and policy makers (accepting uncertainty in decision-making). represented by the checklist pg1 middle) (illustrated pg. 2 of the visual minutes).  Each of these themes are reflected in, and will continue to inform, our approach to evaluation in IDEAS-NET.  Thank you to everyone who attended or was involved in the delivery and organisation to make the event a success. We are overwhelmed by the positive feedback we have received. Watch this space…  Disclaimer:For any individuals or organisations wishing to re-use or distribute materials from The Future of Evaluation in Health & Social Care symposium, please contact the IDEAS Team at ideas@northumbria.ac.uk.