How culture and characteristics can shape your evaluation - lessons from Wicked

As we embark on the beginning of a new school year, consider this: if you were to conduct the same educational evaluation twice, using identical methodologies and research questions, would the outcome of the evaluation be the same? This question struck me during a recent evening at Wicked, having previously seen it several years ago. The show was the same, I enjoyed it as much second time around, but it was not the exact same performance.

Like a musical performed across different venues and casts, each evaluation creates a unique interpretation despite following the same script. The director brings fresh insights, the cast members have changed, the audience brings different energy, and even the theatre itself shapes the experience in subtle ways(summer vs winter in my case).

This demonstrated to me something fundamental about evaluation in education: each assessment is a live performance, shaped by the dynamic interplay of all participants. The written script, our methodology, provides the framework, but the magic emerges from how evaluation culture, stakeholder characteristics, and evaluator expertise work together to bring the story to life.

In applied settings, such as education, the quality of evaluations depends on a range of factors. Some of these factors are directly related to the evaluation itself (e.g., evaluation quality and methodological choices) whereas other factors are indirectly related. Yet, they can still have a significant effect on the overall quality of an evaluation – such as the evaluation culture, stakeholder characteristics, and even evaluator characteristics.

In this blog, we share insights into how these indirect factors can influence an evaluation and things to look out for when running your own evaluations:

Evaluation Culture

The evaluation culture can have a large impact on the implementation of an evaluation and the perception of its usability. Just as a theatre’s atmosphere can enhance or diminish a performance, the institutional environment shapes how evaluation resonates within an organisation. Fortunately, environments that are high learning cultures – such as schools –embrace evaluation, treating it as an analytic process from which to make decisions. These settings provide the ideal stage for evaluation insights to shine and inspire action. Related to this, and as we often see with research schools, a research lead or advocate can further enhance the learnings from evaluations and ensure that findings are embedding within a school longer term. At ImpactEd, we have repeatedly seen this investment from schools and multi-academy trusts first hand, partnering with over 2000 schools a year on evaluation work.

Stakeholder Characteristics

Sometimes stakeholders can have preferences in terms of types of evaluation design and analysis depending on their previous experiences – for example, favouring qualitative approaches over quantitative ones. Like actors who may have preferences for certain theatrical styles based on their training, these preferences can enrich the evaluation. Whilst previous experiences can be useful to inform the scoping of an evaluation, it is always important to remain open to different types of evaluation design and methodology. One way to do this is to, when starting an evaluation, develop or revisit the underlying theory of change model – that is, what change is the programme seeking to bring about – and what specific research questions need to be answered in relation the theory of change. Indeed, all decisions around evaluation design and analysis should be driven by these two things. This is why, at ImpactEd, one of the key elements that underpin all our evaluation work is the development of theory of change models to ensure that we fully understand a programme in terms of the problem statement, inputs, activities, outcomes, impacts, causal pathways and moderating factors. Once we have this level of detail, we can then map out the research questions and evaluation design and analysis options that best meet these needs.

Evaluator Characteristics

Similar to stakeholders, not all evaluators are the same, they vary in terms of their expertise and may approach stakeholder engagement differently. The evaluator, like a director, must possess both technical expertise and the ability to bring out the best performance from every cast member.  After evaluation expertise, the most important characteristic relates to stakeholder engagement – ensuring that there is a trusted relationship between stakeholders and evaluators, and that the expertise of stakeholders feeds into the evaluation process throughout, and lastly, that the deliverables from the evaluation will be accessible and valuable within its setting.  At ImpactEd, we are a team of trained researchers and evaluators, and we consider a fundamental part of our role to be around facilitation and listening so that we understand stakeholders and their needs from conception to completion of an evaluation.  An important way to do this is through the co-production of evaluations and facilitatory methods to enhance the collaborative interpretation of the findings for educators and the sector more widely.

—

Overall, the interaction between evaluation culture, stakeholder characteristics, and evaluator characteristics form the foundations of an evaluation, and often can be set before the evaluation is fully underway. When these elements work together seamlessly, the result is something far more than the sum of its parts.

Therefore, this academic year, we invite you to consider how these factors could, when directed well, enhance the quality of evaluations in your settings. Rather than simply following the methodological script line by line, focus on creating the conditions for a truly collaborative environment which will result in robust evidence and actionable insights to enhance outcomes for young people.

‍