Meet the team: Dr. Annette Zhao

“The role of implementation is becoming more integral to evaluation, working with the implementors makes a big difference to the effectiveness of evaluation and the take up of the evidence”

In this first in the series of Meet the Team we speak to Dr Annette Zhao, Associate Director about her favourite evaluation frameworks, and the importance of partnerships in evaluation design and implementation.

How did you first get into education evaluation? What drew you to it as a field, and was it a deliberate career path or something you fell into?

My postdoc project involved the evaluation of activities to encourage multilingualism among school pupils outside of school and I became really interested in evidence-based decision making and how to evidence the impact of academic research in real life situations and it all started from there. I left academia to work on research and evaluation in international development and education before joining ImpactEd Evaluation last year.

What kinds of evaluations have you led or contributed to?(e.g., program evaluations, curriculum assessments, teacher or school performance reviews, policy impact studies)

I have mostly been involved in the evaluation of different programmes or interventions from those targeting small-scaled individual school/classroom/student cohort to nationwide, large-scaled, standardised learning assessments within and beyond the UK context. Most of these involve a combination of formative and summative assessment, process and impact evaluation methods, as well as cost-effectiveness.

What evaluation frameworks or methodologies do you rely on most?

The CIPP model (Context, Input, Process, and Product) is a helpful framework to keep in mind when working closely with implementors of different programmes as it keeps the evaluation focused on what the programme is trying to achieve. I use a wide range of research and evaluation methods and always try to complement theory (e.g. theory of change) with primary data (e.g. learning assessment or case studies). With my background in quantitative research, I also design and implement randomised controlled trials or other experimental or quasi-experimental studies, often with qualitative elements to highlight student/teacher/parent voices.  

Can you describe a project you're especially proud of? What made it successful, and what impact did it have on the people or institutions involved?

Last year I led on a project with an ambitious and complex design involving a formative evaluation of a programme providing research placements for university students. The project delivered a theory of change, a formative evaluation as well as a plan for potential future evaluations over the course of one year. We encountered several challenges from data availability to implementation and funding changes but were able to deliver the outputs agreed with the partner to their satisfaction. The report fed into ongoing implementation and evaluation and has already helped the partner in this year’s programme implementation.

What's the hardest evaluation you've worked on, and why? Was the challenge methodological, political, interpersonal, or something else?

Some of the more difficult evaluations I have worked on share a range of common characteristics: issues with data availability and implementation fidelity, shifting scope from the partner/funder as our shared understanding of the context changes inevitably, and of course, bigger-picture geopolitical shifts domestically and internationally. All of these relate to the context the evaluation is conducted in and a lack of prior agreement, shared understanding or built-in flexibility for the project.

How do you handle situations where your findings are unwelcome? Have you ever had to deliver results that stakeholders didn't want to hear?

As an evaluator, I understand that evaluations are often commissioned by people who have invested time, money, and identity in a programme and even neutral (or negative) findings can feel like personal criticism and can threaten funding. This makes it extremely important to build a productive relationship with the partners from the beginning to foster collaboration and learning, as well as the evaluating part of the project. When I encounter similar situations, I try to present the evidence (rather than mixing evidence and interpretation) while acknowledging the inevitable complexity of education evaluation. A collaborative relationship is vital in this situation, so that evaluation – whatever it tells you – is seen as a vital feedback tool, to refine and reprioritise resources.

How do you think about the difference between evaluation and research?

In essence, evaluation is more specific than research in general. When evaluating, we normally have a specific programme/intervention/policy in mind and we try to see if it achieves the expected value and what that value is. A research project could be much wider in scope and we try to build evidence beyond a specific tool as much as possible to generate knowledge that applies more widely. The tools and methods employed can be very similar in practice between the two!

How has your approach to evaluation changed over the course of your career? What assumptions did you start with that you've since revised?

I think my understanding of the relationship between the evaluator and the partner/implementor/funder, especially given that evaluation and applied research are often commissioned by the implementor. It is much more complicated than academic research and cultivating a collaborative relationship which embeds learning and flexibility is as important as delivering the evaluation sometimes.

Where do you see education evaluation heading as a field? Are there emerging methods, technologies, or questions you're most excited or concerned about?

The role of implementation is becoming more integral to evaluation and research and working with the implementors rather than alongside them makes a big difference to the effectiveness of evaluation and the take up of the evidence. Implementation fidelity and context need to be at the core of evaluation design. The focus on equity and intersectionality is also here to stay and is becoming a standard consideration in evaluation projects.

--

Keen to work with experts like Annette and understand how ImpactEd Group can support your organisation on all things educational impact and sustainability, do get in touch to speak to the team.

‍