Integration of project management and evaluative thinking.
Is it really working?

Tuesday 11:30am

Vanessa Hood and Heather Aquilina presented and facilitated this 1hr discussion

There were around 60 participants from across Australasia. The room included a mix of evaluation practitioners and commissioners from government, private sector, NGO and academia.

Click here for full abstract.

Summary of group points raised in group discussions

What's working?

 

Top 3

  • Top down expectations / mandating (not enough on own)

  • Program logic / theory of change as a mechanism for communication and discussion

  • Linking design, evaluation and implementation teams

Other 

  • Periodic reflection

  • Implementation team getting early feedback

  • Embedding evaluation in the program cycle

  • Checking early that evaluation was actually embedded

  • Knowing who to go to internally for evaluation assistance

  • Evaluation teams having good relationship with management, implementation and design teams

  • Budgeting up front for evaluation

  • Donor pressure for evaluation

  • Reassurance upfront to reduce fear of external evaluation

What's not working?

Top 3

  • Lack of leadership / champions

  • Lack of resources required to

  • Reports not used / not the end of the cycle

Other 

  • 3 year election cyclesn

  • Processes that are not inclusive or participatory e.g. gender analysis of sex disaggregated data

  • Lack of acceptance that negative findings are just as valuable as positive

  • Pressure of time/funding

  • Evaluation report vs evaluation for a business case

  • Everyone thinks they are an evaluator

  • No evaluation culture in an organisation

  • Outcome driven, not learning stifles innovation

  • Quality of evaluation is NOT evaluated/questioned

  • Rigid quality audits and scoring systems

  • Not doing the basics well enough

  • Imbalance between time for collection vs analysis

  • Inadequate focus on implementation

  • Evaluators not pitching the right incentives to managers

  • Relationship breakdowns

  • Evaluators being interested in all perspectives, vs organisations only wanting success stories

  • Lack of education of the role of evaluators in the organisation / emerging profession

  • Limited evaluation capacity and knowledge

  • Limited evaluation data collection or lots of data but not useable

  • Negative experience of evaluation, therefore don’t value it / don’t see benefits

  • Lack of clarity about different intersecting skillsets and functions, e.g. audit vs evaluation

  • Comparative evaluation between projects

  • Agendas / influence of others within an organisation

  • Poor specification given to the consultant

What can we do to better integrate project management and evaluative thinking?

Top 3

  • CULTURE - Ethos of reflective practice / continuous improvement

  • CAPABILITY - Differentiated skill set (e.g. data skills) – specialist cf everyday evaluation 

  • STRUCTURE / PROCESSES – Points at which you build it in – plans, budgets

Other 

  • Evaluation planning as part of project planning

  • Program logic as part of project planning

  • Resourcing of evaluation in the project budget

  • Championing and leadership

  • Culture of 'If evaluation's not done, that’s a problem'

  • Integration of evaluation 'specialists' across an organisation to build relationships

  • Cross-organisational decisions (and education) about the role and purpose of evaluation in the organisation

  • Repackaging evaluation as a part of the language of ‘risk’ – if no evaluation, ‘you guys are exposed!’  The flipside of risk = key evaluation questions.

  • Using other people's language, rather than evaluation jargon

  • Evaluative thinking as part of the design process – from the business case forward

  • Continuous improvement cycle – evaluation as fuel for this
  • Part of everyone’s job – ‘we are all evaluators here’ – requires resourcing and skill building

  • ‘We all have a contribution to make to evalution processes’ – e.g. to undertake reflective practice