Terms of Reference for the final evaluation of Oxfam’s project “Non-state Actors Promoting Budget Transparency”
Background, rationale and purpose of the evaluation
BTAP is a three-year project funded by the European Committee that runs from March 2015 to March 2018, seeking to address the mismatch between the top-down budget process and bottom-up socio-economic development planning at a local level. The overall objective of the project is to contribute to increase public Budget Transparency, Accountability and People Participation (BTAP) in Vietnam in order to enhance development outcomes for women and men from marginalised communities in the provinces of Hoa Binh and Quang Tri. The specific objective is that empowered civil society organisations (CSOs), community-based organisations (CBOs) and communities in Hoa Binh and Quang Tri are able to effectively influence the public budget process of poverty reduction and development programmes (including on health issues) with wider national impacts on relevant laws and decrees.
Particularly, expected results of the project are:
- Result 1: Poor people, ethnic minorities and women at local level have increased awareness of the budget process and are able to exert influence on this process of poverty reduction and development programmes (including on health issues).
- Result 2: The Budget Transparency Coalition is expanded to include multiple stakeholders and strengthened to promote and advocate for budget transparency, accountability and participation at local and national levels, including in relevant laws and decrees at national level.
- Result 3: Targeted People’s Councils from commune to provincial levels are more effective in the oversight of the budget process, promoting budget transparency, accountability and people’s participation.
- Result 4: Targeted People’s Committees from commune to provincial levels open spaces for civil society’s meaningful engagement in public budget formulation and budget monitoring of poverty reduction and development programmes (including on health issues).
1. The specific objectives of the evaluation
The evaluation will cover the whole project’s interventions at all levels of commune, district and nation. Specifically, the evaluation will seek to assess to what extend and how the identified specific objective and expected results have been achieved, and how these achievements have contributed to the overall objective of increasing public Budget Transparency, Accountability and People Participation in Vietnam, and enhancing development outcomes for women and men from marginalised communities in the selected provinces.
The audience for the final report is Oxfam staff from senior management to front line programme officers, implementing partner organisation staff, beneficiaries and stakeholders to the extent that the country team want to socialize findings, the current donor and future donors (as applicable).
2. Overarching questions for the evaluation
These are the overarching questions that will guide the evaluation. It is expected that these questions will be unpacked and broken down into sub-questions during the inception phase to ensure that the evaluation responds to the information needs of all stakeholders. The evaluation will aim to:
- Assess the overall achievements of the project and how the project has contributed to empowering civil society organisations (CSOs), community-based organisations (CBOs) and communities in Hoa Binh and Quang Tri to be able to effectively influence the public budget process of poverty reduction and development programmes (including on health issues) with wider national impacts on relevant laws and decrees over the project’s lifetime, looking particularly to what awareness has been raised and capacities has been built of poor people, ethnic minorities, women, and targeted organizations that lead to their actions on fiscal justice.
- Assess the effectiveness of the strategies and approaches that Oxfam and partners have used to influence changes towards increased public Budget Transparency, Accountability and People Participation in Vietnam, and enhanced development outcomes for women and men from marginalised communities in the selected provinces. Highlight examples of good practices are and how they could be scaled up and sustained beyond the project.
- Assess the appropriateness and relevance of the project in meeting actual needs of the target groups in the policy environment and country context of Vietnam.
- Assess the sustainability and recommend next steps for the intervention.
3. Approach, methods, and ethical considerations
The evaluation team will propose the evaluation design and methodology and develop the data collection tools. It is anticipated that the evaluation team will utilize both quantitative and qualitative methods of data collection and that participatory tools and techniques for data collection will be employed. Oxfam has used process tracing and contribution analysis extensively in the past, but other evaluation approaches that allow to infer causality are also acceptable. The evaluationis expected to be gender-sensitive, rights-based, inclusive, culturally-sensitive and comply with professional ethical standards. Individuals should not be mentioned in the report. This includes complying with Oxfam’s Responsible Data policy [1]. Innovative approaches that would foster interaction and learning across the partners, that would include capacity building on evaluation, would be welcomed.
The full evaluation design and methodology will be presented in the inception report, and will include as a minimum:
- The evaluator’s understanding of the evaluation questions
- Description of the evaluation approach proposed
- Sources of information for primary and secondary data collection
- Tools to be used for data collection
- Data analysis methods that will be carried out
- Data interpretation approach
- A detailed evaluation workplan.
Stakeholders to be consulted:
- Staff involved in project implementation, implementing partners, other CSOs involved
- Local community actors and citizens engaged in the project, with a balance between gender and age
- Any government ministries or department staff that were engaged in the lifetime of the project.
Information sources to be consulted:
- Founding project documents (contracts, project proposal, governance documents)
- Reports (annual and biannual narrative and financial)
- Project products such as research, social media content, awareness raising materials, etc.
4. Evaluation team qualifications and skills
The following skills and expertise are required:
- Relevant university degree (Masters preferred) in a related field of study (such as research methods, evaluation, social sciences etc.) or equivalent training and work experience.
- Proven track record in undertaking evaluations, including impact assessments.
- Excellent understanding of policy, campaigns and influencing approaches and tools.
- Demonstrable skills and experience in qualitative research methods in general and exploring causal links between activities and policy outcomes. Previous experience using process tracing and contribution analysis is a plus.
- Interest in, understanding of and practical experience of development finance issues such as budget transparency, people’s participation in budget allocation and spending, and civil society activism, campaigning and advocacy.
- Strong understanding of gender and Governance/Active Citizenship
- Excellent communication and facilitation skills.
- Flexibility and openness to work collaboratively with the evaluation manager, the project teams and partners in a co-creative evaluation process.
- Fluency in English, ability to communicate complex information using simple and concise language.
5. Schedule, budget, logistics and deliverables
Schedule:
- January 2018: Selection of consultant(s)
- February 2018: Desk review and Detailed evaluation plan/tools design
- February – mid March 2018: Data collection
- Late March 2018: Draft report and feedback process, Validation exercise
- 31 March 2018: Final report
Budget:
The total amount of funds available for the evaluation is EUR 6,000– 7,000 including tax. It is expected that all travel days, transport, accommodation and any other expenses incurred are incorporated in the stated fees of the evaluators. No additional payments or reimbursements outside the contractually agreed fees will be made.
Payments will be made according to the following payment schedule: 25% upon starting the contract (January), 25% on delivery of the first draft (mid March) and the remaining 50% upon delivery of a satisfactory final report (end of March).
Logistics:
The evaluation would be conducted mainly in Hanoi, with two trips to Hoa Binh and Quang Tri.
Partners, beneficiaries and stakeholders should be included in the design and delivery of the evaluation as far as possible bearing in mind geographical and financial constraints.
Field visits for data collection will be organised to facilitate project sites and face to face discussions with stakeholders. Details of the logistics regarding visits will be managed by CECEM.
Deliverables:
- Evaluation inception report or evaluation plan that will need to be approved before moving on to the next phase (including draft tools)
- Brief progress report from the field work
- Facilitation of a participatory workshop (TBC)
- Evaluation draft report for comments
- Final evaluation report of publishable quality, including an executive summary
- One PowerPoint (up to 10 slides) of evaluation findings and recommendations
All deliverables in English.
6. Dissemination strategy, plan and responsibilities for sharing and using the findings.
The findings will be shared with communities, partners, and key stakeholders (internal and external). The country teams and the reference group will consider and agree what is the most appropriate format and process for sharing the evaluation findings with communities (women, men, boys and girls), partners and key stakeholders.
The evaluation report (or the summary) which is in English may be translated into Vietnamese to facilitate community and stakeholder accessibility In Vietnam. The report will be shared with the donor and a meeting will be organised before project close to discuss the findings.
The full evaluation report or the executive summary and management response will be posted on Oxfam websites and made available for publishing to institutional donors. This is consistent with Oxfam’s commitment to transparency and accountability.
The findings will be used to inform program development, organisational learning, accountability and advocacy. Following the completion of the evaluation, a management response to the findings and recommendations will be developed.
7. Process of the selection of the evaluator or evaluation team and expectations for evaluation proposal
Oxfam invites bids from individuals or teams that combine the experience and skills described above.
Tenders must include:
- A cover letter of no more than 2 pages introducing the evaluator (or members of the evaluation team) and describing how the skills and competencies described above are met, with concrete examples as appropriate, including previous experience working as a team. Please also use this cover letter to indicate consultants’ availability at critical periods.
- A light touch technical proposal of no more than 2 pages explaining the consultant’s approach to conducting this evaluation, including indicative timeframe of availability.
- A 1-page budget covering all major anticipated costs (Oxfam prefers to pay an agreed price for the totality of the work including the field trips, which will be paid in phased instalments).
- A CV detailing relevant skills and experience of no more than 2 pages, including contactable referees.
- One example of a relevant previous evaluation.
Selection criteria to be applied:
- Quality, suitability and feasibility of the methodological proposal
- Profile and competencies of the evaluation team measured in knowledge, experience, composition and other relevant skills as outlined above
- Suitability of the financial proposal for the activities laid out in the methodology within the financial availability of the project.
Tenders should be sent to [email protected] [2] by 17.00pm of Friday 19th January 2018.
Shortlisted candidates will be invited to make an interview with the evaluation reference group in the last week of January 2018.
NOTE: Once the selection process has been completed, the ToR will be updated accordingly.
Recommended outline of an evaluation report
1. Cover page clearly identifying the report as an evaluation and stating:
- Evaluation title
- Program/project title /affiliate identification code
- Geographical coverage: global; region; country(ies)
- Date that the evaluation report was finalised
- Evaluator(s) name(s) and logo (if available)
- Oxfam logo (unless not appropriate)
- Appropriate recognition of institutional donor support.
- Clear statement in case this report can NOT be used externally
2. Table of contents
3. Glossary
4. List of abbreviations
5. Executive summary that can be used as a stand-alone document
6. Introduction, stating objectives of the evaluation and evaluation questions
7. The intervention and context
8. Methodology, including an indication of any perceived limitations of the evaluation
9. Presentation of the findings and their analysis
10. Conclusions
11. Learning and Recommendations
12. Appendices:
- Terms of reference
- Evaluation program (main features of data and activities carried out).
- A list of interviewees (name, function and working environment) and places visited.
- List of documents and bibliography used.
- Details on composition of evaluation team (names, nationality, expertise, working environment).
- Link to Methodological appendices:
- The evaluation proposal
- Evaluation instruments such as questionnaires and interview guides
- Data collected