Consultant for Final Evaluation at Plan International

Consultant for Final Evaluation

4.0 Ethics Approval and Disclosure/Ownership of Information

Full ethical approval will be obtained before the study commences, if required. In accordance with Plan’s Research Policy and Standards, this will be obtained in one of three ways as appropriate:

  • In cases where the proposal is submitted by a University or research institution that has an ethics approval process, then that University or research institution may provide approval
  • In cases where country may require ethics approval from governmental or other regulatory bodies, in such cases ethics approval will be sought by the Consultant(s) in each implementing country, in coordination with the team members

All ownership and copyright for final data collected is held by Plan International. It is understood and agreed that the Consultant(s) shall, during and after the effective period of the contract, treat as confidential and not divulge, unless authorized in writing by Plan International, any information obtained in the course of the performance of the contract. Information will be made available for the consultants on a need‑to‑know basis. Any and all necessary field visits will be facilitated by Plan International staff.

5.0 Child Protection

Plan International is committed to actively safeguarding children from harm and ensuring children’s rights to protection are fully realized. Plan International takes seriously the commitment to promote child safe practices and protect children from harm, abuse, neglect and any form of exploitation as they come into contact with Plan International supported interventions. In addition, we will take positive action to prevent child abusers from becoming involved with Plan International in any way and take stringent measures against any Plan International Staff and/or Associate who abuses a child. Our decisions and actions in response to child protection concerns will be guided by the principle of ‘the best interests of the child’.

As such, the study must ensure appropriate, safe, non-discriminatory participation; a process of free and un-coerced consent and withdrawal; confidentiality and anonymity of participants. The Consultant(s) are required to provide a statement within their proposal on how they will ensure ethics and child protection in the process of data collection and visits in line with Plan International’s Safeguarding Children and Young People Policy. This must also include consideration of any risks related to the study and how these will be mitigated.

6.0 Supervision and Management of the Assignment

The Consultant(s) will be required to work closely with the Plan International Nigeria Office (including Program Management and M&E focal points, to be identified at the beginning of the assignment) as well as Plan International Canada (M&E Advisor supporting BORN project).

The Consultant(s) will provide regular updates on study deliverables to the Plan International Stakeholders in Nigeria and Canada. The Consultant(s) will keep the stakeholders continually informed of progress on the assignment through periodic skype conferences and regular email updates.

The Consultant(s) will be directly accountable to the Head of Health programs

[1] Kindly list the names of the consultant(s)/consulting firm on the cover page only.

Consultant for Final Evaluation

Bauchi Opportunities for Maternal and Neonatal Health (BORN)

· Overview of request for proposal

The Consultant will serve as the lead coordinator for the Final Evaluation of the Bauchi Opportunities for Maternal and Neonatal Health (BORN) project, implemented in Bauchi, Nigeria. A project baseline survey was conducted in 2016, followed by a midterm study in 2018-2019. Nearing the end of the project, Plan International Nigeria is now seeking a qualified consultant to coordinate a Final Evaluation for the project.

The Consultant will be responsible for overseeing all aspects of the Final Evaluation, including: recruitment of study team(s); study design and methodology; training, data collection and quality assurance; data entry, transcription, and analysis; and report writing. The main objective of the consultancy is to produce a robust final evaluation for use by Plan International staff, project implementers, partners, stakeholders, and the donor.

· Project Background

The Bauchi Opportunities for Responsive Neonatal and Maternal Health (BORN) project is a four and a half year (February 2016-June 2020) gender transformative project funded by Global Affairs Canada with a Total value of CAD 13,786,000 (Global Affairs Canada CAD 12,200,000; Plan International Canada cash/in-kind CAD 1,586,000). The project is expected to ultimately contribute to the reduction of maternal and neonatal mortality in 10 Local Government Authorities (LGAs) among marginalized and vulnerable women and new-borns. Its intermediate outcomes are:

  • improved utilization of maternal, neonatal and family planning (FP) services by women of reproductive age (WRA) including adolescent girls, new-borns, and male community members
  • improved quality of maternal, neonatal and FP services for WRA including adolescent girls, new-borns and male community members
  • improved quality of local health governance systems to ensure high quality, gender responsive, adolescent- friendly and results-oriented Maternal and Neonatal Health (MNH)/Sexual and Reproductive Health (SRH) services

By increasing the quality, availability, utilization and accountability of essential MNH/SRH services, BORN aims to scale-up improved health outcomes for 1,003,000 Women of reproductive age (WRA), including 203,000 adolescent girls, as well as approximately 250,000 newborn children, and engage 898,000 men (187,000 adolescent boys) in underserved areas of Bauchi, Nigeria.

The implementation of BORN is led by the Plan International Nigeria country office, in collaboration with Ministries of Health, Ministries of Women’s Affairs/Gender and Local NGOs, with the support of Plan International Canada.

For the evaluation objectives, scope of the evaluation, criteria, methodology, Data collection. sampling strategy, household survey, Health facility questionnaire, purpose of the consultancy, Deliverables, Timeline and qualification of the consultant,

please follow this link: https://drive.google.com/open?id=1FHUJqkYdVFL4s-2wVCUOnRnT49OTlF0o

Deadline for submission of application and proposal package is 20thSeptember,2019 on or before 5;00pm

3.0 Evaluation Objectives

The overall objective of the Final Evaluation is to conduct a robust and objective study in order to assess the project’s performance vis-à-vis expected outcomes within the project Performance Measurement Framework, and vis-à-vis the DAC Criteria for Evaluating Development Assistance. More specifically the objectives of the study are to:

  • Assess the impact of the BORN project’s achievements vis-à-vis the project’s overall objectives/ intended outcomes, referencing baseline and midterm results
  • Assess the project’s performance vis-a-vis the DAC Criteria for Evaluating Development Assistance:relevance, effectiveness, efficiency, impact and sustainability
  • Examine the contribution of partnership and innovation to the results obtained
  • Review best practices in project implementation, and subsequently generate specific recommendations to guide future MNH/SRH program management and design
  • Explore unexpected outcomes of the project activities, or successes that stem from the project expanding beyond its original scope

Plan International Nigeria’s staff in Program Units and the country office, Plan International Canada, implementing and technical partners, and the donor will be the major users of the evaluation results. In addition, key stakeholders in Nigeria such as government line ministries / departments, partner NGOs, local authorities, and communities are interested parties of the findings.

4.0 Scope of the Final Evaluation

5.1 Study Scope

In order to achieve the evaluation objectives, the Final Evaluation will include primary and secondary data collection to assess progress against all indicators included in the BORN Performance Measurement Framework (PMF), as well as to respond to the Evaluation Criteria listed in Section 5.2 below (RelevanceEffectivenessEfficiencyImpact and Sustainability).

Indicators for expected outcomes in the BORN project include:

Intermediate outcome indicators

· % of women aged 15-49 who received antenatal care (ANC) by a skilled health provider at least 4 times during pregnancy (disaggregated by age)

· % of women of reproductive age in union who are currently using a modern method of contraception (dis. by age)

· % of live births attended by skilled health personnel (disaggregated by age)

· % of mothers, and %of babies, who received postnatal care within two days of childbirth (disaggregated by age)

· Average level of satisfaction of WRA (dis. by age) and their male partners with the quality and responsiveness of MNH/SRH services

Immediate outcome indicators

· % of WRA (dis. by age) and their male family members who know at least 2 danger signs and related strategies along the continuum of care

· % of WRA (dis. by age) and their male family members who know at least 2 key gender equality messages related to MNH/SRHR

· % of WRA (dis. by age) and their male partners who know at least 2 benefits of delaying and spacing pregnancies

· % of WRA (dis. by age) and their male partners who know where to obtain modern contraceptives

· % of women who are members of organized community groups (dis. by age and type of group)

· Average level of support provided by male family members for the utilization of MNH/SRH including FP services by female family members (dis. by women, men, and adolescent girls)

· % of health service providers trained who can identify correct usage of modern contraceptive methods (dis. by sex)

· % of health facilities that provide gender and adolescent responsive FP services

· # of targeted health facilities with FP commodities[1]

· # of targeted health facilities with MNH/SRH equipment[2]

· # of Health facilities that utilize environmentally safe waste disposal methods, as per the Government of Nigeria standards (for that type of health facility).

· Average level of satisfaction of female MNH/SRH users (dis. by age) and their male family members with the MNH/SRH referral system

· % of health facilities that provide gender responsive and adolescent friendly MNH/SRH services

· % of QIT membership and leadership positions held by women

The consultant will be expected to design a methodology to identify any intended and unintended changes or outcomes relating to Gender Equality, with analysis by age and sex of respondents. In the absence of a baseline, the methodology will need to harvest outcomes from respondents, triangulating evidence around changes in:

§ Type of support provided by male partner to WRA during, before and after childbirth

§ Male partners’ level of and type of support for exclusive breastfeeding

§ Distribution of decision-making between WRA and their partners

§ Women’s skills/abilities and opportunities related to making community-level decisions

§ Male partners’ level of support towards their female partners’ participation in community groups/committees, assumption of leadership roles in those fora, as well as in community level decision making

§ Level of decision-making/participation of women in committees, and barriers (including household level, community and committee level)

§ Distribution of productive and reproductive labour between WRA and their partners

§ Women’s and adolescent girls’ (relative to men and boys’) level of access to and control over resources and opportunities (income/employment; education; information (including health information); mobile phones, radios or other communication technology; health services; mobility and transport; etc.

Finally, the Final Evaluation is expected to include bivariate and multivariate analysis of expected outcomes by poverty and vulnerability status of respondents, using the same indices applied through household surveys at baseline and midterm in the BORN project:

§ The Progress out of Poverty Index (PPI) developed by the Grameen Foundation, available for Nigeria

§ A Vulnerability Index that categorizes respondents into three levels of vulnerability based on a number of socio-economic criteria[3]

5.2 Evaluation Criteria

The following key questions will guide the Final Evaluation’s assessment of the project against the DAC Criteria for Evaluating Development Assistance:

§ Relevance: The extent to which the project was suited to the priorities of the target beneficiary group(s), stakeholders, and to the donor.

o To what extent are the outcomes of the project still valid to project stakeholders and beneficiaries (i.e. WRA, their partners, children and families; community members; health care providers; government officials; etc.)?

o Was the project relevant to the needs of these beneficiaries, as identified at the project design/inception stage?

o Were the implemented activities and achieved outputs of the BORN project consistent with the intended impacts and effects?

§ Effectiveness: The extent to which the project attained its outcomes.

o To what extent were the outcomes achieved?

o How did the project contribute to the achievement of these outcomes?

o How have the strategies and tools used in the implementation of the project contributed to project results?

o What were the major factors influencing the achievement or non-achievement of the outcomes?

§ Efficiency: The extent to which the project used the least costly resources possible in order to achieve desired results, considering inputs in relation to outputs.

o How economically were resources/inputs (funds, expertise, time) converted to outputs?

o Were resources effectively utilized?

o Were objectives achieved on time and on budget?

o What were the strengths, weaknesses, opportunities and threats to the project implementation process?

o Did the project activities overlap and/or duplicate other similar interventions, funded nationally and/or by other donors?

o To what extent did the project collaborate with national and sub-national partners and stakeholders (technical, advocacy, funding, etc.) to achieve results?

§ Impact: The positive and negative changes produced by the project, directly or indirectly, intended or unintended.

o What has happened as a result of the BORN project – either as intended or unintended, positive or negative?

o According to beneficiaries, what difference has the project made in their lives?

o How many people have been affected?

§ Sustainability: The extent to which the benefits (outputs, outcomes) of the project are likely to continue after donor funding has been withdrawn.

o What is the likelihood of continuation and sustainability of project outcomes and benefits after completion of the project?

§ What commitments (financial, human resources, etc.) have been made by stakeholders to maintain or improve results?

§ How will improvements in stakeholder knowledge, attitudes, capacities, etc., if observed, contribute to maintaining results?

§ To what extent is support available from the external environment to maintain or improve results?

o What are the major external factors that might influence, positively or negatively, the sustainability of project results?

o To what extent has the project’s design, implementation, stakeholder management, etc. contributed to the sustainability of project results?

o To what extent have the project’s exit strategies and approaches to phase out activities contributed and/or hindered the sustainability of results?

These key questions will be further adapted and prioritized to reflect unique implementation modalities of the BORN project, contextual factors, local strategies and partnerships, areas of programmatic interest, etc.[4]

5.0 Evaluation Methodology

The Final Evaluation will be a summative evaluation, employing a quasi-experimental design (towards a pre-post analysis of difference in differences) and a mixed methods approach.

6.1 Data Collection Methods

As one of the objectives of the evaluation is to assess progress towards expected outcomes by comparing results to project baseline (and midterm) studies, the study will include the core components of:

§ A household survey, wherein primary respondents are WRA (15 – 49 years of age) with a live birth in the 24 months / 2 years (index child) preceding the survey. In households that meet this inclusion criteria, a questionnaire will also be administered to male partners / family members who were present during the most recent pregnancy.

§ A health facility assessment including both interviews with key informants (health facility staff) on facility staffing and capacity, service provision, and data management; and observations of facility infrastructure and supplies.

§ A secondary data review, including health facility records and community group records.

The Consultant(s) is expected to propose additional data collection methods appropriate for meeting the evaluation objectives, with the expectation that these will be participatory in nature and should include qualitative methods (i.e. focus group discussions, key informant interviews), additional document review, etc focused on Gender Equality Outcome harvesting.

6.2 Sampling Strategy

The Final Evaluation is expected to employ a quasi-experimental design, towards a pre-post analysis of difference in differences between BORN intervention areas and identified comparison areas. For more information on the population of interest in intervention and comparison areas, see Annex 1: BORN beneficiary population.

A representative sample is expected to be collected for the household survey and health facility assessment; only the household survey is required to be conducted in comparison areas.

6.2.1. Household Survey

It is expected that the household survey will be conducted using a random multi-stage cluster sampling method to ensure cost-effectiveness and efficient survey implementation, as was done at baseline and midterm. It should be noted that final translated household and facility questionnaires have already been developed for BORN, and piloted and used during the baseline and midterm studies.

The household sample size estimations are expected to be determined by the Consultant(s), and the following should be noted:

§ A representative sample is expected to be collected for the household survey and health facility assessment.

§ Of the core components listed above, only the household survey is required to be conducted in comparison areas.

§ Towards an analysis of difference in differences between intervention and comparison areas, it is recommended that the sample estimation for the household survey use the following indicator to power the sample: % of women aged 15-49 who received antenatal care (ANC) by a skilled health provider at least 4 times during pregnancy

Table 1 below outlines key sampling guidance. In the sample calculation, please include an additional 10 percent sample of households to address non-respondents or incomplete questionnaires.

Table 1: Household Survey ANC results

Antenatal Coverage (ANC 4+)

Intervention Baseline

Comparison Baseline

Intervention Midterm

Estimated End line

BORN

57%

58.6%

64.2%

67.0%

In order to maintain a level of effort appropriate for male partners as a secondary respondent at sampled households, we can estimate a sample size equal to that collected at baseline and midterm (400 per arm) for interviews with male partners.

Additionally, no specific strategy is expected to be applied during the sampling distribution to ensure proportionate distribution of male and female index children.

6.2.2. Health Facility Questionnaire

A random sample of health facilities is expected to be drawn from the same clusters where the household survey is administered, to ensure cost-effectiveness and efficient survey implementation,[5] in the intervention area only.

Sample size estimations for the health facility questionnaire are expected to be determined by the Consultant(s), and the following should be noted:

§ A precision-based calculation should be used

§ Table 2 below outlines the total health facilities included in the BORN project

Table 2: Health Facilities as part of the BORN project

Health facilities

BORN

219

The sampling design, sample size estimations and calculations should be fully outlined by the Consultant(s), and the Consultant(s) is expected to propose sampling method(s) for any additional data collection methods (i.e. qualitative methods).

6.0 Purpose and Objectives of the Consultancy

The Consultant(s) will serve as the lead coordinator of the Final Evaluation of the BORN project, and will be responsible for study design and methodology, recruitment and management of the study team, supervision of data collection and entry/transcription, analysis of all data, and report writing.

Specifically, the roles and responsibilities of the Consultant(s)/team are to:

§ Recruit[6] and orient study team, including team leads, enumerators/data collectors, translators, and data entry clerks;

§ Review project documents, including Logic Model, Performance Measurement Framework, Project Implementation Plan (PIP), Gender Assessments, Gender Equality Strategy, project reports, sustainability plans, health investment plans and Baseline and Midterm Study reports;

§ Determine sampling strategies and sample size for intervention and comparison areas;

§ Review and revise existing data collection tools (quantitative tools used in the project’s baseline and midterm studies), and develop new qualitative data collection tools as required;

§ Develop data quality protocols to guide data collection/entry, including spot checking protocols;

§ Write draft and final inception report, including feedback from Plan International Nigeria and Plan International Canada

§ Work with Plan International Nigeria Country Office to obtain any necessary ethics approval;

§ Conduct briefings with study team, with Plan International Nigeria and Plan International Canada prior to enumerator trainings;

§ Ensure all tools are translated into local languages and piloted with women, men, adolescents, health facility staff, and community stakeholders prior to enumerator training;

§ Prepare and facilitate in enumerator training;

§ Oversee data collection, including supervision and spot-checking;

§ Develop data entry guidelines to ensure data quality;

§ Oversee data entry per agreed upon software, and conduct data cleaning as necessary;

§ Verify, process and analyze all raw data;

§ Validate preliminary findings with project stakeholders;

§ Write draft and final evaluation report, incorporating feedback from Plan International Nigeria and Plan International Canada

§ Present findings to Plan International, either in person or remotely

For responsibilities listed above, the Consultant(s) will work in collaboration with Plan International Nigeria Country Office staff and Plan International Canada as appropriate, including on refining and prioritizing evaluation questions; review, development and validation of data collection tools; selection of key informants; and securing necessary approvals for field work in intervention and comparison areas.

7.0 Deliverables and Timelines

8.1 Key Deliverables

The key deliverables expected from the Consultant(s) for this assignment are as follows:

§ Review all relevant documents, studies, and other data sources regarding MNH/SRH issues relevant to the BORN project, particularly baseline (2016) and midterm (2019) studies, with a special focus on sampling methodology utilized in those studies as well as performance indicators identified in the BORN project PMF.[7]

§ Inception Report: a detailed inception report is to be produced including the following:

o Study design and sampling methodology, building on the approach(es) utilized in the baseline and midterm studies

o Detailed work plan that includes all tasks by the Consultant(s) and team members, and incorporating overall study timelines

o Level of effort of each team member

o Reviewed and finalized data collection tools

o Detailed plan and timelines for enumerator training, tool piloting, and field work

o A detailed overview of considerations regarding gender equality and child protection / safeguarding throughout the study, and especially during field work

§ Data collection tools, including:

o Review of data collection tools used at baseline and midterm, with modifications and contextualization as necessary to ensure accurate data is collected.

o Development of new qualitative data collection tools required to meet the study objectives

§ Data Collection and Management Protocol for study team, and a Code Book, as appropriate for newly developed and/or revised data collection tools.

§ Enumerator Guidelines and Protocols for Data Collection

§ Cleaned, complete and validated quantitative data sets, qualitative transcripts, and field notes

§ Analysis of all quantitative and qualitative data in accordance with the evaluation objectives listed above, including critical analysis of the data through statistical treatment and triangulation with other sources and literature review.

§ Draft and final Evaluation Report[8]**

8.2 Timelines

The consultancy is expected to commence in October 2019, with field-based data collection to begin in January 2020. All data collection must be completed by the end of February 2020; and the final report must be submitted to and approved by Plan International Nigeria by early June 2020.

The Consultant(s) selected to complete this assignment will be responsible for the deliverables specified in the table below:

Table 1: BORN Final Evaluation Consultancy – Deliverables and Timelines

Deliverable

Timeline

Contract Awarded

End of September 2019

Orientation for study team and document review

Early October 2019

Detailed inception report

November 2019

Complete ethics approvals (if required)

November 2019

Final inception report, including tools

December 2019

Pre-test, translation and back-translation of tools

January 2020

Enumerator Training

January 2020

Data Collection

January- February 2020

Frequency Tables or quantitative data, and samples of completed qualitative instruments (i.e. translated and coded transcripts)

February 2020

Hard and soft copies of all (cleaned) data collected

March 2020

Draft evaluation report

April 2020

Final evaluation report, incorporating feedback from Plan International Nigeria and Plan International Canada

June 2020

Powerpoint presentation of summary of findings (overall) for Plan International, to be delivered in person or remotely

June 2020

8.0 Qualifications of the Consultant

Plan International is looking for an experienced multi-disciplinary team of consultants (either international or national) with experience in designing and conducting evaluations for MNH/SRH programs in developing country contexts. Both institutions and individuals are eligible to apply for this assignment. However, in the case of a group of consultants who are not associated with an institution, Plan International Nigeria would only assign the agreement with the lead consultant. The team should have the following qualifications:

§ Master’s degree or higher in International Development, Public Health, Gender Studies and/or other Social Sciences, Statistics or related fields (CVs required);

§ Minimum 10 years experience in administering studies, collecting data and producing quality evaluation reports, preferably for international non-profit organizations and/or multilateral agencies

§ Extensive experience working in Nigeria and international experience in assessing health outcomes in developing contexts; experience assessing gender equality outcomes strongly preferred

§ Demonstrated experience in designing project evaluations including proven experience in sound sampling, mixed methods approaches, tool development, enumerator training, quality assurance, etc.

§ Experience using participatory techniques in data collection, sensitive to gender issues

§ Demonstrated experience in quantitative and qualitative analysis

§ Knowledge and experience with MNH and SRH issues, policies and service systems, particularly in development country contexts

§ Familiarity with the social-cultural contexts in Nigeria and any related cultural, political, or religious sensitivities relevant to the completion of this assignment

§ Excellent writing and communication skills in English (Sample of work and references required) is mandatory for team leader, with team member fluency in Hausa required

[1] Number of facilities with commodities to be gathered through project documents

[2] Number of facilities with equipment to be gathered through project documents

[3] This will be provided to the Consultant(s) at the beginning of the assignment.

[4] Specific questions will be provided to the Consultant(s) at the beginning of the assignment.

[5] If required, additional clusters may be selected to achieve the target sample size.

[6] See Section 10.0. Upon request, Plan International Country Offices may provide applicants with recommended research / data collection firms, based on demonstrated expertise in the BORN baseline and/or midterm study.

[7] Copies of the baseline and midterm reports, GE Assessments, GE Strategies, and other relevant documents will be provided to the Consultant(s) at the beginning of the assignment.

[8] A suggested Table of Contents will be provided by Plan International at the beginning of the assignment.

How to Apply

Application Packages and Procedures

Qualified and interested parties are asked to submit the following:

  1. Letter of interest in submission of a proposal (in the form of an email) to [email protected]

Please share this by end of business day (EST) by 20th September 2019.

  1. Detailed technical proposal[1]** clearly demonstrating a thorough understanding of this Terms of Reference and including the following:

a. Description of methodology, including: sample size with detailed calculations (based on information available); and a detailed description of any new data collection tools proposed (beyond those listed in Section 6.1), including topic areas and example questions.

b. Demonstrated previous experience in mixed methods and other qualifications outlined in this ToR

c. Proposed management of study teams and field work

d. A proposed timeframe detailing activities and a schedule/work plan (including a Gantt chart)

e. Proposed approaches to data analysis, in response to overall objectives (see Section 4.0) and evaluation criteria (see Section 5.2)

f. Team composition and level of effort of each proposed team member and indication language skills of team members, including team leads

  1. financial proposal with a detailed breakdown of costs for the study

a. Itemized consultancy fees/costs

b. Estimated field mission expenses

c. Itemized administrative expenses

d. Validity period of quotations

  1. Curriculum Vitae(s) of all proposed team members outlining relevant experience
  2. Names and contact information of three references who can be contacted regarding relevant experience
  3. A copy of at least two previous reports of similar work undertaken on studies on MNCH/SRHR at least one, undertaken in Nigeria
  4. A Consulting Firm profile (if applicable).

The proposal will be scored on both technical (methodology) and financial (budget) aspects weighted at 70% and 30% respectively. Complete applications should be submitted electronically to: [email protected] with the subject line: ‘BORN Final Evaluation Consultancy.’

Closing date for submission of the application package is end of business day (EST) on 20th September 2019.

For development of the proposal, the following will be available upon request:

§ BORN project baseline and midterm study reports

§ BORN Project Annual Reports

§ BORN Gender Equality Assessment and accompanying Gender Equality Strategy

§ Baseline data collection tools

The offer must remain valid for no less than sixty (60) calendar days after the deadline noted above.

I