Factors that Affect the Effectiveness of Monitoring and Evaluation System of Local Non-Governmental Organizations in tracking projects performance. The case of Addis Ababa City Administration

Table of Contents

ACRONYM

ABSTRACT

CHAPTER ONE

1. INTRODUCTION
1.1 Background to the Study
1.2. Non-Governmental Organizations (NGOs) in Ethiopia, Addis Ababa
1.4. Research Questions
1.5. Obj ective of the study
1.6. Hypothesis
1.7. Significance of the Study
1.8. Scope ad limitation of the Study
1.9. Definition of terms
1.10. Organization of the study

CHAPTER TWO
2.1. Introduction
2.2. M&E System
2.3. Information from the M&E system
2.4. A review on theories:
2.4.1. Program Theory( Logical frame approach)
2.4.2. Theory of Change
2.4.3. Evaluation Theory
2.4.4. Effective M&E system
2.5. Factors Affecting the Effectiveness of M&E System
2.5.1. Selection of Tools and Techniques
2.5.2. Budgetary allocation
2.5.3 Technical capability of M &E offcers/experts
2.5.4. Training and technical support
2.6. Review of empirical literature
2.7 Conceptual framework

CHAPTER THREE
3. The Research design ad Methodology
3.1. Description of the Study Area
3.2. Research Design
3.3. The Target Population
3.4. Sampling Strategy
3.5. Data Collection Tools and Techniques
3.6. Pilot Testing the Research Instrument
3.7 Data Analysis ad Presentation
3.8. Ethical Issues

CHAPTER FOUR RESULT AND DISCUSSION
4.1. Validity ad Reliability
4.4.1. Tools and techniques used in M&E system
4.4.2. Budget Allocation
4.4.5. Technical capability of M & E staffs
4.5.1. Multiple Regression Equation

CHAPTER FIVE
5.1. Introduction
5.2. Summary of Research Findings
5.3. Tools and Techniques
5.4. Budget allocation
5.5. Tecnnical competency of staffs
5.6. Training and tecnnical assistance
5.5. Conclusion
5.8. Recommendations
5.9. Recommendation for further research

REFERENCE

Annex

ACRONYMY

Abbildung in dieer Leseprobe nicht enthalten

List of Tables

Table 2.1: General structure of the Log-frame

Table 4.1 Socio- Demographic Characteristics

Table 4.2 organizational M&E system is effectiveness

Table 4.3 responsible party in the managing the M&E

Table 4.4: The availability of standard tools and techniques

Table 4.5: Numbers of standard tools and techniques

Table 4.6: Types of tools and techniques used

Table 4.7: Rate the simplicity of utilizing those tools and techniques

Table 4.8: Rate the contribution of tools and techniques

Table 4.9: Budget allocation for M&E

Table 4.10: Rate contribution of Budget allocation

Table 4.11: M&E educational background

Table 4.12: Rate the factors/statements using the scale given

Table 4.13: Rate the technical capacity

Table 4.14: Training on M&E system

Table4.15: Rate the effectiveness of the training

Table 4.16: Rate the contribution of technical capacity of employees

Table 4.17: Technical assistance of towards effective M&E system

Table 4.18: Rank factors affecting the effectiveness of M&E

Table 4.19: Multiple Regression Analysis

Table 4.23: Regression Model Summary

List of Figures

Figure 2.1 Conceptual frame work

ACKNOWLEDGEMENT

First and foremost; my heartfelt gratitude is to the almighty God who gives me strength and wisdom at every step of my life and during the whole research period. Following, I would also like to express my sincere appreciation to my advisor Dr. Solomon Markos for the guidance, assistance and constant support throughout this study. I also want to acknowledge all

organizations and individuals who gave me data and materials. Last but not least, to my friends those who had contributed assistance, either directly or indirectly, yet their names are not cited here, they deserve my greatest gratitude

ABSTRACT

This study looked at the determinants influencing the effectiveness of M&E systems in LNGO’s within Kolfie Keranio and Nefas Silk Lafto sub cities of Addis Ababa city Administration. Its main objectives was: factors affecting the effectiveness of M&E system of LNGO and expected to help LNGOs, donor agencies and the project management in better understanding of the M&E systems and how to improve them. The population and the sample for this research were 101 and 79 LNGOs which were found within kolfie keranio and Nefas Silk Lafto sub cities of Addis Ababa city administrations. Probability sampling system backed by simple random sampling method through a lottery system was employed to collect primary data from the Executive directors/project managers/program/project coordinators/M&E officers from each LNGO and document review through observation was employed for triangulation purpose. 79 questionaries’ were distributed and 72 of them were collected with a 91.1% response rate. The data collection was conducted after checking for validity and reliability of the questionnaire. SPSS version 20 was used for the data analysis process. The descriptive statistics was used to describe the general result of the variables. The mean average scores and frequencies were presented in a table by considering the four factors that affects effectiveness of M&E considered in the study. The collected data was analyzed using correlation and regression analysis to establish the relationship between the variables using SPSS computer program. The findings indicated that the selection of tools and techniques, training, budget allocation, technical assistance and technical skill/competency to be used in an M&E system determines its success or failure and were also relevant for an effective M&E system. The study concludes and recommends that the LNGOs should be flexible to allow the modification of their M&E systems; an educational policy to be put in place to emphasis on M&E across the sector to have a quantity and quality M&E experts; work with their donors together in developing a database of M&E systems information across the sector, should also develop harmonized training curricula for the M&E staff and conduct training workshops, shall ensure that there is adequate early planning for project M&E activities (including the human and financial resources) and finally, the LNGOs and the charities and societies Agency of Ethiopian Government (CSA) put together and discuss the budget allocation and its consequences by evaluating the pros and cons of them.

Key word; factors of Effectiveness, Monitoring & evaluation, Addis Ababa city Administration

CHAPTER ONE

1. INTRODUCTION

1.1. Background to the Study

Many scholars define project in many ways. Project as “project can be considered to be any series of activities and tasks that have a specific objective to be completed within certain specifications, Has defined start and end dates, Has funding limits (if applicable), consume human and nonhuman resources (i.e., money, people, equipment) and are multifunctional, i.e., cut across several functional lines (Robert K Wysocki, 2000).

He defines it as a sequence of unique, complex, and connected activities that have one goal or purpose and that must be completed by a specific time, within budget, and according to specification. PMI defines project as a temporary endeavor undertaken to create a unique product, service, or result. The temporary nature of projects indicates that a project has a definite beginning and end. The end is reached when the project's objectives have been achieved or when the project is terminated because its objectives will not or cannot be met, or when the need for the project no longer exists (PMBOK, 2005). All projects have a cycle in the way in which projects are planned and carried out follows a sequence beginning with an agreed strategy, which leads to an idea for a specific action, oriented to-wards achieving a set of objectives, which then is formulated, implemented, and evaluated with a view to improving the strategy and further action. In order to manage all the cycles of projects, project management knowledge is essential. Ethiopian management institute gives a definition for project management as “it is the application of knowledge, skills, tools, and techniques to project activities to meet the project requirements Project management is the process of planning, organizing, directing, and controlling of company resources for a relatively short-term objective that has been established to complete specific goals and objectives (World Bank 2012). Furthermore, project management utilizes the systems approach to management by having functional personnel (the vertical hierarchy) assigned to a specific project (the horizontal hierarchy). On the other hand, involves project planning and project monitoring and includes such items as Project planning, Definition of work requirements, Definition of quantity and quality of work, Definition of resources needed, Project monitoring, Tracking progress, Comparing actual outcome to predicted outcome, Analyzing impact and Making adjustments. The potential benefits from project management are Identification of functional responsibilities to ensure that all activities are accounted for, regardless of personnel turnover, Minimizing the need for continuous reporting, Identification of time limits for scheduling, Identification of a methodology for trade-off analysis, Measurement of accomplishment against plans, Early identification of problems so that corrective action may follow, Improved estimating capability for future planning and Knowing when objectives cannot be met or will be exceeded (Harold Kerzner, 2000).

Since Monitoring and evaluation (M&E) is one core phase of project cycle management and is useful to all projects, big or small, it helps in identifying project areas that are on target and those that need to be adjusted or replaced. Different types of projects require different types of M&E systems. However, the most popular M&E systems with projects are the ones developed on M&E Matrix, based on the Logical Framework Approach to monitoring and evaluation (Welsh et al., 2005).

Monitoring is the process of regular and systematic collection, analyzing and reporting information about a project’s inputs, activities, outputs, outcomes and impacts. Monitoring is therefore a way of improving efficiency and effectiveness of a project, by providing the management and stakeholders with project progressive development and achievement of its objectives within the allocated funds (World Bank, 2011). It therefore keeps track of the project work and the management when things go wrong.

Evaluation is a scientific based appraisal of the strengths and weakness of the project (Hunter, 2009). It is therefore a comparison between the actual and the planned. Evaluation is a means of checking efficiency, effectives and impact of a project. Evaluation involves: looking at what the project intended to achieve, assessing progress towards what was to be achieved and impact on targets, looking at the effectiveness of the project strategy, looking at the efficient use of resources, opportunity costs and sustainability of the project, and the implications for the various stakeholders (Hunter, 2009 and Shapiro, 2011).

Monitoring and to some extent evaluation, fall under the control functions of project management. It provides regular feedback that helps the organization track costs, personnel, implementation time, organizational development, economic and financial results and compare what was planned to actual performance (Hunter, 2009). Effective Monitoring and evaluation system mainly determined by the ability to track performance and able to provide instant information for management decision making. Although very essential in improving performance, is also very complex, multidisciplinary and involves skill intensive processes (Hunter, 2009). Building an effective M&E system is a requirement for the growing pressure to improving performance which is also one of the requirements by the NGOs and donors to check on the effective use of the donor funds, impact and benefits brought by the projects. Hence, there is a need for the establishment of rules for constructing minimum parameters for monitoring and evaluation of projects that can be used to track progress and effectiveness (World Bank, 2011).

Adequate skilled staff and financial resources are vital ingredients in developing an effective M&E system (Harold Kerzner, 2000). Failure to ensure a reasonable proportion of resources is spent on this aspect of project management is likely to impede internal learning and result in the poor operation of the M&E system.

1.2. Non-Governmental Organizations (NGOs) in Ethiopia, Addis Ababa

According to the EFDR proclamation number 621/2009 the non-governmental organization is ' private voluntary association of individuals or other entities, not operated for profit or for other commercial purposes but which has organized itself for the benefit of the public at large and having as its objective the promotion of social welfare’.

The proclamation also defines local Non-governmental organizations as ‘those organizations that are formed by Ethiopian residences and got more than 90% of their project fund from abroad’. In Ethiopia context, nongovernmental organizations have been engaged largely in diverse activities across nationwide mainly during the famine of 1970/71 in response of humanitarian crisis and often respond to development problems via projects and all their activities theoretically bounded by defined time frame, need allocation of scarce resources and have objectives to achieve and make use of donor funds for charitable as well as public benefit purposes. Before 2009, there no written proclamation regarding NGOs and the sector was managed through disaster prevention and preparedness agency in which did not have any institutional and legislative framework to govern them.

According to Addis Ababa city finance and economy bureau, there are more than 321 local nongovernmental organizations currently registered, sign agreement and have been providing different services to the community that focused on in different developmental issues including health, education, disability, economic strength, food security and others.

Monitoring and evaluation (M&E) are essential components of project cycle management (Rist, Boily& Martin, 2011).

Building and sustaining effective monitoring and evaluation system is admittedly not an easy task for it requires continuous commitment, champions, time, effort, and resources. In addition, it may take several attempts before the system can be tailored to suit a given governmental or organizational policy, program, or project; but it is doable (Kusek, 2004). 13. Statement of the Problem Monitoring and Evaluation Systems provide important feedback on the progress of programs/projects. That is, the success or failure of projects, programs and policies throughout their respective life cycles. These systems constitute a powerful, continuous management tool that decision makers can use to improve performance and demonstrate results. Monitoring and Evaluation Systems (especially Results based) have a special capacity to add to the learning and knowledge process. These systems provide for learning and knowledge, since by providing continuous feedback to managers, they promote organizational learning through a cycle involving the reflection on progress, learning and Mows for adjustments in the course of programs or projects where need be . (Harold Kerzner, 2000). Information supplied by Monitoring and Evaluation Systems is used as a crucial management tool in achieving results and meeting specific targets.

Such information, which reveals the level of progress, performance and problems, is crucial to managers striving to achieve results.

(Jaszcolt et al. 2010) argue, these systems are actually one of the “techniques” for managing program/project implementation, especially because they provide an early warning to project management about potential or actual problems. Subsequently, when problems are identified, questions about assumptions and strategy behind a given program or project may be raised . This way, they aid development managers make choices and decisions on running projects and programs. Monitorng and Evaluation Systems can also aid in promoting greater transparency and accountability within organizations and government (Elarold Kerzner, 2000).

The Report of Addis Ababa BoFED (2016) exhibited that the monitoring and evaluation practice of local NGOs that are found in Addis Ababa and engaged in social programs is found in its infant stage and where it is done the information is not made public. Though the federal CSA proclamation emphasized the extent to which a development intervention has contributed to the achievement of desired targets should not only be examined after its completion, it requires continues monitoring during its implementation. It also criticized the inadequacy of monitoring practices on development projects implemented by local NGOs operating in the metropolis. The report of Addis Ababa BoFED (2016) also stressed that even if some organizations tried to perform effective M & E system, there was a gap to assign a responsible personnel that regularly follows the activity.

Many local NGOs have been complained the EFDR Proclamation of CSA 621/2009 article 90 in its restriction towards budget assignment for Administrative and Program costs in which it which in turn urge NGOs to minimize or cut out their M&E costs to keep administrative expenses within the 30:70% limit and therefore reserved to hire a qualified M&E experts and in place the M & E activities as expected. Some organizations use a team and the monitoring team feasibly missing the necessary capacity or strength to carry out their work effectively, or they may be approaching their work using incorrect methodologies. They lack the necessary budget, training and technical support/assistance both from the organizational and the donor sides in order to undertake their task effectively and efficiently. Some organizational leaders do not have the required knowledge and awareness about the importance of M & E, rather they think as a waste of time and resources. In addition most NGOs did not have M&E system that guides the determination of measurements, performance indicators, means of measuring and its tools applied through the process.

The four independent variables discussed in this study, namely; utilization of tools and techniques, availability of funds, competency of M&E staff and training and technical support provision stakeholders’ had a high propensity of influencing effectiveness of monitoring and evaluation systems towards tracking project success. There are limited studies on the factors determining effectiveness of monitoring and evaluation systems. Thus, this study sought to fill the gap by undertaking a study on the determinants of effectiveness of a monitoring and evaluation system for projects.

The study aims at establishing the extent to which utilization of tools and techniques, availability of funds, competency of M&E staff and training and technical support provision influence effectiveness of M & E system and then to recommend on how to adopt a result-based M&E system that is more effective and efficient for their project interventions.

1.4. Research Questions

The Researcher was guided by the following key research questions:

- How does the choice and application of M & E guideline, tools and techniques impacts the effectiveness of M&E system to wards tracking project progress.
- How does budget allocation to Monitoring and Evaluation activity affect the
effectiveness of M&E system to wards tracking project progress?
- How M&E knowledge, training and technical support will contribute to effectiveness of M&E system to wards tracking project progress.
- How Technical capability of M & E officers/experts influence the effectiveness of the M&E system to wards tracking project progress.

1.5. Objective of the study

The general objective of the research was to find out the factors affecting the effectiveness of monitoring and evaluation system of local nongovernmental organization within Kolfie keranio and Nefas Silk Lafto sub cities of Addis Ababa city Administration, Ethiopia.

The specific objectives of this research work were:-

i. To evaluate whether the choice and application of M & E guideline, tools and techniques impacts the effectiveness of M&E systems of local nongovernmental organizations in Addis Ababa city Administration
ii. To assess the impact of budget allocation to the implementation of Monitoring and Evaluation activity of local nongovernmental organizations in Addis Ababa city Administration
iii. To examine how M &E Training and Technical support will contribute to effectiveness of monitoring and evaluation system.
iv. To evaluate how the technical capability of M & E officers/experts influence the effectiveness of the M&E systems

1.6. Hypothesis

Hypothesis 1

- The availability and utilization of techniques and tools affect the effectiveness of M&E system in tracking progress of the project intervention.

Hypothesis 2

- The allocation of budget for M&E activity affects the effectiveness of M&E system in tracking progress of the project intervention.

Hypothesis 3

- The competency of M&E staff affects the effectiveness of M&E system in tracking progress of the project interventionA

Hypothesis 4

- The Training and technical assistance provide affects the effectiveness of M&E system in tracking progress of the project intervention.

1.7. Significance of te Study

This study would particularly help the NGO’s staff, donor agencies and project managers a better understanding of the M&E systems and how to improve them to meet the expectations of the stakeholders, as well as provide valuable information for future interventions.

It would inform policies towards setting up of monitoring and evaluation systems, and show how M&E can be used as a powerful management tool to improve the way organizations and stakeholders can achieve greater accountability and transparency.

The study was therefore beneficial to NGOs, donor agencies, project managers and project management students who are involved in the designing and implementation of effective M&E systems. Although this paper looks at the factors affecting the effectiveness of the M&E systems to wards tracking project progress for projects by NGOs within Kolfie Keranio and Nefas silk Lafto sub cities of Addis Ababa city administration, it is also relevant in areas where M&E systems are to be implemented. This study was also contributed to the body of knowledge since it can be used as a reference material by researchers. The study was also identified areas related to M&E field that will require more research, hence a basis of further research.

1.8. Scope and limitation of the Study

The study was conducted within Kolfie Keranio and Nefas Silk Lafto sub cities of Addis Ababa city administration. They are the most populated sub cities having a large number of peoples living under poverty line and hence a number of aid organizations also existed for humanitarian assistance and support. The researcher focused on those two sub cities since more that 31 % of local NGOs that are registered in Addis Ababa city administration were found here. Executive directors/Managers/Project managers/project/Program coordinators/M&E staffs of these organizations were the respondents of this study.

The findings of the research would be more fruitful, if conducted widely by incorporating or including other organizations and parameters which are in similar situation.

Thus, the Study was limited to finding out a clear understanding of how tools and techniques, budget allocation, human resource capacity and training and technical support influence governmental organizations in Addis Ababa city administration.

1.9. Definition of terms

Factors: These are influencing elements.

Effective: successful in producing a desired or intended result

Tracking the performance of mooitoring ad evduaaon systems: is the ability of measured project activities to provide users of the system access to quality and accurate information that can be used for organizational learning and decision making.

M&E System: this is a set of components which are related to each other within a structure and serve a common purpose of tracking the implementation and results of a project

M&E Training: This is the acquisition of practical tools that enhance result-based management by strengthening awareness in M&E. it is also the acquisition of practical tools that enhance result-based management by strengthening awareness in M&E

M&E: This is the process of systematically collecting and analyzing information of ongoing project and comparison of the project outcome/impact against the project intentions.

Budgetary Alocation: Amount of resources allocated for M&E in a particular NGO.

LNGOs(Loca Non-Governmental Organization): It is ' Ethiopian residence voluntary association of individuals or other entities, not operated for profit or for other commercial purposes but which has organized itself for the benefit of the public at large and having as its objective the promotion of social welfare and got more than 90% their fund from abroad.

Project Management: It is a scientific application of modern tools and techniques in planning, financing, implementation, controlling and coordination of activities in order to achieve desired outputs according to the project objectives within the constraints of time and cost Project: This is a specific activity to be carried out, which consumes resources and has a beginning and an end Level of Training: The extent to which an employee is well versed with a particular skill Tools and Techniques: These are methods and procedures used to meet the project’s M&E needs.

1.10. Organization of the study

The study was organized into five chapters. Chapter One consists of the background of the study, statement of the problem, research questions, General Objective, Specific objectives, Significance of the Study, Scope of the Study, Limitations of the Study, Definition of terms and organization of the study.

Chapter Two looks at available literature done on factors affecting the effectiveness of monitoring and evaluation systems by scholars who have studied the subject in other monitoring and evaluation contexts and the conceptual framework which outlines the relationship between the dependent and independent variables identified in the subject of study.

Chapter Three constitutes the research methodology, which is divided into the Research design and Methodology, description of the Study Area, research Design, target Population, sampling strategy, data collection tools and techniques, pilot testing the research instrument, validity and reliability, data analysis and presentation, ethical considerations in the research.

Chapter Four covers data analysis, presentation and interpretation.

Chapter Five covers the summary and discussion of findings, conclusion, recommendations and suggestions for further research.

CHAPTER TWO

2. LITERATURE REVIEW

2.1. Etroduction

Monitoring and evaluation systems have been in existence since the ancient times (Kusek and Rist, 2004), however today, the requirements for M&E systems as a management tool to show performance has grown with demand by stakeholders for accountability and transparency including the government (Gorgens et al2010). Development banks and bilateral aid agencies also regularly apply M&E to measure development effectiveness as well as demonstrate transparency (Briceno, 2010).

2.2.M&E System

Monitoring and Evaluation is a combination of two processes which are different yet complementary (Gorgens and Kusek, 2009). It is therefore a process of systematically collecting and analyzing information of ongoing project and comparison of the project outcome/impact against the project intentions either using logical frame work approach or result based approach (Hunter, 2009). An M&E system, on the other hand is a set of components which are related to each other within a structure and serve a common purpose of tracking the implementation and results of a project (SAMDI, 2007). It is therefore an integrated system of reflection and communication that support project implementation. An M&E system is made up of four interlinked sections, which are: setting up of the M&E system, implementation of the M&E system, involvement of the project stakeholders, and communication of the M&E results (Guijt et al., 2002).

Theoretically, ‘an ideal M&E system should be independent enough to be externally credible and socially legitimate, but not so independent to lose its relevance’ (Briceno, 2010).

2.3. Information from the M&E system

Information got from M&E can be used to serve many purposes. A successful M&E system is therefore measured by the utilization of the information got from it (Briceno, 2010). It should also be able to: clarify the expected impact of the project; show how progress and impact will be assessed; collect and analyze necessary information for tracking progress and impact, give details reasons for success and failure, and show how this information can improve future actions (Welsh et al. 2005). Monitoring and evaluation is an integral part of the project’s design, implementation and completion (Chaplowe, 2008). It is useful to all projects, big or small, since information got from it enables better decision making by helping to identify project areas that are on target and those that need to be adjusted or replaced. Although different types of projects require different types of M&E systems, collection of data and information at all levels of the projects life cycle adds value to every stage of the project by ensuring project targets are met. Weaknesses in the project are also identified on time and collective measures taken (Gorgens et al. 2010).

An effective M&E system also calls for the interaction between the employees, procedures, data, technology and key stakeholders, in order to ensure feasibility and ownership (Chaplowe, 2008). Although monitoring and evaluation are not of inherent value by themselves the information they provide is significant to improving performance (Mackay, 2010), which helps in learning from what/how we are doing or have done by focusing on efficiency, effectiveness, impact, relevance and sustainability (Hunter, 2009). According to the international benchmark, the M&E allocation should be 10% - 12% of the total program cost; however most programs in Ethiopia were seen to allocate less than this (World Bank (2012).

There was also an inconsistency in the choice of performance indicators among the programs which led to incoherent and incomprehensive M&E systems. Most educational projects that were implemented could provide a review team with a logical framework (Wegayehu Huluka, 2014) which rarely infuenced the decision making process being used to inform project and program designs as well as inform policies.

2.4. A review on theories:

2.4.1. Program Theory( Logical frame approach)

the capability of the program to fix a problem by addressing the needs in the need assessment. It also gives tools to determine areas of impact in evaluation (Sethi and Philippines, 2012). Most NGO’s deal with human service programs that are designed to improve the society, which are at times designed and redesigned in due course (Hosley, 2005). The program theory uses logical framework approach as its methodology in detail which can be represented graphically through the logical model. It is used in guiding stakeholders’ engagement, the management and evaluation of outcomes (J-Pal, 2003).

2.4.2. Theory of Change

Theory of change is part of the program theory that emerged in the 1990s as an improvement to the evaluation theory (Stein and Valters, 2012). A theory of change is a tool used for developing solutions to complex social problems. It provides a comprehensive picture of early and intermediate term changes that are needed to reach a long term set goal (Anderson, 2005). It therefore provides a model of how a project should work, which can be tested and refined through monitoring and evaluation.

A theory of change is also a specific and measurable description of change that forms the basis for planning, implementation and evaluation. Most projects have a theory of change although they are usually assumed (CARE, 2013). The theory of changes helps in developing comprehensible frameworks for monitoring and evaluation. It is mainly used by NGOs and donors to articulate long term impact on projects (James, 2011).

2.4.3. Evaluation Theory

Mark (2005), in his discussion on why a theory is important to evaluation practice, states that evaluation theory is a way of synthesizing prior experience.

[...]