Themes

Monitoring and Evaluation (M&E)

Monitoring is a continuous assessment that aims to provide stakeholders with adequate information on the progress or delay of ongoing activities. Evaluation on the other hand is a systematic and objective examination concerning the relevance, effectiveness, efficiency, and impact of activities in the light of specified objectives. M&E can help an organization extract relevant information from past and ongoing activities that can be used as the basis for programmatic fine-tuning, reorientation, and future planning: This includes establishing a baseline or database to monitor against objectives, outputs (throughout the policy process) and outcomes.

M&E serve several purposes. All stakeholders involved can learn to,

  • Determine whether the planned actions are producing the desired results
  • Take corrective action to ensure the desired results are achieved
  • Determine the extent to which initiatives for sustainable development are having a beneficial impact

Most of the M&E reports concentrated on a particular target demographic, such as those residing in certain project communities, rather than on the effectiveness of the intervention. Few M&E reports provided details about how services and results varied within the target population (e.g., disaggregated data on results in poor and marginalized groups, or variation in satisfaction with services or results), or on whether measures reached underserved groups. Additionally, the longevity of results and the continued use of services were rarely addressed, and very few assessments included any sort of savings or economical assessment.

The current emphasis on the use of properly managed services highlights two critical aspects. First, a proper sanitation involves more than just having access to a sanitation facility; it also involves the safe management and usage of a sanitation facility or service over a period. Second, if containment systems need to be emptied, or if faecal sludge must be handled, transported, treated, disposed, or used by service providers and municipal authorities, sanitation services can go beyond the domestic facility. Finally, the total cost of achieving relevant sanitation and hygiene results, as well as the expenses incurred to achieve public health benefits should be examined by M&E systems to enable evaluations of cost efficiency and cost-effectiveness. The degree of cost tracking and impact assessment varies significantly across programs; some only conduct basic cost tracking and quick evaluations of the positive results attained, while others conduct more thorough impact and cost assessments (such as large programs with research components).

India’s current policy statements in urban sanitation is tied to the Swachh Bharat Urban. It focuses on a domestic view that is linked to the objective of behaviour change towards sanitation and does not completely adopt to the range of sanitation within and outside the home, notably in the larger environment.

The following changes need mention:

  • Lack of a clear and uniform understanding of sanitation

    The focus is more on toilet construction and tracking initiatives, but sanitation even includes ending the practice of open defecation. A city or ward is notified as an Open Defecation Free OD(F) city or ward if, at any point during the day, not a single person is seen defecating in the open place.

  • Need for a monitoring framework to track OD(F) on an ongoing basis

    The monitoring framework must be designed to measure and be able to continuously track OD(F) in practice. Since slip-back is very likely, OD(F) status can only be verified after a few years of reliable findings.

  • Measurement challenges beyond OD(F)

    The outcomes of SBM's various components, such as capacity building, behavior change communication, and solid waste management, must be periodically reviewed and reported with a clear knowledge of how these components would drive the overall objectives.

  • Monitoring ‘Swachh Bharat’ (Clean India) beyond the Mission

    This includes defining and evaluating the results of the safe collection, treatment, and disposal of solid and liquid waste, which are crucial elements of the concept of "Swachh Bharat."

  • It is important to look at the survey instruments that are currently in use and their capacity to contribute to research, monitoring, and policy discussions around sanitation in urban and rural India, even though policy struggles to uniformly define OD(F) as a concept. This has impact on the type of data incorporated into sanitation policy discussions and the design of a sanitation matrix. The following characteristics of their designs are evident.
  • Scale and depth of existing survey instruments:

    Low-intensity surveys with inadequate independent verification of interviewed households are conducted and relied on self-declaration. Since this is not a sample, the results of the Census can be used to assess and contrast trends for the entire country.

  • Linking household sanitation infrastructure and OD

    Sanitation is correlated with the provision of sanitary facilities at the household level. Sanitation is expected after all other means of accessing sanitation facilities (home toilets, communal toilets, and public restrooms) have been ruled out.

  • Institutional sanitation infrastructure missing from the equation

    Surveys assume that people work close to their houses so that the toilet at home can meet their need to defecate during the day. However, it cannot be assumed that everyone has access to the bathroom at all times of the day

  • Census information is household-specific

    The census assumes that all family members have an equal status of defecation. Hence, the sanitation status is assigned often based on the interview of only one person and is 'presumed' to be applicable for other family members.

  • Disaggregated information limited

    This communication campaign may be intended to encourage the construction of household toilets. However, the male household members still defecate in the open because they believe that the toilet belongs to women. Information, Education & communication (IEC) programs may be designed to encourage male family members to utilize the restrooms as well.

  • Limited measurement of usage of sanitation infrastructure

    The risk in assessing the presence of sanitary infrastructure that depends entirely on government initiatives is, the evaluation will often become overly concerned with the costs of the infrastructure and any funding shortages rather than the results of such expenditures.

  • Focus on OD(F) at the cost of the other aspects of sanitation

    The behavior aspects of safe sanitation practices are separated from the larger sanitation chain itself when assessment frameworks based solely on OD(F) are used. Environmental issues relating to the release of human feces into the environment without treatment are disregarded in policy and implementation.

Situational Analysis on M&E of Sanitation operations

The National Institute of Urban Affairs in Delhi (NIUA) and the All-India Institute of Local Self Government in Mumbai (AIILSG) have been collaborating with the state Governments to conduct multiple workshops on ODF & inclusive sanitation as well as exposure visits with the goal to educate ULBs on the concepts of ODF, ODF+, and inclusive sanitation.

Challenges in septage management in the state include:

  • Urban regions lack inclusive sanitation understanding, particularly among citizens, service providers, and ULBs
  • Majority of stakeholders are not updated on contemporary technologies, industry-standard building techniques, operational processes, and safety and hygiene precautions
  • Operators and service providers lack adequate knowledge on desludging and do not employ safety equipment when performing their duties
  • Lack of inclusive sanitation-specific service level benchmarks
  • Inadequate funding for building inclusive sanitation infrastructure around the city
  • ULBs are not permitted to levy service fees or sanitary taxes

Dedicated service level standards to assess sanitation operations across all ULBs are established by the states. For instance, San-Benchmark Framework, is established by the State inclusive sanitation Cell and its partners under the guidance of the state inclusive sanitation committee. To achieve these benchmarks by the end of the evaluation time frame, each ULB sets an annual target in their city-level inclusive sanitation strategy for progress and enhancement in service in relation to these benchmarks. These benchmarking and targets need to be in accordance with policy milestones and the timetable for implementation.

The ULBs are graded annually according to the service level criteria for inclusive sanitation, a performance-linked system of rewards and penalties are developed to reward high performing ULBs and motivate ULBs with poor performance to improve. The state inclusive sanitation unit is responsible for developing a framework for assessing performance. Ward councillors, local communities, women's collectives, CBOs, etc. are engaged with inclusive sanitation operation monitoring and compliance activities

A framework for implementation of a state-wide monitoring system

Monitoring the state of fundamental services connected to water supply, sanitation, and solid wastes is vital to assess service level outcomes. The "Service Level Benchmarks (SLB)" framework of the Ministry of Urban Development (MoUD), and Government of India, is the framework for monitoring service levels that is currently used at ULB and state government levels. An operational system for achieving the national and state evaluation and monitoring capability is online monitoring of municipal services, finances, and a benchmarking system placed under the SLB framework This can be accomplished through-

  • State-wide adaptation and customization of online system protocol. to ensures indication reliability, and strengthen information systems at the ULB level
  • Build capacity for state and ULB officials to monitor service targets on an internal level
  • Across Service outcomes for necessities like storm water management, solid waste management, and water supply

Following an evaluation of usage, service, water accessibility, and notably whether informal sector employees have free access to sanitation facilities. .State government implementation of such state-wide systems across all ULBs will assist in establishing a solid foundation for State Finance Commissions (SFCs') future work and assist in determining price and cost recovery for these fundamental services. In accordance with the Fourteenth Finance Commission’s Terms of References (FCC ToRs), it will also enable the linking of expenditures and results. State and urban local governments must take an active part in its implementation.

The WHO/UNICEF Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (JMP) framework defines "safely managed" sanitation services across defined SLBs-

Service Level

Indicator (% of the population)

Sustainable

Utilizing a facility that is safely managed, regularly delivers expected levels of service, is subject to strict regulation, and has an established risk management plan

Safely managed

Utilizing a basic sanitation facility where feces is either properly transported and processed off-site or is safely disposed

Basic

Using an improved sanitation facility* not shared with other households

Shared

Using an improved sanitation facility shared with other households

Unimproved

utilizing a toilet that is shared with other households or that does not hygienically segregate human feces from human contact

No service (OD)

Practicing OD (defecating in bushes, fields, open water bodies or other open spaces)

Note: *Basic sanitation facilities are not shared, and of the following types: flush or pour flush toilets to sewer systems, septic tanks, or pit latrines, ventilated improved pit latrines, pit latrines with a slab, and composting toilets

(Table-1: Emerging ladder for JMP global reporting of progress in access to sanitation at home. Source: Centre for Policy Research)

The sponsors/partners in the project should try to achieve a 360° peer-based partnership in strategic consultations as well as in monitoring activities given the nature of sanitation, its correlation with multiple sectors, and its impact on all levels of government and local communities. A thorough understanding of the following elements should serve as a guide for the inclusion of partners, depending on the scope of sanitation

  • Organization of subnational governance across the country
  • Requirements of national, state, and municipal governments
  • Thematic components of the NUP- Legislation, sub-sectoral urban/territorial policies like mobility/transport, housing and basic services, environment, public space, infrastructure, local economic development, cultural/historical heritage, education etc
  • Selection of the regulatory tools that are needed to improve,
    • Methods used for managing land and administration, as well as the effectiveness and responsiveness of methods used for spatial planning
    • Instruments to increase urban competitiveness and market attractiveness, increased service delivery to rural and urban areas, generation and maintenance of information and databases on urban/territorial dynamics, etc.
  • The effects of current financial institution lending programs on local economies, internal migration, and spatial development
  • To confirm the implementation of gender and youth mainstreaming tools in public decisions, the degree of integration of the principles of gender equality and youth involvement in public policies, as well as the integration of women and youth into monitoring mechanisms, should be promoted.

Any government in charge of the exercise will need to make sure that the monitoring partnership arrangements remain all the way through the evaluation. An element of unbiased external examination may be incorporated if necessary. It is critical to clarify the functions and accountability of the various government entities involved in sanitation delivery process through below mentioned projects.

  • Define M&E responsibilities of implementing partners. Additionally specify the data that must be provided in terms of effects and outcomes, defining precise and definite processes, through use of Terms of Reference (TOR)
  • Establish the M&E responsibilities of key beneficiary/stakeholders. The community's involvement can range from simple monitoring to broader duties, such management of contractors, which can improve implementation efficiency and effectiveness and ownership
  • Provide staff members who are involved with M&E distinct levels of power.

M&E and Supervision of Progress at National level:

In India, sanitation is a state responsibility, and that effective city-level organizations and stakeholders are necessary for its on-the-ground execution and the maintenance of public health and the environment outcomes. Several factors, obstacles, and opportunities are unique to the specific situation of states and cities with respect to sanitation, and institutional variables. Therefore, in overall compliance with the National Policy, each state and city must develop its own sanitation strategy and city sanitation plan.

The central government of India assists to fulfil the goals of the urban sanitation policy by:

  • Engaging states in development of state-level sanitation strategies, and city sanitation plans
  • Assisting with the creation of the Detailed Project Report (DPR) in accordance with the city sanitation plan and funding requirements
  • Encouraging public-private collaboration on important initiatives/projects listed in the city sanitation plan
  • Offering states and cities technical advice and support for raising awareness and boosting capability
  • Periodic evaluation of cities' sanitation performance and establishment of a national award for top achievers
  • Utilizing current programs whenever possible to finance projects

Need for improved M&E of sanitation services

Ideally, monitoring would be crucial to guaranteeing proper implementation of sanitation programs and determining if the allocated funds are achieving sector goals specified by both the central and state governments. Additionally, focus should be on the way monitoring is used in making sanitation specific strategic decisions such as,

  • Creating sector goals and budgeting based on the outcomes of the sanitation data collected and analysed
  • Sharing the outcomes with important stakeholders for their inputs
  • Institutionalizing monitoring with suitable institutional responsibilities for effective implementation

Monitoring the utilization of these funds and the corresponding outputs is a crucial part of carrying out the sanitation program. It is also required to keep track of the reforms committed by the state government and ULBs to receive funding from central government initiatives like the JNNURM and ILCS along with state government programs. Monitoring would help various stakeholders understand the utilization of funds and basis that, the States governments and ULBs can request for funds. The funds would be released based on the progress made on implementation of sanitation process

Management Information System • Clarity regarding monitoring methods and metrics • Integrating monitoring systems and current datasets • Web - based reporting Institutional strengthening and capacity building • Create and manage database along with analysis • Utilize monitored data for decision - making • IT capabilities Information dissemination • Incorporate more WSS sector information to websites • Frequent updates on sector status Performance monitoring and Decision support system • Periodic performance evaluation based on accepted and standardized metrics • Use key performance metrics to inform decisions on funds allocation Improved Monitoring System (Figure-1: Measure to improve the urban sanitation monitoring system. Source: Monitoring Urban Water Supply and Sanitation in Maharashtra - A Paper)

M&E and Supervision of Progress at State and City Levels:

The State government will monitor and evaluate the performance of its cities, therefore procedures for data collection and reporting using outcome indicators must be developed. ULBs must monitor household (and other establishment) adherence to the results and procedural criteria it has established. Consideration will be given to the introduction of citizen report cards, citizen monitoring committees, self-assessment systems, inter-city competitions, etc. Non-Governmental Organization (NGOs) and Community-based organizations (CBOs) will be crucial to M&E.

The M&E of the implementation must be considered as an essential component of the City Sanitation Plan (CSP) by the City Sanitation Task Force and the Implementing Agency. The mechanisms to be used in monitoring implementation include:

  • Administrative information gathered from implementing consultants' and contractors' reports
  • Task Force field visits to several locations throughout the city
  • CBOs/NGOs provide organized feedback to the Task Force and the implementing agency about their own neighbourhoods or through field engagement processes and the implementation progress
  • Independent third-party assessments
  • Concurrent Assessments by Survey Organizations

Having the findings and reports public is a crucial part of M&E so that other stakeholders can provide comments and suggestions. Another method of enlisting the assistance of city stakeholders is to present critical information during news conferences and monthly task force meetings. To measure the success of inclusive sanitation initiatives, state and city governments can work towards defining an indicator metric (e.g., toilet and sewage networks services coverage). These indicators can cover wide range of mandates across government initiatives and indicators such as Swachh Survekshan (list of indicators). Some of the indicators considered for monitoring of sanitation services may include,

  • Coverage of toilets
  • Coverage of sewage networks services
  • Collection efficiency of the sewage network
  • Adequacy of sewage treatment capacity
  • Quality of sewage treatment
  • Extent of reuse and recycling of sewage
  • Efficiency in redressal of customer complaints

To provide a complete picture of change and facilitate learning, M&E systems should combine the following evaluation techniques employed during different time periods. To confirm results and promote changes in large-scale operations, routine monitoring must be complemented by periodic monitoring or assessment (for example, annual sample surveys) and qualitative processes.

Routine monitoring visits

Household surveys

Focus group discussions
  • Ongoing collection of data
  • Often goes from community volunteers (who are tasked with visiting households and gathering data) to local government or program officials
  • Same set of questions are asked to every responder (or group)
  • Measures attitude, knowledge, behavior, intention, and influence on outcomes
  • Open-ended questions are difficult to process, yet included
  • Closed questions typically have a list of options for responses
  • Discussions about perceptions and opinions regarding certain issues, or changes with groups of internal or external stakeholders
  • Requires experienced facilitators and careful planning (to encourage people from speaking up)

Structured observation

Community-wide assessments

Secondary information
  • An observer spends several hours monitoring and documenting household hygiene and sanitation procedures
  • More accurate than self-reported practice, but it is a costly and time-consuming method that is also prone to reactivity (due to the observer's presence)

Like the neighborhood crossings that are a part of some sanitation verification procedures

M&E of sanitation and hygiene may be assisted by a variety of sources of data (such as the collection of population and clinical health data and the use of government poverty markers)

(Table-2: Tools and processes of M&E. Source: Monitoring and evaluation for rural sanitation and hygiene- Guidelines)

Monitoring tools

Real-time tracking mechanisms (Recording and Reporting)

Usage of real-time tracking mechanisms fora compliance and tracking reduces the possibility of illegal discharge and assists in guaranteeing that all the pumped septage makes it to the disposal site. Additionally, certain septage programs may decide to use it as documentation when paying septage carrying subcontractors.

The state engages with technology partners to demonstrate other treatment module advances, such as creating a database of containment users using GIS, enabling GPS for desludging activities, etc. With support from the ward councillors, community groups, women's collectives, CBOs, etc. would be engaged with inclusive sanitatio operation monitoring and compliance.

The Maharashtra cities of Wai and Sinnar, which rely on onsite sanitation systems such as septic tanks, are effectively adopting citywide faecal sludge and septage management plans for the secure collection, transportation, and treatment / disposal of feces. The contracts between the city governments and a commercial service provider specify who oversees performing the planned cleaning of septic tanks. Faecal sludge treatment facilities have also been established for the safe transportation of collected septage. These are likewise handled by a private organization. Cities need monitoring systems to keep tabs on the delivery of these sanitation facilities in view of all this institutional equipment.

To guarantee efficiency and sustainability of operations, the cities are putting in place sophisticated monitoring systems such as,

Case study: SaniTab

The first crucial step in city sanitation planning is the evaluation of the sanitation scenario. SaniTab, a smartphone/tablet-based survey tool, was created in 2015 with the aim of assisting cities in assessing their sanitation situation while gathering the necessary data through household surveys and creating a city level database. It consists of an Android app where forms can be deployed. Surveyors fill out applicable forms, input data, and submit them to create databases. Any type of questionnaire may be readily coded and submitted using the admin panel where the forms are fully customizable. Vernacular languages are also supported.

The app is simple, quick, and may be used by many surveyors simultaneously. For the submission of data gathered on the ground by numerous surveyors, the app is connected to a server. The application supports text-based queries as well as photo and GPS coordinate collection for quick and simple spatial mapping. It is feasible to recognize and concentrate more on the sensitive locations by capturing geographic characteristics. The gathered information is connected to a dashboard for immediate examination. Online dashboards are created using the data given to the server and offer real-time updates so that the city administration may quickly analyse the survey findings and explore numerous relationships.

SaniTab as a monitoring tool- helps in making educated decisions about future service provision and sanitation planning requires access to a comprehensive database of onsite systems. It is extremely rare and expensive to create this kind of database. When scheduled desludging was implemented in cities, every property had to be inspected. This offered a chance to create a distinctive and thorough database of on-site systems in the cities. A dashboard was created for city administration, along with the deployment of a SaniTab form and routine monitoring for desludging. The information gathered includes specifics about the technology, the quantity of properties it serves, the forms and sizes, truck accessibility, etc. of onsite systems.

Property- level data

Performance of desludger

On-site sanitation system database
  • Location details- Locality name, ward and property numbers, GPS location for easy identification
  • Location of the property in a vulnerable area – required to confirm coverage
  • Type of property- Residential, commercial, and institutional
  • Readiness of property to receive service
  • Property Ownership- Either an owner or tenant
  • Number of emptying trips made per property
  • Use of protective gears
  • Challenges faced while emptying
  • Incidence of any septage spill
  • Mandate to leave 2-inch septage during service

Type of Disposal System- Covers aspects such as

  • Individual or shared
  • Number of properties sharing the system
  • Shape, Size & location
  • Accessibility from road
  • Type of access cover
  • Record of the latest emptying of septic tank

(Table-3: Data aspects captured for monitoring the desludging services. Source: Monitoring and evaluation for rural sanitation and hygiene- Guidelines)

Case study: SaniTrack

SaniTrack focused on digitizing the paper forms and customer acknowledgements system. SaniTrack is an online, GIS-enabled monitoring system that was created to track complete desludging activities and offer a much easier way to record daily operations. The desludger can schedule and record daily operations with onscreen signatures using the app and web modules of SaniTrack, which is like an e-commerce app. Web dashboards that display real-time data on geographic coverage, household preparation, safe transportation, etc. are also made available to city managers. To reduce human error and manual calculations as much as possible, it is designed to digitize the paper-based forms and receipts system. SaniTrack may support a variety of sanitation models, such as demand-based and planned models, services delivered by the public or private sector, and single or multiple service providers.

Case study: DIGIT

The Government of Odisha (GoO) has initiated a programme called "SUJOG - Sustainable Urban Services. The DIGIT Sanitation platform, which supports efficient sanitation management, is presently operational in 70 urban local bodies (ULBs) in Odisha. The management of inclusive sanitatio in these locations is the exclusive emphasis of this platform. Through digitalization, the program seeks to bring about a new era in the state's approach to urban governance. Citizens, the ULB administration, multiple government departments, and many other players in the value chain of service delivery are among its beneficiaries. Odisha, one of the top states in the field of urban inclusive sanitatio, has already proven to have efficient city-wide inclusive sanitatio systems in several ULBs with the installation of 108 FSTPs spanning 107 cities by September 2022 and additional expansion across all 115 ULBs in progress. The state has collaborated with eGov Foundation (eGov) to build a Digital inclusive sanitatio platform to handle the ULB-level operations and evaluating the effectiveness of the inclusive sanitatio service delivery to assure the ongoing and effective delivery of inclusive sanitatio services to all its citizens.

Click here to visit the Self-Assessment for this Theme!

© 2024 NFSSM Alliance     Terms of Use     Privacy Policy