Introduction

Monitoring and evaluation (M&E) is critical to ensure the long-term success of climate adaptation initiatives, plans and actions (see Box 1 for definitions of M&E). It can help to demonstrate accountability to stakeholders and communities and show that funds have been allocated, used wisely and have resulted in desired outcomes.  This can also help to leverage any ongoing support for actions and for any further funding that may be required.  Importantly, M&E helps to track whether actions that you have implemented have been effective or not, and can help to determine whether any small tweaks or significant changes may be required to ensure success. M&E has an added significance in climate change adaptation:

• The time frames associated with climate change and adaptation outcomes are likely to be longer than those of traditional management programs. 
• There is uncertainty associated with the magnitude and nature of climate change, particularly at the local level, which can influence monitoring results and the ways in which they are evaluated. 
• A continually changing climate means that traditional approaches to measuring change, such as comparing monitoring results to static baseline conditions, may not be possible and instead you need to consider a moving baseline.

M&E is required to:

• Track the performance of activities undertaken during the development of an adaptation plan (e.g. stakeholder engagement activities). 
• Track pre-identified risk thresholds/trigger levels which identify the point at which to undertake new adaptation actions.
• Determine whether planned outputs and outcomes from adaptation actions have been achieved.

Designing an effective monitoring and evaluation program

mag-glass-320.jpg

Design of an M&E program is critical to ensure that appropriate and current information is used to review and inform decision-making at various stages of the adaptation planning process. M&E can also help to ensure that, if required, appropriate adjustments are made in existing actions (or plans) and reported to stakeholders and decision makers together with the rationale behind the change. Without a clear link between monitoring and decision-making, there is a risk that monitoring activities and resources will be seen as a drain on resources and may be discontinued. Therefore, M&E activities must be designed to show clear links between cause and desired effects, and that there are clearly articulated processes indicating how monitoring results will be analysed and reported, and used to support evaluation.

Monitoring programs should outline approaches to monitor and respond to trigger levels at which changed or new climate adaptation measures will be implemented. They should also contain indicators that will be used to determine the success of adaptation options. 

Monitoring trigger points for implementation of adaptation actions

Uncertainties associated with climate change effects on the coast—including magnitude and time scales of change, scales of impact due to future changes, evolving and contested values of communities and organisations—make it difficult to determine exactly when adaptation actions should be implemented. One way to address this is to sequence adaptation actions based on observed changes or more accurate projections; this enables adaptation actions to be implemented as required triggers or thresholds are met, rather than in a predefined time frame specified in a plan (Step-4 of C-CADS has more discussion on this approach). 

Trigger points can be identified and monitored and, when reached, can prompt the next action in a sequence. In identifying trigger levels, you need to consider the time required for the decision to be made and implemented. This should include the time needed for effective stakeholder engagement, plus any detailed investigation, design and development of actions.

In general, trigger levels should be robust to climate variability (e.g. they should not be based on a single extreme event). The triggers or indicators of triggers must, as much as possible, be observable, measurable and comprehensible to all stakeholders. This will help to get the social license to implement actions.  Trigger levels can either be physical, environmental, social or economic. Some examples include:

• physical: number of flooding events over a certain period
• environmental: number of mangrove seedlings per square metre of saltmarsh
• social: level of satisfaction with beach visits
• economic: costs of insurance premiums.

Additional discussion and information on indicators and trigger levels can be obtained from Monitoring and evaluation.

Monitoring outputs and outcomes of adaptation actions

digger.jpg

Monitoring and evaluation requires an assessment plan to measure the performance of adaptation efforts in terms of relevance, effectiveness, efficiency, results and sustainability of the results (see also C-CADS Step 4). Indicators are a core part of this. Indicators are things that we can measure. They help to determine whether objectives have been achieved for a specific program or project. Therefore, your adaptation plan should contain measurable objectives together with indicators for each of the objectives. Monitoring programs need to be in place so that they can collect appropriate data on each indicator and assess these against a baseline or reference conditions. Depending on the types of indicators, a variety of data will need to be collected and this may include biophysical, social and economic information. 

It is important to consider the measurability of the indicators to ensure that the collected data and information can be analysed appropriately before decisions are made. This also includes determining how to interpret monitoring information. To monitor any indicator it is critical to have baseline information against which the indicator will be compared. You could collect baseline data for comparison or select reference sites (controls) to assess changes against. Seek expert input when developing your monitoring plan; inappropriate monitoring can result in bad decisions being made. The effects of climate change mean that baselines for comparison will not remain the same and will change over time (shifting baselines), making it difficult to compare. 

The outcomes associated with climate change adaptation may not be realised for long periods which means that—over time—there is likely to be pressure to reallocate resources. It is important that from the outset, the need for and approaches used for monitoring are clearly documented and supported by external and internal stakeholders. It is also important that there are processes in place to ensure that monitoring will be continued over the lifetime of the project.  This may involve training of new staff, rewriting consultant briefs to extend or change monitoring programs, and building processes to ensure consistency.

Your monitoring program should include a variety of measurements which will enable you to report to your organisation and stakeholders about your activities. In general monitoring programs should include the following high level points:

• The current status of outputs, such as whether your adaptation action has been implemented or not (e.g. has the seawall been built?)
• Immediate and short-term outcomes that are a consequence of your adaptation action (e.g. has the seawall stopped storm surge inundation of the coastal strip?)
• Provision to re-evaluate the outcomes of your action in the longer term (e.g. after 10 years, is the seawall still stopping inundation of the coastal strip?).

The scale at which your monitoring program will operate is also important and should include consideration of ecological (e.g. ecosystem, species), landscape (e.g. catchment, river, creek), jurisdictional (e.g. local, regional) and management (e.g. policy, program, activity) scales. It is also important to consider time horizons which will determine when data are collected and for how long (duration). Many actions include a time lag before any results (outcomes) from that action can be detected (e.g. change in bank stability due to planting). Analysis of the monitored information also needs to consider the time scale of the measurement to account for variation in data (e.g. the analysis of climate trends occurs over long time horizons; however coastal erosion can happen over shorter time periods). Lastly, you need to take care around the timing of data collection, to ensure data comparability over time and between variables (e.g. both climate and tourism data are seasonal, but the seasons differ).

Importance of collaboration

Depending on the indicators selected, the M&E data may not be collected by a single organisation. In many cases, there is available detailed data collected by the community including businesses, non-government organisations, catchment management authorities or NRM groups, local, state and Commonwealth governments and other groups. Often the most challenging aspect of monitoring programs is to understand what data are already being collected and why, how they might be shared, and whether it is possible to adapt measures in some way.

Evaluation of the plan or project implementation and outcomes

Information from monitoring provides you with the resources to determine whether you are adapting successfully (or not) and also to draw lessons from your approach to help refine it. Your organisation may have existing evaluation approaches that should be used where possible, as this will help with consistency, cost effectiveness and stakeholder engagement with the results.

Evaluation helps you to learn from the results of your planned and implemented activities. Such learning (both positives and negatives) can help to refine your existing planning and implementation approaches in order to get the adaptation outcomes that you or your organisation desire.

In evaluating the success of your planning approaches (including your plan), a number of questions should be addressed.

• Is your adaptation plan being implemented with the necessary support and leadership from your organisation?
• Are you successful in seeking funds or financing mechanisms for your options?
• Are your reporting approaches working and is adaptation being received positively in your organisation?
• Have the objectives of each adaptation option been met?  (These may often be short-term objectives, as long-term objectives may not be possible to achieve as yet.)
• What adaptation options have worked and what did not work?  (This includes considerations of whether multiple benefits associated with objectives have been achieved.)
• Have there been any unforeseen outcomes emerged from implementing your plan?
• Have your stakeholders and community been effectively engaged throughout and do they remain engaged?  Have timeframes allocated for engagement been sufficient?

Logical framework or logic model or logframe

A basic logic model consists of inputs such as the resources (e.g. funding, expert knowledge, information) that go into a program, the activities the program undertakes (e.g. events, research, capacity building), the outputs that are produced through the activities (e.g. information sessions for organisations, new skills and new knowledge), and the impacts of the outcomes (intended and unintended short and medium term effects of initiatives and impacts) from the program.

Theory of change diagram

chalk-diagram.jpg

The theory of change works backwards; a long term goal is first visualised and then a pathway is developed that includes clear outcomes (long, intermediate and short term) that will lead to the goal.  Indicators, thresholds, evidence (if not an assumption) and assumptions are presented in each step as a causal pathway is visualised and a ‘change map’ is developed. All possible causal pathways may be mapped in the change map.

Process based evaluation

This approach is mainly used to assess progress of a plan before implementation of adaptation interventions and thus supports formative evaluation. In other words, process indicators can describe the processes that lead to a successful outcome. For example, a project’s performance can be measured during the planning stage based on interviews with community members to understand the nature of the issue and context of implementation.

Economic evaluation

This approach is mainly used to assess the overall outcome from a project. It is associated with more objective and quantitative evaluation methods such as cost-benefit analysis. Economic evaluation is often driven by organisational requirements to ensure efficiency of projects under limited funding resources and to present accountability to funding agencies. Prioritisation of adaptation options can be based on economic and qualitative evaluations. Complete economic assessments are often costly and time consuming and may be beyond the resources available to local governments.  

Assessing the currency of existing climate change adaptation plans 

Many users will already have plans that include adaptation options for their area of interest. As discussed, it is important to evaluate plans regularly and assess them against the quality, currency and relevance of the information and guidance they provide (but also reflect on the information that underpinned determining the selected and priority options in the first place).

It is also important to evaluate adaptation plans to ensure they contain sufficient information to enable adaptation actions to be implemented effectively. Plans should be fit for the purposes of your organisation and should be tailored in ways that ensure organisational buy-in and do not lead to external and internal challenges. 

Adaptation planning and action should be iterative and plans should reflect this and should be updated over time as organisational needs and capacity change, as new information and technologies become available, and in response to triggers.

Box 4 below contains a series of questions to guide to evaluation of existing plans.

Reporting to stakeholders and community

Having engaged with stakeholders and community throughout, it is essential that you report progress with adaptation actions to these groups. This is important to build political and constituent support for climate adaptation programs and projects. There is often a tendency for monitoring reports to be filled with detailed statistical data, graphs etc. However, audiences will find it easier to identify strengths and weaknesses of a given action if indicators are reported using consistent scale. Scales can be simple descriptors such as ‘very good’, ‘good”, ‘poor’, ‘very poor’, or more general statements such as ‘positive’, ‘of concern’, ‘action required’ etc. There is often good uptake with narratives or ‘stories’ about how actions are tracking and what is being achieved, and what changes are being made in response to the monitoring activities.

Internal stakeholders should not be neglected in your feedback and reporting of monitoring. They need to know what changes and why changes to plans are being made. The staff that are responsible for delivering various actions must be made aware of any changes that affect them.  Importantly, consideration must be given to how the plan is being amended so that the planning cycle can be continued.  It is important to bear in mind that monitoring and evaluation is not the end of the adaptation process, it is a new beginning!

Reviewing insights and direction reflected in other steps of the cycle

Formalising a process of evaluation and review at this stage of the adaptation planning cycle enables reflection on progress and lessons learned in implementing the planning cycle. This is essential as adaptation practices are continually evolving as new information becomes available, and as practitioners gain experience in implementing adaptation actions. This process should involve external and internal stakeholders. Lessons should be clearly documented and actions determined to ensure that changes are implemented going forward.

Checklist for Step 6 of C-CADS