1.    How is monitoring and evaluation of adaptation plans different to regular M&E?

There are three principal differences between M&E for traditional management purposes and for adaptation planning.

• The time frames associated with climate change and the associated adaptation outcomes are likely to be longer than those of traditional management programs. 
• There is uncertainty associated with the magnitude and nature of climate change, particularly at the local level, which can influence monitoring results and the ways in which they are evaluated. 
• A continually changing climate means that traditional approaches to measuring change, such as comparing monitoring results to static baseline conditions, may not be possible, and a moving baseline must be considered.

2.   Selecting indicators and linking them to objectives

During the development of an adaptation plan, you need to select clear and measureable indicators to monitor performance of each planned action. These indicators can help to:

• track the performance of activities undertaken during the development of an adaptation plan (e.g. stakeholder engagement activities)
• track pre-identified risk thresholds/trigger levels which identify when new adaptation actions should be undertaken
• determine whether planned outputs and outcomes from adaptation actions have been achieved.

Guidance on indicator selection can be obtained from Identifying Indicators.  

Evaluation of approaches, activities and outcomes should be done at each stage of planning and implementation.  For example, if developing an engagement strategy in step 1 of a planning cycle, such as C-CADS 1: Identify the challenges, you should include M&E indicators to evaluate the success of your approach. This can help to make any necessary adjustments to the process.

3.    Designing an effective monitoring and evaluation (M&E) program

Design of an M&E program is critical to ensure that information generated through the program is used to:

• inform decision-making
• make appropriate adjustments to adaptation planning (if required)
• report to stakeholders and decision makers. 

Without a clear link between monitoring and decision-making, there is a risk that activities and resources for monitoring will be seen as a drain on resources, which can be discontinued.

Accordingly, M&E should be designed with:

• clear objectives in mind
• clear processes for analysis and reporting of the results of monitoring
• transparent and objective evaluation methods.

Despite the long timeframes associated with climate change, it may be sufficient to monitor indicators in the shorter term to demonstrate that appropriate plans are in place and some of the adaptation actions have been implemented.

Monitoring can be expensive and as a result it is often neglected by decision makers.  A well-designed monitoring program can reduce waste and help ensure that analysis and conclusions are robust.  It is often important to engage experts to assist at the M&E design stage.

4.    Broad categories of adaptation monitoring

In general, three broad categories of monitoring are required in an adaptation process. They are:

4.1    Monitoring of work undertaken in preparing an adaptation plan

There is a variety of activities that need to be undertaken during the development of an adaptation plan (Overview of C-CADS).  The performance of these activities should be assessed and evaluated to ensure they can be improved in future iterations, or to help others learn and build from what has been done before.

As an example, one simple but effective way to make use of monitoring results is to record lessons learned while conducting stakeholder engagement around risk identification, and using the learning to design future consultation in adaptation planning.

Ensuring monitoring is not overly intensive, expensive or time consuming is important if it is to be a help and not a hindrance.

4.2    Monitoring trigger points for implementation of adaptation actions

The uncertainty associated with climate change, including magnitude and timescales of impacts, together with the sometimes conflicting values of communities and organisations, make it difficult to determine exactly when actions should be implemented. This can be addressed by sequencing adaptation actions based on observed changes or availability of more accurate projections, rather than according to a rigid pre-defined time frame.

To implement this approach, you can identify and monitor trigger indicators which, when reached, can stimulate the implementation of the next action in a sequence. In identifying trigger indicators, it is important to consider the time required for the decision to be made and implemented.  Time should be allowed for effective stakeholder engagement, or for any necessary investigation, design and development of actions.

Trigger indicators should be robust to natural climate variability (e.g. not triggered by a single extreme). As much as possible, they should be measurable and understandable to all stakeholders. This will help with getting the social license to implement actions. 

Indicators can be physical, environmental, social or economic. Examples include:

• physical –the number of flooding events over a certain period
• environmental –  the number of mangrove seedlings per square meter of saltmarsh
• social – the level of satisfaction with beach visits
• economic – the costs of insurance premiums.

Additional discussion and information on rationale and use of triggers are provided in Section 8 of Information Manual 4: Costs and Benefits and What is a pathways approach to adaptation? , and Applying a pathways approach.

4.3    Monitoring performance and outcomes of adaptation actions

Adaptation plans should contain measureable objectives together with indicators for each of the objectives.  Monitoring programs need to collect appropriate data on each indicator and assess these against a baseline or reference conditions. Depending on the types of indicators that are being used, a variety of data will need to be collected. This may include biophysical and socio-economic information. Table 1 shows some types of indicators together with an explanation of their purpose.

The selection of indicators and the design of measuring techniques should both consider how the data and information will be analysed to support decision making. It may be possible to collect some baseline information or to select reference sites (controls) for comparison. It may be necessary to seek expert input in developing your monitoring plan, as inappropriate monitoring can result in bad decisions.

Monitoring should include information on:

• outputs – implementation of the planned adaptation action (e.g. has the seawall been built?)
• immediate and short-term outcomes (e.g. has the seawall reduced storm surge inundation to the target levels?)
• longer term outcomes (e.g. after 10 years, is the seawall reducing inundation to the target levels for that date?).

Important considerations in planning M&E for adaptation actions include the following factors.

• Scale of monitoring should include consideration of ecological, landscape, jurisdictional and management scales.  
• Time horizons, which affect when data are collected, and over what timeframe. Many actions include a ‘lag’ before results can be detected (e.g. a change in bank stability due to planting).
• Timing of data collection needs to ensure data comparability over time and between variables (e.g. climate and tourism data are seasonal, but the seasons differ, and this must be accounted for in M&E design).

5.    Undertaking monitoring

Box 2 summarises some key points about M&E program design.

5.1    Collecting the right data

The choice of data to collect will depend on the nature of the project, its intended outcomes and what is feasible and cost-effective to collect.

Monitoring inherently means dealing with data. It is usually assumed that we need precise measurements to inform our decisions.  However, managers are often comfortable with less precision, depending on the type of decision they are making.  In reality, there are often trade-offs between precision and cost and timeliness of data collection.  Environmental managers make accurate estimates against categorical assessment scales based on their experiences (Cook et al. 2014) and this may be sufficiently precise for some types of decisions.

5.2    Issues of scale

Choosing the appropriate scale, or balance between scales of choice, is important when measuring adaptation initiatives, given that this can affect the appropriateness of indicators and the monitoring design. Sometimes, monitoring information from one scale (e.g. individual projects) can be collated to inform evaluation at another scale (e.g. program scale), but sometimes additional or alternative indicators are needed, for example because there are different targets (e.g. a program seeking to build greater institutional or financial support).

Kennedy et al. (2009) identifies multiple scales at which monitoring and evaluation can be conducted, including:

• ecological scale – ecosystem (e.g. rainforest), community (e.g. sub-tropical), species (e.g. cassowary)
• landscape scale – catchment, river, stream, creek
• jurisdictional – international, national, state, region, local government
• management – policy or strategy, program, project, activity
• temporal scale – short, medium, long term

Adaptation actions vary across sectors and by intervention type. Intervention types can be divided into categories of activities: knowledge-based (generating and using relevant information including the assessment of risk and vulnerability), capacity-based (creating the internal capability within target communities and organizations to perceive and evaluate climate risks and to formulate responses), and focal area technically-based (designing and implementing specific measures to manage risk and enhance resilience, both hard and soft measures relevant to the focal area).

Table 1: Example from an adaptation plan showing indicators, associated measures, and an explanation of data that could be collected to assess change over time.

5.3    Gathering and using monitoring information

It may not be necessary for all monitoring and evaluation data to be collected by a single organisation. In many cases, detailed community-collected data exist, as do data collected by business, non-government organisations, Catchment Management Authorities/NRM groups, Federal and state governments and other groups. Often the most challenging aspect of monitoring programs is to understand what data are already being collected, how they might be shared, and whether it is possible to adapt measures in some way to make use of already available resources. For example, a recent review of climate related monitoring and evaluation by local government (Thomsen et al. 2012) found at least five existing reporting requirements in NSW and four in Queensland with data that were directly relevant to climate adaptation monitoring and evaluation.  

5.4    Analysing and using monitoring and evaluation data

The type of analysis required for data will depend on the nature of the data that are being collected and the design of the monitoring program that has been undertaken.  It is important that monitoring results are used appropriately and that they are not used to support poor decision-making.  We strongly recommend that users work with experts to analyse and interpret monitoring results appropriately.  This is particularly important in climate change adaptation where the multidisciplinary nature of the discipline will generally result in a range of qualitative and quantitative variables being collected. Each of these will require different analytical approaches, some simple and some not. 

In considering analysis, it is useful to use approaches that will be clear to stakeholders.  The more complex the analyses, the greater the challenge of communicating results to stakeholders. We provide guidance about working with consultants (see Using consultancies). 

Data analysis needs to be conducted in a way that ensures managers use the results. For quantitative measures, statistical analysis should be used to discern between trends and anomalies, and to be clear, where possible, about cause-effect relationships. Decision support tools such as multi-criteria analysis (MCA) and Bayesian models have been used in climate adaptation planning, but they require more specialist data analysis skills.  Other similarly specialist  techniques include fuzzy-cognitive modelling, or multi-variate data analysis techniques such as pattern analysis. 

In monitoring programs, data and information are often compared to reference or baseline conditions to assess the degree of change resulting from a management action.  A key consideration when doing this is that baselines may not remain the same in a climate affected future.  Changing climate may result in changing reference points.  In designing monitoring programs and undertaking analysis this should be taken into account 

5.5    Evaluation

Evaluation of adaptation planning includes the need to track actions that are being undertaken and provide feedback to relevant stakeholders regarding the success of actions and any adjustments that may be required.  Evaluation also includes considering and assessing a project or program as a whole, and determining whether the objectives of the program were achieved.  Monitoring of indicators provides the data that underpin evaluations, but it is analysis and interpretation of data that develops knowledge.  It is this knowledge that can support decisions about whether objectives in an adaptation plan are being achieved.

There are a variety of ways in which evaluations can be conducted.  It is useful to take an approach that ensures evaluations are done with sufficient rigour and logic, and also considers the data and information that have been collected.  Villanueva (2011) identified four types of evaluation (Table 2):

1. input-output based evaluations/outcome, impact or results evaluation
2. process based evaluation
3. evaluation of behavioural change
4. economic evaluation.

More information on each of these is provided here.

Table 2: Approaches and methodologies for evaluating adaptation interventions. Source: Villanueva 2011, p. 20.

6.    Communicating with stakeholders

Communicating with key stakeholders, including the community, is an important way of building political and constituent support for climate adaptation programs and projects.

Reports are often filled with detailed statistical data and graphs. Without careful explanation, the core findings from these data may be lost to general readers, some of whom may be important stakeholders for the project or program.   

Some useful things to consider in communicating M&E results are using:

• a similar scale to report across an indicator set
• a 3 or 4-point scale.  You could use words like ‘very good’, ‘good’, ‘poor’, ‘very poor’ to describe your scale (as in the Great Barrier Reef Outlook Report 2009), or ‘positive’, ‘of concern’, and ‘action required’ (as in the Sunshine Coast Council Sustainability Indicators Report) or you could develop your own wording (as in NSW State of the Parks Report 2004) (see list of example M&E reports at end for links). 
• colours to provide a simple grading schema, although the choice of colour is important – red may be inappropriate.
• a scale system associated with targets, for example ‘target met’, ‘target not met’ (e.g. the La Trobe University Sustainability Report).

You can report on confidence in your indicator score, for example, the National Sustainability Council report (2013) uses shaded circles to indicate adequate evidence/limited evidence for each indicator.

It’s often tempting to include words such as ‘increase’ or ‘decrease’ within an indicator.  Indicator wording should be neutral, but you can capture change in the way you report.  For example, the National Sustainability Council in its report (2013, for example p. 180) uses an arrow system representing improving, stable, and deteriorating trends.

If you have quantitative data, you might like to present data in a table format.

You might choose to consider summary statements, such as the summaries of impacts on environmental, economic and social values in the Great Barrier Reef Outlook Report (2009). If producing a composite score across a range of indicators, be careful to justify how you came to this score (e.g. whether you are averaging across criteria or whether one criterion is more important than another).