See programmatic comments on coordination projects. A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

The proposal is missing an opportunity to take a more systematic approach to coordination: to think about what the sponsors are really trying to achieve, how they will know if they are achieving it, and how they will adapt to changing circumstances or proactively test new approaches and learn from the outcomes. In several places the statement, “WDFW will monitor and report …” is used. What actually will be monitored? How will the variables monitored be measured? Are there hypotheses about relations between these variables?

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives
The proposed work includes data management (storage, management, and reporting) - 10%, monitoring and evaluation (framework and approach) - 10%, developing and tracking biological objectives - 10%, review of technical documents and processes - 10%, project proposal review - 10%, coordination of projects, programs and funding sources within subbasins - 20%, facilitating and participating in focus workgroups on Program issues - 20%, and information dissemination (technical, policy, and outreach) - 10%.

Significance to regional programs: A lengthy statement lists and describes various projects, workshops, programs and fora that WDFW participates in as part of the implementation of the Fish and Wildlife Program through the research plan, subbasin plans, and MERR plan. Can outcomes from these activities be identified?

Problem statement: This describes the 2010 separation from CBFWA and the need to continue funding for the individual states to continue to participate in the implementation of the Fish and Wildlife Program through the coordination and information transmission that was formerly accomplished through CBFWA. A statement of the problem to be addressed is contained in the first paragraph of the "significance to regional programs" section.

Objectives: The project has specific four objectives. Each objective is stated as a task rather than as a desired outcome.

In general, the Washington Department of Fisheries and Wildlife is one of the active participants in the process of conserving and restoring Columbia Basin fish and wildlife that are affected by the building and operation of the hydro system. “WDFW provides the technical and scientific expertise needed to address mitigation issues related to fish and wildlife management in the Columbia River Basin.” How will the outcomes associated with the four project objectives be measured?

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Explanation of performance: short statement indicating that deliverables have been on schedule.

Major accomplishments: WDFW has participated in the formation of a tri-state coordination effort and continued participation in various fora. Participation in a variety of meetings is enumerated. The proposed plan of work offers few specifics. “Past accomplishments” lists meetings, conference calls, and briefings. This is a good sample to provide information on outcomes and what coordination activities lead to better outcomes.

Management Change: The statement is made, “It should involve stating hypotheses then implementing actions, monitoring, reporting, and evaluating outcomes to provide a clear sequential structure to decisions required in the continuing evolution and implementation of the Program." Expanding this statement into hypotheses for the plan of work would be desirable.

Adaptive Management: An 8-step process for coordination is identified. A reference should be made to BPA [Bonneville Power Administration]. 1997. Wildlife mitigation program final environmental impact statement. DOE/EIS - 0246. U.S. Department of Energy, Portland, OR., “Each of the following eight steps is required to support a transparent, accountable, and effective planning, implementation and evaluation process.” What is presented is a general statement of the principles of adaptive management, without specific indication of how the project will incorporate it in its operations. Could this process be filled in for one of the key coordination issues?

ISRP Retrospective Evaluation of Results

This is a new project, so technically there are no results to evaluate. Historical data on performance is available with the project, “Proposal RESCAT-1989-062-01 - Program Coordination and Facilitation Services provided through the Columbia Basin Fish and Wildlife Foundation.” See the section, “Reporting & Contracted Deliverables Performance.”

Financial performance and history: “Beginning April 1, 2011, WDFW began coordination as an entity independent of CBFWA.” This project has no financial history or review of progress. Previous work was completed under the management of the Columbia Basin Fish and Wildlife Foundation. No explanation of financial performance or financial history is provided.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

Project relationships: A list of related coordination projects is provided without explanation of the specific types of interaction.
Emerging limiting factors: A brief description of strategies employed to address climate change and invasive species, with links to references. Are there regional coordination limiting factors?

Geographic interests: The Columbia Basin and specifically for the State of Washington overall, which encompass the Lower Columbia, Middle Columbia, Upper Columbia, and Snake River Recovery areas.

Value-added: The proposed plan of work offers few specifics. The list of 85 meetings, conference calls, and briefings is a good sample to provide information on outcomes and what coordination activities lead to better outcomes. The value-added describes just a couple of outcomes. Can coordination outcomes be related to "improving the status of fish and wildlife resources in the Columbia Basin"? “Regional coordination includes but is not limited to participation in various forums, technical committees, and workgroups associated with implementing the MERR Plan, Subbasin Plans, and the Council's Research Plan.” These are essentially inputs to coordination. What are the outcomes from these regional coordination activities? How do they improve the status of fish and wildlife resources? How, for example, can the statement, “Increase the efficiency and effectiveness of RM&E efforts by facilitating communication and coordination among project sponsors and funding agencies within the Basin,” be measured and documented? Listed are some 30 statements that might be converted into hypotheses or used as cases for testing hypotheses. What are the most effective ways of organizing and coordinating? What types of coordination activities work best? What percentage of the meetings is facilitated and does this improve outcomes? Does coordination provide value? How would the value of coordination be measured and compared against the costs?
Deliverable 3 is about documentation. Where are the specifics of documentation addressed?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has six deliverables related to the objectives. These include documentation of participation in meetings and types of coordination, and the preparation of reports. A summary list of meetings attended, coordination activities performed and contributions to documents is enumerated. It would be desirable to develop metrics about outcomes and measures of outputs from coordination activities.

Methods and metrics: These are described in greater detail in the "objectives" section than in the "deliverables" section because the objectives are written as tasks. Not much detail is presented on how these activities will be accomplished or metrics to be used to evaluate performance.

Work elements: Two work elements are identified – 122. Provide Technical Review and 189. Coordination-Columbia Basinwide for which no metrics are associated. Can metrics be identified to go with these work elements? Ideally, the hypothesis(es) developed in the proposal would be measured during the course of the coordination activities and results presented in the report on this project. There are many ideas discussed in the proposal that are amenable to this approach. Selecting a few of the most important questions, concerns, or hypotheses and monitoring them is recommended.

The program coordination part of the work elements section provides more specific enumeration of coordination activities. Value added by these activities is generally referenced in the case of fish passage and lamprey recovery but without specific examples of the contribution of WDFW staff. The same comment applies to opportunities to evaluate performance – it is presented as a possibility but without a specific plan.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the two work elements are published but do not provide adequate guidance on the methods and metrics. Guidance is available from ISRP (2007-14:2). Project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives.

Modified by Dal Marsters on 4/17/2012 3:04:04 PM.

A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

The proposal contains so much detail that it is difficult to review. Future proposals would be improved through more summary and synthesis of relevant information.

The proposal provides extensive insight into a scientific perspective on program coordination. A number of hypotheses are presented about the coordination process and its outcomes. The approach provides narrative findings for the experience gained by CBFWA. The insights provide compelling analysis for developing a sound scientific perspective on program coordination early in the evaluation process.

Proposal strengths:

• The proposal is fully documented; methods and accomplishments are exhaustively described.
• The limiting factors statement addresses large-scale issues that have the potential to limit the effectiveness of the project. This is rare among proposals.
• The proposal provides extensive insight into a scientific perspective on program coordination.
• Performance metrics have been identified and used to evaluate project effectiveness.

Weaknesses:

• So much detail is presented that it's difficult for the reviewer to track proposal content. The project is not only complex in itself it is also undergoing significant structural change.
• It is unclear where sturgeon or anadromous fish fit into CBFWA activities.
• It is sometimes difficult for external reviewers to assess the effectiveness of the project.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The CBFWA proposal offers a detailed narrative review of the coordination history from 1989 to the present. It analyzes changes in coordination that have occurred and reasons for them. The proposal also raises a number of policy issues to be addressed by Bonneville and the Council.

Problem statement: The problem statement is overly long, but at its end a summary conclusion adequately states the problem the proposal is designed to address.

Objectives: The proposal is focused around seven objectives, but the implicit overarching objective of this proposal is to coordinate disparate regional coordination projects around subject-matter themes.

Each objective has several deliverables that include development and maintenance; communications; coordinating, implementing, and facilitating; collate and summarize; attend and participate.

Emerging limiting factors: The proposal identifies three limiting factors for effective regional coordination: 1) perception of fairness, 2) participation and buy-in, and 3) adequate funding for both facilitation and participation. The proposal aims to address recent changes in these limiting factors.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Financial performance and history: This section provides an adequate description of a financial history complicated by inconsistencies in reporting and budgeting dates and by a changing project structure.

Performance: An adequate short description is provided.

Major accomplishments: A detailed description of the project's major accomplishments in its former version – the Annual Work Plan. Because it has been coordinating activities since 1989, CBFWA has extensive coordination experience and the proposal lists many insights. Further, the proposal provides detailed discussion on why some members have left CBFWA coordination and facilitation services. The new project configuration will begin with this funding cycle. Metrics of performance are numbers of meeting attendees and qualitative evaluation of outcomes made possible by actions of the CBFWA in various fora. A stakeholder survey was also conducted.

Response to ISRP comments: A complete description of ISRP comments and CBFWA response in terms of developing tools to monitor impact is provided.

Adaptive management: A good description of changes in CBFWA focus and configuration in response to changing circumstances in the region.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

The geographic interests of CBFWA encompass the entire Columbia Basin for both fish and wildlife. CBFWA's goal has been to include all sovereigns and action agencies in the coordination process. In addition, CBFWA encompass the Willamette/Lower Columbia, Middle Columbia, Upper Columbia, and Snake River Recovery areas.

Project relationships: The statement provides a history of the changing configuration of CBFWA and a long list of coordination, monitoring and other programs throughout the region with which it is coordinated.

Tailored questions, data: The proposal provides a long description of the Status of the Resource website and its function. Information provided in the presentation indicates that the project has added tribal data coordinators to the already participating agency representatives. The narrative analysis of the regional coordination problem is excellent and provides useful insights; more attention to identification of a scientific component to the proposal would help to plan for future success.

More findings like this one would be valuable, "These factors illustrate in high relief the Fish and Wildlife Program’s recognition that coordination efforts and funding should be focused through a set of functional activities that need coordination, and not necessarily on the basis of entities desiring coordination funding." This seems to represent a critical principle for organizing coordination activities. Another important set of coordination hypotheses, "solutions intended to increase coordinated efficiencies and effectiveness. This includes developing coordinated synthesis reports, sharing data and information through scientific papers and science/policy forums, holding regular workshops focused on specific species, methods, or geographic areas, and on several topics, the drafting of basin-wide management plans." In this same section, "CBFWA Members recognized the role the organization can play in delivering useful technical, science-based products associated with protection, mitigation and enhancement of the Columbia Basin’s anadromous and resident fish, and wildlife."

The proposal suggests that “the adaptive management framework for which coordination” be used. “Adaptive management” is mention in 4 of the 7 project objectives, in many of the deliverables, and development of an “adaptive management framework” is frequently mentioned. Can this framework be more explicitly and specifically identified? How have the many lessons learned been built into each adaptive management cycle? What is the typical length of an adaptive management cycle? In the adaptive management section a very interesting process is described that suggests that funding arrangements changed the need and approaches to collaboration. This is a very interesting insight. It does not illuminate the adaptive management framework often discussed, but it does indicate that funding is an important driver as to participation in coordination activities. The identification of factors that may limit the effectiveness regional coordination including perception of fairness, participation and buy-in, and adequate funding for both facilitation and participation, is an insightful and useful hypothesis. Did the conduct of a consumer satisfaction survey in 2010 help in assessing these variables?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has 24 deliverables related directly to the seven objectives. Each is described in detail.

Project components: The project has eight components, each described in some detail: Data Management (5%); Monitoring and evaluation (10%); Developing and tracking biological objectives (5%); Review of technical documents and processes (10%); Project proposal review (5%); Coordination of projects, programs, and funding sources within subbasins 20%); Facilitating and participating in focus workgroups on Program Issue (25%); and Information dissemination (20%).

Work elements: CBFWA lists many work elements (11).

Methods and metrics: Methods are described in detail in several different sections. Metrics are also described. Measurement of performance is through numbers at meetings, outcomes of coordination, and a survey of stakeholder satisfaction.

One form of assessing effectiveness is the output of meeting results, documents, and other evidence of outcomes of coordination and facilitation actions. Another way to assess effectiveness is input from the state, federal, and tribal agencies involved in the process, who are well-positioned to assess this effectiveness. Other entities interacting with the program but not formally part of the CBFWA functions are also able to provide input. Some possible approaches to at least showing the degree of success would be, as a minimum, letters from each agency/tribe responding specifically to a series of questions as to how well the CBFWF program is meeting their needs in key areas and how the program might be improved. This request could also be addressed to some outside entities that participate with the workgroups. Some questions should address not only how well the CBFWA is meeting agency and tribal needs, but benefiting the salmon and other basin resources in specific ways that otherwise would not occur. It would also be of interest to know how the program involves entities such as the Oregon and Washington state agencies and the Corps of Engineers, and if more coordination among them and CBFWA entities is possible or can be expected.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the 11 work elements are published but do not provide adequate guidance on the methods and metrics. The project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives. The relative value of “electronic meetings” vs. “face-to-face sessions” would be useful to study. Another worthy topic for review is dimensions of “facilitation.”

What part of “the ISRP for reporting metrics for regional coordination (ISRP 2007-14)” will be implemented? The document suggests (ISRP 2007-14:4), “Metrics of Impact: (e.g., how effective is the project: what is its added value of the coordination project) changes in behavior, value to the members, user evaluation of product utility, lack of redundancy, member assessment of effectiveness and impact, benefits to fish and wildlife of enhanced coordination activities, specific projects or resources benefited by the project, specific effect of coordination on conservation and management.” Where in the proposal are these suggested metrics of impact operationalized? A hypothesis worth testing is whether change in funding has led to decreased regional coordination.

Modified by Dal Marsters on 4/13/2012 12:24:40 PM.

The response includes a detailed description of the types of coordination and facilitation services that CBFWA is or could be providing. It adds information that was missing from the proposal regarding the operational meaning of general coordination terms. The response states that without CBFWA, the BPA, NPCC and the ISRP would find it difficult to staff activities such as holding meetings and providing website services. In addition, the response states that the "Columbia River Basin is dependent on the coordination, administration, and technical services that the CBFWA provides" for two monitoring and evaluation coordination partnerships (PNAMP and CSMEP). CBFWA activities in this regard include subcontracting services, participation in meetings, and website services. In 2005 CBFWA began to further expand its role to data inventory and reporting services. The response further states that the CBFWA role extends beyond coordination of its members to services for non-member entities.



Overall, a better demonstration is needed that CBFWA's services are provided in the most cost-effective manner. The response provides a better description of the association of the $900k budget line to the "annual report", including good detail on the range of products associated with the report. However, questions remain as to whether the costs are reasonable, especially given that a template of the website is already up and running.



The response also provides a description of the withdrawal of the Kalispel and Spokane tribes from membership. It appears that the interests of these two entities were not being addressed at the policy level; however, little explanation is provided as to why this situation exists. Does CBFWA have mechanisms to cope with "under-represented" groups?



The description of performance metrics is useful. As the sponsors indicate, existing performance metrics measure output (e.g. number of meetings, number of participants) but not impact (changes in behavior, value to the members). The table of number of meetings is interesting, particularly the very low number of PNAMP meetings (n=1) relative to other kinds of meeting such as "member meetings." However, evaluating performance on the basis of the number of meetings held, average number of participants, and reports produced is not, as the sponsors acknowledge, sufficient to assess impacts.



As recommended by ISRP, the sponsors conducted a literature review of metrics to assess coordination effectiveness. Review results were not provided but apparently were not considered applicable: "Results from coordination-oriented literature searches provide a broad set of techniques and metrics that are not consistent for coordination efforts, a situation that is comparable to differences that exist among monitoring and evaluation efforts for physical and biological projects."



Regardless of the range of approaches, the ISRP maintains that coordination efforts such as these can be evaluated. The response provides a vigorous defense of the need for the CBFWA, asserting that more coordination will result in better survival and recovery of fish and wildlife populations. However, no quantitative measures are developed for determining the degree to which this is the case. The Status of the Resource Project should provide useful information on key variables such as escapements, but the response does not give much information on project status or data QA/QC. Will Columbia River Basin fish and wildlife agencies rely on the Project for data or will the project duplicate agency data?



The recommended qualification to funding is that the project should develop an approach to monitor its impact in terms of changes in behavior and value to the members. In addition to the PISCES metrics, it would be useful to have CBFWA develop member-feedback instruments to evaluate member assessment of effectiveness and impact. In addition, the new cluster of products included under the Status of the Resource report provides an opportunity for user evaluation of product utility.