See programmatic comments on coordination projects. A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The proposed work includes data management (storage, management, and reporting) - 10%, review of technical documents and processes - 40%, facilitating and participating in focus workgroups on Program issues - 50%. Many meetings are identified and draft documents were prepared. One of the outcomes of coordination was “filling of the gaps” in monitoring. Projects were implemented and data monitoring was improved. The role of coordination in these activities could be clarified, key variables identified and observed, and hypotheses on how coordination improves data management and project implementation formulated. How did these coordination activities add value or achieve desired goals?

On significance to regional programs, a detailed statement describes several regional fora in which the ODFW participates and to which project funding will be applied to enable coordination, assessment, monitoring, and evaluation. Oregon is a participant in the process of conserving and restoring Columbia Basin fish and wildlife that are affected by the building and operation of the hydro system. Oregon’s primary goal is to assure that decisions and actions to recover fish and wildlife populations “are informed by Oregon’s perspective and benefit from Oregon’s expertise.”

The statement about limiting factors does not really identify issues that may limit the effectiveness of the coordination, but instead lists benefits of coordination and notes that the Council's Fish and Wildlife Program is implemented through an adaptive management approach that will ensure that climate change and other sources of uncertainty will be addressed. Could a hypothesis about relevant expertise or "effective communication and collaboration with a myriad of federal, tribal, and other state agencies and other pertinent organizations to coordinate efforts related to the implementation of the Council's Fish and Wildlife Program …ensure that regional decisions on appropriate actions to recover fish and wildlife populations …benefit from Oregon expertise” be limiting?

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Performance: The proposal states that all deliverables were completed on time since ODFW separated from CBFWA. One report is shown as "red"

Major accomplishments: The proposal summarizes participation in meetings, presentations, and contributions to various documents and processes. These are useful for assessing participation. Important for coordination is what were the outcomes from participating in meetings, presentations, and contributions to documents and processes?

Adaptive management: The proposal notes the importance of adaptive management. It contains a very good discussion of the adaptive management process and identifies 8-steps to implement it. The 8-step process is a good beginning for the framing of one or more hypotheses that would show the value of coordination. Transparency, accountability, and effective planning are all variables. Measurement of these variables could be discussed in the section on deliverables. Can measures be proposed and can these variables be related to coordination activities that provide for transparency, accountability, and effective planning. For example, is the face-to-face nature of meetings beneficial for establishing trust and transparency, or are other activities more effective? An assumption is that coordination meetings and activities provide more effective plans. Is there evidence for this relationship? What are ideas for the evaluation process mentioned in step 7? Can a monitoring protocol be identified for evaluating both the occurrence and the effectiveness of this process? There is no discussion of adaptive management approaches taken within this project. Nor does the project have an adaptive management design.

ISRP Retrospective Evaluation of Results

The problem statement describes the 2010 separation from CBFWA and the need to continue funding for the individual states to continue to participate in the implementation of the Fish and Wildlife Program through the coordination and information transmission that was formerly accomplished through CBFWA. This is a new project, so technically there are no results to evaluate.

Historical data on performance is available with the project, “Proposal RESCAT-1989-062-01 - Program Coordination and Facilitation Services provided through the Columbia Basin Fish and Wildlife Foundation.” See the section, “Reporting & Contracted Deliverables Performance.” The proposal contains a very insightful discussion of the changing coordination "landscape," which resulted in Oregon withdrawing from CBFWA in 2010.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

Project relationships: The statement describing the changing landscape of coordination and the different needs and opportunities presented by these changes could be the basis for scientific research on regional coordination. A short summary of the types of coordination that are most needed by ODFW, and a list of projects doing similar work and with which they coordinate is provided and could be built upon.

Regional coordination focus: The geographic interests of the State of Oregon overall encompass the Willamette/Lower Columbia, Middle Columbia, and Snake River Recovery areas.

For situations where, “Some of the changes have diminished the need for regionally-based coordination among the Basin's fish and wildlife managers,” does this mean less effort needs to go into coordination? The insights under “Additional Relationships Explanation” are useful and could serve as the basis for developing hypotheses about what regional coordination is needed and what coordination is no longer relevant. What are the most effective ways of organizing and coordinating? What types of coordination activities work best? What percentage of the meetings is facilitated and does this improve outcomes? Does coordination provide value? How would the value of coordination be measured and compared against the costs?

Under value-added, the proposal makes the point, “Participation in the basinwide coordination resulted in identifying and implementation of projects which filled gaps in the monitoring of listed Snake River Basin spring/summer Chinook salmon and steelhead and other Columbia Basin fish and wildlife populations.” This section summarizes ODFW staff activities enabled by this funding. It includes a summary of outcomes that directly result from the project. Could more insight be included about the specific outcomes with respect to the Fish and Wildlife Program? Are these projects likely to show improvements in recovering listed species?

4. Deliverables, Work Elements, Metrics, and Methods

Work elements: The project has three work elements – 114. Identify and Select Projects, 122. Provide Technical Review, and 189. Coordination-Columbia Basinwide. These work elements have no metrics identified. Can output metrics and methods be identified to go with these work elements? Ideally, the hypothesis(es) developed in the proposal would be measured during the course of the coordination activities and results presented in the report on this project. There are many ideas discussed in the proposal that are amenable to this approach. Selecting a few of the most important questions, concerns, or hypotheses and monitoring them is recommended.

Deliverables and methods: a summary list of meetings attended, coordination activities performed and contributions to documents is enumerated. Metrics are based on inputs, for example numbers of meetings attended, rather than outcomes. What was achieved in the meetings?

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the three work elements are published but do not provide adequate guidance on the methods and metrics. Guidance is available from ISRP (2007-14:2). Project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives.

Modified by Dal Marsters on 4/17/2012 3:03:05 PM.

A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

The proposal contains so much detail that it is difficult to review. Future proposals would be improved through more summary and synthesis of relevant information.

The proposal provides extensive insight into a scientific perspective on program coordination. A number of hypotheses are presented about the coordination process and its outcomes. The approach provides narrative findings for the experience gained by CBFWA. The insights provide compelling analysis for developing a sound scientific perspective on program coordination early in the evaluation process.

Proposal strengths:

• The proposal is fully documented; methods and accomplishments are exhaustively described.
• The limiting factors statement addresses large-scale issues that have the potential to limit the effectiveness of the project. This is rare among proposals.
• The proposal provides extensive insight into a scientific perspective on program coordination.
• Performance metrics have been identified and used to evaluate project effectiveness.

Weaknesses:

• So much detail is presented that it's difficult for the reviewer to track proposal content. The project is not only complex in itself it is also undergoing significant structural change.
• It is unclear where sturgeon or anadromous fish fit into CBFWA activities.
• It is sometimes difficult for external reviewers to assess the effectiveness of the project.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The CBFWA proposal offers a detailed narrative review of the coordination history from 1989 to the present. It analyzes changes in coordination that have occurred and reasons for them. The proposal also raises a number of policy issues to be addressed by Bonneville and the Council.

Problem statement: The problem statement is overly long, but at its end a summary conclusion adequately states the problem the proposal is designed to address.

Objectives: The proposal is focused around seven objectives, but the implicit overarching objective of this proposal is to coordinate disparate regional coordination projects around subject-matter themes.

Each objective has several deliverables that include development and maintenance; communications; coordinating, implementing, and facilitating; collate and summarize; attend and participate.

Emerging limiting factors: The proposal identifies three limiting factors for effective regional coordination: 1) perception of fairness, 2) participation and buy-in, and 3) adequate funding for both facilitation and participation. The proposal aims to address recent changes in these limiting factors.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Financial performance and history: This section provides an adequate description of a financial history complicated by inconsistencies in reporting and budgeting dates and by a changing project structure.

Performance: An adequate short description is provided.

Major accomplishments: A detailed description of the project's major accomplishments in its former version – the Annual Work Plan. Because it has been coordinating activities since 1989, CBFWA has extensive coordination experience and the proposal lists many insights. Further, the proposal provides detailed discussion on why some members have left CBFWA coordination and facilitation services. The new project configuration will begin with this funding cycle. Metrics of performance are numbers of meeting attendees and qualitative evaluation of outcomes made possible by actions of the CBFWA in various fora. A stakeholder survey was also conducted.

Response to ISRP comments: A complete description of ISRP comments and CBFWA response in terms of developing tools to monitor impact is provided.

Adaptive management: A good description of changes in CBFWA focus and configuration in response to changing circumstances in the region.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

The geographic interests of CBFWA encompass the entire Columbia Basin for both fish and wildlife. CBFWA's goal has been to include all sovereigns and action agencies in the coordination process. In addition, CBFWA encompass the Willamette/Lower Columbia, Middle Columbia, Upper Columbia, and Snake River Recovery areas.

Project relationships: The statement provides a history of the changing configuration of CBFWA and a long list of coordination, monitoring and other programs throughout the region with which it is coordinated.

Tailored questions, data: The proposal provides a long description of the Status of the Resource website and its function. Information provided in the presentation indicates that the project has added tribal data coordinators to the already participating agency representatives. The narrative analysis of the regional coordination problem is excellent and provides useful insights; more attention to identification of a scientific component to the proposal would help to plan for future success.

More findings like this one would be valuable, "These factors illustrate in high relief the Fish and Wildlife Program’s recognition that coordination efforts and funding should be focused through a set of functional activities that need coordination, and not necessarily on the basis of entities desiring coordination funding." This seems to represent a critical principle for organizing coordination activities. Another important set of coordination hypotheses, "solutions intended to increase coordinated efficiencies and effectiveness. This includes developing coordinated synthesis reports, sharing data and information through scientific papers and science/policy forums, holding regular workshops focused on specific species, methods, or geographic areas, and on several topics, the drafting of basin-wide management plans." In this same section, "CBFWA Members recognized the role the organization can play in delivering useful technical, science-based products associated with protection, mitigation and enhancement of the Columbia Basin’s anadromous and resident fish, and wildlife."

The proposal suggests that “the adaptive management framework for which coordination” be used. “Adaptive management” is mention in 4 of the 7 project objectives, in many of the deliverables, and development of an “adaptive management framework” is frequently mentioned. Can this framework be more explicitly and specifically identified? How have the many lessons learned been built into each adaptive management cycle? What is the typical length of an adaptive management cycle? In the adaptive management section a very interesting process is described that suggests that funding arrangements changed the need and approaches to collaboration. This is a very interesting insight. It does not illuminate the adaptive management framework often discussed, but it does indicate that funding is an important driver as to participation in coordination activities. The identification of factors that may limit the effectiveness regional coordination including perception of fairness, participation and buy-in, and adequate funding for both facilitation and participation, is an insightful and useful hypothesis. Did the conduct of a consumer satisfaction survey in 2010 help in assessing these variables?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has 24 deliverables related directly to the seven objectives. Each is described in detail.

Project components: The project has eight components, each described in some detail: Data Management (5%); Monitoring and evaluation (10%); Developing and tracking biological objectives (5%); Review of technical documents and processes (10%); Project proposal review (5%); Coordination of projects, programs, and funding sources within subbasins 20%); Facilitating and participating in focus workgroups on Program Issue (25%); and Information dissemination (20%).

Work elements: CBFWA lists many work elements (11).

Methods and metrics: Methods are described in detail in several different sections. Metrics are also described. Measurement of performance is through numbers at meetings, outcomes of coordination, and a survey of stakeholder satisfaction.

One form of assessing effectiveness is the output of meeting results, documents, and other evidence of outcomes of coordination and facilitation actions. Another way to assess effectiveness is input from the state, federal, and tribal agencies involved in the process, who are well-positioned to assess this effectiveness. Other entities interacting with the program but not formally part of the CBFWA functions are also able to provide input. Some possible approaches to at least showing the degree of success would be, as a minimum, letters from each agency/tribe responding specifically to a series of questions as to how well the CBFWF program is meeting their needs in key areas and how the program might be improved. This request could also be addressed to some outside entities that participate with the workgroups. Some questions should address not only how well the CBFWA is meeting agency and tribal needs, but benefiting the salmon and other basin resources in specific ways that otherwise would not occur. It would also be of interest to know how the program involves entities such as the Oregon and Washington state agencies and the Corps of Engineers, and if more coordination among them and CBFWA entities is possible or can be expected.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the 11 work elements are published but do not provide adequate guidance on the methods and metrics. The project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives. The relative value of “electronic meetings” vs. “face-to-face sessions” would be useful to study. Another worthy topic for review is dimensions of “facilitation.”

What part of “the ISRP for reporting metrics for regional coordination (ISRP 2007-14)” will be implemented? The document suggests (ISRP 2007-14:4), “Metrics of Impact: (e.g., how effective is the project: what is its added value of the coordination project) changes in behavior, value to the members, user evaluation of product utility, lack of redundancy, member assessment of effectiveness and impact, benefits to fish and wildlife of enhanced coordination activities, specific projects or resources benefited by the project, specific effect of coordination on conservation and management.” Where in the proposal are these suggested metrics of impact operationalized? A hypothesis worth testing is whether change in funding has led to decreased regional coordination.

Modified by Dal Marsters on 4/13/2012 12:24:40 PM.

The response includes a detailed description of the types of coordination and facilitation services that CBFWA is or could be providing. It adds information that was missing from the proposal regarding the operational meaning of general coordination terms. The response states that without CBFWA, the BPA, NPCC and the ISRP would find it difficult to staff activities such as holding meetings and providing website services. In addition, the response states that the "Columbia River Basin is dependent on the coordination, administration, and technical services that the CBFWA provides" for two monitoring and evaluation coordination partnerships (PNAMP and CSMEP). CBFWA activities in this regard include subcontracting services, participation in meetings, and website services. In 2005 CBFWA began to further expand its role to data inventory and reporting services. The response further states that the CBFWA role extends beyond coordination of its members to services for non-member entities.



Overall, a better demonstration is needed that CBFWA's services are provided in the most cost-effective manner. The response provides a better description of the association of the $900k budget line to the "annual report", including good detail on the range of products associated with the report. However, questions remain as to whether the costs are reasonable, especially given that a template of the website is already up and running.



The response also provides a description of the withdrawal of the Kalispel and Spokane tribes from membership. It appears that the interests of these two entities were not being addressed at the policy level; however, little explanation is provided as to why this situation exists. Does CBFWA have mechanisms to cope with "under-represented" groups?



The description of performance metrics is useful. As the sponsors indicate, existing performance metrics measure output (e.g. number of meetings, number of participants) but not impact (changes in behavior, value to the members). The table of number of meetings is interesting, particularly the very low number of PNAMP meetings (n=1) relative to other kinds of meeting such as "member meetings." However, evaluating performance on the basis of the number of meetings held, average number of participants, and reports produced is not, as the sponsors acknowledge, sufficient to assess impacts.



As recommended by ISRP, the sponsors conducted a literature review of metrics to assess coordination effectiveness. Review results were not provided but apparently were not considered applicable: "Results from coordination-oriented literature searches provide a broad set of techniques and metrics that are not consistent for coordination efforts, a situation that is comparable to differences that exist among monitoring and evaluation efforts for physical and biological projects."



Regardless of the range of approaches, the ISRP maintains that coordination efforts such as these can be evaluated. The response provides a vigorous defense of the need for the CBFWA, asserting that more coordination will result in better survival and recovery of fish and wildlife populations. However, no quantitative measures are developed for determining the degree to which this is the case. The Status of the Resource Project should provide useful information on key variables such as escapements, but the response does not give much information on project status or data QA/QC. Will Columbia River Basin fish and wildlife agencies rely on the Project for data or will the project duplicate agency data?



The recommended qualification to funding is that the project should develop an approach to monitor its impact in terms of changes in behavior and value to the members. In addition to the PISCES metrics, it would be useful to have CBFWA develop member-feedback instruments to evaluate member assessment of effectiveness and impact. In addition, the new cluster of products included under the Status of the Resource report provides an opportunity for user evaluation of product utility.