See programmatic comments on coordination projects. A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

?The proposal is missing an opportunity to take a more systematic approach to coordination: to think about what the sponsors are really trying to achieve and how they will know if they are achieving it.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The proposed work includes data management (10%), review of technical documents and processes (40%), and facilitating and participating in focus workgroups on Program issues (40%). Several documents are identified as being developed. How did participation in workgroups or other coordination activities affect these documents? What coordination process improved their quality?

Significance to regional programs: A list with brief descriptions of the various regional programs and fora for which the coordination project will fund IDFG participation. Idaho is an active participant in the process of conserving and restoring Columbia Basin fish and wildlife that are affected by the building and operation of the hydrosystem. The Idaho Fish and Game Department participates in coordination and consultation efforts related Fish and Wildlife Program, the FCRPS BiOp, the Upper Snake River BiOp and other activities in the Columbia Basin.

Problem statement: The brief statement of the need for coordination and integration of actions should include a research question.

Objectives: The proposal should include a better statement of objectives (separate them from tasks - word as outcomes) and a description of how activities will be evaluated for effectiveness. The project has two objectives.

OBJ 1: emphasizes, “participate, collaborate and communicate effectively and efficiently.” These would be useful variables to measure. What are effective and efficient participation, collaboration, and communication? A deliverable could be testing hypotheses about these relationships and monitoring them in coordination activities.

Mention is made, “Department staff were actively engaged in coordination activities related to review, information development and negotiation related to wildlife impacts …” Could some of these activities be reviewed for their effectiveness and efficiency?

The proposal states, “decisions on actions to recover fish and wildlife populations and mitigate for lost productivity due to construction and operation of the Federal Columbia River Power Supply are informed by Idaho's perspective and benefit from Idaho's expertise. Achieving the objective requires communication, participation, and attendance at regional forums. The outcomes we seek are commonly recognized guidance materials that are considered, used, and followed by the Fish and Wildlife Program and other areas as appropriate.” Could the outcomes mentioned be used to rewrite the objectives in measurable "desired outcome" form?

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

The project began as a stand-alone in 2011. No description of financial performance or history is provided, or expected.

A summary and enumeration of the meetings attended by IDFG staff is provided. Presentations to the NPCC, workshop participation, technical review, tour conduct, and meeting attendance are also listed. What outcomes were achieved in these coordination activities?

Regarding adaptive management, a summary statement identifies research gaps and planned changes in data infrastructure that have resulted from participation in regional coordination contracts. Could the capture of lessons learned and their feedback into coordination activities be more explicitly developed?

ISRP Retrospective Evaluation of Results

“The Idaho Department of Fish and Game intends to formally withdraw from CBFWA April 1, 2012.” This project has no financial history or review of progress. Previous work was completed under the management of the Columbia Basin Fish and Wildlife Foundation. “Additional Relationships Explanation” gives background regarding withdrawal from the CBFWA. This is a useful and insightful analysis.

This is a new project, so technically there are no results to evaluate. Historical data on performance is available with the project, “Proposal RESCAT-1989-062-01 - Program Coordination and Facilitation Services provided through the Columbia Basin Fish and Wildlife Foundation.” See the section, “Reporting & Contracted Deliverables Performance.”

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

Project relationships: The proposal lists a number of regional coordination projects with which it is related, without explanation of specific relationships.

Geographic focus: The geographic interests are stated as basinwide. Examples given reflect primarily on the portion of the Snake River Basin and most examples include the Upper Snake Salmon Recovery area.

Emerging limiting factors: A brief summary of IDFG's participation in various regional fora addressing climate change, non-native species, predation increases and toxics notes the benefit to Idaho from expertise gained through this participation. Are there emerging issues related to regional coordination? Could elements of effective coordination be identified based on these experiences? Could coordination be phrased in an adaptive management framework?

The proposed plan of work identifies many meetings, workshops, technical reviews, forums, tours, discussions, along with other forms of participation. Could the representative list of meetings be used to draft hypotheses and identify variables be used a data points to evaluate effective and efficient coordination during the proposal period? Could these activities be used to develop some hypotheses, lessons learned, and actions to make change to better achieve goals? Did any priorities change? Were insights gained from others? Focus more on results and less on the inputs to get results. What outcomes and relationships might be observed that relate to coordination? What is the value-added as a result of coordination?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has two deliverables: 1: Staff participation in Regional Forums and meetings; 2: Provide Technical and Policy Review. Because the objectives are written as tasks, the deliverables are close to identical to the objectives.

Work elements: Two work elements are identified – 122. Provide Technical Review, and 189. Coordination-Columbia Basinwide. These work elements do not have associated metrics. Can output metrics and methods be identified to go with these work elements? Ideally, the hypothesis(es) developed in the proposal would be measured during the course of the coordination activities and results presented in the report on this project. There are many ideas discussed in the proposal that are amenable to this approach. Selecting a few of the most important questions, concerns, or hypotheses and monitoring them is recommended.

Methods and metrics: Methods of performing the work are described as workshop participation, presentations to the NPCC, participation in the development of various documents and reports, and participation in meetings. A representative list of meeting participation is provided. No metrics are provided to assess effectiveness.

Value-added: a summary description of actions taken that would not have been possible without the support of coordination funds. Can the outcome of these actions be assessed in some systematic way?

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the two work elements are published but do not provide adequate guidance on the methods and metrics. Guidance is available from ISRP (2007-14:2). Project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives.

Modified by Dal Marsters on 4/17/2012 3:05:54 PM.

A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

The proposal contains so much detail that it is difficult to review. Future proposals would be improved through more summary and synthesis of relevant information.

The proposal provides extensive insight into a scientific perspective on program coordination. A number of hypotheses are presented about the coordination process and its outcomes. The approach provides narrative findings for the experience gained by CBFWA. The insights provide compelling analysis for developing a sound scientific perspective on program coordination early in the evaluation process.

Proposal strengths:

• The proposal is fully documented; methods and accomplishments are exhaustively described.
• The limiting factors statement addresses large-scale issues that have the potential to limit the effectiveness of the project. This is rare among proposals.
• The proposal provides extensive insight into a scientific perspective on program coordination.
• Performance metrics have been identified and used to evaluate project effectiveness.

Weaknesses:

• So much detail is presented that it's difficult for the reviewer to track proposal content. The project is not only complex in itself it is also undergoing significant structural change.
• It is unclear where sturgeon or anadromous fish fit into CBFWA activities.
• It is sometimes difficult for external reviewers to assess the effectiveness of the project.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The CBFWA proposal offers a detailed narrative review of the coordination history from 1989 to the present. It analyzes changes in coordination that have occurred and reasons for them. The proposal also raises a number of policy issues to be addressed by Bonneville and the Council.

Problem statement: The problem statement is overly long, but at its end a summary conclusion adequately states the problem the proposal is designed to address.

Objectives: The proposal is focused around seven objectives, but the implicit overarching objective of this proposal is to coordinate disparate regional coordination projects around subject-matter themes.

Each objective has several deliverables that include development and maintenance; communications; coordinating, implementing, and facilitating; collate and summarize; attend and participate.

Emerging limiting factors: The proposal identifies three limiting factors for effective regional coordination: 1) perception of fairness, 2) participation and buy-in, and 3) adequate funding for both facilitation and participation. The proposal aims to address recent changes in these limiting factors.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Financial performance and history: This section provides an adequate description of a financial history complicated by inconsistencies in reporting and budgeting dates and by a changing project structure.

Performance: An adequate short description is provided.

Major accomplishments: A detailed description of the project's major accomplishments in its former version – the Annual Work Plan. Because it has been coordinating activities since 1989, CBFWA has extensive coordination experience and the proposal lists many insights. Further, the proposal provides detailed discussion on why some members have left CBFWA coordination and facilitation services. The new project configuration will begin with this funding cycle. Metrics of performance are numbers of meeting attendees and qualitative evaluation of outcomes made possible by actions of the CBFWA in various fora. A stakeholder survey was also conducted.

Response to ISRP comments: A complete description of ISRP comments and CBFWA response in terms of developing tools to monitor impact is provided.

Adaptive management: A good description of changes in CBFWA focus and configuration in response to changing circumstances in the region.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

The geographic interests of CBFWA encompass the entire Columbia Basin for both fish and wildlife. CBFWA's goal has been to include all sovereigns and action agencies in the coordination process. In addition, CBFWA encompass the Willamette/Lower Columbia, Middle Columbia, Upper Columbia, and Snake River Recovery areas.

Project relationships: The statement provides a history of the changing configuration of CBFWA and a long list of coordination, monitoring and other programs throughout the region with which it is coordinated.

Tailored questions, data: The proposal provides a long description of the Status of the Resource website and its function. Information provided in the presentation indicates that the project has added tribal data coordinators to the already participating agency representatives. The narrative analysis of the regional coordination problem is excellent and provides useful insights; more attention to identification of a scientific component to the proposal would help to plan for future success.

More findings like this one would be valuable, "These factors illustrate in high relief the Fish and Wildlife Program’s recognition that coordination efforts and funding should be focused through a set of functional activities that need coordination, and not necessarily on the basis of entities desiring coordination funding." This seems to represent a critical principle for organizing coordination activities. Another important set of coordination hypotheses, "solutions intended to increase coordinated efficiencies and effectiveness. This includes developing coordinated synthesis reports, sharing data and information through scientific papers and science/policy forums, holding regular workshops focused on specific species, methods, or geographic areas, and on several topics, the drafting of basin-wide management plans." In this same section, "CBFWA Members recognized the role the organization can play in delivering useful technical, science-based products associated with protection, mitigation and enhancement of the Columbia Basin’s anadromous and resident fish, and wildlife."

The proposal suggests that “the adaptive management framework for which coordination” be used. “Adaptive management” is mention in 4 of the 7 project objectives, in many of the deliverables, and development of an “adaptive management framework” is frequently mentioned. Can this framework be more explicitly and specifically identified? How have the many lessons learned been built into each adaptive management cycle? What is the typical length of an adaptive management cycle? In the adaptive management section a very interesting process is described that suggests that funding arrangements changed the need and approaches to collaboration. This is a very interesting insight. It does not illuminate the adaptive management framework often discussed, but it does indicate that funding is an important driver as to participation in coordination activities. The identification of factors that may limit the effectiveness regional coordination including perception of fairness, participation and buy-in, and adequate funding for both facilitation and participation, is an insightful and useful hypothesis. Did the conduct of a consumer satisfaction survey in 2010 help in assessing these variables?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has 24 deliverables related directly to the seven objectives. Each is described in detail.

Project components: The project has eight components, each described in some detail: Data Management (5%); Monitoring and evaluation (10%); Developing and tracking biological objectives (5%); Review of technical documents and processes (10%); Project proposal review (5%); Coordination of projects, programs, and funding sources within subbasins 20%); Facilitating and participating in focus workgroups on Program Issue (25%); and Information dissemination (20%).

Work elements: CBFWA lists many work elements (11).

Methods and metrics: Methods are described in detail in several different sections. Metrics are also described. Measurement of performance is through numbers at meetings, outcomes of coordination, and a survey of stakeholder satisfaction.

One form of assessing effectiveness is the output of meeting results, documents, and other evidence of outcomes of coordination and facilitation actions. Another way to assess effectiveness is input from the state, federal, and tribal agencies involved in the process, who are well-positioned to assess this effectiveness. Other entities interacting with the program but not formally part of the CBFWA functions are also able to provide input. Some possible approaches to at least showing the degree of success would be, as a minimum, letters from each agency/tribe responding specifically to a series of questions as to how well the CBFWF program is meeting their needs in key areas and how the program might be improved. This request could also be addressed to some outside entities that participate with the workgroups. Some questions should address not only how well the CBFWA is meeting agency and tribal needs, but benefiting the salmon and other basin resources in specific ways that otherwise would not occur. It would also be of interest to know how the program involves entities such as the Oregon and Washington state agencies and the Corps of Engineers, and if more coordination among them and CBFWA entities is possible or can be expected.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the 11 work elements are published but do not provide adequate guidance on the methods and metrics. The project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives. The relative value of “electronic meetings” vs. “face-to-face sessions” would be useful to study. Another worthy topic for review is dimensions of “facilitation.”

What part of “the ISRP for reporting metrics for regional coordination (ISRP 2007-14)” will be implemented? The document suggests (ISRP 2007-14:4), “Metrics of Impact: (e.g., how effective is the project: what is its added value of the coordination project) changes in behavior, value to the members, user evaluation of product utility, lack of redundancy, member assessment of effectiveness and impact, benefits to fish and wildlife of enhanced coordination activities, specific projects or resources benefited by the project, specific effect of coordination on conservation and management.” Where in the proposal are these suggested metrics of impact operationalized? A hypothesis worth testing is whether change in funding has led to decreased regional coordination.

Modified by Dal Marsters on 4/13/2012 12:24:40 PM.

The response includes a detailed description of the types of coordination and facilitation services that CBFWA is or could be providing. It adds information that was missing from the proposal regarding the operational meaning of general coordination terms. The response states that without CBFWA, the BPA, NPCC and the ISRP would find it difficult to staff activities such as holding meetings and providing website services. In addition, the response states that the "Columbia River Basin is dependent on the coordination, administration, and technical services that the CBFWA provides" for two monitoring and evaluation coordination partnerships (PNAMP and CSMEP). CBFWA activities in this regard include subcontracting services, participation in meetings, and website services. In 2005 CBFWA began to further expand its role to data inventory and reporting services. The response further states that the CBFWA role extends beyond coordination of its members to services for non-member entities.



Overall, a better demonstration is needed that CBFWA's services are provided in the most cost-effective manner. The response provides a better description of the association of the $900k budget line to the "annual report", including good detail on the range of products associated with the report. However, questions remain as to whether the costs are reasonable, especially given that a template of the website is already up and running.



The response also provides a description of the withdrawal of the Kalispel and Spokane tribes from membership. It appears that the interests of these two entities were not being addressed at the policy level; however, little explanation is provided as to why this situation exists. Does CBFWA have mechanisms to cope with "under-represented" groups?



The description of performance metrics is useful. As the sponsors indicate, existing performance metrics measure output (e.g. number of meetings, number of participants) but not impact (changes in behavior, value to the members). The table of number of meetings is interesting, particularly the very low number of PNAMP meetings (n=1) relative to other kinds of meeting such as "member meetings." However, evaluating performance on the basis of the number of meetings held, average number of participants, and reports produced is not, as the sponsors acknowledge, sufficient to assess impacts.



As recommended by ISRP, the sponsors conducted a literature review of metrics to assess coordination effectiveness. Review results were not provided but apparently were not considered applicable: "Results from coordination-oriented literature searches provide a broad set of techniques and metrics that are not consistent for coordination efforts, a situation that is comparable to differences that exist among monitoring and evaluation efforts for physical and biological projects."



Regardless of the range of approaches, the ISRP maintains that coordination efforts such as these can be evaluated. The response provides a vigorous defense of the need for the CBFWA, asserting that more coordination will result in better survival and recovery of fish and wildlife populations. However, no quantitative measures are developed for determining the degree to which this is the case. The Status of the Resource Project should provide useful information on key variables such as escapements, but the response does not give much information on project status or data QA/QC. Will Columbia River Basin fish and wildlife agencies rely on the Project for data or will the project duplicate agency data?



The recommended qualification to funding is that the project should develop an approach to monitor its impact in terms of changes in behavior and value to the members. In addition to the PISCES metrics, it would be useful to have CBFWA develop member-feedback instruments to evaluate member assessment of effectiveness and impact. In addition, the new cluster of products included under the Status of the Resource report provides an opportunity for user evaluation of product utility.