A report/memo that addresses previous ISRP comments is needed.

A sound scientific proposal should respond to the six questions and related material at the beginning of the regional coordination section.

The ISRP’s FY 2007-09 review commented, "The proposal would be strengthened by including more detail on the benefits to fish and wildlife of enhanced coordination activities. For example, what specific projects or resources are threatened if funding is not provided? How will conservation and management be affected if the funding is not provided?"

The ISRP’s FY 2007-09 review further stated, "sponsors need to provide some measures by which the effectiveness of this coordination can be monitored and evaluated." According to the proposal, "One specific metric that the UCUT Coordination uses to assess the value of our work is to gauge if impacts from a proposed action help one critical natural or cultural resource at the detriment or while causing harm to another critical component (e.g., if an action benefits anadromous fish downriver while causing harm to resident fish upriver. For many issues, stopping the harm is the main short-term objective required, with mutual benefits to all resident and anadromous fish and wildlife being the long-term goal." This is a very worthy metric, but difficult to quantify. What other metrics and methods might be used?

The proposal should be re-written to include a better statement of objectives worded as desired outcomes and separated from tasks. Some text that could serve as the basis for rewritten objectives is already contained in the proposal. The proposal should be more explicit about how adaptive management is conducted within this project, and about how methods of implementation can be measured and evaluated for success. It would be useful to have a more structured and defined approach to measuring effectiveness of methods, and an explanation of how cost-effectiveness is assessed.

The project sponsors raise good questions about the conduct of coordination project evaluations. The proposal is missing an opportunity to take a more systematic approach to coordination: to think about what the sponsors are really trying to achieve, how they will know if they are achieving it, and how they will adapt to changing circumstances or proactively test new approaches and learn from the outcomes? These would be good elements for a research plan.

This proposal identifies a number of very important issues that could be framed into one or more hypotheses that would show the value of regional coordination. Concepts like environmental justice, “ecosystem health, equitable commerce, governance, and sovereignty” are variables. Measurement of these variables could be discussed in the section on deliverables. Can measures be proposed and can these variables be related to regional coordination activities that provide for achievement of UCUT goals.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

Proposed work includes coordination of projects and programs (25%), facilitating and participating (25%), review of technical documents and processes (15%), data management (10%), information and education (10%), monitoring and evaluation (5%), biological objectives (5%), and project proposal reviews (5%).

A claim is made for the benefits of coordination: “Through constant and effective communication, collaboration, and cooperation, the UCUT is able to reduce redundancy, prevent being left out of issues of local-to-regional impact, and increase efficiency and cost effectiveness of the PME efforts of the individual and combined UCUT.” These are worthwhile and useful questions to study. Further, providing evidence for better understanding of tribal views by stakeholders and the increased efficiency and cost effectiveness of the process would be very valuable when it comes to evaluating coordination expenditures.

Significance to regional programs: The project allows member tribes of UCUT to represent their collective issues in various regional programs and to present documents to the Intermountain Province Plan. “The five member tribes of UCUT (Upper Columbia United Tribes) are: the Coeur d'Alene Tribe, the Confederated Tribes of the Colville Reservation, the Kalispel Tribe of Indians, the Kootenai Tribe of Idaho, and the Spokane Tribe of Indians.” UCUT represents its interests and engages in technical and policy tribal, federal, state, and local governments and stakeholders. They relate to the FCRPS and the NWPCC's Program, in order to protect and enhance the UCUT “rights, sovereignty, culture, fish, water, wildlife, and habitat, with scientific validity, and maximized fiscal and resource efficiency and effectiveness.”

Problem statement: The statement emphasizes that since UCUT's 2005 departure from CBFWA its coordination functions have strengthened. UCUT now gets funds from each member tribe as well as BPA and employs a full-time policy analyst. Can the strengthening of coordination functions be measured or illustrated with narratives?

Objectives: The project has four objectives. The objectives are worded as tasks rather than as desired outcomes. However, explanatory text provided with each objective reflects desired outcomes. This text material could be used to restructure the objectives in the form of outcomes.

Objective 4 is about developing a strategic plan. How does the plan relate to the “Common Views” document? The “Common Views” document appears to have outcome measures that might be incorporated into the proposal.

For example: "Increase scientifically valid, effective, and cost efficient outcomes from participation in local, provincial, regional, national, and international decision making processes." “so that diverse decision-making includes outcomes that are consistent with fulfilling PME obligations of the CRPS…" “increase their understanding and support of conservation actions required to fulfill the PME obligations of the CRPS..." reference concepts that could be observed and measured (see Qualifications).

What are the outcomes from “Organized, facilitated, and provided reports,” “frequent computer, phone, and personal contact,” participation in meetings and processes, and “media and web outreach and education …sharing valuable perspectives to tribal and non-tribal local-to-international governments” in terms of achieving the proposal objectives?

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Financial performance and history: Expenditures have been less than budgeted amounts due to delay in filling the policy analyst position. The financial history explains the establishment of a separate BPA funding accord for UCUT that will extend until 2020. Similar long-term funding accords have been developed between BPA and two UCUT tribes.

Project performance: An explanation based on changing contract performance periods is provided for the number of late reports.

Major accomplishments: A long and detailed statement describes contributions to regional documents, organizations, facilitation of and attendance at meetings, document review, outreach and education, all to represent the perspective and position of UCUT on a wide range of issues. Were outcomes from these contributions ones that UCUT expected or wanted to achieve? Can success or lack of success in having UCUT’s position understood be explained?

Response to past ISRP and Council comments and recommendations: The ISRP made two suggestions in their 2007 review. Neither suggestion appears to have been addressed. Rather, a statement of a rationale about why coordination projects are not appropriate for standard scientific review is provided, but it does not refer specifically to ISRP or NPCC comments.

Adaptive management: The statement describes coordination as dynamic and effective at reducing redundancy, ensuring UCUT representation, and increasing cost-effectiveness. However, it does not address how management changes happen or whether active experimentation in new coordination approaches takes place. Further, measuring cost effectiveness would be very useful in justifying funding for regional coordination.

ISRP Retrospective Evaluation of Results

The project financial history goes back to 2007, although no expenditures were made until 2008, when four Columbia Basin tribes left CBFWA. The tribes prefer to develop their own expertise and communicate directly in coordination processes. The 72% report completion rate is stated to be mostly a problem with contracting procedures.

Other historical data on performance are available with the project, “Proposal RESCAT-1989-062-01 - Program Coordination and Facilitation Services provided through the Columbia Basin Fish and Wildlife Foundation.” See the section, “Reporting & Contracted Deliverables Performance.”

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

Project relationships: The proposal describes the primary project relationships as being with the individual UCUT tribes, as well as the relationships between UCUT and entities in the larger region.

Regional coordination focus: The geographic interests are regional to the upper Columbia, national to the responsibilities of the United States toward American Indians, and international, especially regarding Columbia River Treaty negotiations between the US and Canada.

Emerging limiting factors: A detailed statement is provided describing participation in regional and international processes related to climate change, invasive species, northern pike predation and toxics. The statement also describes the inability to propose new needed work as a limiting factor. Could regional coordination activities identify, prioritize, and promote needed work that might increase its likelihood of being funded?

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The project has 5 deliverables. Each deliverable relates to an objective and a short explanation of work that links the deliverable to an objective is provided.

“DELV-2: Educate and communicate with public and relevant stakeholders” seems important for dealing with environmental injustices (OBJ-3). Are these injustices part of the regional coordination process? Do they affect regional coordination outcomes? Are they outside the regional coordination process?

Should DELV-2 be concerned with communication of the “Common Views” document? Can the effectiveness of education messages, methods, and understanding be evaluated? Were the outcomes the ones expected when the education and communication programs were designed?

Five work elements are identified – 99. Outreach and Education, 122. Provide Technical Review, 161. Disseminate Raw/Summary Data and Results, 174. Produce Plan, and 189. Coordination-Columbia Basinwide. Only 99 has metrics, but they are more inputs rather than outcomes. Can output metrics and methods be identified to go with these work elements? Ideally, the hypothesis(es) developed in the proposal would be measured during the course of the coordination activities and results presented in the report on this project. There are many ideas discussed in the proposal that are amenable to this approach. Selecting a few of the most important questions, concerns, or hypotheses and monitoring them is recommended.

Methods and metrics: Detailed descriptions of work performed under each project component are provided in the "project coordination" section. Several assertions of cost-effectiveness, success, and the use of metrics are made, but without specific definition or analysis.
Value-added: The proposal claims that the project results in increased efficiencies and cost-effectiveness but does not provide specific analysis or examples of how this is the case.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocols for the five work elements are published but do not provide adequate guidance on the methods and metrics. Guidance is available from ISRP (2007-14:2).The project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives.

Modified by Dal Marsters on 4/17/2012 2:49:45 PM.

This proposal describes coordination and information provision for the Upper Columbia United Tribes (UCUT) that seems quite useful and productive. A brief but clear section describes the role of the UCUT in coordinating its five member tribes with the Fish and Wildlife Program and with CBFWA. It describes meetings coordinated and information provided to its members, as well as its function in communicating UCUT member positions within the Basin decision arenas.

The proposal provides specific examples of UCUT's role in enabling coordination, communication and participation of its members in regional processes. It makes a good case for the relation of UCUT coordination support to the participation of the upriver tribes in fish and wildlife activities. It describes decreasing levels of UCUT funding from the Bureau of Indian Affairs (BIA), relates the funding declines to a decline in coordination activities, and states that project funding is necessary to maintain UCUT central office functions.

The proposal would be strengthened by including more detail on the benefits to fish and wildlife of enhanced coordination activities. For example, what specific projects or resources are threatened if funding is not provided? How will conservation and management be affected if the funding is not provided?

The proposal has five objectives describing various aspects of coordination, participation, and long-term planning. Work elements are listed for each objective; all are activities that facilitate member tribes' participation in the Fish and Wildlife Program. Work elements are specific and relate well to the objectives. One set of work elements relates to the informing of and involvement in national legislation and international agreements that affect the tribes with regard to salmon and habitat issues and treaty storage water. This seems quite useful and forward-looking.

To strengthen the justification for the proposal, the sponsors should provide specific information on the basis for the following statement made in the proposal: "The upriver Tribes have been innovative leaders in proposing strategies for watershed-based Program management, equitable allocation of fish and wildlife funding, and multiple-purpose river operations."

In addition, because the objective of this project is coordination, the sponsors need to provide some measures by which the effectiveness of this coordination can be monitored and evaluated.