Based on the response, the ISRP finds that the project has adequately addressed concerns #1, #2, #3, and #5, but places conditions on the projects related to #4. In addition, the ISRP again recommends that BPA provide the resources to the proponents to conduct some basic planning and prioritizing for the project to fully develop an implementation strategy beyond the next contract cycle.

The ISRP requests the proponents to provide information on the following two conditions in the next annual report.

1. Coordination with M&E projects. Document the strategy for coordinating with projects conducting M&E in the basin.
2. Project evaluation and adjustment. Develop an explicit schedule for project evaluation and assessment and document their strategy and systematic evaluation process.

In our preliminary review, we requested a response on the topics listed below. Our final comments based on the response are provided after each topic:

1. Summary of projects. In the proponents’ responses to items #1 and #4, the information provided gives some clarity on how projects are selected. This process appears to be informal and based on the general priorities and insights of project partners. The proposal also indicates that projects are prioritized in consultation with District fish biologists, applying some combination of habitat conditions, presence or relative salmonid abundance, and water availability.

The table provided in the response is helpful as an indication of the potential projects that may be implemented during the next contract cycle. As noted in the proposal, all projects are not expected to be implemented, as issues can arise with personnel, cultural resource surveys, or other constraints. Site characteristics for some (but not all) of the projects provide some insight into the nature of the projects (e.g., “1.5 miles of habitat to next upstream barrier”), though no information is provided on what the “Cumulative Expert Panel Prioritization Score” is.

It was reported that future projects will be selected by local district fish biologists based on the anticipated benefits of a project. (The proposal also indicates this process is already occurring, hence the ISRP’s confusion about how projects for the next contract cycle were selected.) However, it still is not clear whether the selection or ranking process is systematic and consistent from year to year. The spreadsheet does not explain how many potential projects were assessed and how the information of rankings, including Low Prioritization Ranking Scores, is documented and retained for future use. As with all projects, the ISRP expected to learn the details about the expected benefits, how they will be characterized, and how they will be used in prioritizing future projects. Given the large number of potential projects (3,411 identified in the response), prioritization for the greatest benefit seems particularly important. Key questions include, for example: How will benefits be characterized? How important are passage projects, relative to pump or gravity screens?

The response provides some general information for evaluating the science behind the project selection and for understanding the scope of projects for the next contract cycle. However, the current processes lack the structure that ISRP expects for programs managing such a large number of projects and a large budget. Simultaneously, the ISRP also recognizes that an adequate budget is necessary for evaluating potential need and priority of projects, and that the proponents currently rely on partners to investigate and prioritize projects. As a result, the ISRP finds that the project could proceed in the coming contract cycle, but more information should be provided in the next annual report, including a clearer summary of the project selection process. The ISRP again recommends that BPA provide additional resources to the proponents to do this basic planning activity.

2. Changing project scope. The response adequately addresses ISRP concerns. Restoration activities are conducted at sites during fish passage/screening improvements and are funded by partners rather than BPA directly. Based on the response and the external source of funds for habitat restoration, it is unlikely that elimination of collaborative habitat restoration activities during fish passage and screening projects would increase the number of screening and passage projects implemented.

3. M&E matrix – support. The ISRP understands that this project does not conduct monitoring, aside from identifying the survival or failure of structures, but encourages the project to consider and clarify ways that data from the Escapement and Productivity project can be used for understanding and improving the screening projects.

4. Project evaluation and adjustment. The ISRP is required to assess monitoring and evaluation for every project. The M&E component is valuable for documenting a project’s value and for evolving and improving the program over time. Having some strategy for program assessment/evaluation is essential, particularly for projects subject to administrative (i.e., budget declines) and hydrologic changes, as this one faces. While the ISRP understands that the project is not funded for monitoring of specific activities, there are funded M&E projects in the basin, as well as options for qualitative and low-cost processes for identifying what projects produce the most benefits, where/how projects and the program can be improved, and where the overall program may need adjustments to accommodate changes. Based on the content provided in the proponents’ response to #5, the work conducted under the Escapement and Productivity project (199801600) seems particularly relevant for evaluating the impact of this program. However, it is not clear how or if this program is using those data to better understand the benefits or limitations of this project. The ISRP finds that this concern has not been adequately addressed and recommends that, in the next annual report, the proponents document their strategy for coordinating with other projects conducting M&E in the basin.

The ISRP appreciates the additional context for how the proponents are considering adjustments to the program, which is organized by key components (i.e., fiscal, prioritization, performance). Like the prioritization approach, the evaluation strategy appears to be informal and opportunistic, with some aspects focusing on individual adjustments and a few reflections aimed at the broader program. The response lists types of decisions made by the project but does not describe a systematic process for evaluation and decision-making. Even with the need to respond immediately to unscheduled events and information, the project would benefit from a regularly scheduled assessment of the collective outcomes of the project for each current work year and documentation of the overall assessment in the next annual report. Such reports are required by BPA and could be readily linked to an adaptive evaluation and adjustment process based on the strategy for addressing needs for fish passage restoration and screening. The ISRP recommends that, in the next annual report, the proponents develop an explicit schedule for project evaluation and assessment and document their strategy and systematic evaluation and management adjustment process.

5. Benefits to fish. The ISRP appreciates the proponents pointing reviewers to the appropriate annual reports and contracts where results of the program are presented. However, as indicated in the directions for proposal preparation, these results are most useful if presented directly in the proposal so that the ISRP can clearly understand the benefits to fish and, thereby, avoid asking for this information in the response loop.

The results presented in the response on results from other studies, and especially data from the Escapement and Productivity project (199801600), provide evidence of the project’s benefits and illustrate the power of collaboration with M&E projects. The ISRP encourages the project to continue to provide information and evidence of the benefits to fish in future annual reports and future proposals. In addition, the ISRP encourages the proponents to continue to look for opportunities to lead the region in understanding the benefits of the various screening projects. For example, the Lemhi study examined the efficacy of gravity diversions for Chinook salmon. The Escapement and Productivity project examined the effectiveness of rotary bypass systems for passing juvenile Spring Chinook salmon and steelhead. Some important questions include: Is there information about the pump screen projects? How might these different systems perform as hydrology changes? The proponents may seek to develop assessments of the relative benefits of pump screen projects with interested monitoring projects in the basin.

Preliminary ISRP report comments: response requested

Response request comment:

This is an important project in the Columbia Basin Fish and Wildlife Program portfolio that should continue to receive support. Screening diversions and increasing access to habitat are essential for the recovery of fish in the basin, and this project appears to be effective at implementing these protections for fish. However, the proposal lacks some important details needed for evaluating the project’s next phase.

The proponents are requested to address the following in a point-by-point response to the ISRP:

1. Summary of projects. A summary of screening projects to be implemented under the new contract, including information used for prioritization and the characteristics of sites selected from the prioritization process. 
   
   
2. Changing project scope. The change of scope to include habitat projects requires further information for scientific review. What actions are being proposed? What strategy is used for selecting and implementing habitat projects? What do “smaller” and “larger” scale projects refer to? Given that there already is not enough budget to adequately support the screening work, the ISRP also seeks to understand how the decision to implement habitat projects was made, and questions if the project title needs to be revised. 
   
   
3. M&E matrix - support. The ISRP is requesting a response from the Escapement and Productivity of Spring Chinook and Steelhead project (199801600) to summarize linkages between implementation and all monitoring projects in the basin. We are asking your project to assist them in creating the summary by providing information to them about what is being monitored for your implementation project, as well as where, when, and by whom the monitoring occurs. A map or maps of locations of monitoring actions would be helpful in this regard. 
   
   
4. Project evaluation and adjustment. The proponents should provide a summary of the strategy for adapting to project changes, independent of RM&E data, including things like how program priorities and objectives, indicators of project performance, and regular assessment are used in making decisions about program resources. 
   
   
5. Benefits to fish. Given the funding level and availability of relevant data, the project should summarize the potential benefits to fish. The number of fish protected could be estimated simply from estimates of run abundance, passage efficiency, and SARs. At a minimum, a subset of actions need to be followed through time to judge effectiveness.

In addition, we direct the Council’s attention again to the same recommendation from the 2013 Geographic Review (ISRP Report 2013-11), resulting in the ISRP “encouraging the Council and BPA to increase funding in order to improve the implementation rate.” The project has cut staff and the number of projects implemented to accommodate the flat budget over the past 15 years, and additional resources are needed from BPA to maintain the productivity and benefits of this important project.

In future proposals, the ISRP requests that proponents include a map and summary of basic characteristics of projects implemented under the prior contract, including basin location and priority, species benefited, and other details. In addition, we recommend that the dashboard (https://projects.nwcouncil.org/ProgramTracker/Modules/Screens/DashboardMap) be updated to include information on the species benefited for each of the mapped screens.

Q1: Clearly defined objectives and outcomes

This project uses BPA funds to install screens and passage structures, and to implement habitat projects at locations where passage is restored. The proposal provides quantitative objectives for each year, though they are not all measurable (i.e., “improve” 10 miles). In addition, they are written as “up to” the targets (e.g., up to 10 gravity fish screens per year). Is there a minimum number of projects to accomplish each year, rather than a maximum? What happens if the project reaches its maximum objective? A minimum desired target would provide a metric to evaluate the project’s intended performance.

The proposal also provides objectives that represent the cumulative outcomes of the project. By 2040, the project aims to construct screens and headgates on 100 diversions, 50 miles of habitat access via passage projects, and restoration of 10 miles of habitat at passage projects. The ISRP notes that these objectives are similar in scope to what was conducted over the past eight years and assumes the reduced scope for the next 20 years is a result of the flat budget that was mentioned throughout the proposal, despite rising costs of labor and materials. Is this correct? This budget issue was raised during the 2013 Geographic Review, and the ISRP re-iterates our same finding from that review. We are once again “encouraging the Council and BPA to increase funding in order to improve the implementation rate.”

Q2: Methods

Regarding screen design, the proponents appear to be applying best practice, using standard designs established by appropriate resource agencies (e.g., NOAA) and go through an extensive review process that involves both BPA and ODFW. Prior reviews have commented on the lack of a robust prioritization plan, and the proponents directly addressed this concern in the proposal. They still do not have a landscape level inventory or comprehensive prioritization plan due to lack of funding. Instead, their approach is to prioritize projects, among sites with willing landowners, by considering “species presence and numbers, habitat quality, water availability, statewide priority, focus areas, etc.” It would have been helpful if the proposal had included summaries of the data used in prioritizing projects. The ISRP recommends that proponents maintain and report data on the prioritization criteria for selected projects under the next phase.

Prioritization is also accomplished through quarterly collaboration with in-basin partners, the use of priority basins from the John Day Atlas, as well as “maps and aerial photography,” though it was not clear how maps and imagery are used. What kinds of maps (e.g., OWRD’s Points of Diversion) are examined, and how is that information used? The narrative suggests that proponents are using data in prioritizing sites, but it is difficult to understand how all of the parts fit together to actually select a project for implementation or to assess if the highest impact projects are selected from among those available. Documentation of this process would also provide some indication of the benefits to fish, as discussed below.

Broadly, given the importance of screening, the ISRP believes that the proponents can be more thorough in structuring and describing their prioritization approach. The summary can include a description of how many screens are currently being considered (rather than a ballpark estimate of how many sites need to be screened), what data are used in prioritization and how, and the characteristics of the sites selected from the prioritization process (e.g., species presence and numbers, habitat quality, and so forth).

Q3: Provisions for M&E

The project has been making operational adjustments over time, most notably with a budget that has not reflected rising wages and material costs. These adjustments include eliminating personnel and reducing the number of projects completed. The proponents have also re-classified a field technician to focus on partnerships, coordination, and prioritization. The proposal also includes a comment that needs further explanation. On page 3, the proposal states that, “After the 2013 categorical review, the Project diversified and has been implementing fish habitat projects on a smaller scale.” It is not clear what smaller scale is referring to, or, after reviewing the 2013 review, what prompted this change. Also, it is unclear how diversification is related to a reduction in spatial scale. More description is needed on what types and scales of restoration are planned, particularly since the project proposes to restore 10 miles of habitat over the next project period and no information is given regarding what habitat projects were completed under the prior contract.

A strength of the proposal is the identification of the project’s primary confounding factor (declining streamflows) and specific strategies for adapting the program to those lower flows.

In summary, the ISRP commends the project for making strategic adjustments as project constraints (e.g., budget, streamflows) change. However, the proposal provides no indication of how these critical decisions are being made and instead indicates the sense that a lack of RM&E makes “it difficult to provide an Adaptive Management strategy.” Adaptive Management can be conducted without quantitative data, but instead be based on a documented strategy, which can include program priorities and objectives, qualitative indicators of project performance (e.g., number of projects completed by type relative to the annual objectives), and regular assessment.

The ISRP also notes that some monitoring is being conducted by partner organizations and it would be helpful to know which partners and to see the results. For instance, some “passage and habitat projects completed with USFS will be monitored by USFS for effectiveness.” Are the projects effective, and what has the USFS found?

Q4: Results – benefits to fish and wildlife

The ISRP notes that no information is provided regarding the projects completed under the prior contract, aside from a summary of the number of types of projects completed. It does not appear that this information is included in the annual report either. The 2020 report was only 2.5 pages long and provided only very general details about screen maintenance activities. Furthermore, the section of the proposal describing Significance to Fish and Wildlife Program and other regional plans is simply a list of relevant pages and documents from the 2014 Program. Thus, the ISRP could not assess the benefits to fish from this program. While the ISRP recognizes that no budget is available for RM&E, some further detail about the projected fish benefits would be helpful in characterizing the impact of the program to fish, rather than just assuming the benefits, particularly given that the benefits are highly variable with screen size and location (Moyle and Israel 2005). Examples of basic information that would not require an RM&E budget could include: a map of the project locations relative to the priority basins for the species of concern, data on the frequency and duration of screen failures and repairs, and/or information that is used in the prioritization process (e.g., species presence and numbers, habitat quality, water availability, statewide priority, focus areas, and so forth). Ultimately, while the project's quantitative objectives can be assessed for success in implementation, the objectives and the proposal do not provide any indication of the benefits for fish. For instance, an expected outcome is clearly an improvement in fish survival. Even a coarse estimate of the annual improvement in survivorship would be enlightening and provide further justification for the program. Based on the estimates of run abundance, passage efficiency, and SARs, the proponents should be able to enumerate a meaningful estimate of the number of fish protected by the screening program.

Reference

Moyle, P. and J. Israel. 2005. Untested assumptions: Effectiveness of screening diversions for conservation of fish populations. Fisheries 30: 20-28.

This is a solid proposal with a long track record. The questions raised may seem insignificant on the surface, but concerns about hatchery straying, effectiveness monitoring, improving passage of non-natives, any active removal of beaver dams, and the lack of any adaptive management leave some lingering concerns.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The Oregon Fish Screen and Passage Project is of significance to regional programs by providing immediate and long-term protection for anadromous and resident fish species in the Deschutes, John Day, Umatilla, and Walla Walla River basins by installation of NOAA Criteria fish screens, irrigation diversions, pump intakes, and barrier removal as well as passage structures including ladders, siphons, culverts at road crossings, and habitat improvements associated with passage plus water efficiency devices. The project and personnel have many years of experience and thereby have developed the technical expertise to carry out the passage installations and modifications.

The Objectives are straightforward and appropriate.

Given the very large number of sites needing treatment a question arises as to what efforts are being used to prioritize work. How many individual projects remain? There are currently 300 diversion sites identified as important. Is there a target date to complete them? Also as projects are added so is maintenance, but the O&M budget is not increasing.

2. History: Accomplishments, Results, and Adaptive Management (Evaluation of Results)

The accomplishments and results to date are as expected, and are assisting the recovery of native fishes.

While there is a good discussion of lessons learned and subsequent program improvement, adaptive management is not being practiced nor is Structured Decision Making. This needs to be corrected by setting quantitative goals and timelines for improving fish passage for each project activity and by articulating hypotheses that can be tested statistically via appropriate monitoring data.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions

The project has a long history of working with landowners, Tribes and agencies – and these relationships appear to be working well. Nevertheless, it is not clear to what extent this program coordinates with others that are working on fish passage restoration, especially correction of culvert passage at road-stream crossings and irrigation diversions. The process for prioritization is described as primarily driven by ODFW with some reference to subbasin plan priorities. Given the wide area covered by the program and the multiple players involved in an array of restoration activities, including passage, it appears that additional coordination is needed.

Further, it appears that some of the instream activities proposed for the John Day River are not well coordinated with the Warm Springs Tribe and their proposed activities. Clarification is needed on this point.

Specific questions:

1) In the Umatilla Basin, it is proposed to construct pool and riffle habitat using instream modifications. Where opportunities exist, work on public, federal, state, tribal and private lands will be conducted to increase the quantity of pools and gravel dominated riffles, as opposed to cobble. What agreements are in place to conduct this work? Is there any overlap with other proposals or projects?

2) The river systems being considered for fish passage improvements contain many native fishes as well as non-natives. Have analyses been conducted to compare the positive versus detrimental effects of improving passage for non-natives? Further, hatchery strays are a serious consideration, especially on the spawning grounds of native salmonids. Have similar analyses been conducted to weigh the advantages and disadvantages of improved passage?

3) Is there any way to use the screens and diversions to restrict the movements of non-native fishes?

4) What is the project’s policy concerning beaver dams?

It is refreshing to see that the project is taking climate change seriously as an emerging limiting factor and planning for it. The reality is that this region is already more than half way through the transition from snow-dominated late season runoff to more winter precipitation as rain.

4. Deliverables, Work Elements, Metrics, and Methods

The program is grounded in screening and passage yet is reliant on personal observation and word of mouth for the identification of many sites. Unfortunately, there is no assessment of irrigation diversions for the Deschutes River basin and there was no discussion as to when that might occur. Additionally, there is no reference to other assessments such as the Forest Service’s comprehensive assessment of fish passage at road-stream crossings.

Project prioritization is accomplished primarily by ODFW personnel using sets of rating criteria. It is not clear if these are only considered or whether they are actually scored for setting priorities. Additionally, there seems to be little if any coordination with restoration activities of other stakeholders which may be an important consideration in establishing a multi-year work plans.

The reporting rate is very low in comparison to other projects. This needs to be improved. Otherwise, the deliverables, work elements, metrics, and methods are appropriate.

In the budget, overhead is charged at 22%. In addition, there is a line item for rent and utilities (water, sewer, power, telephone, postage, office supplies, propane, garbage service, inspections, build). Should this line item be paid through the overhead rate?

Specific comments on protocols and methods described in MonitoringMethods.org

The need for improved Effectiveness Monitoring was previously identified by ISRP. According to the proposal this was not possible at most sites, for a variety of reasons. As a result, three by-pass traps were installed in the John Day watershed and three additional sites were to be established in 2013. There is no discussion of how these sites were selected or any findings, other than the number and species of fish trapped. As well, no location is given for the new sites. It was also mentioned that there are continuing difficulties in funding of effectiveness monitoring. Effectiveness monitoring needs additional attention. Further, there was no mention of ISEMP or AEM or any future program involvement with monitoring.

===========QUALIFICATIONS FOLLOW================

This is a solid proposal with a long track record. The questions raised may seem insignificant on the surface, but concerns about hatchery straying, effectiveness monitoring, improving passage of non-natives, any active removal of beaver dams, and the lack of any adaptive management leave some lingering concerns.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The Oregon Fish Screen and Passage Project is of significance to regional programs by providing immediate and long-term protection for anadromous and resident fish species in the Deschutes, John Day, Umatilla, and Walla Walla River basins by installation of NOAA Criteria fish screens, irrigation diversions, pump intakes, and barrier removal as well as passage structures including ladders, siphons, culverts at road crossings, and habitat improvements associated with passage plus water efficiency devices. The project and personnel have many years of experience and thereby have developed the technical expertise to carry out the passage installations and modifications.

The Objectives are straightforward and appropriate.

Given the very large number of sites needing treatment a question arises as to what efforts are being used to prioritize work. How many individual projects remain? There are currently 300 diversion sites identified as important. Is there a target date to complete them? Also as projects are added so is maintenance, but the O&M budget is not increasing.

2. History: Accomplishments, Results, and Adaptive Management (Evaluation of Results)

The accomplishments and results to date are as expected, and are assisting the recovery of native fishes.

While there is a good discussion of lessons learned and subsequent program improvement, adaptive management is not being practiced nor is Structured Decision Making. This needs to be corrected by setting quantitative goals and timelines for improving fish passage for each project activity and by articulating hypotheses that can be tested statistically via appropriate monitoring data.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions

The project has a long history of working with landowners, Tribes and agencies – and these relationships appear to be working well. Nevertheless, it is not clear to what extent this program coordinates with others that are working on fish passage restoration, especially correction of culvert passage at road-stream crossings and irrigation diversions. The process for prioritization is described as primarily driven by ODFW with some reference to subbasin plan priorities. Given the wide area covered by the program and the multiple players involved in an array of restoration activities, including passage, it appears that additional coordination is needed.

Further, it appears that some of the instream activities proposed for the John Day River are not well coordinated with the Warm Springs Tribe and their proposed activities. Clarification is needed on this point.

Specific questions:

1) In the Umatilla Basin, it is proposed to construct pool and riffle habitat using instream modifications. Where opportunities exist, work on public, federal, state, tribal and private lands will be conducted to increase the quantity of pools and gravel dominated riffles, as opposed to cobble. What agreements are in place to conduct this work? Is there any overlap with other proposals or projects?

2) The river systems being considered for fish passage improvements contain many native fishes as well as non-natives. Have analyses been conducted to compare the positive versus detrimental effects of improving passage for non-natives? Further, hatchery strays are a serious consideration, especially on the spawning grounds of native salmonids. Have similar analyses been conducted to weigh the advantages and disadvantages of improved passage?

3) Is there any way to use the screens and diversions to restrict the movements of non-native fishes?

4) What is the project’s policy concerning beaver dams?

It is refreshing to see that the project is taking climate change seriously as an emerging limiting factor and planning for it. The reality is that this region is already more than half way through the transition from snow-dominated late season runoff to more winter precipitation as rain.

4. Deliverables, Work Elements, Metrics, and Methods

The program is grounded in screening and passage yet is reliant on personal observation and word of mouth for the identification of many sites. Unfortunately, there is no assessment of irrigation diversions for the Deschutes River basin and there was no discussion as to when that might occur. Additionally, there is no reference to other assessments such as the Forest Service’s comprehensive assessment of fish passage at road-stream crossings.

Project prioritization is accomplished primarily by ODFW personnel using sets of rating criteria. It is not clear if these are only considered or whether they are actually scored for setting priorities. Additionally, there seems to be little if any coordination with restoration activities of other stakeholders which may be an important consideration in establishing a multi-year work plans.

The reporting rate is very low in comparison to other projects. This needs to be improved. Otherwise, the deliverables, work elements, metrics, and methods are appropriate.

In the budget, overhead is charged at 22%. In addition, there is a line item for rent and utilities (water, sewer, power, telephone, postage, office supplies, propane, garbage service, inspections, build). Should this line item be paid through the overhead rate?

Specific comments on protocols and methods described in MonitoringMethods.org

The need for improved Effectiveness Monitoring was previously identified by ISRP. According to the proposal this was not possible at most sites, for a variety of reasons. As a result, three by-pass traps were installed in the John Day watershed and three additional sites were to be established in 2013. There is no discussion of how these sites were selected or any findings, other than the number and species of fish trapped. As well, no location is given for the new sites. It was also mentioned that there are continuing difficulties in funding of effectiveness monitoring. Effectiveness monitoring needs additional attention. Further, there was no mention of ISEMP or AEM or any future program involvement with monitoring.

===========QUALIFICATIONS FOLLOW================