This proposal was less comprehensive than project 2010-072-00, which implements projects developed by this project. During the site visit, it was clear that the efforts of this planning and coordination project were well integrated with the implementation proposal. Ideally for review purposes, the two proposals would be combined into one proposal. The program has a strategy for improving salmonid habitat, and it has a monitoring program to evaluate the response of fish and habitat. Therefore, the ISRP considered the greater detail provided in project 2010-072-00, and information provided at the site visit when concluding that this project meets scientific review criteria.

The ISRP does not request a response, but we have identified some issues below that could be addressed in future statements of work and proposals.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

Significance to regional programs is adequately described. More detail is needed on goals of the project for reducing specific limiting factors in the watersheds. The problem statement should include objectives for focal fish species established in subbasin and draft recovery plans, and a time frame for improvement in both habitat and fish. The technical background should provide an indication of how much improvement is needed and the extent to which this project will contribute to those efforts. The narrative for the objectives should include habitat and fish metrics that can be used for project evaluation. For example, how many projects of each category does it plan to facilitate during the five year project period and to what extent will these projects fix the 2,950 water diversions and the 26 of 30 tributaries that are disconnected from the mainstem?

In addition the USBWP, under Project 2007-394-00, supports two long-term monitoring and evaluation efforts. These two long-term projects include 1) groundwater connectivity studies to evaluate management options relative to existing surface water rights in the Lemhi River and 2) a 15-year grazing monitoring plan to study the effects of reduced, late season grazing inside a fence. These activities were not included in the Objectives but are listed as deliverables. These deliverables were quite different from the planning and coordination projects.

2. History: Accomplishments, Results, and Adaptive Management (Evaluation of Results)

The accomplishments section is too brief for detailed ISRP review. However, a more detailed description of accomplishments and results are presented in project 201007200. The sponsor should provide a more detailed description of the scoring system used to evaluate projects, provide the expert panel scores, break the projects down into watershed and type of project, and identify the entity that implemented the project.

Project coordination is certainly important, and providing service to SWCDs and others with planning, permitting, and contracting is valuable. The results section should provide more details on final planning and site selection and alternatives that were considered to address a problem.

The NOAA habitat expert panel is used to judge projects for survival improvement benefits before final selection for implementation, which seems appropriate. However, in the accomplishments and results section, the sponsor states that the expert panel will evaluate project success by the extent limiting factor status has improved. It is not clear whether this evaluation is based entirely on weightings given to projects that have been completed, or whether empirical monitoring from field investigations like CHaMP, and fish monitoring, will contribute to the analysis.

Project sponsors are involved in monitoring efforts to address water flow by measuring wells during the irrigation system and to address grazing in fenced pastures. While these monitoring studies are in the early stages of data collection, the designs should be presented in more detail and the observations from annual field investigations provided.

Adaptive Management: Expert Panel input is changing the type and scope of work undertaken, but the proposal narrative does not explain what those influences might be. The project has transitioned from individual site work to an integrated watershed approach attempting to correct various limiting factors. No examples are provided in the proposal. The proposal identified that the Lemhi watershed is an ISEMP, CHaMP, and IMW site and data from these studies have contributed to restoration planning decisions. More information is needed on outcomes for habitat and fish response, including a description of results to date on the riparian fencing demonstration.

According to the proposal, the principal accomplishment of 2007-394-00 was the development of recovery actions to address limiting factors to protect, enhance, and restore anadromous and resident fish habitat and achieve a balance between resource protection and human use in the Upper Salmon River Basin. From 2006 until 2012, the USBWP staff has developed 35 projects and from 2007 to 2012 have implemented 29 projects through the Lemhi Soil and Water Conservation District and other entities. Ground water level has been monitored in wells and fences have been erected to protect riparian areas. Specific outcomes were not described. Implementation of specific projects since 2009 was described in 2010-072-00.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions

Project Relationships: A schematic would be helpful showing how all the projects work together to restore MPGs of steelhead and spring Chinook by restoring and enhancing habitat. Many projects are noted, but what they do in relation to 2007-394-00 is not always clear. This project reportedly provides a planning/design/permitting function, and whether this is provided to all associated projects is not clear. Many Upper Salmon proposals mentioned this project, but this proposal focuses on Lemhi restoration only.

Tailored Questions, Monitoring and Large Habitat: A link to the demo site for project comparison and evaluation is provided. A worthwhile adaptive management question is how does this habitat restoration selection scheme work in comparison to others, and how well are the selected projects meeting subbasin and watershed objectives? Considerable thought has been given to the planning scheme, but whether it is working for the benefit of fish is uncertain. The proposal identifies that site specific projects are no longer a priority, and that whole watershed integration of flow, temperature, sediment, and passage is now the principal driver of developing projects within a single watershed. The proposal states this requires hydrologists, planners, construction experts, and biologists. Explaining how this integration takes place would be useful. Once a project is selected for development, it is not clear who initiates development of alternatives and a specific design plan.

The sponsors have developed a web-based evaluation and tracking tool to complement their selection process. It would be worthwhile for the team to compare their approach with Science Base and Tools for Evaluating Stream Engineering, Management, and Restoration Proposals, NOAA Technical Memorandum NMFS-NWFSC-112 and the web based evaluation tools already developed and peer reviewed. In the presentation to the ISRP, an explanation of the different functions for the projects tracking tool and the River Rat tracking tool was provided. A written explanation should be incorporated into the proposal and projects web tracking tools documentation.

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables are generally consistent with the objectives and discussion of project selection, design, and proposal development. However, since there are no funds attached to each deliverable it is difficult to assess the sufficiency of what is being proposed with the extent of habitat restoration that needs to take place.

Metrics and Methods: The proposal states there are no RM&E methods or protocols. However, one deliverable is monitoring of wells to determine the status of groundwater recharge and another deliverable is monitoring of grazing effects. These monitoring activities should be discussed in the RM&E section. The experimental designs should be made public so they can be reviewed.

This proposal was less comprehensive than project 2010-072-00, which implements projects developed by this project. During the site visit, it was clear that the efforts of this planning and coordination project were well integrated with the implementation proposal. Ideally for review purposes, the two proposals would be combined into one proposal. The program has a strategy for improving salmonid habitat, and it has a monitoring program to evaluate the response of fish and habitat. Therefore, the ISRP considered the greater detail provided in project 2010-072-00, and information provided at the site visit when concluding that this project meets scientific review criteria.

The ISRP does not request a response, but we have identified some issues below that could be addressed in future statements of work and proposals.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

Significance to regional programs is adequately described. More detail is needed on goals of the project for reducing specific limiting factors in the watersheds. The problem statement should include objectives for focal fish species established in subbasin and draft recovery plans, and a time frame for improvement in both habitat and fish. The technical background should provide an indication of how much improvement is needed and the extent to which this project will contribute to those efforts. The narrative for the objectives should include habitat and fish metrics that can be used for project evaluation. For example, how many projects of each category does it plan to facilitate during the five year project period and to what extent will these projects fix the 2,950 water diversions and the 26 of 30 tributaries that are disconnected from the mainstem?

In addition the USBWP, under Project 2007-394-00, supports two long-term monitoring and evaluation efforts. These two long-term projects include 1) groundwater connectivity studies to evaluate management options relative to existing surface water rights in the Lemhi River and 2) a 15-year grazing monitoring plan to study the effects of reduced, late season grazing inside a fence. These activities were not included in the Objectives but are listed as deliverables. These deliverables were quite different from the planning and coordination projects.

2. History: Accomplishments, Results, and Adaptive Management (Evaluation of Results)

The accomplishments section is too brief for detailed ISRP review. However, a more detailed description of accomplishments and results are presented in project 201007200. The sponsor should provide a more detailed description of the scoring system used to evaluate projects, provide the expert panel scores, break the projects down into watershed and type of project, and identify the entity that implemented the project.

Project coordination is certainly important, and providing service to SWCDs and others with planning, permitting, and contracting is valuable. The results section should provide more details on final planning and site selection and alternatives that were considered to address a problem.

The NOAA habitat expert panel is used to judge projects for survival improvement benefits before final selection for implementation, which seems appropriate. However, in the accomplishments and results section, the sponsor states that the expert panel will evaluate project success by the extent limiting factor status has improved. It is not clear whether this evaluation is based entirely on weightings given to projects that have been completed, or whether empirical monitoring from field investigations like CHaMP, and fish monitoring, will contribute to the analysis.

Project sponsors are involved in monitoring efforts to address water flow by measuring wells during the irrigation system and to address grazing in fenced pastures. While these monitoring studies are in the early stages of data collection, the designs should be presented in more detail and the observations from annual field investigations provided.

Adaptive Management: Expert Panel input is changing the type and scope of work undertaken, but the proposal narrative does not explain what those influences might be. The project has transitioned from individual site work to an integrated watershed approach attempting to correct various limiting factors. No examples are provided in the proposal. The proposal identified that the Lemhi watershed is an ISEMP, CHaMP, and IMW site and data from these studies have contributed to restoration planning decisions. More information is needed on outcomes for habitat and fish response, including a description of results to date on the riparian fencing demonstration.

According to the proposal, the principal accomplishment of 2007-394-00 was the development of recovery actions to address limiting factors to protect, enhance, and restore anadromous and resident fish habitat and achieve a balance between resource protection and human use in the Upper Salmon River Basin. From 2006 until 2012, the USBWP staff has developed 35 projects and from 2007 to 2012 have implemented 29 projects through the Lemhi Soil and Water Conservation District and other entities. Ground water level has been monitored in wells and fences have been erected to protect riparian areas. Specific outcomes were not described. Implementation of specific projects since 2009 was described in 2010-072-00.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions

Project Relationships: A schematic would be helpful showing how all the projects work together to restore MPGs of steelhead and spring Chinook by restoring and enhancing habitat. Many projects are noted, but what they do in relation to 2007-394-00 is not always clear. This project reportedly provides a planning/design/permitting function, and whether this is provided to all associated projects is not clear. Many Upper Salmon proposals mentioned this project, but this proposal focuses on Lemhi restoration only.

Tailored Questions, Monitoring and Large Habitat: A link to the demo site for project comparison and evaluation is provided. A worthwhile adaptive management question is how does this habitat restoration selection scheme work in comparison to others, and how well are the selected projects meeting subbasin and watershed objectives? Considerable thought has been given to the planning scheme, but whether it is working for the benefit of fish is uncertain. The proposal identifies that site specific projects are no longer a priority, and that whole watershed integration of flow, temperature, sediment, and passage is now the principal driver of developing projects within a single watershed. The proposal states this requires hydrologists, planners, construction experts, and biologists. Explaining how this integration takes place would be useful. Once a project is selected for development, it is not clear who initiates development of alternatives and a specific design plan.

The sponsors have developed a web-based evaluation and tracking tool to complement their selection process. It would be worthwhile for the team to compare their approach with Science Base and Tools for Evaluating Stream Engineering, Management, and Restoration Proposals, NOAA Technical Memorandum NMFS-NWFSC-112 and the web based evaluation tools already developed and peer reviewed. In the presentation to the ISRP, an explanation of the different functions for the projects tracking tool and the River Rat tracking tool was provided. A written explanation should be incorporated into the proposal and projects web tracking tools documentation.

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables are generally consistent with the objectives and discussion of project selection, design, and proposal development. However, since there are no funds attached to each deliverable it is difficult to assess the sufficiency of what is being proposed with the extent of habitat restoration that needs to take place.

Metrics and Methods: The proposal states there are no RM&E methods or protocols. However, one deliverable is monitoring of wells to determine the status of groundwater recharge and another deliverable is monitoring of grazing effects. These monitoring activities should be discussed in the RM&E section. The experimental designs should be made public so they can be reviewed.

Modified by Dal Marsters on 6/11/2013 2:04:49 PM.

The proposal and response materials for this longstanding project (now downsized to include only the Lemhi SWCD area) report the tasks accomplished but do not give reviewers a clear picture of the extent to which those tasks have improved habitat conditions and/or fish populations. Also still unclear is how far along they are in meeting their long-term goals, how much have they accomplished, and how much needs to be done.

In response to past ISRP comments, project staff in a previous proposal committed to develop a more unified monitoring and evaluation program. Yet the current proposal and response make it clear that project personnel are struggling with M&E, as discussed below.

The question of where they are in the overall model watershed plan has not been satisfactorily answered, especially in any quantitative sense. Projects ready for implementation in FY 07 should be funded as well as administrative efforts focused on monitoring, including developing and using an analysis approach that would allow a substantive assessment of the entire project's success in terms of benefits to fish. Technical lessons learned should be summarized. Funding beyond FY 07 should be dependent upon evidence that the project is focused on realizing the greatest benefit for the resources invested and is using appropriate effectiveness monitoring through the analysis and adaptive management phases. Comments below are intended as constructive criticism. Reviewers note that considerable assistance in dealing with these M&E problems should be forthcoming from the Integrated Status and Effectiveness Monitoring Program (ISEMP) project 200301700 that is doing work in the Upper Salmon. Its scope is the design and evaluation of monitoring tools for salmon populations and habitat in the Interior Columbia River Basin.

The Project Monitoring Report for FY 05 that was included in the response did provide some specifics that contrast with the generalities in the proposal. The project has largely divorced itself from fish, using the rationale that anadromous fish in the watershed are controlled by out-of-basin factors. "Habitat" becomes the surrogate, perhaps not inappropriately. But rather than being applied to aquatic habitat that is valuable for resident salmonids (and thus for anadromous fish rearing in future if runs increase), the term "habitat" has become a nebulous entity. The key attributes for salmonids that are vital and easily measured (like maximum water depth and bank shading) were not recorded. The Project Monitoring Report examined 16 sites funded by BPA. Most were riparian fencing, presumably a subsample of the approximately 50 miles of fencing that the proposal indicates have been installed since 1994. Information was gathered by photo monitoring, greenline survey, and "datasheets." This approach appears reasonable if amended as described above. It was clear from the report that such monitoring is in its embryonic stage. Absent was any summary of what worked and what did not, and any discussion of why. Reviewers could see no evidence that such a report was integrated into the project to help direct future efforts.

The current plan includes some pre-project monitoring, implementation monitoring in year 1, then monitoring every 5 years for 10 and 15 year contracts. This means only a few views of a project. No end-of-project monitoring is described, nor any planned response if results are not satisfactory, or if unanticipated opportunities arise. Page 18 of the response says, "Analysis has not yet been determined." Yet this is the key element of adaptive management, suggesting that the entire point of monitoring has been missed.

Salmon data provided do not show clear separation between wilderness stream redds (Big Creek) and the Lemhi but this is the kind of comparison that should help provide an assessment of the habitat treatment protocols used in the Lemhi Basin. The sponsors produced what seems to be an honest assessment of data for fish abundance before and after the habitat work was implemented. Comparison of redds in other non-treated basins and the Lemhi Basin is not perfect in that they cannot eliminate the possibility that out-of-basin effects are different for populations in these basins; assessment efforts should include consideration of the probability of this alternative. In addition, the fish data show no benefit of the habitat work, so at least three alternative explanations are possible; (1) the habitat work has not been effective in increasing productivity, (2) the work that has been done is nowhere near enough to cause increased productivity, or (3) the wrong changes were implemented. Sponsors have the responsibility to sort out these and other explanations for the apparent absence of a response.