This is a complex and high-impact project with many partnerships, connections, and reports. The ISRP greatly appreciates the time and attention dedicated to organizing the proposal to effectively communicate the project’s results and the details of the plan for moving forward. Some of the proposal and project strengths are:

• The proposal provides a through context for the regional and tribal history, landscape characteristics, ecological focus, limiting factors, monitoring programs, and relevant management plans for the project.
• The proposal provides a high level of detail in the narrative of the text, and then provided direct links to documents where more information could be found.
• Including hot links to annual reports, umbrella, monitoring, and other reports are always appreciated since it saves the ISRP time in locating the documents.
• The proposal uses maps and figures effectively to communicate results and strategy.
• The project proponents have developed a diverse, scientifically robust, and collaborative approach to developing and implementing monitoring in the basin, and they have demonstrated a commitment to managing data produced by those efforts.
• The highly collaborative nature of the project, from monitoring to data management to the common ground restoration approach, is a core strength.
• The 15-year synthesis is of great value and, once complete, will be an important contribution both to the John Day basin and Columbia River restoration more broadly.

The ISRP notes that the prioritization and adaptation processes are not as linear and structured as for other (and often simpler) projects. However, the processes reflect a thoughtful strategy and appear to be serving the project. Thus, a highly structured, formal decision-making framework does not seem essential to continuous learning for this project.

The project meets scientific review criteria, but the ISRP makes the following suggestions for project improvement. Actions toward addressing these suggestions should begin immediately and the results can be described in future work plans, annual reports, and proposals.

1. SMART Objectives: The proponents should clearly identify the project goals and objectives for the next project period. The objectives should be written as SMART objectives (see proposal instructions). These goals and objectives can be provided in the next annual report. 
   
   
2. M&E matrix - support. As habitat projects and monitoring projects are not presented as part of an integrated proposal or plan, the need for a crosswalk to identify the linkages between implementation and monitoring is extremely important for basins or geographic areas. The ISRP is requesting a response from the Escapement and Productivity of Spring Chinook and Steelhead Project (199801600) to summarize the linkages between implementation and monitoring projects in the basin. During the response loop (September 24 to November 22, 2021), we ask your project to assist them in creating the summary and provide information to them about what is being monitored for your implementation project and where, when, and by whom the monitoring occurs. A map or maps of locations of monitoring actions would be helpful in this regard. In most cases, we are asking a regional coordination project like yours to develop the summary, but during the project presentation discussions, the ODFW project agreed that it would be more efficient for them to lead the effort and rely on your project and others for supporting information and assistance.

Q1: Clearly defined objectives and outcomes

The proposal describes a comprehensive and thoughtful list of goals and objectives to guide the project, but they were not organized in a way that made it easy for the ISRP to understand what exactly is being planned. The goal of the program is to “protect, manage, and restore aquatic habitats” and objectives supporting that goal are provided (pages 2 and 25), but they are not SMART. Each objective does have a “physical benchmark,” which could theoretically be measurable but, as written, they include vaguely-defined benchmarks, such as “functioning appropriately.” Most of the objectives are more like project goals and lack quantitative outcomes with explicit time frames. SMART objectives provide a framework for evaluating the trajectory of the outcomes and are not intended as contractual guarantees or constraints on future funding. On page 27, the proposal includes one outcome for the project (long-term increasing trend in fish populations), a SMART goal for that outcome, and Table 3, which the ISRP understands is to provide some linkages between actions and measuring the progress toward meeting that goal. Table 3 seems to cover a reasonable set of actions and maps them directly to locations, measures of success, and monitoring strategy. However, the table is not easy to interpret, and the ISRP has difficulty understanding what the measurable outcomes might be.

The ISRP finds it difficult to follow the many different levels of goals, objectives, outcomes, and then objectives again. For example, how does Objective 1A1 (in Table 3) relate to SMART goal 1.1? Are they the same? And how does Outcome 1 relate to the objectives listed in pages 2 and 25? The ISRP called for “quantitative objectives and measures of progress towards those objectives” in the 2017 Umbrella Review, and the need for improvement in this project area remains.

Q2: Methods

This project is responsible for identification, development, implementation, and monitoring of restoration projects. The ISRP review of the project monitoring is discussed in the following section. For identification and development, during the next project period, the JDWR Project will use a combination of the:

• CTWSRO Strategy for metrics like restoration potential benefit, limiting factors, and identified target and focal actions.
• John Day Basin Partnership’s Atlas for prioritizing basins for project implementation.
• Structured Implementation and Monitoring Framework (SIM) for setting the implementation timeline.

Appendix B is a map of planned project locations for FY 2023-27 that, in combination with Tables 2 & 3, gives a general sense of what types of projects will be pursued. Given that the projects were not selected at the time the proposal was submitted, the text lacks some details on what exactly will be performed where. This limits the ISRP to a review of the process rather than the projects. Based on the information available in the proposal and in the linked documents, the framework for selecting projects, which applies the Strategy, Atlas, and SIM, appears to reflect meaningful ecological metrics. As well, the framework appropriately relies on collaboration with basin partners.

Q3: Provisions for M&E

Collectively, the monitoring program aims to address both project-level effectiveness and basin-level fish responses. Given the diverse collection of projects and partners, as well as the need for collaboration due to budget limitations, the proponents rely on different study plans and strategies across subbasins and projects. For example, in Fox Creek, the proponents developed a monitoring plan that is based on the BACI design, one that will be implemented over multiple years. In addition, the project uses their Implementation Effectiveness Monitoring (IEM) to revisit specific types of projects after implementation to identify maintenance needs. They are also participating in the Structured Implementation Monitoring framework (SIM) with ODFW for projects in the Middle Fork John Day, as well as the Intensively Monitored Watershed (IMW) and BPA’s Action Effectiveness Monitoring (AEM) program.

While details of the monitoring plans are a bit scarce in the proposal, the proponents provided links to several monitoring plans. These plans include a) SMART monitoring objectives that are measurable, b) hypotheses about what response is expected, c) sampling locations, timelines, and protocols, and d) details on data management. In short, the ISRP is impressed at the robustness and careful design of the monitoring plans and commends the proponents for this comprehensive effort. Furthermore, the ISRP was also impressed by the efforts made in centralizing data management, the development of the web-based project tracker, and the decision to hire a data manager to support that effort.

Regarding project adjustments, the proposal links to the Partnership’s Theory of Change approach and also provides some specific examples of how monitoring feeds back into project decision making and design. The details of the Theory of Change framework provide both narrative examples and a complex figure (Figure 2 of Appendix D) to help illustrate justifications for project selection and the anticipated outcomes of actions, but do not provide much insight on project adjustments. However, the proposal’s specific examples of how monitoring data have been applied in decision making are very informative. For example, the proposal outlines key findings from the IMW related to the important role of temperature, shading, and tributaries (rather than groundwater) and how those findings have impacted their restoration work and prioritization. In addition, the proposal describes pre-determined checkpoints (e.g., 25% canopy coverage after 5 years) in the IEM plans that trigger project review and potential modifications. Another example describes how monitoring data are used to delay project implementation following a very poor return of spring Chinook. In summary, while the proposal did not include the summary of a structured decision-making framework for adaptive management, the ISRP finds that the proponents are effectively collecting and utilizing data about the project to inform decisions and are satisfied that this process serves the project’s needs.

Q4: Results – benefits to fish and wildlife

The proposal includes some informative graphics on results from the past twenty years, including an online map, a table that covers the entire project, and a time series of restoration metrics (miles restored, miles accessed, number of LWD structures, acres of juniper removed, number of riparian plants planted). The results are impressive, though it is not clear to what degree these actions have directly benefited fish. The proponents identify another project that has been responsible for data on recovery of fish populations, though connection of the physical metrics and population data to examine causality is not reported. Given that the data are not designed to be used in that way, it is appropriate that such an analysis is not included. Nevertheless, the ISRP looks forward to what proponents will learn from the targeted monitoring that is planned and underway on how the activities are influencing targeted fish populations.

The proposal also discusses outreach and engagement efforts in multiple places. Perhaps most notable is a short film that highlights project efforts, particularly around collaboration between the Tribes and ranchers along Fox Creek. The film is professional and inspiring, and high impact. Since it was published in Oct. 2019, it has over 1,700 views, has been included in multiple film festivals and will be shown in classrooms throughout Oregon.

This proposal, largely conceptual in format, has two distinct aspects: habitat implementation and project prioritization and selection. It is intended to develop an implementation strategy, including stakeholder and advisory committees, development of scientific scoring of biological integrity, and a feasibility scoring system to guide the selection and completion of suites of habitat restoration projects for 2014-2018. Overall, this project has a successful record of accomplishments, especially related to improving fish passage. The discussion of plans for restoration and desired elements of a restoration strategy including protect and maintain highest quality habitat areas, manage land to ensure ecological integrity and function and restore highest priority watersheds and habitat are presented but are not thoroughly incorporated into the proposal.

The project, as written, intends to be an umbrella project for fish habitat restoration in the John Day basin. However, it was not clearly indicated how much support the sponsor’s strategy for the basin will have from other entities doing work in the subbasin and operating independently for decades. What is the overall plan for the basin? How does this proposed project mesh with other basin activities? What is the exact nature of the cooperation and how are the sponsors going to include all the managers, Oregon Department of Fish and Wildlife (ODFW), the soil and water conservation districts, and other stakeholders in their strategic planning? The sponsors should bring the TAC in early in the process to assist with a strategic plan for implementation and monitoring.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

This project is consistent with multiple tribal, federal, and state agency regional and subbasin recovery plans. The problem is clearly defined. The sponsors concisely discuss the major factors limiting fish production in the John Day subbasin and the kinds of restoration actions that should be taken to remediate them. Based on the discussion in Project History, it appears that significant progress has been made in the John Day subbasin in improving fish passage, habitat, and land management.

The sponsors provided a detailed description of a formalized Implementation Strategy that they are in the process of developing to prioritize restoration activities. This Strategy apparently was developed in response to a recommendation in the ISRP’s 2006 project review. Development of the Strategy is the first objective in the proposal. The remainder of the objectives pertain primarily to protection of high quality habitats and restoration of degraded habitats prioritized by the Implementation Strategy and thus these objectives are contingent on successful completion of Objective 1, "Develop Strategy Document,” which the sponsors say will be completed in 2014.

The ISRP commends the sponsors for developing what appears to be a rational, systematic procedure for project site selection and action. This approach could serve as a model for other restoration planning efforts in the subbasin, however many entities are working in the John Day Basin. Will they participate in the Strategy and follow the priority listing of projects? If so, how will they participate? There is a need to avoid duplication of effort in planning. The proposal states that "The Tribes would like to coordinate with basin partners and technical experts to leverage existing scientific data, physical information, and stakeholder input for the development of a strategic, prioritized restoration implementation strategy”. With the extensive planning efforts that have already been undertaken within the subbasin, including the Subbasin Plan and the Mid-Columbia Steelhead Conservation and Recovery Plan, it seems that much of what the Strategy proposes to do, that is determine fish use of stream reaches by life stage, limiting factor identification, site prioritization, appropriate remedial actions should have been completed some years ago. Why are partnerships still being built after 5 years? It seems they should have been already in place. It is not clear how well this proposed work is coordinated with ODFW, Soil and Water Conservation districts, and other basin entities. Why is prioritization only occurring now? It seems as though it should have been done prior to ongoing enhancement actions. The implementation phase of this work seems to be getting ahead of the coordination. It would have been helpful if the sponsors were more explicit about why their strategic approach is needed in lieu of other subbasin planning efforts. What will it provide that other planning documents have not?

As criteria for site selection, the sponsors may want to consider the locations of other restoration sites in the basin and proximity to high quality habitats.

The project objectives are actually goal statements and lack quantitative description of desired products or specified dates for completion. These need to be provided.

2. History: Accomplishments, Results, and Adaptive Management (Evaluation of Results)

Restoration actions undertaken so far by this project are primarily passage improvement, juniper removal, riparian planting, LWD placement, and installation of cattle exclosures. Results consist primarily of descriptions of projects that have been undertaken to date. Few quantitative results were presented. The proposal could have been improved if the sponsors had discussed in more detail what sort of M&E program is currently in place, what kind of monitoring data has been collected, and whether the data have been analyzed and utilized.

The sponsors discuss extensively the Middle Fork John Day Intensively Monitored Watershed (IMW) project in which they are a cooperator. It is unclear, however, how this project is related to the work set forth in the current proposal. The sponsors should have clearly identified how they will make use of the results from the IMW project in their proposed work and what role it has in development of the sponsor’s Implementation Strategy.

The sponsors consider development and implementation of the Strategy to represent adaptive management.

Restoration in the John Day has been ongoing for 30 years. Past ISRP comments (2006) suggested the need for clear criteria to prioritize projects, more M&E, development of an accomplishments report and review, and additional detail to be included in work elements. It appears that no retrospective analysis of past actions has been done. There is limited discussion of lessons learned and their application into program design or operation. A positive aspect is that there has been some upslope work that includes juniper treatment to improve streamflow. Unfortunately, there was no mention of the extent of this treatment needed to actually result in measurable increases in flow.

There are no clearly established criteria for prioritizing projects and there is little detail provided regarding key designs or considerations for work elements. There has been additional staffing for effectiveness monitoring.

To understand project significance at the landscape scale, the sponsors need to conceptualize at a wider scale than the reach scale. This is because many important processes, potentially affecting habitat quantity and quality, operate at broader than the reach scale. A geomorphologist should be included on the TAC for the project.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions

There are a large number of projects pertaining to both fish and habitat on-going and planned in the John Day basin as well as IMW and other ISEMP projects. Many of these projects appear to be taking place in similar parts of the subbasin and some have different objectives than others. One of the major questions is how all of these projects coordinate their restoration and monitoring activities so as to be complementary and not duplicative, and maximize the probability that the projects, taken together, have a positive cumulative impact on fish and habitat. For example, is project site selection done cooperatively with all major entities involved? It seems that the proposed Implementation Strategy could be used cooperatively by all entities working in the subbasin. Are the monitoring efforts consistent among projects in terms of the monitoring design, data collected, and analyses conducted? The ISRP recognizes that answering these questions should not solely be the responsibility of the sponsors of this project but rather it should be a joint response by all cooperators in the subbasin.

The sponsors discuss climate change as a potential problem and maintain that their habitat restoration work will help to mitigate climate change impacts especially to the extent that the restoration actions reduce water temperatures. No potential effects on lamprey are discussed. Additionally, there is no discussion of forest health and potential effects of major fires or disease outbreaks on aquatic habitat.

4. Deliverables, Work Elements, Metrics, and Methods

The first five Deliverables pertain to development of the Implementation Strategy which will prioritize project locations and is scheduled to be completed in 2014. Many of the remaining Deliverables are nearly restatements of the Objectives. Specific project locations are not identified in the Deliverables. They will be selected based on the outcome of the Implementation Strategy process. This approach is reasonable and should not delay commencement of the projects beyond 2014.

The work in public education and outreach is a positive element and it appears that a wide range of activities have been developed and implemented in the past few years.

Specific comments on protocols and methods described in MonitoringMethods.org

There is limited discussion on specific monitoring changes since the last ISRP review. There is no mention of future needs to become involved in ISEMP and AEM.

===========QUALIFICATIONS FOLLOW================

This proposal, largely conceptual in format, has two distinct aspects: habitat implementation and project prioritization and selection. It is intended to develop an implementation strategy, including stakeholder and advisory committees, development of scientific scoring of biological integrity, and a feasibility scoring system to guide the selection and completion of suites of habitat restoration projects for 2014-2018. Overall, this project has a successful record of accomplishments, especially related to improving fish passage. The discussion of plans for restoration and desired elements of a restoration strategy including protect and maintain highest quality habitat areas, manage land to ensure ecological integrity and function and restore highest priority watersheds and habitat are presented but are not thoroughly incorporated into the proposal.

The project, as written, intends to be an umbrella project for fish habitat restoration in the John Day basin. However, it was not clearly indicated how much support the sponsor’s strategy for the basin will have from other entities doing work in the subbasin and operating independently for decades. What is the overall plan for the basin? How does this proposed project mesh with other basin activities? What is the exact nature of the cooperation and how are the sponsors going to include all the managers, Oregon Department of Fish and Wildlife (ODFW), the soil and water conservation districts, and other stakeholders in their strategic planning? The sponsors should bring the TAC in early in the process to assist with a strategic plan for implementation and monitoring.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

This project is consistent with multiple tribal, federal, and state agency regional and subbasin recovery plans. The problem is clearly defined. The sponsors concisely discuss the major factors limiting fish production in the John Day subbasin and the kinds of restoration actions that should be taken to remediate them. Based on the discussion in Project History, it appears that significant progress has been made in the John Day subbasin in improving fish passage, habitat, and land management.

The sponsors provided a detailed description of a formalized Implementation Strategy that they are in the process of developing to prioritize restoration activities. This Strategy apparently was developed in response to a recommendation in the ISRP’s 2006 project review. Development of the Strategy is the first objective in the proposal. The remainder of the objectives pertain primarily to protection of high quality habitats and restoration of degraded habitats prioritized by the Implementation Strategy and thus these objectives are contingent on successful completion of Objective 1, "Develop Strategy Document,” which the sponsors say will be completed in 2014.

The ISRP commends the sponsors for developing what appears to be a rational, systematic procedure for project site selection and action. This approach could serve as a model for other restoration planning efforts in the subbasin, however many entities are working in the John Day Basin. Will they participate in the Strategy and follow the priority listing of projects? If so, how will they participate? There is a need to avoid duplication of effort in planning. The proposal states that "The Tribes would like to coordinate with basin partners and technical experts to leverage existing scientific data, physical information, and stakeholder input for the development of a strategic, prioritized restoration implementation strategy”. With the extensive planning efforts that have already been undertaken within the subbasin, including the Subbasin Plan and the Mid-Columbia Steelhead Conservation and Recovery Plan, it seems that much of what the Strategy proposes to do, that is determine fish use of stream reaches by life stage, limiting factor identification, site prioritization, appropriate remedial actions should have been completed some years ago. Why are partnerships still being built after 5 years? It seems they should have been already in place. It is not clear how well this proposed work is coordinated with ODFW, Soil and Water Conservation districts, and other basin entities. Why is prioritization only occurring now? It seems as though it should have been done prior to ongoing enhancement actions. The implementation phase of this work seems to be getting ahead of the coordination. It would have been helpful if the sponsors were more explicit about why their strategic approach is needed in lieu of other subbasin planning efforts. What will it provide that other planning documents have not?

As criteria for site selection, the sponsors may want to consider the locations of other restoration sites in the basin and proximity to high quality habitats.

The project objectives are actually goal statements and lack quantitative description of desired products or specified dates for completion. These need to be provided.

2. History: Accomplishments, Results, and Adaptive Management (Evaluation of Results)

Restoration actions undertaken so far by this project are primarily passage improvement, juniper removal, riparian planting, LWD placement, and installation of cattle exclosures. Results consist primarily of descriptions of projects that have been undertaken to date. Few quantitative results were presented. The proposal could have been improved if the sponsors had discussed in more detail what sort of M&E program is currently in place, what kind of monitoring data has been collected, and whether the data have been analyzed and utilized.

The sponsors discuss extensively the Middle Fork John Day Intensively Monitored Watershed (IMW) project in which they are a cooperator. It is unclear, however, how this project is related to the work set forth in the current proposal. The sponsors should have clearly identified how they will make use of the results from the IMW project in their proposed work and what role it has in development of the sponsor’s Implementation Strategy.

The sponsors consider development and implementation of the Strategy to represent adaptive management.

Restoration in the John Day has been ongoing for 30 years. Past ISRP comments (2006) suggested the need for clear criteria to prioritize projects, more M&E, development of an accomplishments report and review, and additional detail to be included in work elements. It appears that no retrospective analysis of past actions has been done. There is limited discussion of lessons learned and their application into program design or operation. A positive aspect is that there has been some upslope work that includes juniper treatment to improve streamflow. Unfortunately, there was no mention of the extent of this treatment needed to actually result in measurable increases in flow.

There are no clearly established criteria for prioritizing projects and there is little detail provided regarding key designs or considerations for work elements. There has been additional staffing for effectiveness monitoring.

To understand project significance at the landscape scale, the sponsors need to conceptualize at a wider scale than the reach scale. This is because many important processes, potentially affecting habitat quantity and quality, operate at broader than the reach scale. A geomorphologist should be included on the TAC for the project.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions

There are a large number of projects pertaining to both fish and habitat on-going and planned in the John Day basin as well as IMW and other ISEMP projects. Many of these projects appear to be taking place in similar parts of the subbasin and some have different objectives than others. One of the major questions is how all of these projects coordinate their restoration and monitoring activities so as to be complementary and not duplicative, and maximize the probability that the projects, taken together, have a positive cumulative impact on fish and habitat. For example, is project site selection done cooperatively with all major entities involved? It seems that the proposed Implementation Strategy could be used cooperatively by all entities working in the subbasin. Are the monitoring efforts consistent among projects in terms of the monitoring design, data collected, and analyses conducted? The ISRP recognizes that answering these questions should not solely be the responsibility of the sponsors of this project but rather it should be a joint response by all cooperators in the subbasin.

The sponsors discuss climate change as a potential problem and maintain that their habitat restoration work will help to mitigate climate change impacts especially to the extent that the restoration actions reduce water temperatures. No potential effects on lamprey are discussed. Additionally, there is no discussion of forest health and potential effects of major fires or disease outbreaks on aquatic habitat.

4. Deliverables, Work Elements, Metrics, and Methods

The first five Deliverables pertain to development of the Implementation Strategy which will prioritize project locations and is scheduled to be completed in 2014. Many of the remaining Deliverables are nearly restatements of the Objectives. Specific project locations are not identified in the Deliverables. They will be selected based on the outcome of the Implementation Strategy process. This approach is reasonable and should not delay commencement of the projects beyond 2014.

The work in public education and outreach is a positive element and it appears that a wide range of activities have been developed and implemented in the past few years.

Specific comments on protocols and methods described in MonitoringMethods.org

There is limited discussion on specific monitoring changes since the last ISRP review. There is no mention of future needs to become involved in ISEMP and AEM.

===========QUALIFICATIONS FOLLOW================