Columbia Basin Fish and Wildlife Program Columbia Basin Fish and Wildlife Program
RSS Feed for updates to Project 2016-001-00 - BPA Project Action Effectiveness Monitoring (AEM) Programmatic Follow this via RSS feed. Help setting up RSS feeds?

Project Summary

Project 2016-001-00 - BPA Project Action Effectiveness Monitoring (AEM) Programmatic

Please Note: This project is the product of one or more merges and/or splits from other projects. Historical data automatically included here are limited to the current project and previous generation (the “parent” projects) only. The Project Relationships section details the nature of the relationships between this project and the previous generation. To learn about the complete ancestry of this project, please review the Project Relationships section on the Project Summary page of each parent project.

Project Number:
2016-001-00
Title:
BPA Project Action Effectiveness Monitoring (AEM) Programmatic
Summary:
BPA and its partners have funded thousands of habitat improvement actions throughout the Columbia River Basin under the Northwest Power and Conservation Council (NPCC) Fish and Wildlife Program for the benefit of anadromous and resident fish species. For more than three decades these actions have been monitored and evaluated individually by many different Fish and Wildlife Program partners using different methods and protocols. Data were difficult to obtain and analyze due to the differing approaches, and conclusions regarding the habitat and fish response to habitat improvement actions were difficult to formulate. In 2013, the BPA Action Effectiveness Monitoring Program (AEM) was developed to establish a comprehensive, consistent, efficient, and cost effective programmatic approach to monitor and evaluate this large quantity of habitat improvement actions implemented throughout a large area, the Columbia River Basin, with a large quantity of program partners including Tribes, Federal agencies, states, county and local governments, non-profit organizations, and universities. Although the majority of BPA’s Fish and Wildlife Program partners are no longer conducting effectiveness monitoring individually for their projects, the AEM Program collaborates with many partners, including the Shoshone Bannock Tribe, Columbia River Inter-Tribal Fish Commission, Nez Perce Tribe, Confederated Tribes of the Umatilla Indian Reservation, Confederated Tribes of the Warm Springs, Idaho Department of Fish and Game, and Oregon Department of Fish and Wildlife.

The Habitat improvement actions evaluated under the AEM Program include fish passage (complete and partial barrier removal and replacement), instream enrichment (large wood structure installation), floodplain enhancement (levee modification, floodplain reconnection, and channel reconstruction), and riparian restoration (planting and invasive plant species removal). An extensive post-treatment (EPT) design is used to evaluate actions completed prior to 2014. The EPT design, which has been used to evaluate complete barrier removal and replacement, wood structure installation, riparian planting, and invasive plant removal, samples a large number of paired treatment and control reaches one time, after the habitat improvement action has been implemented. A multiple before-after control-impact (MBACI) design is used to evaluate actions planned for implementation beginning in 2015 and beyond. The MBACI design, which has been used to evaluate partial barrier removal and replacement and floodplain enhancement, samples sites for two years pre-implementation and multiple years post-implementation (years 1 and 2, or years 1, 3, and 5).

This project was competed in 2017 and Cramer Fish Sciences was selected by BPA to implement and adaptively manage the AEM Program, including both the MBACI and EPT protocols, beginning in 2018.
Proposer:
Proponent Orgs:
Cramer Fish Sciences (Private)
Natural Systems Design (Private)
Starting FY:
2016
Ending FY:
2022
BPA PM:
Stage:
Implementation - Project Status Report
Area:
Province Subbasin %
Basinwide - 100.00%
Purpose:
Programmatic
Emphasis:
RM and E
Focal Species:
Chinook - All Populations
Chinook - Mid-Columbia River Spring ESU
Chinook - Snake River Fall ESU
Chinook - Snake River Spring/Summer ESU
Chinook - Upper Columbia River Spring ESU
Chinook - Upper Columbia River Summer/Fall ESU
Other Anadromous
Steelhead - All Populations
Steelhead - Middle Columbia River DPS
Steelhead - Snake River DPS
Steelhead - Upper Columbia River DPS
Species Benefit:
Anadromous: 100.0%   Resident: 0.0%   Wildlife: 0.0%
Tags:
None
Special:
None

No photos have been uploaded yet for this project.

Summary of Budgets

To view all expenditures for all fiscal years, click "Project Exp. by FY"

Expense SOY Budget Working Budget Contracted Amount Modified Contract Amount Expenditures *
FY2018 (Previous) $1,089,988 $1,089,988 $981,017 $470,888 $598,588

BiOp FCRPS 2008 (non-Accord) $1,089,988 $981,017 $470,888 $598,588
FY2019 (Current) $411,633 $0 $470,888 $252,117

BiOp FCRPS 2008 (non-Accord) $411,633 $0 $470,888 $252,117
FY2020 (Next) $0 $0 $39,241 $0

BiOp FCRPS 2008 (non-Accord) $0 $0 $39,241 $0

* Expenditures data includes accruals and are based on data through 31-Mar-2019

Decided Budget Transfers  (FY2018 - FY2020)

Acct FY Acct Type Amount Fund Budget Decision Date
FY2018 Expense $1,089,988 From: BiOp FCRPS 2008 (non-Accord) FY18 SOY Budgets 07/17/2017
FY2019 Expense $411,633 From: BiOp FCRPS 2008 (non-Accord) Nov 21st FY19 SOY Upload 11/23/2018

Pending Budget Decision?  No


Actual Project Cost Share

Current Fiscal Year — 2019
Cost Share Partner Total Proposed Contribution Total Confirmed Contribution
There are no project cost share contributions to show.
Previous Fiscal Years
Fiscal Year Total Contributions % of Budget
There are no cost share summaries to display from previous years.

Contracts

The table below contains contracts with the following statuses: Active, Complete, History, Issued.
* "Total Contracted Amount" column includes contracted amount from both capital and expense components of the contract.
Expense Contracts:
Number Contractor Name Title Status Total Contracted Amount Dates
78579 SOW Cramer Fish Sciences 2016-001-00 EXP BPA PROJ ACTION EFFECTIVENESS MONITORING Issued $981,017 2/23/2018 - 2/29/2020



Annual Progress Reports
Expected (since FY2004):5
Completed:4
On time:4
Status Reports
Completed:19
On time:7
Avg Days Late:8

Historical from: 2011-008-00
Earliest Subsequent           Accepted Count of Contract Deliverables
Contract Contract(s) Title Contractor Start End Status Reports Complete Green Yellow Red Total % Green and Complete Canceled
51379 55712, 59477, 67656 2011-008-00 EXP TECHNICAL SUPPORT FOR BIOP RM&E Tetra Tech, Inc. 01/2011 01/2011 History 20 21 0 0 0 21 100.00% 0
Project Totals 54 73 3 0 1 77 98.70% 3


Historical from: 2012-011-00
Earliest Subsequent           Accepted Count of Contract Deliverables
Contract Contract(s) Title Contractor Start End Status Reports Complete Green Yellow Red Total % Green and Complete Canceled
46273 REL 47 46273 REL 85, 46273 REL 98 2012-011-00 EXP TRIBUTARY HABITAT RM&E BIOP REPORTING National Oceanic and Atmospheric Administration 07/2012 07/2012 Closed 15 17 0 0 0 17 100.00% 3
Project Totals 54 73 3 0 1 77 98.70% 3


Earliest Subsequent           Accepted Count of Contract Deliverables
Contract Contract(s) Title Contractor Start End Status Reports Complete Green Yellow Red Total % Green and Complete Canceled
71969 74809 PROJECT ACTION EFFECTIVENESS MONITORING Cramer Fish Sciences 01/2016 01/2016 Closed 8 15 0 0 1 16 93.75% 0
72029 74745 2016-001-00 EXP PROJECT ACTION EFFECTIVENESS MONITORING (AEM) Natural Systems Design 01/2016 01/2016 Closed 8 16 0 0 0 16 100.00% 0
78579 2016-001-00 EXP BPA PROJ ACTION EFFECTIVENESS MONITORING Cramer Fish Sciences 02/2018 02/2018 Issued 3 4 3 0 0 7 100.00% 0
Project Totals 54 73 3 0 1 77 98.70% 3


The table content is updated frequently and thus contains more recent information than what was in the original proposal reviewed by ISRP and Council.

Review: 2018 Research Project Status Review

Independent Scientific Review Panel Assessment

Assessment Number: 2016-001-00-ISRP-20181115
Project: 2016-001-00 - BPA Project Action Effectiveness Monitoring (AEM) Programmatic
Review: 2018 Research Project Status Review
Completed Date: 11/15/2018
Final Round ISRP Date: 9/28/2018
Final Round ISRP Rating: Meets Scientific Review Criteria (Qualified)
Final Round ISRP Comment:

1. Objectives


The ultimate goals of AEM are to 1) quantify localized reach-scale improvements in habitat and juvenile steelhead and Chinook salmon abundance resulting from restoration actions implemented in the Columbia River Basin and 2) guide future restoration efforts to ensure that BPA is investing in effective restoration and habitat improvement techniques. Specifically, AEM is designed to answer the following questions:

1. What is the effect of different action categories on habitat at the reach scale?
2. What is the effect of different action categories on fish and other biota at a reach scale?
3. Within an action category, why are some projects more successful than others in producing physical and biological improvements? For example, what is the relationship between the amount or intensity of habitat improvement and fish response?
4. Are there differences among geographic areas (ESUs) in the physical and biological success of action types?

The first two questions are achievable and testable at the reach scale, whereas the other two are more challenging, the last owing to the need for comparisons across regions. For the Summary Report, the proponents transformed the third question from a "why" question into a "what" question to make it more quantifiable. The fourth question on differences in the physical and biological success of various types of habitat restoration among ESUs is not addressed in their report.


The answers to the questions are important and represent a first step in a fuller understanding of how habitat restoration benefits focal fish species across spatial and temporal scales. For watershed scale restoration, a combination of treatments, both instream/floodplain and those occurring upslope (road decommissioning, silvicultural treatments) are typically applied for accomplishment of restoration objectives. Also, other approaches will likely be needed to better understand whether there are differences in habitat and population effects, within action categories, due to differences in project design, intensity and location. Other monitoring approaches will be required for the landscape-scale predictions that will be needed, for example, to inform life cycle models.


Milestones and end dates were clearly defined and apparently on track, given the changes to a single contractor.

2. Methods


For the first two questions, the MBACI vs. Extensive Post Treatment (EPT) designs are well explained and the analyses generally were appropriate for the scale of investigation. Several critical aspects of the methods, however, are not explained in adequate detail.


Though characteristics of control and treatment sites are described in the supporting documentation, it is unclear what design was used to select sites for either the barrier removal or LWD studies. An ideal design involves creating a database of all possible sites, then selecting a subset for sampling by some stratified random method (e.g., Generalized Random-Tessellation Stratified design, GRTS). Instead, the reader is left with the impression that the sites were selected haphazardly and geographically clumped (e.g., barrier removal projects cluster near Seattle and in northern Idaho). If so, does this sampling design provide strong inference for extrapolating across basins or regions?

Additionally, it was not clear how controls are selected. Does the AEM project identify many sites with similar geomorphic features as treatments and then randomize? Some of this is described in the annual reports, but a synopsis in the Summary Report is needed. It may be possible to improve the EPT design by using multiple control reaches at each site. This would provide information on reach-to-reach variation and may improve statistical power, which could be quantified.


The Summary Report does not indicate whether bankfull flows or greater have occurred in restored sites. It is important that sites have successfully experienced design flows when determining whether they passed fish.


Reach lengths for investigating physical or biological responses differed by type of restoration action. The full barrier removal sites are small streams and most of them lacked anadromous salmonids, which limits their benefits for the Fish and Wildlife Program. Will sampling of 50 m of habitat be sufficient to yield strong inference about potential differences after barrier removal? Paired sites had similar habitat and fish abundance (see Clark et al. MS in supporting documents), but numbers of pools were few in all cases, and even fish abundance was often low. For example, only about a third of the sites had more than 20 fish of all species in either the treatment or control sites, suggesting that comparisons may suffer from low statistical power.

Results for the two types of habitat restoration assessed relied on different methods of estimating abundance, ranging from three-pass removal estimates, which account for capture probability, to snorkel counts, which do not. As a result, these measure only relative abundance with unknown confidence. Future reports and work should address how estimates of relative abundance can be calibrated to estimate error and bias. In the best case, future work would employ methods to estimate capture probabilities.

Table 1 in the barrier report (document P149035) indicates that three-pass electrofishing and snorkel survey data are mixed, which may be valid in a paired design. However, more worrisome is that the raw counts range from < 10 to close to 100. Differences then cannot come from a single distribution and the paired-t-test on the raw counts is not appropriate. Similarly, were pairs with no fish deleted prior to analysis? It would be better to analyze log(ratio) or use Poisson ANOVA to deal with these issues. Only p-values but no estimates of effect sizes are reported (with standard error/confidence interval). The 2017 report describing analysis of wood restoration (document P158833) includes an analysis of log(ratio) values. Estimates and confidence intervals are presented along with p-values. This type of analysis should be used for the barrier removal analysis.


The proponents report that their measurements of summer abundances of juvenile anadromous salmonids are consistent with estimates of total smolt production, based on Ogston et al. (2015).1 However, limiting factors that occur during winter may have stronger effects on ultimate smolt production than summer habitat. More detail on this relationship in final reports and publications would be highly useful for managers and researchers.

Chelgren and Dunham (2015)2 provide a sophisticated analysis of the abundance of fish species after barriers are removed for coastal Oregon streams. Their method accounts for probabilities of capture and detection for multiple species. This might be useful as a reference and when considering next steps but was not cited in the Clark et al. manuscript.


The report indicates that evaluation of fencing (i.e., livestock exclusion) was dropped because of inadequate sample size. This needs better explanation because bank stabilization and fencing are two of the most common river restoration actions. Why can the project not find more sites to create a more powerful experimental design? The summary report should describe the cause for the low number of potential study sites and explain why the restoration action should be dropped from their project.

Question 3 (why are some projects more successful?) is open ended. The approach for identifying the range of factors that can affect project success is not thoroughly described either in the report or the background documents. The wide array of restoration methods, site conditions, fish assemblages, timing of implementation, geomorphic and hydrologic histories, and land uses present challenges for unbiased analysis of factors responsible for success of restoration. For example, in the project’s analysis of the success of wood restoration practices, they would need to assess important variables such as size of material, whether anchored or not, design and intensity/density of placement, or location of placement. The proponents should provide additional information on how they will frame their assessments of factors responsible for differences in success between projects and the specific limits of their inferences or conclusions for practitioners.


Question 4 focuses on the ESU scale for assessing restoration project success, but it lacks a clear description of how the spatial scales of the ESU will be addressed by the research design. The proponents need to address how their sampling design across space can be used to extrapolate results to watershed and basin scales, and what caveats may be necessary about this inference if no statistical sampling design was used to select sites. The 2013 ISRP review (ISRP 2013-2) recommends stratification of sites by stream size or discharge. Presentation of the results and conclusions will require clear description of the scope of inference for the studies and the spatial scales for which the results are applicable.

3. Results


The report presented results for the studies of barrier removal and large wood restoration projects. Studies of other restoration projects are ongoing and will be reported in the future. The researchers encountered challenges finding sites that met the requirements of the MBACI designs, which may require future modification of the study.


The proponents have made solid progress in evaluating the overall effectiveness of several basic categories of restoration actions. One of the strengths of the AEM Project is its publication of literature reviews, project results, and research protocols. The research both provides documentation of the scientific background and specifics of the research design, but it also informs many other participants in the Fish and Wildlife Program and associated state programs within the Columbia River Basin.

While the objectives generally are clear, the project is incomplete. The EPT evaluations of barrier removal and LWD are complete, but the MBACI portion of the project appears to have had implementation issues and was re-contracted. Results will not be available for a few more years. Assuming no further problems, this should be feasible.


As noted above, the project has completed initial sampling and analysis of two restoration types: fish passage restoration at full barriers (32 paired sites) and LWD placement (29 paired sites). The results are useful as a first step in addressing the management question of whether removing barriers to movement or adding large wood has any detectable effects at the reach scale on summer daytime abundance of juveniles of anadromous salmonids. Whether this translates into higher survival and more smolts produced per redd is the next question, requiring much more work to assess.


Results of the barrier removal study includes many zeros (Table 3), and presumably these values are removed prior to analysis, though this is not stated in the methods. This leaves a single site for comparing coho abundance, and consequently there is little information on the impacts on this species. Only four sites can be used for analysis of Chinook salmon, one of which had a single fish. Some of the values from Table 1 in the detailed analysis of the barrier removal project differ from Table 3 in the summary. The report concludes that there are "no significant differences," but this is a meaningless statement. With such small sample sizes, only large effects could be detected. Proponents should clarify how zero abundances were addressed in the analysis and present effect sizes and measures of uncertainty (SE/CI).


The third and fourth questions were not addressed or discussed. The report on LWD (Clark and Roni 2017, AEM 2017 Annual Report) briefly mentioned a result that addressed the fourth question (i.e., p. 16-17; "Fish response did not differ by ESU for any species [ANOVA; p > 0.20]").

4. 2017 Research Plan uncertainties validation


The proponents’ statement clearly identified linkages to critical uncertainties in the 2017 Research Plan. In several cases, the linkages are valid, but the contributions of the restoration actions may be relatively small. For example, it is difficult to identify a physical process that would cause restoration of large wood to cool streams.

_____________________________

1 Ogston L, S Gidora, M Foy, and J Rosenfeld. 2015. Watershed-scale effectiveness of floodplain habitat restoration for juvenile coho salmon in the Chilliwack River, British Columbia. Canadian Journal of Fisheries and Aquatic Sciences 72(4):479-490.
2 Chelgren ND and JB Dunham. 2015. Barriers to passage of aquatic organisms at stream road crossings are a major cause of habitat fragmentation in stream networks. Ecological Applications 25:1357-1372.

Qualification #1 - Additional Information Needed
The AEM project uses reach-level comparisons to evaluate how habitat and fish populations respond, in general, to restoration actions grouped into categories. The proponents have integrated two separate projects by third-party contractors into a single project. Several components of the originally proposed research have been completed or modified, but several subprojects are still being adjusted. More information is needed on how the Multiple-Before-After-Control-Impact (MBACI) sites will be refocused. Also, the proponents need to provide information on the Adaptive Management strategy for identifying and selecting alternative approaches for modified subprojects as they complete the remaining research. Additional information is needed on how estimates of fish abundance based on different sampling methods can be calibrated to estimate error and bias, and possible future methods to estimate capture probabilities. The proponents need to provide a more specific implementation plan, analytical framework, and explicit description of the nature of inferences that will be possible for the last two questions that AEM is designed to answer (see Objectives below). They also need to address how their sampling design across space can be used to extrapolate results to basin scales. The proponents need to provide information on a public outreach strategy and the mechanics of how findings are being communicated to managers.
Documentation Links:
Review: 2019-2021 Mainstem/Program Support

Independent Scientific Review Panel Assessment

Assessment Number: 2016-001-00-ISRP-20190404
Project: 2016-001-00 - BPA Project Action Effectiveness Monitoring (AEM) Programmatic
Review: 2019-2021 Mainstem/Program Support
Proposal Number: NPCC19-2016-001-00
Completed Date: None
First Round ISRP Date: 4/4/2019
First Round ISRP Rating: Response Requested
First Round ISRP Comment:

Response requested comment:

The ISRP requests responses to the following issues with the understanding that some of the issues may take more time to fully address than provided in the month-long response loop (e.g., issues 4 and 5). In those cases, the response should discuss an approach to address the issues in the near future.

1.      Explain the process for selecting the additional 39 sites for the riparian planting study to be sampled in 2019. Develop the framework for a randomized, stratified selection of study sites that accounts for major factors anticipated to influence restoration success (e.g., stand age, site history and land use). What are the final metrics for project success?

2.      Explain the process for selecting additional sites for the floodplain restoration study. Clarify the range of practices included in the floodplain restoration projects category (e.g., levee removal or setback, channel reconnections, floodplain elevation modification, large wood placement, vegetation planting, hyporheic reconnection). What is the primary set of actions required before a site can be considered for inclusion in the floodplain restoration study? How will the analyses adjust for the variability in types of practices included?

3.      Explain why the AEM project will complete the study of partial barrier removals with only three sites. The three sites will not address the full range of treatment types (road removal, open bottom arch, bridge) or provide the needed statistical power to draw strong conclusions. Why would the three case studies warrant an additional four years of field measurements?

4.      The effect sizes measured are potentially biased by (a) the original practitioners selecting the highest-quality sites for restoration and (b) the proponents selecting the most tractable sites and controls for comparison. Each of these could result in the benefits of restoration being overestimated. How can these potential biases be addressed with the data that are available or could be collected?

5.      Explain the types of restoration actions that are included and excluded in each of the five types of restoration actions being studied. Discuss how differences in watershed conditions (fully functional, intermediate, severely impaired), which can alter the observed responses to restoration actions, are addressed in the analysis. Are the types of watershed conditions evenly distributed across the projects or can the analyses stratify by these covariates? Explain how the criteria for reference and treatment sites potentially influences the effect sizes observed. In general, sites with best M&E are accompanied by the best design. Does the elimination of sites with more variable or mixed site conditions and practices increase effect size?

6.      Develop an informative regional context for understanding the strengths and limitations of the AEM studies and for effectively applying findings. Describe both the appropriate regional applications and the limitations of the AEM findings. How will basinwide impacts be analyzed? Will the regional analysis be able to measure the cumulative impacts of several projects on a stream (e.g., 20 projects vs 200 projects)? Explain biases inherent in the sizes of streams and fish communities represented within the regions, the lack of study of responses in winter, the way practitioners chose sites for restoration, and the effect of excluding sites where commonly used restoration practices did not meet implementation criteria. Address variations in treatment design and implementation important in influencing project success. Also, discuss the potential effects of combinations of restoration treatments in a given area (riparian planting, floodplain restoration, LWD placement) on the performance of an individual treatment type. Discuss lessons learned for both restoration and monitoring from the types of sites excluded from the study as well as those that were included. Provide a context to assist practitioners and decision makers to implement more effective restoration practices and identify gaps in our understanding for future study.

Comment:

The AEM Project is well organized and focused on assessment of the effectiveness of reach and local-scale restoration actions. The proponents are on schedule for completing their deliverables. The AEM Project provided thorough and thoughtful responses to the Research Review in 2018. They have published their protocols and research results and are working to increase their outreach to on-the-ground practitioners in the region.

The project has an opportunity to adjust two study elements (e.g., riparian plantings and partial barrier removal) at this midpoint. ISRP review prior to additional field work would provide beneficial coordination (see Issues 1-3).

The AEM Project is likely to produce a product that meets their first goal: (1) what are the effects of different action categories on fish and habitat at the reach scale? They will have difficulty with the second and third goals: (2) why some projects within an action type are more effective than others and (3) whether there are differences in project effectiveness among regions (see Issues 4 and 5).

The AEM Project faced many challenges in sites it inherited from other programs and sites it selected. The range of stream sizes and characteristics, river network locations, aquatic community composition, terrestrial plant communities, geology, land forms, land uses, land types, designs, and intensities of restoration creates biases inherent in any study of this spatial extent within the constraints of funding and logistics. The sampling design leaves open the possibility that the results are biased, to an unknown extent, leading to either a false sense of security that effects are more positive than is true, or a false sense that there are not effects. The leaders of the AEM Project have extensive experience in these types of analyses and are well qualified to develop a framework for interpreting and applying their results, but caution is needed in communicating about the results to practitioners who may not understand potential biases and limitations in the study design and datasets.

1. Objectives, Significance to Regional Programs, and Technical Background

The proponents lay out timelines and clear objectives to measure (a) the effects of five types of habitat restoration on habitat and fish at the local (reach scale), (b) to determine why some projects are more effective than others, and (c) to measure whether these effects are different across regions (ESUs of salmonids in the Columbia River Basin). The proposal includes an update of quantitative and qualitative results and describes the time frames for completion of planned activities. It would be beneficial to have guiding hypotheses for individual components of the AEM analyses.

The measured fish and habitat responses are local (site and reach scale) and are not designed to answer the global question of the long-term watershed-scale impacts of improved habitat, such as possible bottlenecks later in the life-cycle.

The proponents have the technical training to conduct the analyses. The ISRP would like to learn more about efforts being made for the ongoing training and retention of staff. This would benefit the ISRP in understanding a potential widespread issue in the Basin.

2. Results and Adaptive Management

Results to date are well articulated. The ISRP looks forward to seeing the complete set of results and analyses when the project is finished. Lessons learned are summarized in the proposal and in an impressive number of publications. However, the potential to guide future habitat improvement and restoration efforts is hampered by several factors that limit the ability to generalize the results so that they can be applied in other locations.

For example, the results presented show that anadromous salmon ascended upstream of barriers after they were treated. However, the effect sizes measured are not likely to be accurate because the sites were not sampled with a statistically valid randomized sampling design.

There are similar limitations for the estimates of effects caused by large woody debris (LWD) placements. They may be less than the project predicts because sites deemed unsuitable are less likely to be sampled than when a rigorous random (or stratified random) sampling design is used. Other studies in the region have found that LWD was not as effective as anticipated, possibly reflecting the influence of the study site selection on effect size.

An additional bias that is likely to result from EPT designs is that LWD placement is more likely at sites where effects will be greater and less likely at the sites chosen for controls, making the results more favorable. This is avoided in a true experiment where treatments are assigned randomly after pairs of sites are chosen.

Given these potential biases, the results are most appropriately viewed as providing evidence for what CAN happen under best-case scenarios of habitat restoration, but not what WILL happen, on average, when sites are subjected to these habitat actions. As such, simulations that roll up these results to the basin-scale are likely too optimistic.

The lack of detail describing initial conditions limits the use of the results for guiding the planning and implementation of future restoration work. Examples of additional detail include watershed and stream settings, past disturbance history at the sample sites (flood, wind and fire) and design and treatment details for restoration treatments at sampled projects. Given the relatively small sample size in each treatment category, this detail could be provided without a great deal of additional effort.

No formal adaptive management strategy is provided (a request from past ISRP reviews). There is, however, a lengthy discussion of various lessons learned and resulting project and activity adjustments that resulted. The proponents have shared their results with the scientific community and recognized the need to share the results with non-academic managers in the Basin. The ISRP agrees this type of outreach is an important use of resources for this project.

The ISRP strongly agrees (see p. 20) that "limiting factors that occur during winter may have stronger effects on ultimate smolt production than summer habitat. More detail on this relationship in final reports and publications would be highly useful for managers and researchers." Perhaps this can be highlighted in reports to the Council/BPA so that winter monitoring and restoration actions can be prioritized for future activities.

We reviewed the AEM website but found it hard to figure out how to access the AEM data. Are the primary and summary data available online? If not, why not? The ISRP believes that these data should be freely available to the public.

3. Methods: Project Relationships, Work Types, and Deliverables

The mBACI and EPT designs are appropriate analyses for these projects. The analysis methods are well described. However, given the range of constraints, the results can be best used as a case example of the results for a specific set of sites but cannot be used to generalize results across sites in the Columbia River Basin as managers are expecting. For example, the results for LWD placement likely represent a best-case scenario, owing to unknown biases of (a) practitioners selecting the best sites in which to do the work and (b) the proponents selecting the most tractable sites for measuring the results.

Several questions need to be addressed, including: Can (or should) estimates of the benefits of these types of restoration be extrapolated to estimate basinwide benefit? Can the authors suggest a wide-scale monitoring plan to evaluate effectiveness at larger scales? What type of data would be needed to accomplish this?

The riparian planting study is not complete, and 39 more sites will be measured in 2019. Preliminary results indicate that shrub abundances increase but not tree abundances. The results did not include important covariates for the projects, such as the ages of the stocks used for riparian plantings nor the planting techniques, so it is difficult to discern whether a response in tree cover would have been expected. Results will be more informative if the relevant context of other factors that influence restoration success is analyzed and provided.

Analysis of partial barrier removal was initiated by another project, and AEM took over the study in 2018. Only nine sites were identified originally, and six were dropped because of lack of implementation. The project plans to continue monitoring the three sites through 2022. It is not clear why this study is continuing when the number of sites is so small. This number of sites cannot even include the number of basic treatment types (crossing removal, open bottom arch, bridge, and multi-plate culvert) most often used in passage restoration. This assessment of three case studies does not seem consistent with the statistical approach of the AEM Program.

Similar and additional challenges are associated with the study of floodplain restoration, which was taken over by AEM in 2018. Only six sites meet the study criteria. In this case, AEM is going to supplement the analysis with 20 sites using EPT analysis by 2022. However, floodplain restoration activities are not defined. Typically, this treatment category includes a number of treatments including vegetation planting, increasing floodplain roughness, re-accessing of side channels and off channel ponds/wetlands, levee removal or setback, etc. Currently there does not appear to be a clear description of the treatments that are included in this category.

Bias in the responses to restoration is likely at several levels:

Level 1: Practitioners select sites at which to remove barriers, install LWD, or enhance floodplains, which are not a random subset of all such sites. For example, sites selected to install LWD are likely to be among the most suitable sites for such treatments with watersheds and stream segments, not average sites. Given this, the control sites selected post-hoc in and EPT design are not likely to be similar to the treatment sites originally chosen and may have lower fish abundance pre-treatment. If so, the estimated effect of the treatment will be overestimated. Randomization of treatment locations is required to estimate this effect.

Level 2: From the set of sites that practitioners treated, the proponents selected a subset of sites to measure effects. Sites were rejected where adequate data did not exist.Given this, the effect of habitat treatments could be overestimated so that the results represent the best-case scenario rather than the average effects that managers are interested in knowing. A statistically valid randomized (or stratified random) sampling design is needed to avoid bias.

A related problem is that a statistically valid sampling design was not used to account for different regions (strata), and sample sizes of several types of habitat restoration were small. This means that the question of whether effects differ by region or ESU cannot be answered with confidence.

Given these potential biases, the ISRP requests that the project proponents carefully consider how they can be addressed with the data that are available or could be collected.

Documentation Links:

Project Relationships: This project Merged From 2012-011-00 effective on 10/1/2015
Relationship Description: 2016-001-00 is merged from 2011-008-00 and 2012-011-00.

This project Merged From 2011-008-00 effective on 10/1/2015
Relationship Description: 2016-001-00 is merged from 2011-008-00 and 2012-011-00.


Name Role Organization
David Kaplowe Project Manager Bonneville Power Administration
Phil Roni Project Lead Cramer Fish Sciences