1. Objectives

The ultimate goals of AEM are to 1) quantify localized reach-scale improvements in habitat and juvenile steelhead and Chinook salmon abundance resulting from restoration actions implemented in the Columbia River Basin and 2) guide future restoration efforts to ensure that BPA is investing in effective restoration and habitat improvement techniques. Specifically, AEM is designed to answer the following questions:

1. What is the effect of different action categories on habitat at the reach scale?
2. What is the effect of different action categories on fish and other biota at a reach scale?
3. Within an action category, why are some projects more successful than others in producing physical and biological improvements? For example, what is the relationship between the amount or intensity of habitat improvement and fish response?
4. Are there differences among geographic areas (ESUs) in the physical and biological success of action types?

The first two questions are achievable and testable at the reach scale, whereas the other two are more challenging, the last owing to the need for comparisons across regions. For the Summary Report, the proponents transformed the third question from a "why" question into a "what" question to make it more quantifiable. The fourth question on differences in the physical and biological success of various types of habitat restoration among ESUs is not addressed in their report.

The answers to the questions are important and represent a first step in a fuller understanding of how habitat restoration benefits focal fish species across spatial and temporal scales. For watershed scale restoration, a combination of treatments, both instream/floodplain and those occurring upslope (road decommissioning, silvicultural treatments) are typically applied for accomplishment of restoration objectives. Also, other approaches will likely be needed to better understand whether there are differences in habitat and population effects, within action categories, due to differences in project design, intensity and location. Other monitoring approaches will be required for the landscape-scale predictions that will be needed, for example, to inform life cycle models.

Milestones and end dates were clearly defined and apparently on track, given the changes to a single contractor.

2. Methods

For the first two questions, the MBACI vs. Extensive Post Treatment (EPT) designs are well explained and the analyses generally were appropriate for the scale of investigation. Several critical aspects of the methods, however, are not explained in adequate detail.

Though characteristics of control and treatment sites are described in the supporting documentation, it is unclear what design was used to select sites for either the barrier removal or LWD studies. An ideal design involves creating a database of all possible sites, then selecting a subset for sampling by some stratified random method (e.g., Generalized Random-Tessellation Stratified design, GRTS). Instead, the reader is left with the impression that the sites were selected haphazardly and geographically clumped (e.g., barrier removal projects cluster near Seattle and in northern Idaho). If so, does this sampling design provide strong inference for extrapolating across basins or regions?

Additionally, it was not clear how controls are selected. Does the AEM project identify many sites with similar geomorphic features as treatments and then randomize? Some of this is described in the annual reports, but a synopsis in the Summary Report is needed. It may be possible to improve the EPT design by using multiple control reaches at each site. This would provide information on reach-to-reach variation and may improve statistical power, which could be quantified.

The Summary Report does not indicate whether bankfull flows or greater have occurred in restored sites. It is important that sites have successfully experienced design flows when determining whether they passed fish.

Reach lengths for investigating physical or biological responses differed by type of restoration action. The full barrier removal sites are small streams and most of them lacked anadromous salmonids, which limits their benefits for the Fish and Wildlife Program. Will sampling of 50 m of habitat be sufficient to yield strong inference about potential differences after barrier removal? Paired sites had similar habitat and fish abundance (see Clark et al. MS in supporting documents), but numbers of pools were few in all cases, and even fish abundance was often low. For example, only about a third of the sites had more than 20 fish of all species in either the treatment or control sites, suggesting that comparisons may suffer from low statistical power.

Results for the two types of habitat restoration assessed relied on different methods of estimating abundance, ranging from three-pass removal estimates, which account for capture probability, to snorkel counts, which do not. As a result, these measure only relative abundance with unknown confidence. Future reports and work should address how estimates of relative abundance can be calibrated to estimate error and bias. In the best case, future work would employ methods to estimate capture probabilities.

Table 1 in the barrier report (document P149035) indicates that three-pass electrofishing and snorkel survey data are mixed, which may be valid in a paired design. However, more worrisome is that the raw counts range from < 10 to close to 100. Differences then cannot come from a single distribution and the paired-t-test on the raw counts is not appropriate. Similarly, were pairs with no fish deleted prior to analysis? It would be better to analyze log(ratio) or use Poisson ANOVA to deal with these issues. Only p-values but no estimates of effect sizes are reported (with standard error/confidence interval). The 2017 report describing analysis of wood restoration (document P158833) includes an analysis of log(ratio) values. Estimates and confidence intervals are presented along with p-values. This type of analysis should be used for the barrier removal analysis.

The proponents report that their measurements of summer abundances of juvenile anadromous salmonids are consistent with estimates of total smolt production, based on Ogston et al. (2015).¹ However, limiting factors that occur during winter may have stronger effects on ultimate smolt production than summer habitat. More detail on this relationship in final reports and publications would be highly useful for managers and researchers.

Chelgren and Dunham (2015)² provide a sophisticated analysis of the abundance of fish species after barriers are removed for coastal Oregon streams. Their method accounts for probabilities of capture and detection for multiple species. This might be useful as a reference and when considering next steps but was not cited in the Clark et al. manuscript.

The report indicates that evaluation of fencing (i.e., livestock exclusion) was dropped because of inadequate sample size. This needs better explanation because bank stabilization and fencing are two of the most common river restoration actions. Why can the project not find more sites to create a more powerful experimental design? The summary report should describe the cause for the low number of potential study sites and explain why the restoration action should be dropped from their project.

Question 3 (why are some projects more successful?) is open ended. The approach for identifying the range of factors that can affect project success is not thoroughly described either in the report or the background documents. The wide array of restoration methods, site conditions, fish assemblages, timing of implementation, geomorphic and hydrologic histories, and land uses present challenges for unbiased analysis of factors responsible for success of restoration. For example, in the project’s analysis of the success of wood restoration practices, they would need to assess important variables such as size of material, whether anchored or not, design and intensity/density of placement, or location of placement. The proponents should provide additional information on how they will frame their assessments of factors responsible for differences in success between projects and the specific limits of their inferences or conclusions for practitioners.

Question 4 focuses on the ESU scale for assessing restoration project success, but it lacks a clear description of how the spatial scales of the ESU will be addressed by the research design. The proponents need to address how their sampling design across space can be used to extrapolate results to watershed and basin scales, and what caveats may be necessary about this inference if no statistical sampling design was used to select sites. The 2013 ISRP review (ISRP 2013-2) recommends stratification of sites by stream size or discharge. Presentation of the results and conclusions will require clear description of the scope of inference for the studies and the spatial scales for which the results are applicable.

3. Results

The report presented results for the studies of barrier removal and large wood restoration projects. Studies of other restoration projects are ongoing and will be reported in the future. The researchers encountered challenges finding sites that met the requirements of the MBACI designs, which may require future modification of the study.

The proponents have made solid progress in evaluating the overall effectiveness of several basic categories of restoration actions. One of the strengths of the AEM Project is its publication of literature reviews, project results, and research protocols. The research both provides documentation of the scientific background and specifics of the research design, but it also informs many other participants in the Fish and Wildlife Program and associated state programs within the Columbia River Basin.

While the objectives generally are clear, the project is incomplete. The EPT evaluations of barrier removal and LWD are complete, but the MBACI portion of the project appears to have had implementation issues and was re-contracted. Results will not be available for a few more years. Assuming no further problems, this should be feasible.

As noted above, the project has completed initial sampling and analysis of two restoration types: fish passage restoration at full barriers (32 paired sites) and LWD placement (29 paired sites). The results are useful as a first step in addressing the management question of whether removing barriers to movement or adding large wood has any detectable effects at the reach scale on summer daytime abundance of juveniles of anadromous salmonids. Whether this translates into higher survival and more smolts produced per redd is the next question, requiring much more work to assess.

Results of the barrier removal study includes many zeros (Table 3), and presumably these values are removed prior to analysis, though this is not stated in the methods. This leaves a single site for comparing coho abundance, and consequently there is little information on the impacts on this species. Only four sites can be used for analysis of Chinook salmon, one of which had a single fish. Some of the values from Table 1 in the detailed analysis of the barrier removal project differ from Table 3 in the summary. The report concludes that there are "no significant differences," but this is a meaningless statement. With such small sample sizes, only large effects could be detected. Proponents should clarify how zero abundances were addressed in the analysis and present effect sizes and measures of uncertainty (SE/CI).

The third and fourth questions were not addressed or discussed. The report on LWD (Clark and Roni 2017, AEM 2017 Annual Report) briefly mentioned a result that addressed the fourth question (i.e., p. 16-17; "Fish response did not differ by ESU for any species [ANOVA; p > 0.20]").

4. 2017 Research Plan uncertainties validation

The proponents’ statement clearly identified linkages to critical uncertainties in the 2017 Research Plan. In several cases, the linkages are valid, but the contributions of the restoration actions may be relatively small. For example, it is difficult to identify a physical process that would cause restoration of large wood to cool streams.

_____________________________

¹ Ogston L, S Gidora, M Foy, and J Rosenfeld. 2015. Watershed-scale effectiveness of floodplain habitat restoration for juvenile coho salmon in the Chilliwack River, British Columbia. Canadian Journal of Fisheries and Aquatic Sciences 72(4):479-490.
² Chelgren ND and JB Dunham. 2015. Barriers to passage of aquatic organisms at stream road crossings are a major cause of habitat fragmentation in stream networks. Ecological Applications 25:1357-1372.

The AEM project uses reach-level comparisons to evaluate how habitat and fish populations respond, in general, to restoration actions grouped into categories. The proponents have integrated two separate projects by third-party contractors into a single project. Several components of the originally proposed research have been completed or modified, but several subprojects are still being adjusted. More information is needed on how the Multiple-Before-After-Control-Impact (MBACI) sites will be refocused. Also, the proponents need to provide information on the Adaptive Management strategy for identifying and selecting alternative approaches for modified subprojects as they complete the remaining research. Additional information is needed on how estimates of fish abundance based on different sampling methods can be calibrated to estimate error and bias, and possible future methods to estimate capture probabilities. The proponents need to provide a more specific implementation plan, analytical framework, and explicit description of the nature of inferences that will be possible for the last two questions that AEM is designed to answer (see Objectives below). They also need to address how their sampling design across space can be used to extrapolate results to basin scales. The proponents need to provide information on a public outreach strategy and the mechanics of how findings are being communicated to managers.

Response requested comment:

The ISRP requests responses to the following issues with the understanding that some of the issues may take more time to fully address than provided in the month-long response loop (e.g., issues 4 and 5). In those cases, the response should discuss an approach to address the issues in the near future.

1.      Explain the process for selecting the additional 39 sites for the riparian planting study to be sampled in 2019. Develop the framework for a randomized, stratified selection of study sites that accounts for major factors anticipated to influence restoration success (e.g., stand age, site history and land use). What are the final metrics for project success?

2.      Explain the process for selecting additional sites for the floodplain restoration study. Clarify the range of practices included in the floodplain restoration projects category (e.g., levee removal or setback, channel reconnections, floodplain elevation modification, large wood placement, vegetation planting, hyporheic reconnection). What is the primary set of actions required before a site can be considered for inclusion in the floodplain restoration study? How will the analyses adjust for the variability in types of practices included?

3.      Explain why the AEM project will complete the study of partial barrier removals with only three sites. The three sites will not address the full range of treatment types (road removal, open bottom arch, bridge) or provide the needed statistical power to draw strong conclusions. Why would the three case studies warrant an additional four years of field measurements?

4.      The effect sizes measured are potentially biased by (a) the original practitioners selecting the highest-quality sites for restoration and (b) the proponents selecting the most tractable sites and controls for comparison. Each of these could result in the benefits of restoration being overestimated. How can these potential biases be addressed with the data that are available or could be collected?

5.      Explain the types of restoration actions that are included and excluded in each of the five types of restoration actions being studied. Discuss how differences in watershed conditions (fully functional, intermediate, severely impaired), which can alter the observed responses to restoration actions, are addressed in the analysis. Are the types of watershed conditions evenly distributed across the projects or can the analyses stratify by these covariates? Explain how the criteria for reference and treatment sites potentially influences the effect sizes observed. In general, sites with best M&E are accompanied by the best design. Does the elimination of sites with more variable or mixed site conditions and practices increase effect size?

6.      Develop an informative regional context for understanding the strengths and limitations of the AEM studies and for effectively applying findings. Describe both the appropriate regional applications and the limitations of the AEM findings. How will basinwide impacts be analyzed? Will the regional analysis be able to measure the cumulative impacts of several projects on a stream (e.g., 20 projects vs 200 projects)? Explain biases inherent in the sizes of streams and fish communities represented within the regions, the lack of study of responses in winter, the way practitioners chose sites for restoration, and the effect of excluding sites where commonly used restoration practices did not meet implementation criteria. Address variations in treatment design and implementation important in influencing project success. Also, discuss the potential effects of combinations of restoration treatments in a given area (riparian planting, floodplain restoration, LWD placement) on the performance of an individual treatment type. Discuss lessons learned for both restoration and monitoring from the types of sites excluded from the study as well as those that were included. Provide a context to assist practitioners and decision makers to implement more effective restoration practices and identify gaps in our understanding for future study.

Comment:

The AEM Project is well organized and focused on assessment of the effectiveness of reach and local-scale restoration actions. The proponents are on schedule for completing their deliverables. The AEM Project provided thorough and thoughtful responses to the Research Review in 2018. They have published their protocols and research results and are working to increase their outreach to on-the-ground practitioners in the region.

The project has an opportunity to adjust two study elements (e.g., riparian plantings and partial barrier removal) at this midpoint. ISRP review prior to additional field work would provide beneficial coordination (see Issues 1-3).

The AEM Project is likely to produce a product that meets their first goal: (1) what are the effects of different action categories on fish and habitat at the reach scale? They will have difficulty with the second and third goals: (2) why some projects within an action type are more effective than others and (3) whether there are differences in project effectiveness among regions (see Issues 4 and 5).

The AEM Project faced many challenges in sites it inherited from other programs and sites it selected. The range of stream sizes and characteristics, river network locations, aquatic community composition, terrestrial plant communities, geology, land forms, land uses, land types, designs, and intensities of restoration creates biases inherent in any study of this spatial extent within the constraints of funding and logistics. The sampling design leaves open the possibility that the results are biased, to an unknown extent, leading to either a false sense of security that effects are more positive than is true, or a false sense that there are not effects. The leaders of the AEM Project have extensive experience in these types of analyses and are well qualified to develop a framework for interpreting and applying their results, but caution is needed in communicating about the results to practitioners who may not understand potential biases and limitations in the study design and datasets.

1. Objectives, Significance to Regional Programs, and Technical Background

The proponents lay out timelines and clear objectives to measure (a) the effects of five types of habitat restoration on habitat and fish at the local (reach scale), (b) to determine why some projects are more effective than others, and (c) to measure whether these effects are different across regions (ESUs of salmonids in the Columbia River Basin). The proposal includes an update of quantitative and qualitative results and describes the time frames for completion of planned activities. It would be beneficial to have guiding hypotheses for individual components of the AEM analyses.

The measured fish and habitat responses are local (site and reach scale) and are not designed to answer the global question of the long-term watershed-scale impacts of improved habitat, such as possible bottlenecks later in the life-cycle.

The proponents have the technical training to conduct the analyses. The ISRP would like to learn more about efforts being made for the ongoing training and retention of staff. This would benefit the ISRP in understanding a potential widespread issue in the Basin.

2. Results and Adaptive Management

Results to date are well articulated. The ISRP looks forward to seeing the complete set of results and analyses when the project is finished. Lessons learned are summarized in the proposal and in an impressive number of publications. However, the potential to guide future habitat improvement and restoration efforts is hampered by several factors that limit the ability to generalize the results so that they can be applied in other locations.

For example, the results presented show that anadromous salmon ascended upstream of barriers after they were treated. However, the effect sizes measured are not likely to be accurate because the sites were not sampled with a statistically valid randomized sampling design.

There are similar limitations for the estimates of effects caused by large woody debris (LWD) placements. They may be less than the project predicts because sites deemed unsuitable are less likely to be sampled than when a rigorous random (or stratified random) sampling design is used. Other studies in the region have found that LWD was not as effective as anticipated, possibly reflecting the influence of the study site selection on effect size.

An additional bias that is likely to result from EPT designs is that LWD placement is more likely at sites where effects will be greater and less likely at the sites chosen for controls, making the results more favorable. This is avoided in a true experiment where treatments are assigned randomly after pairs of sites are chosen.

Given these potential biases, the results are most appropriately viewed as providing evidence for what CAN happen under best-case scenarios of habitat restoration, but not what WILL happen, on average, when sites are subjected to these habitat actions. As such, simulations that roll up these results to the basin-scale are likely too optimistic.

The lack of detail describing initial conditions limits the use of the results for guiding the planning and implementation of future restoration work. Examples of additional detail include watershed and stream settings, past disturbance history at the sample sites (flood, wind and fire) and design and treatment details for restoration treatments at sampled projects. Given the relatively small sample size in each treatment category, this detail could be provided without a great deal of additional effort.

No formal adaptive management strategy is provided (a request from past ISRP reviews). There is, however, a lengthy discussion of various lessons learned and resulting project and activity adjustments that resulted. The proponents have shared their results with the scientific community and recognized the need to share the results with non-academic managers in the Basin. The ISRP agrees this type of outreach is an important use of resources for this project.

The ISRP strongly agrees (see p. 20) that "limiting factors that occur during winter may have stronger effects on ultimate smolt production than summer habitat. More detail on this relationship in final reports and publications would be highly useful for managers and researchers." Perhaps this can be highlighted in reports to the Council/BPA so that winter monitoring and restoration actions can be prioritized for future activities.

We reviewed the AEM website but found it hard to figure out how to access the AEM data. Are the primary and summary data available online? If not, why not? The ISRP believes that these data should be freely available to the public.

3. Methods: Project Relationships, Work Types, and Deliverables

The mBACI and EPT designs are appropriate analyses for these projects. The analysis methods are well described. However, given the range of constraints, the results can be best used as a case example of the results for a specific set of sites but cannot be used to generalize results across sites in the Columbia River Basin as managers are expecting. For example, the results for LWD placement likely represent a best-case scenario, owing to unknown biases of (a) practitioners selecting the best sites in which to do the work and (b) the proponents selecting the most tractable sites for measuring the results.

Several questions need to be addressed, including: Can (or should) estimates of the benefits of these types of restoration be extrapolated to estimate basinwide benefit? Can the authors suggest a wide-scale monitoring plan to evaluate effectiveness at larger scales? What type of data would be needed to accomplish this?

The riparian planting study is not complete, and 39 more sites will be measured in 2019. Preliminary results indicate that shrub abundances increase but not tree abundances. The results did not include important covariates for the projects, such as the ages of the stocks used for riparian plantings nor the planting techniques, so it is difficult to discern whether a response in tree cover would have been expected. Results will be more informative if the relevant context of other factors that influence restoration success is analyzed and provided.

Analysis of partial barrier removal was initiated by another project, and AEM took over the study in 2018. Only nine sites were identified originally, and six were dropped because of lack of implementation. The project plans to continue monitoring the three sites through 2022. It is not clear why this study is continuing when the number of sites is so small. This number of sites cannot even include the number of basic treatment types (crossing removal, open bottom arch, bridge, and multi-plate culvert) most often used in passage restoration. This assessment of three case studies does not seem consistent with the statistical approach of the AEM Program.

Similar and additional challenges are associated with the study of floodplain restoration, which was taken over by AEM in 2018. Only six sites meet the study criteria. In this case, AEM is going to supplement the analysis with 20 sites using EPT analysis by 2022. However, floodplain restoration activities are not defined. Typically, this treatment category includes a number of treatments including vegetation planting, increasing floodplain roughness, re-accessing of side channels and off channel ponds/wetlands, levee removal or setback, etc. Currently there does not appear to be a clear description of the treatments that are included in this category.

Bias in the responses to restoration is likely at several levels:

Level 1: Practitioners select sites at which to remove barriers, install LWD, or enhance floodplains, which are not a random subset of all such sites. For example, sites selected to install LWD are likely to be among the most suitable sites for such treatments with watersheds and stream segments, not average sites. Given this, the control sites selected post-hoc in and EPT design are not likely to be similar to the treatment sites originally chosen and may have lower fish abundance pre-treatment. If so, the estimated effect of the treatment will be overestimated. Randomization of treatment locations is required to estimate this effect.

Level 2: From the set of sites that practitioners treated, the proponents selected a subset of sites to measure effects. Sites were rejected where adequate data did not exist.Given this, the effect of habitat treatments could be overestimated so that the results represent the best-case scenario rather than the average effects that managers are interested in knowing. A statistically valid randomized (or stratified random) sampling design is needed to avoid bias.

A related problem is that a statistically valid sampling design was not used to account for different regions (strata), and sample sizes of several types of habitat restoration were small. This means that the question of whether effects differ by region or ESU cannot be answered with confidence.

Given these potential biases, the ISRP requests that the project proponents carefully consider how they can be addressed with the data that are available or could be collected.

The proponents have a good track record of addressing science review concerns and the ISRP appreciates their responses to our questions. The explanation of the relationship between this project and others in the region is very helpful. Comments below are intended to help the proponents and their partners as the project goes forward.

The project proponents have provided a more detailed explanation of the link between the project and ongoing effectiveness monitoring programs such as ISEMP/CHaMP and OBMEP. The goals of this work in relation to the objectives of Upper Columbia River (UCR) salmon recovery plans have been made clearer. The project is intended to fill information gaps that have been identified by the UCR Technical Team as being critical to calculation of VSP parameters for focal species, and will help determine why at-risk salmon and steelhead populations are continuing to decline in spite of the extensive regional investment in restoration.

The proponents clearly identify the critical gaps addressed in the proposed work as monitoring of all projects in the Upper Columbia for 1) implementation/compliance monitoring, and 2) reach-scale effectiveness monitoring. They propose adding an additional 36 sites in the Upper Columbia to the effectiveness monitoring being conducted, but further information about sampling sites would have made the proposal a little clearer. The ISRP appreciates that additional detail was provided with respect to treatment-control reaches for evaluating restoration effectiveness. The methods for locating sites (randomly selected, but stratified for restoration category) seem appropriate. One ISRP concern has to do with reach length. Study reaches of 150-500 meters may not be sufficient to detect changes in target species densities or other response metrics (reach lengths of 30 times the average channel width have often been recommended in fisheries work), so project proponents may wish to revisit the size criteria for some locations.

A more complete rationale for the work was presented in the Project Significance and Problem Statement, and the Emerging Limiting Factors sections. Descriptions of the response metrics and their importance were improved over the original proposal. The categories of restoration projects, which form the basis for stratification of study pairs, are also better described, although it was not clear how the 6 site pairs in Panel 2 will address all 7 Salmon Recovery Funding Board project categories.

The response was a little weak in addressing the ISRP's request for information about the status and trends in habitats and fish populations in areas where restoration is taking place. We were hoping for a few examples to illustrate where restoration has had, or has failed to produce, a measurable effect on target species, including some insight into why the monitoring was or was not successful. The history of restoration activity in the Upper Columbia is long enough that there should be some useful case studies.

The response states that data sharing between CHaMP, OBMEP, and this project will occur to "coordinate and share resources related to data management and QA/QC." It was not clear from the revision how this coordination would occur and who would act as the central clearinghouse for shared data. We guess it would be ISEMP, but that may not be the case.

The preliminary list of implementation/compliance metrics given in Table 4 is quite extensive and will need to be winnowed down to the most cost-effective and applicable ones. An attempt to select the most potentially useful compliance metrics should take place prior to initiation of post-project sampling. Likewise, the QA/QC guidelines should be completed prior to initiation of field measurements.

The response correctly points out that time constraints may limit some BACI-design evaluations to a single year of pre-treatment data. While this is not necessarily the most desirable scenario, the ISRP stresses the importance of completing at least one full year of pre-treatment sampling for those treatment/control pairs where a BACI study design will be used.

The ISRP feels that additional details are needed before it can complete a scientific assessment of this proposal, and for this reason we are asking for a response. The proposal should provide a clear description of relationships with other monitoring efforts in the Upper Columbia, including ISEMP/CHaMP and the large monitoring program planned for the Okanogan Subbasin (OBMEP). Monitoring is expensive and resources for conducting these projects are limited. Therefore, close coordination among related projects is critical to the efficient and effective use of monitoring funds. The description should go beyond box-and-arrow diagrams and explain how funding and manpower resources would be shared. The proposal needs to provide sufficient information about objectives, adaptive management, deliverables and work elements. More explanation could have been presented on the background and rationale for this project, as well as a summary of existing monitoring programs that have pointed to certain factors as being limiting to salmonid productivity. Most importantly, however, the proposal should provide more technical details about both the post-treatment implementation monitoring and the reach scale effectiveness monitoring. A brief list of metrics and citations is not enough; the ISRP needs to see a discussion of why certain metrics have been selected, information about sampling schedules, where monitoring sites will be located in relation to the restoration actions (at least in general terms), what types of sites will serve as controls, who will carry out the monitoring work, and how will the data be managed. Once this information is provided we will be able to assess the merits of the proposal. 1. Purpose, Significance to Regional Programs, Technical Background, and Objectives Valuable information could be produced by expanding current implementation and effectiveness monitoring efforts in the Upper Columbia to encompass all restoration projects (according to the presentation on September 13, 2010, many projects are not currently monitored). The proposal should include an expanded discussion of the threats to focal species in the Upper Columbia area and a summary of the major categories of habitat restoration that are taking place which the proposed monitoring would cover. It would also be helpful to summarize the existing implementation and effectiveness monitoring efforts currently in place, where they are located, what types of restoration projects they are monitoring, what the key findings to date have been especially with respect to limiting factors, and where significant knowledge or geographical gaps exist. More details should be given for the objectives. Most objectives are described in 1-2 sentences followed by an identical list of work elements (some of which are repeated multiple times) for each objective. This made it difficult to understand how the objectives were separated, other than by looking at the title. It was not clear in this section whether the proposal was primarily for post-implementation compliance monitoring or for reach scale effectiveness monitoring, although it became clear later in the proposal that the request was for both. 2. History: Accomplishments, Results, and Adaptive Management This project proposes to expand an existing set of monitoring sites. However, little information is provided on results that have been generated to date from the existing program. Some indication that the methodologies and design of the program have been effective would provide support for the plan to expand the number of sites monitored using this process. The project will expand the number of monitored sites to 30 control-impact pairs. A statement was made in the proposal that this number of sites was selected after conducting a power analysis, yet no information about this analysis is provided. As the ultimate value of the information generated by this project will be dependent on the statistical power of the experimental design, a reasonable description of this analysis should be included in the proposal. One must assume that this power analyses was based on monitoring results that have been generated from the existing study sites, further emphasizing the value of including project results in this proposal. Adaptive management is implied rather than directly addressed in the proposal. The statement is made that adaptive management in the Upper Columbia will be enhanced by the inclusion of more sites in the monitoring program, but no description of this adaptive management process is provided. There are several statements that monitoring data will be used by the Action Agencies’ expert panel process. Is this panel considered to be the core of the Upper Columbia adaptive management process? A more thorough explanation of how information generated by this project will be used to inform and change habitat restoration and fish management policies and practices in the Upper Columbia should be incorporated into the proposal. 3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (Hatchery, RME, Tagging) A thorough explanation of the relationship between this proposed effort and other monitoring programs in the Upper Columbia should be incorporated into this proposal. The proposal does indicate several other monitoring programs, including ISEMP, but fails to indicate how this program will augment efforts already underway. The proposal also fails to mention a large monitoring effort planned for the area under the CHaMP program. The Colville Tribe has proposed a very large monitoring program for the Okanogan (OBMEP), with many of the same objectives as this proposal. The Colville monitoring program and the one proposed here should be carefully coordinated. Cooperation among these various monitoring efforts is essential to ensure that information is collected efficiently and in a compatible format. The proposal authors make the statement that this project will not interfere with other monitoring programs in the Upper Columbia. Avoiding interference should not be the objective. Rather all these monitoring efforts should be coordinated such that they provide complimentary information. A thorough discussion of the linkages among the various monitoring efforts in the area, one that clearly illustrates the coordination among these programs, should be added to the proposal. The statement is made that, “Various efforts are underway to resolve some of the confounding issues, including climate change-related projects in the Okanogan and Methow sub-basins to develop better streamflow forecasting tools.” This is useful to know because extreme low flows likely constitute a significant environmental limitation to salmonid productivity in many of the watersheds. However, more details are needed on how potentially confounding factors such as climate change will be addressed in this monitoring program. The proposal should be more explicit in this regard. 4. Deliverables, Work Elements, Metrics, and Methods Deliverables basically were often described in a single sentence. More description of what these deliverables will include should be added. The work elements were sometimes unclear and duplicative for different objectives. They were presented as a list of variables (e.g., length, width, bankfull height, etc.) with an associated reference, which provided the methods to be used in measuring this variable, but the same reference was often listed multiple times for a given parameter. The description of these elements in the proposal does not provide enough information to conduct a technical review. Under post-implementation compliance monitoring, the list of metrics should be given, even if it is still preliminary. For each provisional metric, sampling schedules (including years) and sample sizes (if possible) should be stated, as well as who would do the work and where the monitoring data would be housed. An explicit connection between each metric and one or more limiting factors should be given so that the relevance of the metric is clear. How, specifically, will QA/QC issues related to the metrics be addressed? Under SFRB reach scale effectiveness monitoring, will the 30 treatment-control pairs, plus the 6 pairs already requested, be stratified according to restoration type (e.g., fish passage improvement, increased in-stream flows, physical habitat restoration, nutrient management)? How long will the pre-treatment sampling period be? It would be very helpful to give more details on the SFRB effectiveness monitoring protocols and why post-treatment sampling will occur in years 0, 1,3, 5, and 10 (as opposed to a different schedule). Fish density is the only biological metric mentioned in the proposal, but some restoration actions could affect other parameters (e.g., fish growth or migration). Will other demographic properties of focal species be estimated?