The sponsors have outlined their objectives more clearly in the response than in the proposal, especially those for assessing effects of Libby Dam. They have also demonstrated that they are conducting ongoing data analysis, and improving methods for analysis in many cases.

Additional suggestions that may assist in optimizing studies for the Libby Dam/Kootenai River include:

1. The revised objectives make the proposal much clearer. Setting up clear hypotheses and alternatives is an important step. However, is there no expectation about the direction of these? The statement of non-directional alternative hypotheses indicates that little is known about the consequences of actions proposed. However, with the benefit of past data and experience, it should be possible to formulate more specific hypotheses for more powerful tests.

In addition, the sponsors should become versed on the current thinking about the drawbacks of hypothesis testing vs. model selection, so they can couch their results in appropriate terms. A priori hypotheses are a good thing, but how they are tested and reported are important for acceptance by the scientific communities.

2. Will production of phytoplankton and zooplankton be estimated, or simply density or volume? 

3. It appears that the sponsors will fit models of Didymo thickness and other metrics as a function of various covariates such as phosphorus. It would be informative to compare these models and covariates using the methods developed for "model selection and multi-model inference" by Burnham and Anderson (2002). These methods are believed by many to be superior to strict hypothesis testing, and traditional methods of developing one reduced model from the global model by deleting variables that are not significant. We urge the sponsors to look into these newer methods.

4. Under Objective 11, it was unclear for what biological group the richness and evenness would be calculated.

5. The ISRP appreciates the details presented for the early attempts of analysis of survival for bull trout, sturgeon, and rainbow trout using MARK. Although past data often may be too sparse for robust estimates, this design and analysis method should prove useful for future work, and analysis of some past data. Again, we urge the sponsors to approach experts with MARK if questions arise. They are often glad to help.

6. As for depletion estimates of bull trout abundance, the sponsors argue that the software that Montana uses to analyze data for population estimates has only certain options available. Nevertheless, working with an expert like Dr. Paul Lukacs at the University of Montana would allow using even newer methods that can combine data across bull trout samples in various tributaries or regions of Montana to estimate more robust capture probabilities. Likewise, new estimators allow including length as a continuous covariate instead of forcing separate estimates by size classes. This would improve estimates and confidence intervals for this and other projects.

7. As to topography of stream mouths, the hypothesis presented seems rather uninformative, especially if the goal is to determine fish passage upstream or downstream. Would it be possible to determine a measure of minimum habitat needed for fish passage (thalweg depth?) and to compare this through time, in addition to overall topography?

8. It is not entirely clear why evaluations of condition, growth, CPUE, and contributions of hatchery and wild sturgeon for fish captured in Montana will be compared to fish collected in Idaho and British Columbia. How will this information be used to inform management decisions?

9. More details for the adaptive management plan should be provided for contracting and Council consideration. For example, what are the implications for decisions about dam operations resulting from various outcomes of the project? Who will be involved in making those decisions and who will have final authority? How will dam operation decisions influence habitat restoration and fisheries management planning? What are the consequences to other aspects of the project if Didymo control efforts fail?

10. The sponsors should provide evidence of a strong commitment to sharing successes and lessons learned with others in the region and beyond through workshops, meetings, agency reports, and peer reviewed scientific publications. The peer-reviewed publications provide the opportunity to learn from other scientists who review the work, as well as to create a permanent record of the research and management decisions that is easily accessible in other places and future times.

ISRP Retrospective Evaluation of Results

This project has been ongoing for six years, during which the flow regime has been changing in the Kootenai River but has been more stable in the Flathead River. Partly as a result of recent changes, project results are further advanced for the Flathead River system than the Kootenai. Key questions about effects of ramping rates and reservoir levels on bull trout habitat use and benthic invertebrates have been answered in the Flathead River, although further questions remain about potential effects on native mountain whitefish and an invasive diatom (Didymo). These are the subject of ongoing study. In contrast, data on the effects of the new regime in the Kootenai River have been collected for several years since the flow regime change in 2008, but it will take more years before sufficient data are available to compare to previous regimes. Five years of data after the regime change would be a minimum to allow useful analysis. Improved designs are being developed for analysis of these before-after comparisons to capitalize on the substantial data being collected in the Kootenai River.

 

The research in the Kootenai River needs more effective integration through time. The ISRP recommend that the project sponsors develop an integrated conceptual model of the system, which would facilitate identification of key hypotheses to be tested and the presenting and publishing of results when these questions are answered. The ISRP will be looking for this in the next project review.

Likewise, the ISRP will be looking for dissemination of new knowledge and insights in the form of presentations at scientific or regional management meetings; workshops held to present information on problems common to these and other similar regulated rivers; and especially in the form of peer-reviewed publications to make a permanent record of the findings.

This project will evaluate the changes in drawdown limits and ramping rates at Libby and Hungry Horse dams, the two dams in Montana where flows can be altered to potentially benefit resident native fishes including those with fluvial and adfluvial life histories. There is a long history of dam operation changes, with current changes at Libby Dam coming online in 2008.

The proposal makes clear that dam operations affect the entire river ecosystem, with effects cascading downstream and upstream, partly through the altered movements and migrations of fishes. For the Kootenai River, the proposal describes that large amount of data collected both before and after the 2008 operation changes on hydrology, geomorphology of tributary deltas, benthic algae, and fish populations. For the Flathead River, where flow regime changes and research of them have been ongoing longer, the proposal describes evaluation of regime effects on habitat for juvenile bull trout, PIT tagging of bull trout in North Fork tributaries, and recent work on movement of mountain whitefish.

Overall, there were several deficiencies with the proposal itself, and several objectives and deliverables that were not well justified. First, we list deficiencies with the proposal which should be corrected in further proposals, followed by identification of responses requested to address problems with objectives and deliverables.

A. Deficiencies with the proposal:

1. Objectives were not well organized into key components to be measured. For the Kootenai River, Objective 1 was too broad, and aimed at monitoring many physical and biotic components from river flows to fish survival, whereas 2 through 5 were much more specific. In the future, objectives could be divided into more logical key components, such as 1) physical changes to reservoirs and flow regimes, 2) changes in nutrients, periphyton, and invertebrates, 3) demographics and viability of listed fish species, 4) population analysis of other native and non-native species, and 5) potential for invasions by non-native fishes.

2. Within objectives, deliverables should focus on key questions phrased as hypotheses. Examples might include: 1) Is survival rate and population growth rate (lambda) of listed bull trout increasing, stable, or decreasing? And 2) are brook and brown trout invading above Kootenai Falls? Focused questions to be addressed were a key component missing in the proposal for the Kootenai River work.

3. Sections on accomplishments need much greater synthesis for the work on the Kootenai River. This section was long and lacked the critical analysis and synthesis needed. No questions were presented, and no statistical analysis was completed or presented. Although the ISRP understands that these data are recent, and often too few data have been collected since the flow regime change in 2008 to analyze appropriately, the work needs to provide useful summaries rather than so much relatively detailed data. Some parts, such as the figures of aggradation at delta mouths, were unreadable, and so unusable. Not all reviewers will be able to delve into the annual reports.

B. Deficiencies with objectives and deliverables that require response in this cycle:

1. Appropriate study designs were lacking for much of the work on the Kootenai River, although baseline data collected provide useful information to develop such designs. Given this background, future work needs to focus on specific questions, rather than attempting to address all aspects of the changes caused by the flow regimes. Specific points raised by the ISRP that need to be addressed in this cycle include:

a. Bull trout survival and population growth – in both river systems, much effort is being expended to capture and mark bull trout with PIT tags, but no information was given about how these data would be analyzed to yield robust estimates of survival or rates of population change (lambda). These are two key vital rates that must be evaluated to make management decisions. Program MARK (see website of Dr. Gary White at Colorado State University) provides a flexible method for analyzing these data to estimate these rates. It offers much better power at detecting differences between time periods or locations than, for example, the analysis described in the annual report for the Kootenai River (Sylvester and Stephens 2010, FY10 Annual Report). Examples of such integrated MARK analyses for mark-recapture data on fishes in major river systems are available in Bestgen et al. (2007) and Zelasko et al. (2010), both in Transactions of the American Fisheries Society. Expertise to conduct such analyses can be gained by consulting with experts such as Drs. Paul Lukacs (U of MT), Gary White or Kevin Bestgen (Colorado State U), and Paul Conn or Brett McClintock (NOAA Fisheries, Seattle), and/or attending one of the periodic workshops on MARK (after some self-study, the intermediate-level workshop might be best). Given the large effort and excellent datasets being collected, appropriate design and analysis are a key component missing from the proposed work. Without them, much effort and funding could be wasted. With them, the work could provide landmark information to help fuel sound management decisions.

Response requested: Provide information about how data will be analyzed to yield robust estimates of survival or rates of population change for bull trout.

b. Similar data will be collected for white sturgeon in the Kootenai River, and can profit from a similar design, although the ISRP recognizes that recaptures may be too few for robust analysis. Nevertheless, without a suitable design, it may be difficult to prove that survival is indeed low, or to place confidence intervals around this, or test the evidence for it.

Response requested: Provide information about how data will be analyzed to yield robust estimates of survival or rates of population change for white sturgeon.

c. Problems were reported about estimating survival of rainbow trout, based on concerns about inaccurate ageing using scales. The method proposed for estimating survival in the FY 2010 Annual Report is akin to using catch curves, which is based on many assumptions. It seems like it would be much more efficient to simply PIT tag rainbow trout, avoiding the ageing issues altogether, and estimate survival directly using MARK. This could be done within size classes of interest. Given this, some deliverables need to be modified or dropped.

Response requested: Provide an evaluation of using PIT tagged rainbow trout to estimate survival within size classes. How would using this approach affect currently proposed deliverables?                                                                       

d. Population estimation for salmonids can also be improved markedly with new methods in MARK (see Saunders et al. 2011 TAFS for an example), which can integrate analysis across all size classes and even among different locations or sites. This improves power greatly by estimating capture probabilities for a much larger sample of fishes, with a continuous covariate for fish length (Huggins model) and other covariates for sites or times. In contrast, the method proposed and commonly used of estimating abundance for several size classes of fish at each site separately produces estimates with wide confidence intervals because they are each based on small sample sizes.

Response requested: Provide an evaluation of using the new methods in MARK for population estimation.                                                                      

e. Questions about factors affecting Didymo invasions should be developed as alternative plausible models or hypotheses, which can be tested using model selection (see Burnham and Anderson 2002). This method allows evaluating the weight of evidence for alternative models, and estimating parameters using multi-model inference to more fully use the information gained. Expertise in these methods can also be gained from the people listed above. Moreover, there are several proposals from this region addressing the same topic, and all have substantial budgets, so the ISRP is interested for clarification on how the workload will be shared.

Response requested: Provide alternative models or hypotheses about factors affecting Didymo invasions and outline how current and future data could be used to evaluate the alternatives.                                                                      

f. The information on delta aggradation and potential blockage of bull trout migrations is very interesting and important, but no questions were developed, and no methods or statistical analyses discussed for evaluating these changes.

Response requested: Develop questions and identify methods of analysis to evaluate changes due to delta aggregation and potential blockage of bull trout migrations.                                                                      

g. Adaptive Management: Most material in the text is a small component of adaptive management. It is suggested that the sponsors develop an adaptive management process that is responsive to the resources as well as to the public and to management needs.

Response requested: Provide an Adaptive Management plan to describe a process that is responsive to the resource, public, and management needs.                                                                       

h. Emerging Limiting Factors: The sponsors seem to have a good understanding of most emerging factors. However, changes in winter icing conditions were not mentioned, and could become increasingly important with time.

Response requested: Identify how winter icing conditions would affect the proposed project objectives.

i. Data Management: What percentage of the total budget is dedicated to data management? How is data QA/QC addressed for specific projects activities? Please describe anticipated improvements in the data management system expected to occur over the next few years (e.g., cloud computing, new software, equipment upgrades).

Response requested: Provide a response addressing issues raised in item i.                                                                       

j. The sponsors need to improve their collective productivity by publishing peer-reviewed articles in professional journals.

Response requested: Describe plans to publish in peer-reviewed journals.                                                                       

2. On the Flathead River system, where work is apparently more advanced, several main points require response:

a. Analysis of bull trout survival and rates of population growth require a more robust approach using current capture-recapture analysis methods like those included in Program MARK. Please see point 1.A. above.

Response requested: Provide information about how data will be analyzed to yield robust estimates of survival or rates of population change for bull trout on the Flathead River system.

b. The accomplishments presented about whitefish tracking in the Flathead River were useful information, but no study plan of focused questions was presented for the future, and there was no indication about how these data will be analyzed to answer specific questions.

Response requested: Develop questions and identify methods of analysis to evaluate whitefish tracking data.                                                                      

c. Points 1.G., and 1.I. from above also need to be addressed for the Flathead River system.

Response requested: Provide an adaptive management plan to describe a process that is responsive to the resource, public, and management needs for the Flathead River system. Respond to the questions posed in item I above with respect to the Flathead River system.                                                                       

Retrospective Evaluation of Results: This project has been ongoing for six years, during which the flow regime has been changing in the Kootenai River, but has been more stable in the Flathead River. Partly as a result of recent changes, project results are farther advanced for the Flathead River system than the Kootenai. Key questions about effects of ramping rates and reservoir levels on bull trout habitat use and benthic invertebrates have been answered in the Flathead River, although further questions remain about potential effects on native mountain whitefish and an invasive diatom (Didymo). These are the subject of ongoing study. In contrast, data on the effects of the new regime in the Kootenai River have been collected for several years since the flow regime change in 2008, but it will take more years before sufficient data are available to compare to previous regimes. Five years of data after the regime change would be a minimum to allow useful analysis. Nevertheless, better designs are needed for analysis of these before-after comparisons to capitalize on the substantial data being collected in the Kootenai River.

Modified by Dal Marsters on 4/16/2012 10:54:15 PM.

Modified by Dal Marsters on 4/17/2012 12:41:03 PM.

This is a well-prepared proposal that rates high marks for all ISRP review criteria. The project is well justified and deserves continued funding.

The ISRP previously reviewed this proposal; see ISRP 2004-6, Second Review of Proposal to Evaluate the Biological Effects of the Council's Mainstem Amendments on the Fisheries Upstream and Downstream of Hungry Horse and Libby Dams; www.nwcouncil.org/library/isrp/isrp2004-6.htm.

In that report the ISRP recommended continued support for the project and had some suggestions to improve the project:

1. More explicitly plan the strategy for using the existing data and models with updated data and models, and
2. Identify key indicators of trends in biological responses for early judgments about the nature and magnitude of biological effects.

The ISRP is pleased to note that the project sponsors responded to these ISRP suggestions by revising Objectives 1 and 2 to more explicitly include the model simulations of reservoir trophic responses and river habitat availability to provide the most immediate comparisons for assessing the biological consequences of the Council's operation strategy per Mainstem Amendments. The proposal (Objective 5) also emphasizes incorporating benthic community productivity (recolonization rate) into the river models to help inform policy on dam operations designed to benefit the fishery in many river systems affected by hydropower operations. The radio telemetry study (Objective 7) is designed to test the null hypothesis that hourly and daily discharge variation does not influence fish movement. This will be another key indicator of a relatively quick time-sensitive biological response to changes in discharge within the Kootenai and Flathead rivers. They anticipate that a before-after and control comparison could be used as the experimental design to test the null hypothesis.

Other comments:

Project history: Extensive details were provided in this section and results indicate that the project appears to have achieved many of its objectives. However, we would have expected more for a project that has been continuing this long. A list of technical products and peer-reviewed papers produced would be helpful in this section (e.g. like the list of references given in the preceding rationale section describing the interactions with Dr. Taper's lab, but including the full citations).

Have the reservoir models been peer reviewed and published? We couldn't find anything other than a BPA Report, where they are cited.

Tasks (work elements) and methods: On page 34, the paragraph at the bottom states a null hypothesis that seems to be unrealistic. The statistical analysis seems inappropriate for the situation. Wouldn't the objective be more appropriately stated as measurement of the effects of discharge variation on behavior of fish? What would be an appropriate statistical test? The proposal says that distances moved would be the measurement used. Perhaps the initial observations might suggest that distance is not as important as location of movement - from where to where? This deserves further thought, particularly from the standpoint of developing recommendations for modification of discharge patterns. So what if the fish do move further? Would we want to do anything about that?

On page 41, there is a typographical error in the top line. The word "no" has been omitted from the statement about requirements for Objective 8.