Response requested comment:

StreamNet is a long-standing project that has evolved into a key component of the Basin's anadromous fish management and reporting program. It has provided data management and dissemination for more than 20 years. Although there appears to be room for continued improvement and efficiency, the project is well organized and effective at meeting an increasing array of goods and services to its users. With continued focus on reducing expenditures for most projects and given the importance of the project to basinwide management and reporting, it appears careful review of future work plans and approaches will be needed to effectively address increasing demands for data management and dissemination.

The proponents are asked to respond to the following issues as well as additional issues identified in the review comments.

1.      The ISRP requests that the proponents develop an adaptive management (AM) plan, one that encompasses both internal and external AM. This plan was also requested in past ISRP project reviews. The ISRP notes that while an internal AM process is not described in the proposal, the proponents do provide internal goals and objectives. However, these are only part of an AM process. Please present the internal AM process for ISRP review.

2.      As described in the 2006 ISRP Review, "The project should have in place a system for monitoring and evaluating its performance. The program still needs to develop more in-depth measures of monitoring effectiveness and assess its impact in terms of user satisfaction. Use of the services should be documented, and more focus should be placed on outputs rather than inputs. A systematic way of evaluating effectiveness is needed. Who are the users? Were these users satisfied? Is tracking software used (e.g., Web Trends)? The sponsors should provide some evaluative performance information to address these questions."

The proponents still need to respond to a previous qualification from the 2012 ISRP review, specifically:

3.      That the proponents: "Provide a report describing in detail the data quality assurance and quality control (QA/QC) procedures used by StreamNet." If this is in another StreamNet publication, please provide the link. Otherwise, please provide a response detailing the QA/QC procedures.

Comment:

1. Objectives, Significance to Regional Programs, and Technical Background

There are three primary objectives listed in the 2019 proposal that address priority work items identified in the Five-Year Plan for Coordinated Assessments. These objectives are qualitative and do not lend themselves to tracking accomplishments with given timelines. The ISRP believes that the StreamNet project needs to establish quantitative objectives and timelines as well as interim milestones for meeting them.

The ISRP recognizes that the StreamNet Project is an important component among regional agencies and programs, especially its role in providing web-based, standardized, and comprehensive information for anadromous fish in the Basin. The need for regionally coordinated and readily accessible data has been consistently identified by the Northwest Power and Conservation Council (NPCC), the Bonneville Power Administration (BPA), and the National Oceanic and Atmospheric Administration Fisheries Program (NOAA). StreamNet began in 1995 and is now the primary data source for a number of agencies. A five-year Strategic Plan is in place and the current priority is identified as the Coordinated Assessments Project. StreamNet helps lead implementation of the Coordinated Assessments Project, in partnership with the Pacific Northwest Aquatic Monitoring Partnership (PNAMP).

The technical background of personnel involved appears to be strong.

2. Results and Adaptive Management

The project has been very effective in developing a data management and dissemination system focused primarily on anadromous fish. It provides consistent and transparent data-sharing among formal members as well as other cooperators and partners. Due to past and current work priorities and limited funding, StreamNet does not provide much organization or support for resident fish or wildlife data.

The use of supported technical staff located and working within member agencies is a useful way of developing ownership and technical expertise. However, the ISRP acknowledges that supervision and oversight of quality and work timelines may be difficult with a dispersed work group. According to the proponents, these technical staff work to develop the data management infrastructure within specific agencies to first satisfy the internal need for this information, and secondly to standardize and coordinate sharing of this information across state and agency boundaries.

The ISRP is impressed by the proponent's contributions to the development of database systems and approaches for improving the efficiency of data management and dissemination, their work enhancing agencies' capacities for data management, and their participation in a variety of teams of data management professionals from states, tribes, and agencies that coordinate regional data sharing.

There is no formal plan or discussion of adaptive management (AM). It is noted that a new organizational structure was put in place in 2014 to help improve project direction and performance. An Executive Committee was formed to direct the project, and it is comprised of policy level members of the StreamNet group as well as representatives of regional management agencies that fund the program and/or use the information. However, how the Executive Committee addresses AM issues is not discussed, and the ISRP would like more information on the mechanics of how AM is implemented.

An activity that helped drive adaptive changes to the project was the development and implementation of an online survey involving fisheries data management professionals in spring 2015. The survey asked respondents for opinions on data management questions. Discussions were then held with the StreamNet Steering Committee, BPA, NPCC, and others. One result was adoption of a five-year plan to guide activities. The plan was reviewed and updated by the Committee again on September 2017 and November 2018.

Some specific comments:

1.      The recommendation on pages 4-5 of the latest annual report represented a thoughtful level of synthesis and strategic thinking.

2.      The ISRP notes that many projects generate data in diverse formats and StreamNet brings them together nicely. This is a vitally important aspect of the project.

3.      The ISRP was surprised that photographs are no-longer stored. For habitat restoration work, photographs provide visual evidence of changes over time. Are these superseded by other formats? StreamNet seems like a logical place to store images from photo-points recorded over time.

3. Methods: Project Relationships, Work Types, and Deliverables

Well-developed database methods are used by StreamNet. There is a comprehensive description of project relationships, work types, and deliverables. Given declining funding it appears that maintaining the current mix of work types and level of activity will be increasingly challenging. The StreamNet Steering Committee and other partners will need to engage in a careful review of future work plans and approaches to ensure that the project is able to maintain its important basin-wide management and reporting responsibilities.

The proponents rightfully recognize that "Threats and limiting factors for StreamNet are primarily related to leadership, direction, and the data management capabilities within the data source agencies, rather than biological or ecological factors." Do the proponents have suggestions on how the Fish and Wildlife Program could assist in addressing these important threats and limiting factors?

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The StreamNet project has clearly played an important role in providing information to regional programs and meeting regional objectives. However, the proposal would be improved by inclusion of a description and evaluation of project significance with respect to regional coordination of data management under the Council's Draft Monitoring Evaluation Research and Reporting Plan, Subbasin Plan objectives, Council's 2006 Research Plan, and other regional plans.

The problem statement focuses on the difficulty that StreamNet has in acquiring data from the Basin's management agencies. If regional data networks develop, however, the need for a central facility like StreamNet may decrease. A long-term goal is to move regional data dissemination toward a distributed “Exchange Network” model. The proposal would be improved by a more detailed description of this model. As those capabilities are developed, StreamNet is working with the Coordinated Assessment project to provide critical metrics, and the ISRP supports this effort. 

The primary objective of StreamNet, to provide easily assessable regional data for agencies and others, is highly important. StreamNet appears to be succeeding in this objective, but it could have provided statistics on numbers of users of the database from various agencies (see Qualification #2). Although these data are reportedly located in annual reports, these data should have been summarized in the proposal. 

The StreamNet website encourages submission of datasets that may be of interest to others. Although the proposal mentioned a number of other dataset projects, it was not clear to what extent StreamNet datasets might overlap with other datasets that are made available online. The proposal would be improved by inclusion of a dataflow chart showing how all of the datasets and database organizations integrate among themselves and avoid duplication.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

The sponsors note that Pisces has some errors in their accounting of project reports, and that they have had difficulty in meeting deadlines for the annual reports and quarterly status reports because "input has to be obtained and consolidated from six subcontracting agencies." 

The proposal lists a number of accomplishments, and it describes how it has responded to previous deficiencies. They collaborate with agencies to help standardize data collection so that the data can be stored more readily. StreamNet has responded to most past ISRP comments (see Qualification #3), and they are attempting to fix problem areas as described in detail in the proposal. 

It would be useful to obtain information from active agency participants on StreamNet about the percentages of data from their agencies that find their way into the program each year, as well as their perceptions of how successful and useful the project is and what can be done to increase the quantity and quality of data entered (see Qualification #2).

StreamNet has focused on maintaining and updating a set of fish related data over many years and seems to have done a good job with those data and the information technology available. Better results could have been obtained if data storage in StreamNet had been viewed as higher priority among the states and tribes.

One important negative result relevant to the ISRP’s ongoing resident fish review is that provision of resident fish data to StreamNet is still not an agency priority. The sponsors state, "We are unable to change this situation, given that even when the NPCC Chair requested that we include more resident fish data, he also stated outright that there would be no additional funding to support the effort."

StreamNet devised an internet-based approach to disseminating data that are standardized and georeferenced across agency lines. StreamNet is, however, labor intensive, and with current staffing they have to focus on updating existing data sets, and have little time available to work to locate and standardize additional types of data.

StreamNet is evolving in response to input from agencies and user groups, demonstrating adaptive management for example by working with the Coordinated Assessments Project. However, this will result in delays by three agencies (WDFW, IDFG, and ODFW) in the updating of primary StreamNet databases (see Qualification #1). The ISRP concurs with the sponsors’ statements that a regional discussion is needed on which datasets, if any, to completely eliminate.

About three fourths of the project consists of sub-projects with states and tribes to develop data and databases and make these data available via StreamNet. StreamNet is proposing to expand the project to include the Colville, Shoshone-Bannock and CRITFC member tribes to the project. However, CRITFC’s StreamNet budget was moved to Columbia Basin Fish Accords. The current relationship with CRITFC is unclear. Has CRITFC data not been stored on StreamNet in the past? The proposal would be improved by addressing these issues.

Retrospective Evaluation of Results

StreamNet’s primary past contribution to the Fish and Wildlife Program has been to provide access to summarized, interoperable fisheries datasets collected by the Basin’s fisheries agencies. Data are provide via the project's website (www.streamnet.org) through an online data query system and interactive map applications. The primary data sets include:

• Anadromous fish distribution (generalized)
• Resident fish distribution (generalized)
• Adult abundance in the wild, redd counts
• Adult abundance in the wild, spawner counts
• Adult abundance in the wild, dam/weir counts
• Adult abundance in the wild, estimates of spawner population
• Hatchery returns (anadromous).

StreamNet’s goal of providing updated data within a year of data collection in the field has not always been met because of delays from internal reviews and in release of data by the agencies that collected the data.

StreamNet also develops and disseminates a variety of other data types including stream network hydrography, fish barriers, protected areas, hatcheries, dams, and other structural facilities, and fish age data. StreamNet also disseminates independent data sets that do not fit the StreamNet data exchange format and are archived in the Data Store (www.streamnet.org/datastore_search.cfm), where they are searchable and downloadable, along with metadata and functions as a data archive, as suggested by the ISRP (ISRP 2000-3). They also provide source documents for all data contained in the StreamNet database to the StreamNet Library.

Additional past contributions include:

• Initiating development of internal database systems in some partner agencies;
• Responding to data-related requests from participants in the Fish and Wildlife Program, for example their lead role in developing an initial draft Data Exchange Template for the Coordinated Assessments project and hiring specialists to assist agencies in describing data management gaps and needs, which was used as a template to help the state and tribal agencies determine their capacity to locate and provide the specified indicators and metrics (www.pnamp.org/sites/default/files/ca-lessons_learned_report-2011-05-17.pdf);
• Redesign and ongoing implementation of an online data query application for the StreamNet database (http://test.streamnet.org/); and
• A guide on data sharing, Considerations for Regional Data Collection, Sharing and Exchange (Schmidt and the StreamNet Steering Committee (2009), and a condensed ‘top ten list’ format (StreamNet, 2010). Many of the concepts discussed in the data sharing guide were adopted by the Coordinated Assessments project and in new data-related requirements for BPA contracting.

StreamNet results can be evaluated in part by review of their responses to issues raised in the past at workshops and by groups like NED, PNAMP, CBFWA and the ISRP, as follows:

• Timeliness of data updates: Addressed in part by encouraging and supporting agencies to develop internal database systems and initiate agency-wide approaches to data management.
• Lack of data from some tribes: Addressed in part by initiating work with the CRITFIC member and other tribes.
• Need for additional types of data: Addressed in part by assisting agencies with developing internal database capacity to allow more efficient data management and sharing.
• Lack of derived data: Addressed in part by assisting agencies in development of a data exchange network approach, whereby agencies other than StreamNet provide derived estimates and supporting data via web services.
• Lack of standardized field sampling: Addressed in part by collaborating with management agencies and regional scale entities to coordinate what is monitored and how, e.g., the Coordinated Assessments project.
• Not enough resident fish data: Not addressed due to a lack of funding to support the effort.
• Lack of standardization in data collected, collection methods, and data standards: Addressed in part because data collection issues were not addressed, but data standards were addressed through the Data Exchange Format (www.streamnet.org/reports_pubs.cfm.
• Unclear priorities for the types of data provided through StreamNet: Addressed in part by organizing a workshop with CBFWA in fall 2006 and prioritizing abundance data to support the Status of the Resource report, but regional consensus recommendations were not developed.
• Lack of clarity of who uses StreamNet, site use, and user satisfaction: Addressed in part by reporting site usage in annual reports, but user satisfaction could not be determined via an online user survey.
• Lack of description of QA/QC procedures: Addressed only by a brief description in the 2012 proposal.
• Lack of adequate metadata: Addressed in part by working through the PNAMP Metadata Work Group, but limited primarily to general descriptions due to lack of original metadata with the data submitted by data-collection agencies.
• Justification of the amount of staff and infrastructure: Addressed in the 2012 proposal.
• Description of the project interface: Addressed in StreamNet's Web Query System User’s Guide (www.streamnet.org/wqs_guide.html) and user guide for the map interfaces (map.streamnet.org/website/bluesnetmapper/HelpFile.htm).

The StreamNet strategic plan (Schmidt 2009) emphasizes developing internal database capabilities within the data source agencies and a distributed network for dissemination of data. As the distributed network develops, the need for a central location for data management should be evaluated by the agencies and entities collecting, disseminating, and using the data, as centralized coordination and management of such a distributed network will require considerable resources. 

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

The primary relationships described include those with PNAMP, CBFWA, and CRITFC, and there appears to be good coordination and collaboration with these entities. This section of the proposal would be improved by more information on BPA-funded projects and project numbers, including data collection projects, that are necessary for successful completion of proposed StreamNet objectives and deliverables. The sponsors note that agencies are the major limiting factor for StreamNet, that is, the efficiency with which data can be located and accessed within agencies and converted to regional standards. Clearly, provision of data to StreamNet is not always a high priority for management agencies.

The ISRP supports the StreamNet strategic plan (Schmidt, 2009) that emphasizes providing more support for developing internal database capabilities within the data source agencies. However, progress has been slow in some agencies, because it is not viewed as high priority. Although not discussed in this proposal, the Tribal Data Network proposal (#200850700) appears to be the companion proposal to facilitate this move.

The ISRP supports the increased focus on providing derived estimates and assisting data source agencies with development of internal data systems for storage and dissemination of data.

The sponsors describe important emerging limiting factors with respect to regional-scale data coordination in the proposal. Concerns are being addressed to at least some extent by the Coordinated Assessments project. Nevertheless, the ISRP concludes that there is a need for improved coordination of data management at the regional scale that will necessarily involve discussions with the Council and BPA, as well as agreement and support from the states, tribes, and other agencies and entities involved in providing data to StreamNet.

4. Deliverables, Work Elements, Metrics, and Methods

The ISRP supports StreamNet’s proposed new work in several areas: (1) to help develop indicator and metric data for the Coordinated Assessment project; (2) to collaboratively establish data needs and priorities, agree on standardized formats and definitions for sharing, and initiate sharing of the selected data as routine operations; and (3) to revise the data query system to improve user friendliness, increasing speed, and linking tabular and GIS data.

The ISRP is concerned that some data currently collected (Deliverable #2) will be put on hold until a distributed network can be established (see Qualification #1). Because the primary focus of regional data coordination is the Coordinated Assessment project, it is not likely that Deliverable #2 can be met within the period covered by this proposal. Although this is reflected in a reduction in funding for this deliverable, as discussed earlier in the proposal there is a need for regional consensus on this issue and whether some Streamnet datasets should be completely eliminated.

Among the three work elements listed for this project, none has metrics. The guidance given on the proposal submission site emphasizes an “emphasis on outcomes,” discussion of hypotheses, quantitative (and qualitative) measures and metrics, summary tables and graphs, and trends. Data management activities are amenable to scientific analysis. Key questions, hypotheses, relationships, data gathering and analysis, reporting of results, and revisions based on what is learned are expected. Greater emphasis on trying to measure outcomes and include in the proposal an adaptive management framework for designing, implementing, evaluating, and revising data management activities is recommended.

A log-in system to StreamNet might allow the program to more effectively evaluate public usage.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The project is 100% RM&E and Data Management with three work elements:

None of these work elements is associated with metrics in MonitoringMethods.org; however, it would be useful for retrospective evaluation of project results to develop quantifiable metrics or these work elements that could be used to track trends in data management project results.

Qualifications

In summary, StreamNet’s objectives are clearly stated, and most deliverables appear to be on track to meet the objectives (see Qualification #1). The project is of benefit to the Fish and Wildlife Program. However, an effectiveness monitoring plan needs to be developed and implemented (Qualification #2), and QA/QC methods are not documented in sufficient detail (Qualification #3).

The ISRP supports the project’s shift in focus to increased emphasis on derived estimates, such as indicators and metrics to support regional scale reporting under the ESA, as per the Coordinated Assessment (CA) project (www.pnamp.org/project/3129). Acquiring data from the tribes is a major step forward. The ISRP supports the strategic plan (Schmidt, 2009) that emphasizes developing internal database capabilities within the data source agencies and a distributed network for dissemination of data. However, the need of a central location for data should be revaluated as a distributed network system is developed. Coordination and management of such a distributed network will require considerable resources. The ISRP concludes from information in the proposal that there is substantial room for improvement in regional coordination of data management in the Columbia River Basin. This will necessarily involve discussions with the Council and BPA, as well as agreement and support from the states, tribes, and other management agencies and entities involved in collecting and providing data to StreamNet.