Response requested comment:

A response is requested to provide the following:

1.      Development of quantitative objectives/deliverables and metrics that can be used to measure and evaluate project accomplishments and outcomes.

2.      Description of a more formal process for adaptive management. This is particularly important for long term project success given PNAMP'S expanding geographic scope and increased user base.

3.      A strategic vision for PNAMP outlining where the project is headed in the next 5 years. This should include a brief description of how the collection of current activities help move the project towards that vision. Also, it would be helpful to see how these activities are part of broader, long term thinking about future PNAMP contributions, especially given expansion to the national arena.

4.      An explanation of the reasons behind the lags in publishing methods and protocols, and how the issue will be remedied in the very near future (i.e., 1-2 years).

Comment:

This project has a long history of impressive accomplishments. It continues to provide a wide range of technical tools and services, and it acts as a forum to bring a wide range of agencies, tribes, and other entities together for improved coordination, consistency, and collaboration in aquatic monitoring. There are project areas where improvements are desirable, and they are primarily included in the categories where a response is requested. They include developing quantitative objectives; gauging user needs and satisfaction/project effectiveness; developing metrics for measuring the impact of the PNAMP program, including major accomplishments and outcomes and the development of long-term project goals for PNAMP and a strategic framework to help move towards them. The number of reports and workshops, while impressive, is not a clear measure of whether the program is actually increasing the quality or efficiency of monitoring or producing new knowledge to support decision making. Finally, it should be noted that the topics, where a response is requested, are all discussed in the programmatic section of this ISRP report and apply to most data management projects reviewed.

1. Objectives, Significance to Regional Programs, and Technical Background

The PNAMP project plays an important role in coordinating aquatic monitoring efforts in the Pacific Northwest (PNW), from Canada to Northern California. It has broad significance to regional programs and has provided sustained leadership to support effective monitoring and data sharing efforts across the Pacific Northwest. The PNAMP group has the technical expertise to guide a successful endeavor.

The project has four primary objectives; all are qualitative and broadly stated with no dates for accomplishment. This makes it very difficult to evaluate actual accomplishments and outcomes. Some additional details were provided under each of the 11 deliverables, but many are still vague. Current descriptions leave many questions unanswered. Some include: How do the proponents know if the workshops and coordination are having an impact? Being able to describe more quantitative results would help answer this question. The proposal discusses workshop reports, white papers, annual reports, website and tools, etc., but is there some way to track the impact of these activities? Or are they just available online but underutilized? The ISRP recognizes that the impacts of coordination can be hard to measure. However, the objectives and deliverables could be written to be more specific to facilitate quantitative evaluation to determine if an objective was met or a deliverable was produced and what the associated outcomes were.

2. Results and Adaptive Management

Overall results appear to be outstanding. The number of activities reported in the proposal is admirable. PNAMP has provided regular annual reports describing an impressive array of products and accomplishments. Some of these include assistance in development of study and sample designs for charter members' monitoring projects; outreach and training including workshops to webinars; assistance in the development of regional high-level habitat indicators; coordination of effectiveness monitoring and data management; sharing of "best practices"; and operation and maintenance of the MonitoringResources.org website. It is apparent that impressive and important enhancements have been made to MonitoringResources.org, despite the relatively low number of protocols and methods that have been published.

Although the project does not appear to have a formal adaptive management (AM) process, it can point to several changes in work and organization that are a result of past lessons learned. Proponents describe PNAMP collaboration and coordination functions as responsive to the needs of partners. This is an indication of the adaptive management philosophy of PNAMP, and it seems to be implementing it reasonably well with numerous meetings and information feedback loops. Currently, the primary tool for assessment of overall project effectiveness is the PNAMP Steering Committee (SC). The proponents note that there are also periodic strategic planning retreats and annual user surveys. A summary of findings/results of these efforts and how they informed decisions on project design and operation was not provided and would have been helpful.

There are no metrics provided that can be used to gauge project outcomes/impacts for key activities nor is there any strategic vision or framework to guide longer term program development. In a 2007-2009 solicitation review, it was noted in discussing PNAMP'S effectiveness that, "To assess the effectiveness of this facilitation an audit or poll of participating agencies should be conducted within two years. Adaptive management and course corrections within the PNAMP framework could be realized if direct feedback from the participating agencies were obtained." The proposal would be improved by documentation of this feedback as well as by a better description of whether a particular model of coordination is being used." Although user feedback is provided by the Executive Committee, it does not appear that a formal audit or poll has been completed or has there been a summary of findings and description of how they have been incorporated into the current project. This feedback information would clarify, for example, how practitioners in the basin become aware of PNAMP and its resources, the effectiveness of dissemination of information about various training opportunities, MonitoringResources.org, and such. Articulating a strategic dissemination plan, if it does not already exist, could be useful in informing this audit and increasing the impact of the program.

Another ISRP concern, especially with the expansion of PNAMP activities outside the Pacific Northwest, is that there continues to be prioritization of activities to ensure continued service to the Fish and Wildlife Program and Tribes. While expansion can bring great opportunities, there is a need to remain focused on the important tasks within the region. The ISRP assumes that the expansion will enhance programmatic functions and analyses, but, if so, the proponents need to describe in what ways this will happen and what the possible positive outcomes might be.

A final concern relates to publishing monitoring protocols (p. 22). Only 219 of the 1,166 are published. Methods have a better rate of publication, but there are still several hundred waiting. Please explain why the lags are occurring. The ISRP hopes that the efforts expended toward project expansion are not delaying the publications.

3. Methods: Project Relationships, Work Types, and Deliverables

There is a wide range of work types and services provided by the project. These range from data management, operation and maintenance of a website, public outreach, and user training sessions and webinars. A general description of project methods, in key project areas, is provided. All provide good examples of how a project should function and how to prepare a well written proposal.

There are 11 deliverables listed. They are described in some detail but do not provide quantitative measures or metrics to measure activity completion or effectiveness. An example would be tracking the activities organized under the fourth objective of "develop and maintain web resources." This type of description is very general and makes the actual assessment of what activities were completed and their outcomes nearly impossible to evaluate. Also, while the proposal indicates that PNAMP participants and administrators will show responsiveness to the needs of partners by identifying the tools that are most technically appropriate and durable over the long term, it would have been particularly helpful to have a strategic vision and framework for the future that establishes the broad project direction and provide a general description of expected results. Given the changes in project scope, this vision is particularly relevant. Having this information will assist the ISRP and the Council/BPA to understand how needs will be met in the future. It is acknowledged that the proponents have reported that such a strategic document is in preparation and a review draft ready soon. The ISRP is interested in participating in a review of this document to provide feedback on PNAMP's vision and strategy going forward.

198810804 - StreamNet - Coordinated Information System (CIS)/ Northwest Environmental Database (NED)

The issues raised in this review can be addressed during contracting. No response to the ISRP is required.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The continuation of PNAMP activities, particularly web-based coordination and standardization of study protocols and field methods, is beneficial to the region. 

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

The sponsors effectively describe the evolving history of PNAMP and provide an effective summary of accomplishments. The proposal describes PNAMP collaboration and coordination functions as reactionary processes that are responsive to the needs of partners. This is one indication of the adaptive management philosophy of PNAMP. The proposal focuses on shifting more PNAMP effort to web-based resources and tools as another indication of adapting to new information from changing situations. 

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

The proposal summarizes project relationships by stating that, "PNAMP provides a voluntary forum for coordination and collaboration for new and existing monitoring programs and projects in the Pacific Northwest." The relationships are additionally documented by having 20 signatories to the PNAMP charter. The proposal identifies how involvement with PNAMP varies among signatories depending on the activity.

PNAMP's approach to limiting factors is reactionary to the needs of partners as new information about threats to focal species arises. There may be a valuable role for PNAMP to identify limiting factors for discussion among partners before threats arise.

PNAMP has developed the web-based resource, MonitoringMethods.org, to support data management and sharing. Feedback from users of MonitoringMethods.org should be actively solicited and used to improve the resource. Other web-based tools have been developed or are proposed. The ISRP supports these efforts. 

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables are mostly clearly identified and linked to project objectives. One exception is that the utility of the geodatabase mentioned in Deliverable 16 is not clear. More explanation of how integrating the geodatabase with other web resources will be beneficial. This should be specified during contracting.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The ISRP supports the continued development of the standardized protocols and methods in MonitoringMethods.org.