Qualifications:

The ISRP recommends that the proponents describe their responses to the ISRP's comments and suggestions below in their upcoming annual report covering FY 2019 accomplishments.

1.      Objectives need to be quantitative with specific timelines for attaining clearly stated milestones and criteria for success.

2.      Provide an Adaptive Management (AM) process description for ISRP review.

3.      Provide the ISRP documentation on the project mission and out-year work plan, i.e., the plan developed to guide future activities. The documentation needs to include the strategic approach or activity list, as well as the timeline to support multi-year implementation.

4.      The proponents adequately addressed some qualifications from the previous ISRP review(2012-6), but some were not addressed. The ISRP would like to discuss Qualifications No. 1 and No. 3 (and possibly Qualifications 2 and 4) from the previous (2012-6) review in a face-to-face meeting. No. 1 relates to objectives being restated in terms of desired outcomes rather than tasks and No. 3 relates to defining the success criteria used to determine whether each of the five project objectives will have been met specified milestones. The ISRP believes that these can be accomplished as long as there is a common understanding of what is expected.

Comment:

This project is challenging in that it provides support for upgrading and enhancing data management for a group of tribes with varying degrees of support and enthusiasm for the effort. It appears that there has been major progress and that it has resulted in tribal members' active participation and data sharing with broader regional efforts. The broader efforts include implementation of the Columbia Basin Fish Accords, recovery planning under the ESA, tribal co-management needs regarding U.S. v. Oregon and the Pacific Salmon Treaty. Although designed as an interim project, funding reductions are likely to extend the time needed for full implementation. The ISRP notes that that data management is a full-time effort and requires an appropriate level of financial support.

1. Objectives, Significance to Regional Programs, and Technical Background

The proposal includes a strong and clearly stated goal that centers around effective data management, and the text has a comprehensive and explicit explanation of what effective data management means. The proponents appear to have the technical expertise and appropriate leadership to execute the activities to achieve the goal. However, some objectives are not written in a way that progress toward them can be evaluated. Simple changes from words like "ensure" or "enhance" toward more measurable goals could help. With that said, some of the text beneath the objectives did provide measurable objectives, so it may just be a proposal structure issue. For instance, "Facilitate routine (e.g.; 6 times per year) ITMD coordination phone calls between tribal data stewards and attend occasional (e.g.; once per year) site visits to share information and transfer technology." Nevertheless, as stated, the objectives (p. 5) are very general and lack timelines for completion. They are really work statements rather than quantifiable objectives.

The text on significance to regional programs clearly defines that the project is a resource to help the Tribes manage and share data. The proposal does indicate that the project is responsive to some other efforts in the basin (e.g., BiOp), but (appropriately) does not attempt to extend project significance more broadly to all efforts across the basin.

Overall, the ISRP believes that the project is highly relevant to member tribes as well as to other regional data management programs. The proponents have the technical skills to be successful. Nevertheless, the ISRP is concerned whether the project is threatened by personnel issues (p. 16) without having adequate funding to acquire and retain skilled staff, to train and educate staff (continuing education and conferences), and to overcome the difficulties in recruiting skilled professionals to remote tribal locations.

2. Results and Adaptive Management

The project has provided important support and encouragement for improving member Tribes' data management and information sharing capabilities. It has resulted in a wide range of deliverables ranging from increased infrastructure development, improved coordination and communication, and enhanced data transfer support. A major accomplishment occurred in 2018. With the help of an EPA grant, the tribes were able to install centralized data management systems and load a limited number of data sets.

The project has enabled data sharing for important regional projects including recovery plans and U.S. v. Oregon. Although the project is intended as an interim effort, it is limited by the need for improved data management staffing. Two tribes now have full time data stewards who are rapidly acquiring data management skills. The remaining tribes have identified individuals for the role of a part time data steward despite the proponent's observation that a part-time steward is not sufficient to fully support tribal fisheries programs.

The ISRP notes that a project mission and out-year work plan has been developed to guide future activities. However, there is no mention of any documentation of a strategic approach or activity list and timeline to support multi-year implementation. The ISRP mentions this because the proponents do not feel that this project needs an adaptive management (AM) process. The proposal states, "The ITMD project is not the type of project that requires an adaptive management plan of its own per the specific definition." It does acknowledge that the "project does adapt to ever changing policy guidance on data management from the tribes, CRITFC, and Columbia Basin fish and wildlife resource management programs." Despite lacking an AM process, there is an excellent discussion of lessons learned and recommendations for change contained in the 2018 Annual Report. Nevertheless, the ISRP feels that the project could improve efficiency by having a clearly articulated AM process for both internal and external issues.

The ISRP was pleased to see that "At the close of 2018, all the tribes and CRITFC possess the required infrastructure to be able to share data regionally. Experiments in uploading and downloading data have taken place to several regional repositories. The expectation is that by the close of 2019 some tribal and CRITFC data will be available across the region." This is a very positive development.

As well, the ISRP supports the continued efforts made toward training personnel and seeing that each tribe has the technology and skills to successfully participate in the data management project. The proponents understand that a serious threat toproject success (p. 17) is not having adequately trained personnel and have established a process for maintaining that expertise over time.

While the proponents provided honest responses to previous (2012-6) ISRP qualifications, it would be good to discuss Qualifications No. 1 and No. 3 (and perhaps Nos. 2 and 4) in a face-to-face meeting. No. 1 relates to objectives being restated in terms of desired outcomes rather than tasks and No. 3 relates to defining the success criteria used to determine whether each of the five project objectives will have met specified milestones. The ISRP believes that these can be accomplished if there is a common understanding of what is expected.

The ISRP agrees only in part with the statement (p. 13) that "Good decisions are based on quality of data, quantity of data (over space and time), and on real-time data flowing quickly through data management systems (for those decisions that require a quick turn-around time)." We also believe that good decisions are based on appropriate analyses of good information and having the experience to interpret the results accurately (wisdom). The overall impression is that abundant data are being collected by each Tribe and processed through the project, but less emphasis is given to analyses and interpretations. In the future, the proponents should add analysis and interpretation to the training of skill sets.

3. Methods: Project Relationships, Work Types, and Deliverables

The project is guided by the overall data management strategies developed for each of the Tribes and CRITFC. The ISRP notes that "Since the last ISRP review, the project has been gradually transitioning from supporting individual project data systems to more common data systems to support tribe-wide and regional data sharing and reporting. This is linked to the demand for broader data sharing on a regional scale. Work types include infrastructure development, skills and technical capacity development for tribal staff, information sharing and review and application of new technical developments, most recently to support field data entry."

While the nine deliverables are quite detailed, they are generally qualitative in describing activities but not outcomes, as recommended by the 2012 ISRP review

The ISRP is concerned that several data stewards are only part-time positions. A discussion with the proponents and the Council/BPA is warranted to see if the positions can become full-time. Part-time positions, ones that have responsibilities elsewhere, do not bode well for long-term success. That said, could these positions also include responsibilities for advanced data analyses and interpretation?

The ISRP is still not completely clear on how the proposed data management activities are related to data management activities of other programs in the Basin, for example, the AEM activities proposed by the Yakama Tribe, PNAMP, StreamNet, and others. The ISRP would appreciate understanding how much database sharing and overlap occurs.

Objective 1 and Deliverable 2 (p. 24) are seriously hampered by current BPA funding rules for travel for meetings and conferences, which restrict access to continuing education opportunities. Additional support and funding for continuing education, information sharing, and outreach appears justified for this project.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

Project significance to regional programs and technical background were adequately addressed. Objectives are stated as tasks, for example “Providing data management services to the tribes” rather than as desired outcomes. The sponsors need to define the success criteria used to determine whether the project’s objectives have been met. The proposal uses many undefined acronyms and technical jargon, and would be improved by providing a list with definitions of acronyms and technical terminology.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

The sponsors list a number of project accomplishments, but this section of the proposal would be improved by describing each result in terms of value-added, specifically with respect to the Council's Fish and Wildlife Program and the region, results of user/member assessment of effectiveness and impact of the work accomplished, and how results of this assessment have modified previous and proposed activities over time to increase value of this work.

 The sponsors provide some useful examples of how project results are used for adaptive management.

ISRP Retrospective Evaluation of Results

This is a relatively new project, initiated in FY 2009, to continue support for personnel and infrastructure to allow the CRITFC tribes to collect, house, and distribute data from the projects funded by the Accords, that is, fish and habitat monitoring data for the reservations, ceded lands, and key co-management areas.

The Tribal Data Network’s (TDN) primary goal is to ensure the availability and sharing of accurate and timely monitoring data among CRITFC member tribes and with other agencies to meet the reporting needs of the Accords and BiOp while also building capacity within tribes to support informed policy management decisions and tribal co-management needs.

Overall, the project appears to be on track to meet its objectives.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

Tribal Data Network (200850700) and StreamNet (198810804) will work synergistically to integrate data management and sharing across the Basin consistent with the Columbia River Basin Collaborative Data Sharing Strategy. What are plans for checking accuracy of data? Will there be peer review of methods for analysis of data? What are the plans for updating data, for example the CHaMP project? Will this project store and disseminate data from all tribes, that is, both CRITFC and non-CRITFC tribes, in the Columbia Basin? 

As described in the TDN 2011 workshop report, there seem to be several limiting factors related to data management, not adequately discussed in the proposal, for example, data sharing with NOAA and software/server compatibility. Although this project involves 25% coordination, tailored question for coordination were not addressed.

4. Deliverables, Work Elements, Metrics, and Methods

Is it not possible to get SARs with confidence intervals directly from DART for any set of PIT tagged fish?  The sponsors stated that DART may provide some SARs.  The sponsors need to check whether estimates are the same.

It is not clear exactly what data will be housed in this Tribal Data Network; for example, is habitat data for intensively monitored watersheds from the Columbia Habitat and Monitoring Project (CHaMP) project to be included? Is this the only place where CHaMP data are stored? Later, it is stated that CHaMP data will be downloaded.

The sponsors need to describe the percentage of project time that will be devoted to work elements, explain what metrics will be used to assess effectiveness and impact of the work accomplished, describe key personnel duties on the project, including the hours they will commit to the project, and provide a more detailed description of QA/QC procedures.