Comment:

The Columbia Basin Bulletin (CBB) is a unique source for information sharing in the Basin. It is designed to "provide information about fish and wildlife issues important to Columbia River Basin fish and wildlife policy development" and receives broad use that is reportedly increasing steadily. It is a major outlet for dissemination of information about restoration and environmental developments in the Basin.

Currently the CBB is scheduled to be terminated in FY 2020. Funding for FY 2019 is about 50% of the level of previous years. Because there are no other similar information/technical transfer products with the timeliness, public accessibility, and range of subjects provided by the CBB, its termination of the CBB will have adverse effects on program users and the general effectiveness of information sharing in the Basin. The ISRP strongly recommends that policy makers consider either (1) abandoning current plans that call for the dissolution of the CBB, or (2) developing replacement tools for comprehensive and timely, basinwide information sharing before terminating the current project.

1. Objectives, Significance to Regional Programs, and Technical Background

No specific objectives were listed or apparently required for the 2018 project contract. Nevertheless, the CBB is described in the proposal narrative as a tool for use in information sharing and technical transfer of a wide range of information relevant to the Fish and Wildlife Program and other recovery programs for fish and wildlife in the Basin. This description is quite broad and makes it difficult to measure completion or outcomes. Nevertheless, it is clear that the overriding objective of this weekly bulletin is to find and publish online articles that are of interest to Fish and Wildlife stakeholders in the Columbia Basin. The proposal does not provide metrics that describe anticipated results or benefits beyond the reference to having 9500+ current subscribers. Given the many ways that people access information these days, the proponents need to be able to demonstrate that it is "stakeholders' key source for objective, complete, timely information about Columbia Basin Fish and Wildlife issues."

The significance to regional programs appears to be strong, and much information is provided to demonstrate many of the ways that the CBB aligns with various priorities for the Fish and Wildlife Program and the Basin. To maintain and strengthen this strong linkage, the ISRP suggested in the 2012 review (ISRP 2012-6), that the Council might consider using an independent scientific survey of members/users to evaluate the CBB in terms of regional coordination of outreach and education. It is important to understand how users view the CBB. This was not addressed in the proposal.

The discussion on Technical Background is satisfactory. However, the CBB should be actively considering new and emerging techniques for communication as ways to improve its readership and usefulness (if it is not already being done).

2. Results and Adaptive Management

Overall the results are outstanding. The CBB has been produced on a weekly schedule for more than 20 years and has provided information summaries to a wide range of users. It has served as a unique information sharing resource for the Fish and Wildlife Program, its participants and other interested parties and publics. It has been a useful tool to disseminate up-to-date information across the entire region. Unfortunately, the proposal does not include additional information that attempts to describe specific outcomes from CBB or any metrics to gauge efficiency in information transfer of the newsletter and website. It is reported by the proponents that the subscriber base and website visits have steadily grown over the years, but the reported number of subscribers has remained constant in proposal documents.

The proposal does not present lessons learned or indication of how the program has evolved. The proposal would be improved by a discussion of factors that might limit the project's success at meeting project objectives. Since there does not appear to be a requirement for the project to provide annual reports, it is difficult to track any adaptive management that has occurred. It was noted in the 2012 ISRP review that "No significant changes in project direction have occurred, although based on requests from a reader survey conducted by the CBB, the sponsors have included more articles on research in the Columbia Basin or relevant research conducted outside the Basin." It was also suggested in the 2012 review that there be: a future, independent scientific survey of members/users to evaluate the CBB in terms of regional coordination of outreach and education; a greater emphasis on trying to measure outcomes; and including in the proposal an adaptive management framework for designing, implementing, evaluating, and revising coordination activities and a discussion of the factors that might limit the project's success at meeting coordination objectives. There is no evidence that any of this has occurred.

It seems like the impact of the project could be substantially larger. The basin and technology have changed significantly since 1998, but it is not clear that the Bulletin has done the same. There are a number of questions regarding change. These include:

• Are email blasts the most effective way to disseminate the information? Given how many list serves people receive every day, it is likely that some of the emails are going directly to the trash. What other mechanisms (e.g., Twitter, others) could be used to increase the effectiveness of sharing information?
• Who exactly is the audience? Under a limited budget, it may help to identify who is reading the emails and how content could be focused.
• It sounds like readership survey was conducted before the 2012 review, but it may be useful to revisit that survey. For example, it seems like they would need to know how many of the 9500+ subscribers actually read the emails, better understand how the subscribers navigate the information (what broad themes are they interested in? How do they search for/reach those themes, etc.), and get feedback on how the newsletter impacts their work and decisions. It sounds like the project has some website analytics, but it is not clear how those results are being used to improve the dissemination of the information. For example, those analytics could be used to better understand how subscribers are accessing the information, which should lead to improvements of the emails and website.

3. Methods: Project Relationships, Work Types, and Deliverables

Producing the CBB requires considerable coordination and interaction with a wide range of individuals and agencies across the basin. The proponents provide a comprehensive list; however, it did not appear to include any political offices (e.g., governors or state legislators), an important audience.

The Columbia Basin Bulletin (CBB) continues to provide an important communication and education service to a wide range of people in the Columbia River Basin. The CBB's unaligned information encourages improved coordination by promoting better understanding of complex and contentious issues. Based on past project performance, the CBB is meeting its objectives. No significant changes in project direction have occurred, although based on requests from a reader survey conducted by the CBB, the sponsors have included more articles on research in the Columbia Basin or relevant research conducted outside the Basin.

The CBB provides regional coordination of information dissemination as outreach and education. Information in the proposal, however, was not sufficient for the ISRP to evaluate the CBB's effectiveness and impact on regional coordination. While beyond the scope of this proposal, the ISRP recommends a future, independent scientific survey of members/users to evaluate the CBB in terms of regional coordination of outreach and education.

The guidance given on the proposal submission site emphasizes outcomes, discussion of hypotheses, quantitative and qualitative measures and metrics, summary tables and graphs, and trends. Coordination activities are amenable to scientific analysis. Key questions, hypotheses, relationships, data gathering and analysis, reporting of results, and revisions based on what is learned are expected. Greater emphasis on trying to measure outcomes and include in the proposal an adaptive management framework for designing, implementing, evaluating, and revising coordination activities is recommended.

1. Purpose: Significance to Regional Programs, Technical Background, and Objectives

The Columbia Basin Bulletin is a valuable resource on news related to Columbia Basin fish and wildlife issues.

Significance to regional programs: As a neutral information provider to projects and programs throughout the Basin, the project meets the Council's goal for program-level coordination of information dissemination including technical, policy, and outreach components. 

Problem statement: The statement notes the complexity and controversy of many fish and wildlife mitigation issues and the corresponding need for a timely and neutral source of information that promotes coordination among stakeholders. The technical background is brief and no references, beyond the newsletter itself, are cited. This might be improved by describing the project's web site, newsletter, and their content. Providing more background on the history and development of the CBB as well as a brief summary of the work of key project personnel on similar past or current efforts would be beneficial. 

Objectives: The proposal interprets the question as referring to biological objectives only, so does not list specific objectives. The CBB is in a unique position as an information provider. Based on the proposal the objective might be to provide “unbiased information about fish and wildlife issues important to the Columbia Basin.” The deliverable is a weekly newsletter.

Emerging limiting factors: None is listed. The proposal would be improved by a discussion of the factors that might limit the project's success at meeting coordination objectives.

2. History: Accomplishments, Results, and Adaptive Management (ISRP Review of Results)

Financial performance and history: a budget history of the past 6 years is provided. Annual budgets have varied in size but have not been exceeded. Cost share is listed. A description of effort adjustments in response to budget variability is provided.

Performance: Past performance at delivering a weekly newsletter is outstanding; weekly electronic newsletters and website posts have been delivered on time during the entire 14-year project history. However, status reports have been delivered late, on average 31 days late. The sponsors acknowledge the need for improvement in this regard. 

Major accomplishments: The CBB has been published weekly since June 1998. The proposal notes the growth in the number of CBB subscribers from 300 to over 9500 over the history of the project. The number of subscriptions continues to grow. The amount of content and frequency of delivery has also grown. Web links are provided to all stories. Website use has also grown. Documentation of the trend in subscribers would be useful for assessing accomplishments. 

In the program coordination section, there is no description of Past Accomplishments with work, value added, or user assessment of effectiveness and impact all marked NA. The accomplishments section provides a brief project history that would be better placed in the previous, technical background, section.  The sponsors need to more thoroughly explain past results of regional coordination, evaluate these results in terms of their coordination objective(s), and briefly summarize improvements made to the CBB based on past results in terms of adaptive management. The sponsors mention a reader survey that they conducted, and it would be useful to have a more detailed summary of the survey questions and results.  The sponsors state, "The CBB is now stakeholders’ key source for objective, complete, timely information about Columbia Basin Fish and Wildlife issues." How was this determined? For example, the number of website users might or might not be a relevant metric of use by stakeholders as their key source of information.

Response to ISRP comments: This section provides details about the 2006 comments and their response, including conduct of a reader survey.

Adaptive management: None is listed. However, the sponsors’ explanation of how they responded to their reader survey is a good example of adaptive management. In the "response to ISRP" section it describes the outcome of a readership survey and the adaptation of CBB content to reflect reader requests. The financial history section describes adaptations to budget variability.

3. Project Relationships, Emerging Limiting Factors, and Tailored Questions for Type of Work (hatchery, RME, tagging)

The CBB is a Northwest regional project that informs many people interested in the Fish and Wildlife Program in the Columbia Basin.

Project relationships: the primary relationship is as an information link between projects and people throughout the Basin. The proposal provides a long list of entities that comprise its readership and with whom they work to collect and disseminate information related to Basin fish and wildlife activities, projects, and biological opinions. However, it would be useful to have a breakdown of the number of CBB articles per year by entity, as well as a chart of annual trends to help identify gaps in coordination of dissemination of information/education.

4. Deliverables, Work Elements, Metrics, and Methods

Deliverables: The statement describes in detail how the weekly newsletter is composed and delivered. While the work of the CBB is 100% information dissemination, more specific attention to identifying a scientific component to the proposal would help to plan for future success. The proposal guidelines ask how the proposed work and accomplishments “contribute to or inform Program policy development; lead to broad-scale implementation; and be reported back to the Council?” A well informed public is critical to the Fish and Wildlife Program process. More details on the subscriber base and trends in subscribers and online users would be useful. Further, effort to get responses from users and subscribers might help the CBB play a stronger role in the Basin.

The proposal says that the Bulletin has “diverse and representative subscribers who are interested in supporting this collaborative and integrated approach." Are more data available about the subscriber base? The proposal also suggests that the CBB is independent thus making the news is more comprehensive and less biased. Can this be documented?

If the goal is to communicate broadly, then one question is why there is a charge to review the Bulletin’s archives? It appears that the charge may prevent users from searching the history of an issue and increasing their knowledge of a recurring topic.

Work elements: The entire effort of the CBB is information dissemination to stakeholders. Outreach and education are the main work elements. One work element is identified, 99. Outreach and Education.

The CBB proposal offers may testable hypotheses about the importance of a communication mechanism like the Bulletin. Evidence would be useful in supporting statements like, "To terminate the CBB as a FWP supported, easily accessible, on-line stakeholder information tool -- and not considering the value of several years’ investment in building stakeholder trust and use -- would eventually, due to lack of access of important, timely information, lead to a demand among stakeholders, particularly those who cannot afford or have the time to attend the myriad of meetings related to Basin fish and wildlife mitigation and recovery -- to create an information product that does just what the CBB is doing now.”

Metrics: Primary metrics used by CBB sponsors to evaluate their results are the number of subscribers, website hits, visits, page views, or story reads. More detailed explanation of these metrics and analyses of annual trends in these metrics would be useful. Direct methods for members to provide feedback on articles and issues, for example through website tools or an online letters-to-the editor forum, might further assist in evaluation of CBB's educational and outreach performance.

How do we know that the CBB is viewed “as a trusted and well-used stakeholder information tool …?" Do other news outlets pick up CBB stories? The expectation of feedback contained in the statement, “it would not take long for feedback to the Editor and others to make clear that something was awry" seems to be a rather passive approach. What kinds of questions were asked in the Survey Monkey project in 2007? Of the 800 respondents how many are in the 950 subscribers?

Methods: The description of methods, for example attendance at meetings, telephone interviews, in-person interviews, use of research reports, studies, policy letters, memoranda and other documents, is very general. The proposal does not provide enough information on membership fees and how these are used. The ISRP notes from reading the webpage that to become a CBB member, a $5 fee must be paid, and membership fees range from $10 for one month to $60 for one year. These fees likely limit access by users who cannot afford the costs and might diminish the effectiveness of the CBB as a regional education/outreach tool. The educational/outreach methods of the CBB website might also be improved if all articles included direct links to all information sources used to write the articles, as well as additional information sources for readers who want to read and learn about the issues in more detail. A page on the CBB website devoted to a calendar of Columbia Basin regional meetings and events, as well as a page with links to primary information sources would be useful to readers/members.

4a. Specific comments on protocols and methods described in MonitoringMethods.org

The protocol for one work element is provided, but does not provide adequate guidance on the methods and metrics. The best guidance available is from the ISRP (2007-14:2) and the overview provided in this report. The project sponsors can strengthen the science in proposals by developing methods and metrics for the most important project objectives.

Modified by Dal Marsters on 4/17/2012 2:30:52 PM.

This proposal is to continue support for the Columbia Basin Bulletin (CBB). The proposal clearly and concisely describes the need for trusted, neutral, and timely information on Columbia Basin fish and wildlife issues and references earlier Council support for its work. In the eight years of its operation, the CBB has become a widely used and proven source of timely and reliable information. Dissemination of information to stakeholders and agencies in the Columbia River Basin is critical. The CBB seems to be a cost-effective mechanism for disseminating technical and policy information about fish and wildlife in the Columbia Basin.

The proposal stresses the CBB's role as a coordinator of information in the Columbia Basin. It makes the point that trustworthy information is the basis for collaboration on the complex and contentious issues of the Columbia Basin. The proposal makes a convincing argument for the CBB's communication value beyond the fish and wildlife interests to the broader group of river interests and for the benefit of having a neutral provider of information.

The proposal has a single objective to provide summary information related to fish and wildlife to the Basin in order to assist policymaking and help achieve restoration goals. Methods are briefly but adequately described as the routine tasks of information gathering and newsletter production. It is not clear how the CBB staff decide which scientific papers they will profile. One suggestion, if they are not already doing this, would be for staff to scan the American Fishery Society publication website which cites "most downloaded (or read)" papers. If particular papers deal with Columbia River Basin issues, they might be worth mentioning in the CBB.

The proposal history is a short summary of the evolution of the CBB from a web-based product in 1998 to the current email delivery product with a subscriber list of 5000. Summary usage information is provided. Thorough monitoring of outreach and information provision would require a specific analysis to see if the CBB is increasing stakeholder/agency knowledge about Columbia Basin fish and wildlife issues relative to other web sites, and print/visual media. However the increasing number of hits and story reads indicates the CBB is increasing its popularity as an information source. Earlier ISRP review comments had requested that information regarding quality control mechanisms be included in the proposal, and this information should continue to be provided.