Fishing Effort Survey Research and Improvements

Fishing Effort Survey Study and Path Forward

• We have concluded data collection and are wrapping up data analysis as part of a large-scale study conducted throughout 2024 to test improvements to the Fishing Effort Survey (based on results from prior studies)
  • "Evaluating Measurement Error in the MRIP Fishing Effort Survey" is one of several studies NOAA Fisheries conducted to evaluate potential sources of bias in the FES as part of our continuous improvement process. This report was published in August 2023.
• The 2024 tested design changed the order of two fishing effort questions and also increased the administration of the survey from every two months to monthly
• Monthly sampling provides future flexibility to produce more frequent estimates
• The 2024 FES study results analyzed thus far are generally consistent with the results from the aforementioned pilot study published in 2023. The tested design produces recreational fishing effort estimates that are lower on average than those generated by the current FES, with similar variability by state and fishing mode. Preliminary results also support the hypothesis that the current FES design may lead to survey respondents reporting trips outside of the questioned timeframes based on how the fishing activity questions are ordered, which could lead to overestimation of fishing effort. The revised FES is designed to address this issue.
• NOAA Fisheries’ guidance to fisheries managers and stock assessors who currently use FES data is to continue using existing FES data until the new design is peer reviewed and the historical estimates have been calibrated (anticipated in spring 2026). We recognize there are important regional considerations (as outlined in the hyperlinked guidance document).

Anticipated Timeline (contingent on continued available resources and peer review results)

2024

• Administered the revised design alongside the current design throughout all of 2024
• Worked across NOAA Fisheries to provide interim fisheries stock assessment and management guidance to assessors and managers
• Worked with Atlantic, Gulf, and Hawaii partners on preliminary planning for potentially transitioning to a new FES design

2025

• Ongoing: 
  • Develop and refine the calibration model to rescale the historical effort estimates produced from the current FES to the scale of the revised FES
  • Support partner requests for interim fisheries management guidance; work with partners on transition planning to implement a revised design; and provide updates at relevant Council and Commission meetings, as requested (noting full study results won’t be available until late summer/early fall; and calibrated catch estimates won’t be available until spring 2026)
• Summer: 
  • Finalize study report/revised survey design
  • Independent expert peer review of the study report/revised design
• Late summer/early fall: 
  • Publish and communicate final, peer-reviewed study report and results
  • Conduct an independent expert peer review of the calibration model through a virtual peer review workshop, which will be open to the public
• Fall: 
  • Finalize plan with partners to transition to a revised FES design

2026

• January: Implement the revised FES design
  • This is the earliest a new survey design could be implemented and is contingent upon results from the peer review and successful completion of a transition plan 

• Spring: Produce calibrated historical estimates for use in fisheries management and stock assessments

   

When will fisheries managers and stock assessment scientists have updated estimates for use?

The historical calibrated total catch estimates (updated to reflect outputs from a revised survey design) are anticipated to be available in spring 2026, as outlined in the timeline above.

It's important to note that the FES only produces private boat and shore fishing trip/effort estimates. All species catch information is collected through dockside/public-access interviews (Access Point Angler Intercept Survey). The effort information is generally multiplied by the catch information to estimate total catch for a species. Therefore, we won't know species-specific impacts until we have the calibrated estimates in spring 2026.

NOAA Fisheries’ guidance to fisheries managers and stock assessors who currently use FES data is to continue using existing FES data until the new design is peer reviewed and the historical estimates have been calibrated. We recognize there are important regional considerations (outlined in the hyperlinked guidance document). 

What were the key pilot study findings from the 2023 pilot study report that led to this 2024 follow-up study? 

The current FES asks respondents to report their fishing activity over a 2-month period and a 12-month period. In the previous pilot study, we changed the question order and first asked about fishing trips in the previous 12 months.

Switching the sequence of questions resulted in fewer reporting errors and illogical responses, and effort estimates that were generally lower for shore and private boat modes than estimates produced from the current design. However, results varied by state and fishing mode. In addition, based on anecdotal information from angler interviews, we suspect that anglers are eager to report fishing activity, so they may do so at the earliest possible opportunity when questioned, which may lead to reporting trips outside of a given timeframe.

To verify these results and also determine the impacts of increased survey administration, the team determined a follow-up study was necessary. In addition, the pilot study was conducted over 6 months with a smaller sample size than the current FES sample size. The 2024 follow-up study allowed for a full year of side-by-side comparison between the current FES and a revised design, as well as a larger sample size.

How was the FES question order designed in the current survey?

The current FES was developed over several years through a series of pilot studies and was peer reviewed. During the FES development, we also conducted several rounds of cognitive interviews with anglers to identify possible sources of confusion in the questionnaire. Question order bias occurs when responses to a prior question on a questionnaire affect responses to a subsequent one. In some survey designs, questions may need to be asked in a particular order or grouped in a logical way to minimize confusion for respondents.

Peer review of the FES included three separate reviews: a panel of independent survey statistical experts recruited by the Survey Methods Section of the American Statistical Association; five members of the MRIP independent consultant team who had no involvement with the FES development; and the National Academies of Sciences, Engineering, and Medicine’s 2017 MRIP review committee. 

The sequence of questions in the current FES is based on a well-researched, standard practice to ask easier questions prior to more challenging questions. This strategy generally improves response rates and response accuracy. In the case of the FES, recalling fishing activity that occurred during the most recent two months should be easier in theory than recalling fishing activity that occurred during the previous 12 months. The pilot studies conducted show this may not always be the optimal question order design. 

As with all of the surveys we administer, we have extensively tested the FES for various other common survey biases that can impact data accuracy, such as non-response and adequate coverage of the recreational fishing population.

Why We Made the Change to a Mail Survey From a Telephone Survey

In 1979, the Coastal Household Telephone Survey (CHTS) began to collect data about fishing effort by dialing a random sample of residential households in Hawaii and along the Atlantic and Gulf coasts. While random digit dialing (RDD) was a standard sampling methodology for conducting household surveys, there were known limitations to this approach. As these limitations grew more pronounced over time—due in large part to a decline in the use of landline telephones—the accuracy of the survey’s estimates began to suffer.

In its 2006 review of the methods we used to collect recreational fishing data and report recreational catch, the National Research Council recommended fundamental changes to our data collection techniques and acknowledged the limitations of collecting information through random-digit dialing. In response, the Marine Recreational Information Program (MRIP) began to explore ways to improve how we understand and estimate the number of fishing trips taken by shore and private boat anglers in Hawaii and on the Atlantic and Gulf coasts. In 2018, we adopted a new Fishing Effort Survey (FES).

Coverage Error

Since the early 2000s, the percentage of adults living in homes with landline telephones has steadily declined. According to estimates from the 2018 National Health Interview Survey, more than half of all American homes don’t have landline phones. Among homes with landline and wireless phones, 42% receive all or almost all of their calls on a mobile device. In other words, 15% of American homes can be considered wireless-mostly, and 70% have effectively excluded themselves from being sampled by landline-based telephone surveys.

From 2000 on, the exclusion of wireless-only households from the pool of households we sampled meant our samples were growing less and less representative of the general population. Landline-only households, for example, report older residents and fewer children, and are more likely to be composed of single women and to report health and mobility problems. These demographic differences are correlated with differences in fishing activity: the landline-only households—the only households that could be reached by the CHTS—are much less likely to report fishing than the general population.

Measurement Error

The CHTS contacted households without prior notice; asked initial respondents a series of screener questions to determine whether anyone in the household fished during the two-month reference wave; and expected initial survey respondents to immediately describe household-level fishing activity without the benefit of memory cues. Most initial respondents were women, who were much less likely to report household-level fishing activity than men. Under this “gatekeeper effect,” the people who were answering our calls did not always accurately report—and in many cases, underreported—their households’ fishing effort.

Pilot Studies and Peer Review

Between 2007 and 2013, MRIP conducted a series of pilot studies to identify a more accurate and efficient way to estimate fishing effort. These pilot studies provided the basis for the design of the FES.

Methods Tested

• A telephone survey that used fishing license information rather than random-digit dialing as its sampling frame.
• A dual-frame telephone survey that used fishing license information and random-digit dialing as its sampling frame.
• A dual-frame mail survey and a dual-frame, mixed-mode telephone and mail survey that sampled anglers from state databases of licensed saltwater anglers and residential address frames maintained by the U.S. Postal Service.
• A mail survey whose design was revised in an effort to address weaknesses identified in prior studies, increase response rates, and eliminate biases resulting from inaccurate matches to the address frame.

Assessing Non-response Bias

In 2013, a non-response follow-up study demonstrated no significant differences in fishing activity between those who initially responded to the FES and those who responded to a follow-up questionnaire. This suggests non-response to the FES is not a significant source of bias. Routine comparisons between preliminary and final estimates of fishing prevalence also show no significant differences between early and late survey responders.

Results

• The FES provides a more representative sample of the population we survey.
• The FES is less susceptible to bias resulting from non-response and non-coverage.
• The FES gives more household members more time to provide complete answers, which is believed to produce more accurate responses to questions about fishing activity.
• The FES is a more efficient method of collecting fishing data and a superior approach for monitoring recreational fishing effort.
• Differences between CHTS and FES estimates can largely be attributed to differences in fishing activity between the households in each survey’s sample frame. These differences in fishing activity are correlated with differences in demographic characteristics, such as age, gender, and number of children at home.

Peer Review

In 2014, Development and Testing of Recreational Fishing Effort Surveys: Testing a Mail Survey Design (PDF, 56 pages) underwent rigorous peer review. Reviewers included staff from NOAA Fisheries’ Office of Science and Technology, three independent experts selected by the American Statistical Association’s Survey Research Methods Section, and five members of an external expert consultant team. Reviewers provided comments on the methods, results, and conclusions described in the report, and concurred with the overall findings and recommendations to implement a single-phase mail survey design.

In 2015, the design of the FES was certified (PDF, 6 pages) as a scientifically sound and suitable replacement for the CHTS.

Transition from the Telephone Survey to Mail Survey

To minimize the potential for disruptions to fisheries science and management, a cross-disciplinary Transition Team of state and federal partners, scientists, stock assessors, and managers was established to develop a process for transitioning from the CHTS to the FES. The team’s Atlantic and Gulf Subgroup prepared a transition plan and timeline (PDF, 34 pages).

Benchmarking

From 2015 through 2017, the FES and CHTS were conducted side-by-side. During this period, estimates produced by the CHTS were considered the best available for use in scientific assessments. The FES Transition Progress Report (PDF, 10 pages) describes the results from the first full year of this benchmarking period.

Developing a Calibration Model

Between 2016 and 2017, NOAA Fisheries staff and independent expert consultants worked to develop a calibration model to re-estimate statistics produced by the CHTS, which would soon be discontinued as a legacy survey design.

Calibration is a critical step in transitioning to a new recreational fishing survey. Because our work to foster healthy, productive, and sustainable marine fisheries requires a consistent, long-term time series of recreational catch statistics, calibration must be used to place historical estimates into the scale of the new survey design to allow for meaningful comparisons to be made.

Discontinuing the CHTS and Implementing the FES

On December 31, 2017, the CHTS was discontinued. As of January 1, 2018, the FES is used to produce all federal estimates of fishing effort.

In March 2018, Transition Team co-chairs led a NOAA Central Library Brown Bag seminar about the transition process. In July, they delivered a presentation (PDF, 34 pages) via webinar about the transition process and our work to calibrate historical effort estimates.

Re-estimating Historical Recreational Fisheries Statistics

Once our calibration model was peer reviewed and approved, we converted effort estimates dating back to 1981 to the scale of the FES. (A similar peer-reviewed process was used to adjust historical catch estimates following the 2013 transition to an improved Access Point Angler Intercept Survey (APAIS) sampling design.) In 2018, federal stock assessments began to incorporate these calibrated recreational fishing statistics.

Technical Workshops and Reports

In 2019, we participated in a multi-day workshop with the South Atlantic Fishery Management Council’s Scientific and Statistical Committee, held to explain the differences between CHTS and FES-produced estimates. Following the workshop, the committee agreed the FES marks an improvement in our work to measure fishing activity and endorsed the use of estimates calibrated to and produced by the FES in assessing stocks.

Similar presentations were delivered to the Atlantic States Marine Fisheries Commission's Management and Science Committee, and to the Mid-Atlantic Fishery Management Council's Scientific and Statistical Committee.

Implications from the FES Transition

For stocks assessed to date, this increase in effort from the FES in historical catch estimates has generally resulted in a retrospective increase in estimates of fish stock abundance, especially for those fisheries with large recreational components. Learn how recreational catch estimates inform stock assessments, and how stock assessments inform fisheries management.

More Information

• Recreational Fishing Surveys
• FES At-a-Glance
• FES Annual Reports
• FES Outreach and Educational Resources
• MRIP Survey Design and Statistical Methods Manual