USGS - science for a changing world

USGS Coastal and Marine Geology Program

South Carolina Coastal Erosion Study, Data Report for Observations, October 2003 - April 2004


Data Processing

Skip navigational side bar

Data processing was conducted using the proprietary software for each instrument, and (or) specialized software developed by the USGS. The proprietary software was often used to download data from the instruments and export the data to ASCII files. Post-processing of the raw binary or ASCII files was accomplished using USGS software developed in the Matlab® (http://www.mathworks.com/) programming language. Most Matlab® M-files used for post-processing are available via the World Wide Web (WWW). Those that are not available via the WWW are included in this report (see the Matlab® m-files page).

Data were first decoded and calibrated from instrument-specific formats and units to the EPIC-standard NetCDF format (http://www.pmel.noaa.gov/epic/) and scientific units. Data were carefully checked for instrument malfunctions and then edited. The beginning and end of each data series were truncated to remove data collected out of water. The data were carefully checked at each stage of processing. After final editing, the best basic version of the data file includes all variables recorded at the basic sampling interval. Best basic versions of most data files in NetCDF format are included in this report (see the Digital Data Files page). Some data types were only processed to the calibration version (such as ABS and sonar). We are developing best basic version methods for these data types.

Low-pass Filter

A low-pass filter was used to remove tidal and higher-frequency fluctuations from the time-series data that sometimes mask smaller fluctuations in the time-series plots that are driven by winds and the density field. The filter, called PL33 (Flagg and others 1976; Beardsley and others, 1985), operates on hourly data values. The digital filter replaces each point with a weighted average of the 33 points on either side of the central point (fig. 47). The filtering reduces the total length of the time series by 66 hours (33 hours on each end). The filter preserves signals at unreduced amplitude that have periods longer than about 50 hours (fig. 48). The half-amplitude point of the filter is at 33 hours and the half power point is at 38 hours. The filter removes more than 99% of the amplitude at the semidiurnal tidal periods and more than 90% of the amplitude at the diurnal tidal periods (table). Although low-pass filtered data is not included in this report, several plots of filtered time series data are available from the Time Series Plots page.

Click on the following figures for larger images in PDF format.

Figure 47. Filter weights applied to the time-series data in the low-pass filter PL33.
Figure 47. Filter weights applied to the time-series data in the low-pass filter PL33.

Figure 48. Amplitude transfer function for the low-pass filter PL33, showing the amplitude response as a function of frequency.
Figure 48. Amplitude transfer function for the low-pass filter PL33,.

Jump to:RD Instruments ADCPTM
            Nortek Aquadopp AP
            SonTek Argonaut-XR
            SonTek ADV
            Paroscientific Digiquartz Pressure Sensor
            Sea-Bird SEACAT
            Sea-Bird MicroCAT
            D&A Instruments OBS
            Aquatec ABS
            Imagenex Rotating Sonar

RD Instruments (RDI) Acoustic Doppler Current Profiler (ADCPTM)

The RDI ADCPTMwater flow observations were processed using Matlab®-based USGS software (available at http://woodshole.er.usgs.gov/operations/stg/Pubs/ADCPtools/adcp_index.htm). The ADCPTM's were normally configured to record data in beam coordinates (rather than earth coordinates). Upon recovery, the ADCPTM data were transferred to a personal computer using a PCMCIA flash memory card. These data were converted to NetCDF format using software available from the ADCPTM Toolbox (above). Matlab® routines were used to check for data quality, flag wild values, truncate the data to remove out of water data at the beginning and end of the deployment, and discard bins that were always exposed above the water surface. Some near-surface bins were not discarded even though the side-beam reflection at times of low tide renders these data invalid, so near-surface ADCPTM data must be interpreted with care. Normally, a 4-beam solution was used to rotate the data to earth coordinates. When one beam was flagged as bad, a 3-beam solution was used. If 2 or more beams were bad, the output file has a fill value. The end result of processing is an EPIC-standard NetCDF data file.

The ADCPTM wave observations were processed using the RDI proprietary software, WavesMon (see www.rdinstruments.com and RDI, 2003), and Matlab®-based Wave Data Processing Toolbox software (available at http://pubs.usgs.gov/of/2005/1211/). The WavesMon software package was used to process the raw binary ADCPTM file and produce a series of data files from which the directional and non-directional wave energy spectra were calculated. These spectra, in addition to the time series of wave parameters, were output as a series of ASCII files. These files were converted to NetCDF format using the Wave Data Processing Toolbox. This toolbox used a series of Matlab® routines to load the data from the WavesMon-generated ASCII-files, flag wild values, truncate data to remove out of water data at the beginning and end of the deployment, and convert the data to an EPIC-standard NetCDF file.

Nortek AquadoppTM Acoustic Profiler (AP)

The Nortek AquadoppTM AP water flow observations were processed using the proprietary AquaPro software from Nortek (see www.nortek-as.com), and Matlab®-based software (available on the Matlab® m-files page) developed by the USGS. AquaPro was used to read the raw binary data file from the instrument and export a series of ASCII files containing instrument setup information, sensor altimetry information, and water flow and pressure data. Matlab® routines were used to check for data quality, flag wild values, convert data from beam to geographical coordinates, truncate data to remove out of water data at the beginning and end of the deployment, and convert the data to an EPIC-standard NetCDF file.

The Wave Data Processing Toolbox (available at http://pubs.usgs.gov/of/2005/1211/) was used to process wave observations from the AquadoppTM. This toolbox loaded the ASCII data generated in AquaPro into Matlab® and performed PUV analysis (see www.nortek-as.com/technotes/PUVWaves.pdf and Gordon and Lohrmann, 2001) to estimate wave parameters. Wild data values were flagged, the data was truncated to remove out of water data at the beginning and end of the deployment, and the data was converted to an EPIC-standard NetCDF file.

SonTek Argonaut-XR (Extended Range) Acoustic Doppler Profiler (ADP)

The SonTek Argonaut-XR ADP water flow and wave observations were processed using the proprietary ViewArgonaut software from SonTek (see www.sontek.com), and Matlab®-based software (available on the Matlab® m-files page) developed by the USGS. ViewArgonaut read the raw binary data file from the instrument and exported a series of ASCII files containing instrument setup information, sensor altimetry information, and water flow, pressure, and wave measurement data. For water flow data, Matlab® routines were used to check for data quality, flag wild values, convert data from beam to geographical coordinates, truncate data to remove out of water data at the beginning and end of the deployment, and convert the data to an EPIC-standard NetCDF file. Wave measurement data were processed using the Wave Data Processing Toolbox (available at http://pubs.usgs.gov/of/2005/1211/). This software is a toolbox to congregate processed files into a standard NetCDF format, document metadata, and allow dissemination of the data in a common format.

SonTek Acoustic Doppler Velocimeter (ADV)

The SonTek ADV water flow data were also processed using the Hydratools Toolbox- developed by the USGS. The software organized the data files, gathered metadata, converted the data from the SonTek raw binary format to NetCDF format, checked data quality, removed noise and bad data, rotated the water flow data from beam to geographic coordinates, and converted the data to an EPIC-standard NetCDF file.

Paroscientific (Paros) Digiquartz® Pressure Sensors

The Paros Digiquartz® Pressure sensors deployed for this report were external sensors that were attached to the ADVs (above) and PCADPs (above). As such, their data were processed simultaneously with the ADV and PCADP data. The USGS Hydratools software checked the pressure data for quality and removed any data that were bad. The data were then written to the EPIC-standard NetCDF files for the instrument (ADV or PCADP) to which the sensors were attached.

Sea-Bird SEACAT 16

SEACAT data are stored internally. After recovery of the SEACAT, SEASOFT programs (Sea-Bird Electronics, Inc.) were used to read the data into a file on a personal computer, convert the data to calibrated oceanographic units, calculate salinity and density, and write the data to ASCII files. ASCII files were translated to EPIC-standard NetCDF files, and the data were edited for bad data and truncated to remove out of water data at the beginning and end of the deployment.

Sea-Bird MicroCAT

MicroCAT data were processed in the same manner as SEACAT data (above). After recovery of the MicroCAT, SEASOFT programs (Sea-Bird Electronics, Inc.) were used to read the data into a file on a personal computer, convert the data to calibrated oceanographic units, calculate salinity and density, and write the data to ASCII files. ASCII files were translated to EPIC-standard NetCDF files, and the data were edited for bad data and truncated to remove out of water data at the beginning and end of the deployment.

D&A Instruments Optical Backscatter Sensor (OBS)

The D&A Instruments OBS-3s deployed for this report were external sensors that were attached to the ADVs (above) and PCADPs (above). As such, their data were processed simultaneously with the ADV and PCADP data. The USGS Hydratools software checked the OBS data for quality and removed any data that were bad. The data were then written to the EPIC-standard NetCDF files for the instrument (ADV or PCADP) to which the sensors were attached.

Aquatec Acoustic Backscatter Sensor (ABS)

The Aquatec ABS data were processed using Matlab®-based software (available on the Matlab® m-files page) developed by the USGS. This software decoded the Aquatec raw binary file, flagged wild data values, truncated out of water data at the beginning and end of the deployment, and converted the data to an EPIC-standard NetCDF file.

Imagenex Rotating Sonar

The Imagenex rotating sonar data were processed using Matlab®-based software (available on the Matlab® m-files page) developed by the USGS. This software decoded the Imagenex raw binary file, created images from the data, and converted the images to an EPIC-standard NetCDF file. The images were then animated using the VideoMach software (available at www.gromada.com).

adobe reader To view files in PDF format, download free copy of Adobe Reader.

Jump to top of page
Return to Biological Fouling. Go on to Data Files Identifiers.

Skip Navigation

Accessibility FOIA Privacy Policies and Notices

U.S. Department of the Interior | U.S. Geological Survey | Coastal and Marine Geology

URL: cmgds.marine.usgs.gov/publications/of2005-1429/WEBPAGES/data_processing.html
Page Contact Information: CMGDS Team
Page Last Modified: Wednesday, 06-Dec-2017 12:52:08 EST