The data validation process employed on the files in this database is consistent across data types. The quality-control (QC) checks are performed as part of the processing and errors are rectified before submission to the database manager for review prior to inclusion in the database.
Procedures for Quality Control
The following protocols are used to ensure data quality:
- Instrument identifications, sampling configuration, calibrations, locations, and timing marks are checked with mooring logs and, if necessary, personnel that deployed the instruments.
- Documented procedures and software are used to decode and calibrate data, to derive secondary variables (statistics,salinity, etc.), and to convert data formats.
- The time base of the data is checked for consistency with records of instrument deployment and recovery; data are checked for time gaps. Data recorded out-of-water or during instrument placement or recovery are removed.
- Data sets are examined by scanning statistics and plots of raw data for outliers, sensor or recorder malfunction or impairment, or other artifacts. Causes of sensor and recorder malfunctions are determined to the extent possible in collaboration with the scientific and technical staff that prepared and deployed the instruments.
- Documented procedures and software are used to replace erroneous values with interpolations or pad values, as appropriate. Where possible, fixes for known errors are implemented and documented.
- Data are compared with other data sets to verify time base, calibrations, and sensor function.
- All data-processing steps are documented and recorded, as are a record of any data values that were edited, replaced or deleted, the reasons for these actions, and the algorithms used for interpolation.
The data are reviewed twice prior to publication in this database. Prior to designation as a Best Basic Version (BBV), a data set is reviewed by the database manager. This review verifies:
- Metadata are complete and accurate.
- Data format standards are met.
- Processing records are complete.
- Processing procedures, software policies were followed.
- Identification, editing, and interpolation over outliers is appropriate.
- Sensor and system malfunctions are documented.
Before inclusion in the database, BBV data are subject to a second review by project scientists who exercise the data in combination with other project data, to verify that:
- Data ranges, patterns, and trends are reasonable. If unexpected patterns resulting from instrumentation malfunctions or processing oversights are found, they are investigated and, if necessary, corrected.
- Data and metadata formats are error-free so that data can be used in analysis.
Data are continually assessed as they are used in analysis by USGS and other scientists. If errors or inconsisencies are discovered, the BBV is updated.