Cleaning Dirty Data
Acoustic Doppler velocimeters (ADVs) are used for measuring velocity in streams. They emit a sonic signal and measure the reflections off particles passing through a small volume. When measuring near the bottom of cobble-bed streams, spurious reflections can occur from nearby cobbles. This produces negative spikes in the record as shown below. Such measurements would usually rejected, but if you are studying the interaction of flow and biota near the bed, they may be the only measurements that are possible, so you must use them somehow. In the plot below, the spikes look bad, but they comprise only 836 points out of 30,000.
The method we came up with to identify the spikes uses concepts from three different areas of science:
- Differentiation enhances spikes (from signal processing theory);
- Phase-space plots can be used to display the relationship between derivatives (from chaos theory);
- Gaussian
distributed data have an expected maximum of
, called the Universal threshold, where n is the
number of data and 
is an estimate
of the standard deviation (from statistics theory).
When we take a signal with spikes like the one above, differentiate twice, and plot in 3-D phase-space, we find that the good data lie in a cloud within an ellipsoid defined by the Universal threshold and the spikes lie outside of the ellipsoid, as shown below.
Identifying the spikes in this way allows us to remove them and produce the cleaned version of the signal below.
This method can be used for a wide variety of data types. The main requirement is that the data must be roughly Gaussian white noise. This may require pre-processing of the data to remove low-frequency oscillations.
For more details please see:
Goring, D. G.; Nikora, V. I. 2002: Despiking
acoustic Doppler velocimeter records. ASCE Journal of Hydraulic Engineering
128: 117-126.
Goring, D. G.; Nikora, V. I. 2003: Closure to:
Despiking acoustic Doppler velocimeter records. ASCE Journal of Hydraulic
Engineering 129: 487-488.
Last Updated: 17 November 2003
For more information contact: d.goring@mulgor.co.nz