ARCHIVES

Abstract: Anomaly detection (aka outlier analysis) is a step-in data mining that identifies data points, events, and/or observations that deviate from a dataset's normal behaviour. Anomalous data can show critical happenings, such as a technical difficulty, or potential opportunities, for instance a variation in consumer behaviour. Anomaly detection in high dimensional data is becoming a fundamental research problem that has various applications in the real world. However, many existing anomaly detection techniques fail to retain sufficient accuracy due to so-called “big data” characterised by high-volume, and high-velocity data generated by variety of sources. This phenomenon of having both problems together can be referred to the “curse of big dimensionality,” that affect existing techniques in terms of both performance and accuracy. To address this gap and to understand the core problem, it is necessary to identify the unique challenges brought by the anomaly detection with both high dimensionality and big data problems.