Disentangled Sticky Hierarchical Dirichlet Process Hidden Markov Model for Statistical Anomaly Detection

Abstract

Hidden Markov Models (HMMs) are widely utilized for their strengths in processing sequential time series data across various fields. However, a significant challenge with this approach is the necessity for manual determination of the number of states in advance, as this represents the unknown pattern of data. To address this problem, this paper applies a Bayesian nonparametric approach to outlier detection based on a hidden Markov model with a Disentangled Sticky hierarchical Dirichlet process. Disentangled Sticky Hierarchical Dirichlet Process Hidden Markov Model (DS-HDP-HMM), an extended version of the Hidden Markov Model, adds an additional parameter to the HDP prior on transition matrix to separate the self-persistence probability and the transition probability of the state. These modifications are often more flexible and provide significant improvements in modeling data. This paper aims to develop a Bayesian nonparametric approach for anomaly detection. The results of the experiment demonstrate that the proposed DS-HDP-HMM approach not only automatically estimates the number of states for input data but also yields superior detection rates compared to traditional HMM-based methodologies.