Robust and Transferable Log Anomaly Detection - Liu, Si Tong (My University, 2023)
Maintaining the stability and performance of software and computer system has always been an ongoing question. As system logs are present in almost all computer systems and software, log anomaly detection has become a major method towards troubleshooting system failures and conducting in-depth analysis to identify the underlying causes. In recent years, even though there has been lots of research done on log anomaly detection, most of them ignored two major problems: the presence of noise when acquiring and processing logs, and the cold-start problem when handling a newly onboarded system. These are two practical problems that usually coexist in a real-world scenario and hasn’t been addressed together effectively yet. There were a few works proposed to apply transfer learning in log anomaly detection so that knowledge from a source log dataset can be transferred to a target log dataset, thus alleviating the cold-start problem when handling new systems being onboarded. However, without first solving the noise issue within the log dataset, these methods become impractical in real-world settings. The existence of noise within the source system can greatly impair feature extraction process from the log dataset, leading to a decreased performance of model when detecting anomalies in the target system dataset which might contain noise as well. In this paper, we propose a novel robust and transfer-learning-based method, called LogRT. LogRT utilizes an Attention-based Bidirectional-Long-Short-Term-Memory model during the feature extraction process to extract the contextual information in log sequences where importance of different log events can be learned by the model automatically even if there are noises present in the dataset. Through combining this robust module with a state-of-the-art transfer learning method, domain adaptation, LogRT can apply the valuable information learned from a noisy source system into a target system and provide high performance on detecting anomalies in logs even with the presence of multiple types of noise in both source and target systems. Extensive experimental evaluations demonstrate that LogRT has competitive performance in real-world scenarios.