A Host-based Anomaly Detection Approach by Representing System Calls as States of Kernel Modules

S. S. Murtaza, W. Khreich, A. Hamou-Lhadj, M. Couture , "A Host-based Anomaly Detection Approach by Representing System Calls as States of Kernel Modules," In Proc. of the 24th IEEE International Symposium on Software Reliability Engineering (ISSRE), Pasadena, CA, USA, 2013.

Despite over two decades of research, high false alarm rates, large trace sizes and high processing times remain among the key issues in host-based anomaly intrusion detection systems. In an attempt to reduce the false alarm rate and processing time while increasing the detection rate, this paper presents a novel anomaly detection technique based on semantic interactions of system calls. The key concept is to represent system calls as states of kernel modules, analyze the state interactions, and identify anomalies by comparing the probabilities of occurrences of states in normal and anomalous traces. In addition, the proposed technique allows a visual understanding of system behaviour, and hence a more informed decision making. We evaluated this technique on Linux based programs of UNM datasets and a new modern Firefox dataset. We created the Firefox dataset on Linux using contemporary test suites and hacking techniques. The results show that our technique yields fewer false alarms and can handle large traces with smaller (or comparable) processing times compared against the existing techniques for the host based anomaly intrusion detection systems.