Monitoring System Calls for Anomaly Detection in Modern Operating System

Shayan Eskandari, Wael Khreich, Syed Shariyar Murtaza, Abdelwahab Hamou-Lhadj, Mario Couture, "Monitoring System Calls for Anomaly Detection in Modern Operating Systems," In Proc. of the 24th IEEE International Symposium on Software Reliability Engineering (ISSRE), Pasadena, CA, USA, 2013.

Host-based intrusion detection systems monitor systems in operation for significant deviations from normal system behaviour. Many approaches have been proposed in the literature. Most of them, however, make assumptions about the running environment that are not necessarily valid in modern operating systems. One common assumption is that new security prevention mechanisms that are activated by default on modern
operating systems, such as Address Space Layout Randomization and Data Execution Prevention, are not being considered in the analysis. This work is an exploratory study to investigate the impact of novel attacks (trying to overcome these prevention mechanisms) at the system call level.