Monitoring System Calls for Anomaly Detection in Modern Operating System

Host-based intrusion detection systems monitor systems in operation for significant deviations from normal system behaviour. Many approaches have been proposed in the literature. Most of them, however, make assumptions about the running environment that are not necessarily valid in modern operating systems. One common assumption is that new security prevention mechanisms that are activated by default on modern
operating systems, such as Address Space Layout Randomization and Data Execution Prevention, are not being considered in the analysis. This work is an exploratory study to investigate the impact of novel attacks (trying to overcome these prevention mechanisms) at the system call level.