Tool to automate popular security technique earns distinguished paper

A new method to automate the setup for a common security technique earned a distinguished paper award at the 2020 Object-Oriented Programming Systems, Languages and Applications Conference (OOPSLA 2020). Working with researchers from the University of Texas at Austin, Prof. Xinyu Wang co-authored “Automated Policy Synthesis for System Call Sandboxing,” which demonstrates how to automatically build sandboxing policies to keep a computer system safe.

Sandboxing is a software management strategy that isolates applications from critical system resources and other programs. It provides an extra layer of security that prevents malware or harmful applications from negatively affecting a system, preventing vulnerabilities in one component from compromising others.

One common sandbox approach is to restrict which system calls different applications can make, since attacks commonly make atypical system calls while compromising a system. Different operating systems have built-in tools to define policies for system calls, specifying which access patterns are allowed. Applications that break the rules have their execution terminated early.

These tools greatly reduce the attack surface of a system, but manually constructing policies is time-consuming and prone to errors. As a result, many applications, including some that accept untrusted user input, opt not to use a system call sandbox.

To make these security practices more accessible, Wang and collaborators developed a technique to automatically construct system call policies for an application.

The group’s technique makes use of static code analysis and program synthesis to construct policies, taking care to ensure they never incorrectly terminate an application while still restricting its system call access as much as possible. They first develop an approximate abstract policy based on an analysis of a program and the system calls it needs to make during an execution, and then translate that into a policy compatible with an operating system’s sandboxing tool.

The tool, called Abhaya, was able to synthesize policies for 674 different Linux and OpenBSD applications. Abhaya’s automatically synthesized policies successfully blocked nearly all known privilege escalation vulnerabilities for the Linux kernel in the past 5 years. The group’s automated policies were also shown to be competitive with policies written by developers, and in some cases were able to reveal developer-confirmed bugs in the original manually-written policies.