Many compilers are built to optimize generated code execution speed, some also try to optimize generated code size, the quality of error reporting or even the correctness of the compilation process.
In this talk, we present an other class of compilers: compilers that optimize an ill-defined metric, code obfuscation level. These compilers are generally called code obfuscators and they aim at making it difficult to reverse-engineer the compiled code, while keeping its execution time and its code size within reasonable bounds that may be control by user.
Depending on the identified threat (static analysis, dynamic analysis, execution in a virtual machine, in a debugger etc) various counter-measure can be implemented at the compiler level, through direct manipulation of the LLVM-IR, as showcased during the talk:
- opaque predicates - call graph, control flow graph flattening - anti-debugging / anti-vm: small code sequences that detect if, for example, the process is being debugged - integrity checks, checksums - generate intricate, difficult to simplify code sequence in place of simpler ones
Of course, these obfuscations may have an important impact on execution time, and some also require informations or features that are not directly available at the LLVM level. This talks walks though various challenges found when building an LLVM bitcode obfuscator:
- how to inject small to medium size code written in C without relying on an explicit runtime library - how to insert end-of-function marker in the code (e.g. to scan for breakpoints, compute code integrity) - how to inject checks that depends on information produced by the back-end (relocations, final code, etc) - how to create a cross-platform linker wrapper - how to cope with incorrect handling of rare patterns (inst combine, vectorization, APInt)
This talk is both an introduction to LLVM bitcode obfuscation and a showcase of the limitation/possible improvements of LLVM to support that goal. Its is based on the four years of experience gathered when building an industrial tool to obfuscate C/C++/Objective-C code obfuscator.
