Software Security: Execution Semantics and Buffer Overflows

Topics

  • Basic Execution Model
    • Compiler workflow
    • Register/memory move and addressing modes
    • Control flow
    • Program Memory Organization
    • Functions
  • Basics of buffer overflow attacks

Goals

  • Simulate the execution of x64 assembly

  • Identify and exploit a basic buffer overflow

Execution Semantics

  • How does high-level code become machine code?
  • What is the basic processor execution model?
  • What are the different register addressing modes?
  • How can you distinguish AT&T vs Intel syntax?
    • AT&T: Look for percent signs, dollar signs, and parentheses
    • Intel: Look for square braces and fewer annotation symbols
  • Memory addressing
    • Endianness
    • How can you read and interpret control flow in assembly?
    • Memory layout
  • What problems do calling conventions solve?
    • Just a convention!
    • Details of System V AMD64 ABI
  • Need to be able to read assembly and simulate execution

Buffer Overflows

  • What is a control flow highjack?
  • What enables buffer overflows?
  • What can they accomplish?
  • How does a basic buffer overflow work?
  • What makes buffer overflows more challenging in practice?
    • How can attacker overcome these challenges