Software Security: Execution Semantics and Buffer Overflows

Topics

• Basic Execution Model
  • Compiler workflow
  • Register/memory move and addressing modes
  • Control flow
  • Program Memory Organization
  • Functions

Goals

• Simulate the execution of x64 assembly

Execution Semantics

• How does high-level code become machine code?
• What is the basic processor execution model?
• What are the different register addressing modes?
• How can you distinguish AT&T vs Intel syntax?
  • AT&T: Look for percent signs, dollar signs, and parentheses
  • Intel: Look for square braces and fewer annotation symbols
• Memory addressing
  • Endianness
  • How can you read and interpret control flow in assembly?
  • Memory layout
• What problems do calling conventions solve?
  • Just a convention!
  • Details of System V AMD64 ABI
• Need to be able to read assembly and simulate execution