Insecurity in Software
CS 3IS3
Ryszard Janicki
Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada
Acknowledgments: Material based on Information Security by Mark Stamp (Chapters 12.1-12.2)
Ryszard Janicki
Insecurity in Software 1/23
Software Reverse Engineering (SRE)
Software Reverse Engineering
Also known as Reverse Code Engineering (RCE)
Or simply “reversing” Can be used for good…
Understand malware
Understand legacy code . . . or not-so-good
Remove usage restrictions from software Find and exploit flaws in software
Cheat at games, etc.
We assume. . .
Reverse engineer is an attacker Attacker only has exe (no source code) No bytecode (i.e., not Java, .Net, etc.)
Attacker might want to Understand the software Modify (“patch”) the software
SRE usually focused on Windows So we focus on Windows
Ryszard Janicki
Insecurity in Software 2/23
Software Reverse Engineering Tools
Disassembler
Converts executable to assembly (as best it can)
Cannot always disassemble 100% correctly
In general,it is not possible to re-assemble disassembly into working executable
Debugger
Must step through code to completely understand it Labor intensive – lack of useful tools
Hex Editor
To patch (modify) executable file
Process Monitor, VMware, etc.
IDA Pro – good disassembler/debugger
Costs a few hundred dollars (free version exists)
Converts binary to assembly (as best it can) OllyDbg – high-quality shareware debugger
Includes a good disassembler
Hex editor – to view/modify bits of executable
UltraEditis good – freeware
HIEW – useful for patching executable Process Monitor – freeware
Ryszard Janicki
Insecurity in Software 3/23
Why is Debugger Needed?
Disassembly gives static results Good overview of program logic
User must “mentally execute” program Difficult to jump to specific place in the code
Debugging is dynamic Can set break points
Can treat complex code as “black box” And code notalways disassembled correctly
Disassembly and debugging both are required for any serious SRE task
Ryszard Janicki
Insecurity in Software 4/23
SRE Necessary Skills
Working knowledge of target assembly code Experience with the tools
IDA Pro – sophisticated and complex OllyDbg – good choice for beginners
Knowledge of Windows Portable Executable (PE) file format Boundless patience and optimism
SRE is a tedious, labor-intensive process!
Ryszard Janicki
Insecurity in Software 5/23
SRE Example – From The Textbook
We consider a simple example
This example only requires disassembly (IDA Pro used here) and hex editor
Trudy disassembles to understand code Trudy also wants to patch (modify) the code
SRE Example
For most real-world code,would also need a debugger (e.g., OllyDbg)
Program requires serial number Program requires serial number
But Trudy doesn’t know the serial number… But Trudy doesn’t know the serial number. . .
Can Trudy get serial number from exe? Trudy get serial number from executable?
Ryszard Janicki
Insecurity in Software 6/23
Part 4 Software 112
Again, IDA Pro disassembly Again, IDA Pro disassembly
Again, IDA Pro disassembly
And hex view… And hex view. . .
And hex view…
“test eax, eax” is AND of eax with itself
Part 4 Software 115 Part 4 SoSftow,azrero flag set only if eax is 0 115
If test yields 0, then jz is true
Trudy wants jz to always be true
Can Trudy patch exe so jz always holds?
Ryszard Janicki
Insecurity in Software 7/23
SRE Example
Can Trudy patch exe so that jz always true? • Can Trudy patch exe so that jz is always true?
Assembly
test eax,eax xor eax,eax
xor
jz always true!!!
Hex
85 C0 … 33 C0 …
Part 4 Software 117
Ryszard Janicki
Insecurity in Software 8/23
Can edit serial.exe with hex editor • Trudy can edit serial.exe with hex edior
serial.exe
serialPatch.exe
SRE Example
• And save as serialPatch.exe Save as serialPatch.exe
Part 4 Software 11
Any “serial number” now works!
Any “serial number” now works!
Very convenient for Trudy
Very convenient for Trudy
8
Ryszard Janicki
Insecurity in Software 9/23
Part 4 Software
SRE Example
Back to IDA Pro disassembly… • Back to IDA Pro disassembly. . .
serial.exe
serialPatch.exe
Part 4 Software 120
Ryszard Janicki
Insecurity in Software 10/23
SRE Attack Mitigation
Impossible to prevent SRE on open system
Can we make such attacks more difficult?
Anti-disassembly techniques
To confuse static view of code
Anti-debugging techniques
To confuse dynamic view of code
Tamper-resistance
Code checks itself to detect tampering
Code obfuscation
Make code more difficult to understand
Ryszard Janicki
Insecurity in Software 11/23
Anti-disassembly
Anti-disassembly methods include Encrypted or “packed” object code False disassembly
Self-modifying code
Many other techniques Encryption prevents disassembly
Anti-disassembly Example
Anti-disassembly Example
But need plaintext decrypto to decrypt code!
Suppose actual code instructions are Anti-disassembly Example
Same problem as with polymorphic viruses
Suppose actual code instructions are Suppose actual code instructions are
inst1jmp junk inst3inst4… a “dumb”
inst 1
jmp
junk
inst 3
What disassembler sees
WWhathat“daum“bd”udmisabss”emdbislear sesees mbler sees inst1 inst2 inst3 inst4 inst5 inst6 …
inst 4
…
inst 1
inst 2
inst 3
inst 4
inst 5
inst 6
…
This is example of “false disassembly”
This is example of “false disassembly”
This is example of “false disassembly”
Persistent attacker will figure it out
Persistent attacke Ryszard Janicki
r will figure it out
Insecurity in Software 12/23
Persistent attacker will figure it out
Anti-debugging
IsDebuggerPresent()
Can also monitor for
Use of debug registers Inserted breakpoints
Debuggers don’t handle threads well Interacting threads may confuse debugger. . . . . . and therefore, confuse attacker
Many other debugger-unfriendly tricks
Ryszard Janicki
Insecurity in Software 13/23
Anti-debugger Example
o
Again, clever attacker can figure this out
Anti-debugger Example
inst 1
inst 2
inst 3
inst 4
inst 5
inst 6
Suppose when program gets inst 1, it pre-fetches inst 2, Suppose when program gets inst 1, it pre-
…
inst 3, and inst 4
fetches inst 2, inst 3, and inst 4
This is done to increase efficiency
o TShuipspiossedownheentodienbcurgegaesreeexfefcuictieesnicnyst 1, it does not
SupposAe wnhtein-deebubgugegrgeexercuEtexsainmst p1,leit Can we use this difference to confuse the debugger?
pre-fetch instructions
does not pre-fetch instructions Cansh
we u
inst 1
et
inst 2
is di
inst 3
ijnusntk4 ffer
ence
inst 5
to c
inst 6
…
onfuse the
debugger?
Suppose inst 1 overwritesinst 4 in memory
Suppose inst 1 overwrites inst 4 in memory
Part 4 SoTftwharen program (without debugger) will b125e OK inst 4 at same time as inst 1
Then program (without debugger) will be OK since it fetched
since it fetched inst 4 at same time as inst 1
Debugger will be confused when it reaches junk where inst Debugger will be confused when it reaches
4 is supposed to be
junk where inst 4 is supposed to be
Problem if this segment of code executed more than once!
Problem if this segment of code executed Also, self-modifying code is platform-dependent
more than once!
Again, clever attacker can figure this out
o Also, self-mRoydsziafrdyJianigcki c
dIenseicsuritpy linaStoftowarem-dependen1t4/23
Tamper-resistance
Goal is to make patching more difficult
Code can hash parts of itself
If tampering occurs, hash check fails
Research has shown, can get good coverage of code with small performance penalty
But don’t want all checks to look similar Or else easy for attacker to remove checks This approach sometimes called “guards”
Ryszard Janicki
Insecurity in Software 15/23
Code Obfuscation
Goal is to make code hard to understand Opposite of good software engineering Spaghetti code is a good example
Much research into more robust obfuscation Example: opaque predicate
int x,y
:
if((x-y)*(x-y) > (x*x-2*x*y+y*y)){…} The if() conditional is always false
Attacker wastes time analyzing dead code
Code obfuscation sometimes promoted as a powerful security technique
Diffie and Hellman’s original idea for public key cryptography was based on code obfuscation
But public key cryptography didn’t work out that way
It has been shown that obfuscation probably cannot provide strong, crypto-like security
Obfuscation might still have practical uses
Even if it can never be as strong as cryptography
Ryszard Janicki
Insecurity in Software 16/23
Authentication Example
Software is used to determine authentication
Ultimately, authentication is 1-bit decision
Regardless of method used (pwd, biometric, . . . )
Somewhere in authentication software, a single bit determines success/failure
If Trudy can find this bit,she can force authentication to always succeed
Obfuscation makes it more difficult for attacker to find this all-important bit
Ryszard Janicki
Insecurity in Software 17/23
Obfuscation
Obfuscation forces attacker to analyze larger amounts of code Method could be combined with
Anti-disassembly techniques Anti-debugging techniques Code tamper-checking
All of these increase work/pain for attacker But a persistent attacker can ultimately win
Ryszard Janicki
Insecurity in Software 18/23
Software Cloning
Suppose we write a piece of software
We then distribute an identical copy (or clone) to each customers
If an attack is found on one copy, the same attack works on all copies
This approach has no resistance to “Break Once, Break Everywhere” (BOBE)
This is the usual situation in software development
Ryszard Janicki
Insecurity in Software 19/23
Metamorphic Software I
Metamorphism is sometimes used in malware Can metamorphism also be used for good?
Suppose we write a piece of software Each copy we distribute is different
This is an example of metamorphic software Two levels of metamorphism are possible
All instances are functionally distinct (only possible in certain application)
All instances are functionally identical but differ internally (always possible)
We consider the latter case
Ryszard Janicki
Insecurity in Software 20/23
Metamorphic Software II
If we distribute N copies of cloned software One successful attack breaks all N
If we distribute N metamorphic copies, where each of N instances is functionally identical, but they differ internally. . .
An attack on one instance does not necessarily work against other instances
In the best case, N times as much work is required to break all N instances
We cannot prevent SRE attacks
The best we can hope for is BOBE resistance
Metamorphism can improve BOBE resistance Consider the analogy to genetic diversity
If all plants in a field are genetically identical, one disease can rapidly kill all of the plants
If the plants in a field are genetically diverse, one disease can only kill some of the plants
Ryszard Janicki
Insecurity in Software 21/23
Cloning vs Metamorphism
Suppose our software has a buffer overflow Cloned software
Same buffer overflow attack will work against all cloned copies of the software
Metamorphic software
Unique instances – all are functionally the same, but they differ
in internal structure
Buffer overflow likely exists in all instances
But a specific buffer overflow attack will only work against some instances
Buffer overflow attacks are delicate!
Ryszard Janicki
Insecurity in Software 22/23
Metamorphic Software
Metamorphic software is intriguing concept
But raises concerns regarding. . . Software development, upgrades, etc.
Metamorphism does not prevent SRE, but could make it infeasible on a large scale
Metamorphism might be a practical tool for increasing BOBE resistance
Metamorphism currently used in malware So, metamorphism is not just for evil!
Ryszard Janicki
Insecurity in Software 23/23