Operating Systems and Security II
CS 3IS3
Ryszard Janicki
Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada
Acknowledgments: Material based on Information Security by Mark Stamp (Chapter 13.3)
Ryszard Janicki
Operating Systems and Security II 1/17
Trusted Operating System and Trust vs Security
An OS is trusted if we rely on it for Memory protection
File protection Authentication Authorization
Every OS does these things
But if a trusted OS fails to provide these, our security fails ———————————————————————————–
Trust implies reliance Trust is binary
Ideally, only trust secure systems
All trust relationships should be explicit
Security is a judgment of effectiveness
Judge based on specified policy
Security depends on trust relationships
Ryszard Janicki
Operating Systems and Security II 2/17
Trusted Systems and Trusted OS
Trust implies reliance
A trusted system is relied on for security
An untrusted system is not relied on for security
If all untrusted systems are compromised, your security is unaffected
Ironically, only a trusted system can break your security
OS mediates interactions between subjects (users) and objects (resources)
Trusted OS must decide
Which objects to protect and how Which subjects are allowed to do what
Ryszard Janicki
Operating Systems and Security II 3/17
General Security Principles and OS Security
Least privilege – like “low watermark” Simplicity
Open design (Kerchoffs Principle) Complete mediation
White listing (preferable to black listing) Separation
Ease of use
But commercial OSs emphasize features Results in complexity and poor security
Any OS must provide some degree of Authentication
Authorization (users, devices and data) Memory protection
Sharing
Fairness
Inter-process communication/synchronization OS protection
Ryszard Janicki
Operating Systems and Security II 4/17
OS Services
OS Services
users
User interface
Operating system
Part 4 Software 232
Synchronization Concurrency Deadlock Communication Audit trail, etc.
Data, programs, CPU, memory, I/O devices, etc.
Ryszard Janicki
Operating Systems and Security II 5/17
Trusted OS
A trusted OS also provides some or all of User authentication/authorization Mandatory access control (MAC) Discretionary access control (DAC) Object reuse protection
Complete mediation – access control Trusted path
Audit/logs
Ryszard Janicki
Operating Systems and Security II 6/17
Trusted OS Services
users
Synchronization Concurrency Deadlock Communication Audit trail, etc.
User interface
Authentication
Operating system
Data, programs, CPU, memory, I/O devices, etc.
Part 4 Software 234
Ryszard Janicki
Operating Systems and Security II 7/17
Trusted OS Services
OS Services (up) vs Trusted OS Services (down)
User interface
Operating system
User interface
Authentication
Operating system
OS Services
Synchronization Concurrency Deadlock Communication Audit trail, etc.
Part 4 Software 232 —————————————————————-
Data, programs, CPU, memory, I/O devices, etc.
ices
Synchronization Concurrency Deadlock Communication Audit trail, etc.
Data, programs, CPU, memory, I/O devices, etc.
Part 4 Software
Ryszard Janicki
234
Operating Systems and Security II 8/17
users
Trusted OS Serv
users
MAC and DAC
Mandatory Access Control (MAC)
Access not controlled by owner of object
Example: User does not decide who holds a TOP SECRET clearance
Discretionary Access Control (DAC) Owner of object determines access Example: UNIX/Windows file protection
If DAC and MAC both apply, MAC wins
Ryszard Janicki
Operating Systems and Security II 9/17
Object Reuse Protection
OS must prevent leaking of information Example
User creates a file
Space allocated on disk
But same space previously used “Leftover” bits could leak information
Ryszard Janicki
Operating Systems and Security II 10/17
Trusted Path
Suppose you type in your password What happens to the password?
Depends on the software!
How can you be sure software is not evil? Trusted path problem:
“I don’t know how to to be confident even of a digital signature I make on my own PC, and I’ve worked in security for over fifteen years. Checking all of the software in the critical path between the display and the signature software is way beyond my patience. ”
− Ross Anderson
Ryszard Janicki
Operating Systems and Security II 11/17
Audit
System should log security-related events Necessary for postmortem
What to log?
Everything? Who (or what) will look at it? Don’t want to overwhelm administrator Needle in haystack problem
Should we log incorrect passwords? “Almost” passwords in log file?
Logging is not a trivial matter
Ryszard Janicki
Operating Systems and Security II 12/17
Security Kernel
Kernel is the lowest-level part of the OS Kernel is responsible for
Synchronization Inter-process communication Message passing
Interrupt handling
The security kernel is the part of the kernel that deals with security
Security kernel contained within the kernel Why have a security kernel?
All accesses go through kernel
Ideal place for access control Security-critical functions in one location
Easier to analyze and test Easier to modify
More difficult for attacker to get in “below” security functions
Ryszard Janicki
Operating Systems and Security II 13/17
Reference Monitor
Reference Monitor
The part of the security kernel that deals
The part of the security kernel that deals with access control
with access control oMMedeidatiaetseascaccecsessosfosfubsujebcjtesctos toobjoebcjtescts
o Tamper-resistant Tamper-resistant
o Analyzable (small, simple, etc.) Analyzable (small, simple, etc.)
Objects
Subjects
Part 4 Software
241
Reference monitor
Ryszard Janicki
Operating Systems and Security II 14/17
Trusted Computing Base (TCB) and Its Implementation
TCB – everything in the OS that we rely on to enforce security
If everything outside TCB is subverted, trusted OS would still be trusted
TCB protects users from each other
Context switching between users Shared processes
Memory protection for users
I/O operations, etc.
Security may occur many places within OS
Ideally, design security kernel first, and build the OS around it
Reality is usually the other way around Example of a trusted OS: SCOMP
Developed by Honeywell
Less than 10,000 LOC in SCOMP security kernel Windows 10 has more than 60,000,000 lines of code!
Ryszard Janicki
Operating Systems and Security II 15/17
Poor and Better TCB Designs
Poor TCB Design
Poor TCB Design
Hardware
OS kernel Operating system User space
Security critical activities
Problem: No clear security layer
Problem: No clear security layer
Part 4 Software 244 ————————————————————-
Better TCB Design
Better TCB Design
Hardware Security kernel Operating system User space
Security kernel is the security layer
Security kernel is the security layer Part 4 Software 245
Ryszard Janicki
Operating Systems and Security II 16/17
Trusted OS Summary
i
mplies reliance rusted computing
Trust implies reliance
s everything in OS
TCB (trusted computing onforbasee)cisuevreirtytyhinginOS
rything outside
subverted,
we rely on for security
subverted, we still
If everything outside TCB
is subverted, we still have
usted system
trusted system
If TCB subverted, security
y is broken
ware 246
is broken
OS
OS Kernel
Security Kernel
Ryszard Janicki
Operating Systems and Security II 17/17
t i
y
r
B t