COMP 3000 Operating Systems
File Systems and Storage Management (part 1)
Lianying Zhao
Memory vs. Storage
• RAM has (relatively) low capacity, despite higher speed • In-RAM content disappears when powered off
• Why can’t a program run directly on your hard drive?
• Storage device abstractions are not part of a computer architecture
• It’s I/O!
Persistence!
COMP 3000 (Winter 2021)
Source: www.flashmemorysummit.com
2
Why is a Driver Needed?
• Storage devices as an example (simplified):
COMP 3000 (Winter 2021) 3
Types of “Files”
• Regular file
• Directory
• Symbolic link
• FIFO (named pipe)
• Socket
• Device file (block, character)
COMP 3000 (Winter 2021) 4
What is a File Descriptor (fd)?
• A value (non-negative integer), pointing to a data structure in the kernel
• Like indices to arrays of structs
• We are not talking about resource handles in computing in general
• HANDLE, in Windows
• So stdin, stdout and stderr are just special ones among them COMP 3000 (Winter 2021) 5
Tracing down the File Access
• The file descriptor table • perprocess
• The open file table • system-wide
• The i-node table • In-memory copy
COMP 3000 (Winter 2021)
Credit: Michael Kerrisk 6
So What really is an inode?
• A POSIX (VFS) concept
• In some sense, the inode is the file
• Identified by an inode number (unique within a file system)
• inode types: • directory
• regular file
• char device
• block device • (named) pipe • symbolic link • socket
COMP 3000 (Winter 2021) 7
COMP 3000 Operating Systems
Misc.
COMP 3000 (Winter 2021) 8
File Mode Notation (Permission Bits)
• Symbolic mode: e.g., rwx
• Octal (numeric) mode: e.g., 7→111
• Just conversion from octal to binary
• Counting in setuid and setgid
• Mask for setuid: 4000
• When there’s x for u: lower case s
• When there’s no x for u: upper case S
• Mask for setgid: 2000
• When there’s x for g: lower case s
• When there’s no x for g: upper case S
COMP 3000 (Winter 2021) 9
Memory Allocation
• A process gets allocated some space with the initial address space • For dynamic allocation, this is called the heap
• Ways to ask for more memory at runtime, from the OS • Via system calls
• In-process memory management
• The C runtime does it, oftentimes
• malloc() and free() – from the heap, and?
• Write a custom memory allocator, if interested
COMP 3000 (Winter 2021) 10
The Program Break – brk()
• What is the break?
• The address of the first byte beyond the data segment — the location of the
end of the “heap”
• Actually reflecting the size of the data segment
• Two system calls: • brk() – absolute • sbrk() – relative
COMP 3000 (Winter 2021) 11
mmap()
• Very powerful
• Map files or devices into memory
• File mapping – access files like memory
• Device access, depending on the driver
• Anonymous mapping – allocate new space for the process
• Independent of the original heap, but can be treated as a heap by the C runtime
COMP 3000 (Winter 2021) 12
Linking Options and System Calls
• Static linking:
• To avoid dependency issues at the cost of space (disk + memory) • So, no library calls should be made by design
• Dynamic linking:
• Reuse of common functions
• The expected versions must be ensured
COMP 3000 (Winter 2021) 13
Dynamic Library Dependencies
• Symbols
• Just variable names and function names
• Symbol tables
• Various ways to list symbols • readelf
• nm
• objdump
COMP 3000 (Winter 2021) 14
Matching System Calls with Library Calls
• Not 1/1 mapping
• How the runtime library does the job via system calls
• System calls are invoked when any OS service is needed
• In general, I/O-intensive functions tend to generate more system calls • Expensive
• Various ways to do it
• Even including reading documentation
COMP 3000 (Winter 2021) 15
More about Program Binaries
• Relocatable object files (.o) → (archive, ar) → Static libraries (.a) • ↓ (linking, ld)
• Shared libraries (.so) – ELF
• Executable files – ELF
COMP 3000 (Winter 2021) 16