ECS 150 – Filesystem Abstraction
Prof. Joël Porquet-Lupine
UC Davis – 2020/2021
Copyright © 2017-2021 Joël Porquet-Lupine – CC BY-NC-SA 4.0 International License /
1 / 19
Introduction
Long-term data storage requirements
User User
(Very) large amounts of non-volatile data
Easy way to find data
Concurrent access from processes OS Controlled sharing between users
Performance CPU Reliability
Processes
Survive power-off, OS crash, process termination
Filesystems (FS)
Abstractions for maximizing the usage of non-volatile storage
Persistent, named data (files)
Hierarchical organization (directories, subdirectories) Performance
Access control
Crash and storage error tolerance
… RAM I/O
Mass-storage
???
Mass-storage
2 / 19
/
Filesystem concepts
File
Abstraction that provides persistent, described data Logical storage unit
Metadata
File attributes added and managed by the OS
Size, creation/modification time, owner, permissions, etc.
Metadata
– size: 42 KiB
– created: 1970-01-01 …
Data
010111011001
101001110001
110111011000
….
File’s content location (index structure)
Data
What a user puts in the file Array of untyped bytes
3 / 19
/
Filesystem concepts
Directory
Directories provide names for files
Groups of named files or subdirectories
Mapping from human-readable file names to file metadata locations
File name
“foo.txt”
directory
music/ 320 work/ 219 foo.txt 871
File number
871
index structure
Storage sectors
4 / 19
/
Filesystem concepts
Naming conventions
Depends on filesystem
Case
Windows traditionally case-insensitive UNIX traditionally case-sensitive
Length
Before, could be quite limited (11 with MS-FAT) Today, usually up to 255 characters
Extension
Before, separated from filename, and meaningful Today, usually part of the filename, and just a hint
$ file innocent_text_file.txt
innocent_text_file: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked
$ file scary_executable_file.x scary_executable_file.x: ASCII text
5 / 19
/
Filesystem concepts
Digression about executables
How does the OS recognize executable files?
Binary executables
Magic number at very beginning of file, to identify its format Example with an ELF executable
Scripts
Shebang as first line of script (#!)
Tells OS which interpreter should be used to run the script Line ignored by interpreter because it’s a comment
Example with a shell script
$ xxd /usr/bin/firefox | head -1
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000 .ELF…………
$ cat test_script.sh #!/bin/sh
echo Hello
$ ./test_script.sh Hello
$ /bin/sh test_script.sh Hello
6 / 19
/
Filesystem concepts
Path
Absolute
Path of file from the root directory
e.g., /home/jporquet/project1/solution
Relative
Path from the current working directory (CWD) CWD stored in process’ PCB
Special entries
2 special entries in each UNIX directory
.: current directory ..: parent directory
7 / 19
/
Filesystem concepts
Hard link
Link from name to metadata location
File name
“foo.txt”
directory
File number
871
index structure
Storage sectors
music/ work/ foo.txt
320 219 871
“bar.txt”
movie/ 523 tmp/ 990 bar.txt 871
8 / 19
/
Filesystem concepts
Hard links and cycles
No difference between an “original” file name and a hard-link to it
$ touch test
$ ls -li test
21892004 -rw-r–r– 1 joel joel 0 2017-03-02 17:15 test
$ ln test test2
$ ls -li test*
21892004 -rw-r–r– 2 joel joel 0 2017-03-02 17:15 test 21892004 -rw-r–r– 2 joel joel 0 2017-03-02 17:15 test2
Creation of a directory loop would make any file tree walker error-prone (e.g. du or fsc k)
Unless keeping track of all the inodes that are being traversed Hard-links to directory are forbidden
$ mkdir mydir
$ ln mydir mydirlink
ln: mydir: hard link not allowed for directory
Solution: special type of files dedicated for links.
9 / 19
/
Filesystem concepts
Soft link (aka symbolic link)
Link from name to alternate name
File name
“foo.txt”
directory
File number
index structure
Storage sectors
music/ 320 871 work/ 219
foo.txt 871
“bar.txt”
movie/ 523 tmp/ 990 bar.txt 042
042
Symlinks can be well-identified
File tree walkers can safely ignore them
POSIX limit: #define _POSIX_SYMLOOP_MAX 8
/path/to/foo.txt
10 / 19
/
Filesystem concepts
Volume
A volume is a collection of physical storage resources that form a logical storage device containing a file-system.
A volume can be: (a) A whole disk
(b) A partition on a disk (c) Multiple disks
(a) (b) (c)
11 / 19
/
Filesystem concepts
Disk partitions
MBR (old) GPT (new)
12 / 19
/
Filesystem concepts
Drive letter assignment (Windows)
Each volume holds a fully independent tree, in its own namespace Letter assignment typical order:
, B:: first and second floppy disk drives
: first active primary partition of the first physical hard drive
Then, first active primary partition of subsequent physical hard drives, etc.
A:
C:
13 / 19
/
Filesystem concepts
Mounting (UNIX)
HDD partition 1 HDD partition 2 trek-1.ucdavis.edu
///
grub
initrd.img
vmlinuz
bin
boot
etc
home …
usr var
Virtual File System
aaa aab
abc …
xyz z99
/
Mountpoints: bin
hda2 => /
hda1 => /boot boot trek-1.ucdavis.edu => /home
etc
home …
usr var
/
grub
initrd.img
vmlinuz
aaa aab
abc …
xyz z99
14 / 19
/
Filesystem concepts
Mounting (UNIX)
Mounting multiple volumes arbitrarily in a single logical hierarchy
$ cat /etc/fstab /dev/sda1 /dev/sda2 /dev/sdb1 /dev/cdrom /dev/sdc1 10.0.0.1:/shared
/ ext3
/boot ext3
/home ext3
/mnt/cdrom iso9660
/mnt/usbkey vfat
/srv/shared nfs
# the following entry would not make much sense (duplicate) but is possible #/dev/sdb1 /mnt/home ext3
15 / 19
/
Filesystem software layers
Application
Application
Application
C Standard Library (Libc)
User-space Kernel-space
to network
Syscall layer
to kernel structures
Virtual File System (VFS)
Network FS
NFS
Block-based FS
Ext2/3/4 VFAT
Pseudo-FS
proc sys
Block buffer cache
Block device driver
Block device driver
SSD
Hardware
Disk
16 / 19
/
Filesystem typical API
Create and delete files
create(filename, mode)
create new (empty) file, including metadata and name in directory
link(existing_filename, new_filename)
create new name for same underlying file as existing filename
unlink(filename)
remove name for a file from its directory
int fd = open(“test”, O_CREAT, 0644); // also creat() but deprecated close(fd);
link(“test”, “test2”);
unlink(“test”);
$ ls -l # After open()+link()
-rw-r–r– 2 joel joel 0 2018-05-22 11:28 test -rw-r–r– 2 joel joel 0 2018-05-22 11:28 test2 $ ls -l # After end of program
-rw-r–r– 1 joel joel 0 2018-05-22 11:44 test2
17 / 19
/
Filesystem typical API
Open and close
open(filename)
prepare to access specified file and return file descriptor close(filename):
release resources associated with open file
*Image from “The Linux Programming Interface” by Michael Kerrisk
18 / 19
/
Filesystem typical API
File access
, write(): sequential reading/writing
: change file’s current position for random access
read()
seek()
fd = open(…)
lseek(fd, 42, SEEK_CUR)
/* Write in open file at offset 42 */ char c = ‘a’;
write(fd, &c, 1);
close(fd);
mmap()
: create mapping between file’s content and memory : destroy mapping
munmap()
fd = open(…)
char *address = mmap(0, len, PROT_WRITE, MAP_SHARED, fd, 0) address[42] = ‘a’;
munmap(address, len);
close(fd);
19 / 19
/