Practical 10
Working with Files and Processes
Introduction
This practical will take you through some essential skills for working with files in Linux. You will make use of text editors to create and edit files. You will learn various commands for displaying and searching files as well as replacing and reformatting text using regular expressions. We will then cover creating symbolic links (like shortcuts in windows). Finally, you will learn how to view what processes are running on the system, as well as how to stop them.
Lab Activities
· Lab Activity 1: Text editors
· Lab Activity 2: Dealing with Files – View file content and manipulation
· Lab Activity 3: Regular expressions and sed
· Lab Activity 4: Symbolic Links (symlinks)
· Lab Activity 5: Process management
Lab Activity 1: Text editors
Learning to use a text editor is very important because text (just content without formatting) is used ubiquitously in various areas, such as code, markup, configuration files. You need to learn to use at least one text editor to be to effectively edit configuration files and manage the system. In Unix/Linux, you need to use Linux command line text editors to edit configuration files, create user instructions, view and edit files on remote server. Two of the popular choices are:
· Vim is a powerful command line-based text editor that has enhanced the functionalities of the old Unix Vi text editor. It is one of the most popular and widely used text editors among System Administrators and programmers. That is why many users often refer to it as a programmer’s editor. It enables syntax highlighting when writing code or editing configuration files.
· Nano is an easy to use text editor especially for both new and advanced Linux users. It enhances usability by providing customizable key binding. Nano has the following features:
· Highly customizable key bindings
· Syntax highlighting
· Undo and redo options
· Full line display on the standard output
· Pager support to read form standard input
There are other alternatives, such as GNU Emacs, and other GUI based text editor. They won’t be discussed here.
Look at the man pages for these two tools and spend 15 minutes familiarising yourself with each of the tools. To get a substantial amount of text to work with, we will save a copy of the kernel messages to a file for this exercise using the dmesg command:
$dmesg > ~/ifn507.log
$vi ~/ifn507.log
Alternative you can create a file and name it ifn507.log, use vi then save and quit using:
$ vim flie.txt
:w (to save the file)
:q (to quit)
Try insert, delete, search, replace and save the file contents with following commands
Starting VI
vi filename
Edits filename
vi -r filename
Edits last save version of filename after a crash
vi + n filename
Edits filename and places curser at line n
vi + filename
Edits filename and places curser on last line
vi +/string filename
Edits filename and places curser on first occurrence of string
vi filename file2 …
Edits filename, then edits file2 … After the save, use :n
Ending VI
ZZ or :wq or 😡
Saves and exits VI
:w
Saves current file but doesn’t exit
:w!
Saves current file overriding normal checks but doesn’t exit
:w file
Saves current as file but doesn’t exit
:w! file
Saves to file overriding normal checks but doesn’t exit
:n,mw file
Saves lines n through m to file
:n,mw >>file
Saves lines n through m to the end of file
:q
Quits VI and may prompt if you need to save
:q!
Quits VI and without saving
:e!
Edits file discarding any unsaved changes (starts over)
:we!
Saves and continues to edit current file
Inserting Text
i
Insert before cursor
I
Insert before line
a
Append after cursor
A
Append after line
o
Open a new line after current line
O
Open a new line before current line
r
Replace one character
R
Replace many characters
Deleting Text
x
Delete character to the right of cursor
X
Delete character to the left of cursor
D
Delete to the end of the line
dd
Delete current line
:d
Delete current line
Search for strings
/string
Search forward for string
?string
Search back for string
n
Search for next instance of string
N
Search for previous instance of string
Replace
:s/pattern/string/flags
Replace pattern with string according to flags.
g
Flag – Replace all occurences of pattern
c
Flag – Confirm replaces.
&
Repeat last :s command
Files
:w file
Write to file
:r file
Read file in after line
:n
Go to next file
:p
Go to previos file
:e file
Edit file
!!program
Replace line with output from program
Repeat the exercise with nano
$nano ~/ifn507.log
Alternative you can create a file names ifn507.log then save and quit using:
$nano ifn507.log
:w (to save the file)
😡 (to quite)
Try insert, delete, search, replace and save the file contents with following commands
Nano Shortcuts
CTRL-R
Read file
CTRL-O
Save file
CTRL-X
Close file
ALT-A
Start marking text
CTRL-K
Cut marked text or line
CTRL-U
Paste text
ALT-/
End of file
CTRL-A
Beginning of line
CTRL-E
End of line
CTRL-C
Chow line number
CTRL-W
Find
ALT-W
Find next
CTRL-\
Search and replace
Q1. What is your preferred editor and why?
Lab Activity 2: View and Manipulate File Contents
It is a common task to view the contents of a plain text file. A number of commands/tools allow you to do this:
· cat short for concatenate ~$ cat filename.txt
· more a program to display a page of text at a time ~$ more filename.txt
· less a program to display a page of text at a time, with more features than more ~$ less filename.txt
· head displays a number of lines from the start (head) of a file ~$ head filename.txt
· tail displays a number of lines from the end (tail) of a file ~$ tail filename.txt
Look at the man pages for these five tools, they are all useful for some specific tasks. You can manipulate the displayed results with piping (taking the output of one command and using it for the input of the next command) and additional commands as well.
Q1. Name two differences between more and less?
· One scroll upward and downward
· Less is faster ep. With larger files
· It has many more features
· Less does not need to read the entire file before reading
· Less can run on a variety of terminal bcz it uses termcap or terminfo
Q2. How would you display the first 5 lines of a file /etc/passwd?
/$ head –n5 filename.txt
Q3. How would you display the last 100 lines of a file, and any following lines that get appended?
/$ tail -f –n100 filename.txt
Text Searching
Searching for text within a file is another common task that is often needed to be performed. A number of tools exist for text manipulation and searching. In this activity you will be using grep.
Begin by browsing the man pages for grep. For the time being, it is not necessary to know about extended expressions or regular expressions.
After browsing the man pages, do the following steps:
Q4. Display all lines that contain the word db in the file /etc/nsswitch.conf
$ grep “db” /etc/nsswitch.conf
Q5. Find lines containing monthly in /etc/logrotate.conf, making sure to print 2 lines before and 2 lines after the matching line.
$ grep ‘monthly’ –n2 /etc/logrotate.conf
Q6. Print the number of lines containing false in /etc/passwd
$ grep –c ‘false’ /etc/passwd
Q7. Create (touch) a sequence of filename using the touch command and curly braces, starting with an alphabet character (a to z) and followed by two digit number 00 to 99. Check the total number of file created with wc command.
$ touch {a..z}{0..9}{0..9}
$ ls * | wc –l
Q8. Search syslog (/var/log/syslog) for all activities related to USB. Display only the activity time and date. Hint: you can use cut command
$ grep USB /var/log/syslog | cut –c 1-15
IFN507 Network Systems
P10: Working with Files and Processes
2
Lab Activity 3: Regular Expressions and sed
A regular expression is a formal yet flexible language for describing a set of strings. Regular expressions provide a very fast and concise means of analysing text. Log analysis, file parsing, report generation all benefit from being able to describe strings as regular expressions.
The power of regular expressions for text processing is in pattern matching. Using regular expressions for search and replace is extremely powerful. For example, the command grep cat will let you find every occurrence of the word cat in a file. Regular expressions will let you find every occurrence of “The cat sat on the mat” and replace it with “The dog sat on the log”. This may seem a simple search and replace, but a carefully constructed regular expression will also be capable of handling:
· “The cat sat on the furry, red mat” becomes
“The dog sat on the furry, red log”
· “Sam the cat jumped on the blue mat” becomes
“Sam the dog jumped on the blue log”
Or for more system administration related purposes:
· “131.181.116.0/26” becomes
“131.181.116.0/255.255.255.192”
· “00:12:3F:50:5A:30” becomes
“00;12;3F;50;5A;30”
· “doej:x:1000:1000:John Doe:/home/doej:/bin/bash” becomes
“John Doe has the username doej, Home directory /home/doej, and uses the bash shell”
Regular expressions are expressed in a particular format that could be considered a regular expression language. What really happens is the regular expression is passed to a regex parser which performs the operation. The advantage is many programming and scripting languages provide regex parsers, so learning the basic formation of regular expressions will be useful in any language.
Regular expressions are able to be used in shell scripts with the sed. Because regular expressions are quite complex, only some simple exercises will be done in this practical. Use https://en.wikipedia.org/wiki/Regular_expression a guide for writing regular expressions. Please note that there are a few different regex syntaxes, in this unit we will use POSIX Basic Regular Expression Syntax. Take particular notice of the options to match globally, and for case insensitive matching. Also, take note of the various POSIX character classes.
Introduction to sed
sed is a stream editor. In this practical we will be only using the most basic of sed functionality. Please refer to the lecture slides and the web page http://www.gnu.org/software/sed/manual/sed.html for details. http://www.grymoire.com/Unix/Sed.html is also a resource with some syntax examples that may be of help to you.
A very simple inline replacement example
sed can operate on a file, or stdin (Standard In) via piping. Use echo to pipe the phrase “The cat sat on the mat” into sed to change the phrase to say “The dog sat on the log”. When using piping, it is often best to build the command, one section at a time, for example:
1. First, get the echo to work:
user@host:$ echo “The cat sat on the mat”
The cat sat on the mat
2. Then pipe into sed to do one search and replace:
user@host:$ echo “The cat sat on the mat” | sed -e ‘s/cat/dog/g’
The dog sat on the mat
3. Lastly, add a second expression to change mat to log:
user@host:$ echo “The cat sat on the mat” | sed -e ‘s/cat/dog/g’ -e ‘s/mat/log/g’
The dog sat on the log
4. Alternatively, you can use a single, focussed regular expression:
user@host:$ echo “The cat sat on the mat” | sed -e ‘s/\(.*\) cat \(.*\) mat/\1dog \2log/g’
The dog sat on the log
5. Try and change the phrase “The fat cat sat on the dirty, old mat” using both methods. echo “The cat sat on the mat” | sed -e ‘s/cat/fat cat/g’ -e ‘s/mat/dirty, old mat/g’
echo “The cat sat on the mat” | sed -e ‘s/\(.*\)cat\(.*\)mat/\1fat cat\2dirty, old mat/g’
Using sed on a file
sed can also operate on a file. However, the output will still go to stdout. You can redirect that output into a new file if you wish, or you can have sed make any required changes inline (in the file that is being read).
1. Create a text file using vim or nano with “Red apples grow on trees and are not as nice as green apples which also grow on trees”
2. Using sed, change Red to Green and green to red making sure that the changes are done inside the file created in the previous step. Remember sed and regular expressions are case sensitive:
user@host:$ sed -i -e ‘s/Red/Green/g’ -e ‘s/green/red/g’ file
3. View the file and see if the changes have occurredNote: the -i option stands for inline. This option will update the file with your changes rather than print the changed text to the terminal. Be careful with this option as it will change the original file.
Extracting text from a line
Simple search and replace is not the best use of regular expressions. Regular expressions can be used to extract text from a wider body of text.
Consider the following email signature:
John Doe – j. .au – 3138 4000
It is possible to turn that (or any signature in that format) into:
Name: John Doe
Email: j. .au
Phone: 3138 4000echo “John Doe – j. .au – 3138 4000” | sed -e ‘s/\(.*\) – \(.*\) – \([0-9]\{4\} [0-9]\{4\}\)/Name: \1\nEmail: \2\nPhone: \3\n/g’
Further Regex and sed Exercises
Q1. Create a new file called dog.txt and insert the following text:
I throw the ball and the dog chases it
My cat does not like to fetch the ball because it is not a dog
I threw the ball to the dog and the dog brought the ball back to me
Construct a regular expression that changes every occurrence of ball into frisbee and displays it to the terminal without modifying the file.~$ sed -e ‘s/ball/frisbee/g’ temp.txt
Q2. Download the file order_and_chaos.txt from Blackboard. It is easiest to download from Firefox inside your virtual machine rather than try and copy it across from your host PC.
Using sed and appropriate regular expressions, change every occurrence of order into ORDER, and ever occurrence of chaos into CHAOS. Note, this search and replace should work for mixed capitalisation (ie, be case insensitive). Also, make sure that the original file is not modified. The new document should be written to a new file called new.txt.~$ sed -e ‘s/order/ORDER/gI;s/chaos/CHAOS/gI’ order_and_chaos.txt > new.txt
Lab Activity 4: Symbolic Links (symlinks)
Unix/Linux file systems allow the creation of file ‘links’. These are commonly referred to as symlinks even though file links can be either ‘hard’ or ‘symbolic’ (soft). In most cases a symbolic link is the type of link you want to use, only use hard links when necessary.
A symlink allows you to create one or more pointers to a single real file or directory. When an application or process tries to access a symlink, the filesystem automatically follows the link to the real file and proceeds as normal. The utility of symlinks will become more apparent as you work through the semester.
Try the following simple exercise to demonstrate how symlinks work. Take a look at the man pages for ln
Q1. Using a text editor such as vi/nano, create a file called realfile.txt in your home directory containing any piece of text you like. Exit from vi/nano ensuring that you save the file.
~$ vi realfile.txt
Press i to go to insert mode. Then write anything
Press Esc to go back to command mode
:w to save the document
:q to exit
~$ nano realfile.txt
😡 to quit
~$ touch realfile.txt
Q2. Use the ls command to verify the file was created. If created successfully, using the ln tool to create a symlink to realfile.txt called linkfile.txt within the same directory:
~$ ln –s realfile.txt linkfile.txt
Q3. Use the ls command to verify the symlink was created.
~$ ls
Q4. Use the cat or less command to view the contents of linkfile.txt and realfile.txt. The contents should be exactly the same as linkfile.txt is only a symbolic link to the realfile.txt.
~$ cat linkfile.txt
~$ cat realfile.txt
Lab Activity 5: Process Management
In Linux, all programs and applications are made up of one or more processes. Processes can also spawn, or fork, child processes. In this exercise, you will notice that all processes on the system inherit from init because init is responsible for starting the system, and command interpreters, etc.
A number of commands are used to display the list of all running processes, as well as for killing processes. Look at the man pages for the following commands:
1. ps
2. top
3. kill
4. killall
5. renice
The ps and top commands are used to query the system about currently running processes. Mastering the ps command is vital in managing a Linux system. Each process also has a priority associated with it, called niceness in Linux. It is possible to adjust the priority of a process by using the renice command.
Q1. Display a full listing of processes running under your user account.
-f full-format
-U real user id or name (or –USER
ps –fU ifn507
Q2. Show a long format listing for all entries in the process table for the cron process.
-l long format
-C command name
ps –l -C cron or ps -elf | grep cron
Q3. Display a full, long listing for all processes on the system.
-e all process
ps -fle
Q4. Show a full listing of all process on the system sorted by UID.
–sort=uid specify sort order as: [+|-]key
ps –ef –-sort=uid
Q5. View a listing of only the PIDs for all bash processes
-o pid (or –-format) user-defined format
ps –C bash –o pid
Q6. Explain the meaning of the UID, PID, PPID, and CMD fields for the entries in the ps command. identify two processes that have a parent-child relationship. How did you determine this relationship?
UID: user ID
PID: process ID
PPID: parent process ID
CMD: command with all its arguments as a string
Q7. Display the most processor intensive process on the system in a form that continuously updates the information. Which is the most CPU intensive task? How did you determine that? top
Q8. Open a terminal window, run the following command three times:
$ftp &
Display the process table and check that three entries exist for the ftp process. Using a single command, killall the instances of ftp.
ps –C ftp
killall -9 ftp
Q9. Open a terminal window, run $ ftp & command once. What is the nice value? Set the priority to 10.
ftp &
ps –lU ifn507
renice –n 10 –p (pid)
End of the Practical