Transcript The Shell

Unit -3
Shell Programming
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63
By Narinder Kaur, Asstt. Prof
What is the Shell
• Shell is a user level program.
• It is an command language interpreter that executes commands read
from the standard input device (keyboard) or from a file.
• It is not part of system kernel, but uses the system kernel to execute
programs, create files etc.
• In MS-DOS, Shell name is COMMAND.COM which is also used for
same purpose, but it's not as powerful as Linux Shells are!
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Common Shells
• Sh (1971) – Developed by Ken Thompson
- Introduced the concept of pipes, filters and redirection
Lacked the ability to script.
• Bourne (1977) – Developed by Stephen Bourne
- Introduced the concepts of scripting, control flows, string literals and
command substitution
- Lacked the ability to define functions
• Korn (1978) – Developed by David Korn
(also called ksh)
- Includes features from other shells
- Includes advanced features from Ruby and Python languages like
floating point arithmetic and arrays
- File name completion
- History feature
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Common Shells
•
C (1978) – Developed by Bill Joy
- Provides C language feel
- Advanced feature is Command history
(also called csh)
•
Tenex C (1983) – Developed by Ken Greer
- Command line editing feature
- Command completion
(also called tcsh)
•
Bourne Again Shell – Developed by Brijan Fox
- It combines the features of Korn and C shells
- Introduced some environment variables
- Default shell in Linux.
(also called bash)
- specific shell variables.
- some built-in commands like set and man.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
/etc/shells
The file /etc/shells lists the shells available under Linux.
 It contains a list of all the installed shells, available to all users
 The first entry acts as the default shell
Note : that each shell does the same job, but each understand a different
command syntax and provides different built-in functions.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Login/Logout Scripts
Shell
Bash
Tcsh
Csh
Scripts
.bash_profile: Login initialization file
.bashrc: BASH shell configuration file
.bash_logout: Logout script
.login: Login Initialization File
.tcshrc: TCSH shell configuration file
.logout: Logout file
.cshrc
.login (executed only at login)
.logout (executed only at logout)
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell: Responsibilities
1.
Command Interpretation
2.
Providing a scripting language
3.
A process that provides runtime environment
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Simple architecture of a Shell
User
Lexical
Analysis
Expansion
Execution
Kernel
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Interpretation cycle for shell commands
The Shell
1. Issues the prompt and waits for a command to be
entered
2. Analyses the input and parses it
3. Scans the command line for meta-characters and
performs expansions to give a simple command
4. Passes command line to the kernel for execution
5. Waits for command to execute
6. On completion gives the prompt again; and the cycle
continues
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell: Variables
• Control the environment provided by the shell
• Set via any of the following activities
 After user logs in through login scripts
 As a result of scripts executed after login
 Interactively by user
• Also known as Environment / System variables
• .bash_profile initialization file contains definitions and
assignments of parameter variables.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell: Variables
• Each shell variable is used for a specific purpose
• A shell variable is typically entered in uppercase
• To get its value, you precede the shell variable name with a
$, as in $PWD
• The set command shows the complete list
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Learning about shell variables
Examples of shell variables:
 PWD – the most recent working directory set with the cd
command
 OLDPWD – the previous working directory set with the
cd command
 BASH – the full path of the bash shell
 HOME – each user’s home directory, typically
/home/username, where username is a user’s login name
 HOSTNAME – the current hostname of a Linux system
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Expansion
• Expansion is the process of using meta characters and
special symbols to change the given text
• Simple examples:
 Variable Expansion
$HOME expands into current user’s home directory,
$BASH expands to give the full path of the bash shell
• The types of expansion discussed:
 Parameter and variable
 Command substitution
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Parameter and Variable Expansion
• Parameter expansion substitutes values for parameter or
variable names
• Parameters and variables are names that contain data values
• Example:
 echo $x will display the contents of a variable named x
 Notice=“Meeting will be held tomorrow”
 Echo $Notice
Use of braces {…} allows one to mix variables and
numbers. Example:
echo ${EmployeeName} will display an employee’s name
on the screen
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Expansion – Command Substitution
Command substitution allows you to substitute the output of a
command in place of the command itself
Two forms:
 Use $(command)
Example: echo "Today is " $(date)
 Surround the command with a single back quote as in
`command`
Example : Mycmd = `ls *.c`
echo $Mycmd
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell Script
• Normally shells are interactive i.e. shell accept command from
you (via keyboard) and execute them.
• But if required a set of related commands can be run by
storing them to text file and telling the shell to execute this
text file.
• This is known as shell script.
• Definition:
 "Shell Script is series of command written in plain text
file. Shell script is just like batch file is MS-DOS but have
more power than the MS-DOS batch file."
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Why to write shell script?
• Shell script can take input from user, file and output them on screen.
• Useful to create our own commands.
• Multiple commands can be bound by means of control constructs to
define a logical flow of control.
•
Shell scripts execute commands sequentially.
•
You can alter the sequential order with decisions, loops and
functions
•
Shell scripts
•
combine programming logic with operating system commands
•
help in automating job duties
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Creating shell scripts
• Use any editor like vi or gedit to write shell script.
• After writing shell script set execute permission for your
script as follows
• Syntax:
 chmod permission script-name
 Examples:
$ chmod +x script-name
$ chmod 755 script-name
• Note: This will set read write execute(7) permission for
owner, for group and other permission is read and execute
only(5).
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Executing shell scripts
syntax:
$ sh script-name or
$ ./script-name
Examples:
$ sh scr
$ ./scr
or
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
User defined variables
Syntax:
variable name=value
Rules for naming variables:
 Variable name must begin with Alphanumeric character or
underscore character (_)
 Don't put spaces on either side of the equal sign when
assigning value to variable
 Variables are case-sensitive, just like filename in Linux
 NULL variable (variable which has no value at the time of
definition) can be defined as:
$ vech=
 Do not use ?,* etc, to name your variable names
Variable value can be accessed using $vname
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell Arithmetic
Syntax:
 expr op1 math-operator op2
Examples:
$ expr 1 + 3
$ expr 2 - 1
$ expr 10 / 2
$ expr 20 % 3
$ expr 10 \* 3
$ echo `expr 6 + 3`
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Exit Status
In Linux when a particular command/shell script is executed, it returns
two type of values, signifying success or failure
This value, returned via return statement, is known as Exit Status.
These values can be used to check whether command or shell script
executed successfully or not.
 If return value is zero (0), command is successful.
 If return value is nonzero, command is not successful or some
sort of error executing command/shell script.
$? can be used to find the exit status of last executed command.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Exit status : example
For e.g. (This example assumes that unknownfile does not exist )
$ rm unknownfile
It will show error as follows
rm: cannot remove `unkownfile': No such file or directory
and after that if you give command
$ echo $?
it will print nonzero value to indicate error
The value of $? can be used in scripts for decision making
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell Grammar - I
A blank is a space or tab used to separate items
A word (token) is a sequence of characters considered to be a
single unit to the shell
A name (identifier) is a word consisting of letters, numbers and
underscore
A meta character is a character that, when unquoted, separates
words
A Control Operator is a token that performs control function.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell Grammar - II
Metacharacter: Examples
 | - the pipe symbol allows you to pass output to another




command
& - the ampersand allows you to run a process in the
background
; - the semicolon allows you to sequence commands
< - less than redirects input
> - greater than redirects output
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Shell Grammar - III
Control operators:
 || - executes a command depending upon failure of
another
 && - executes a command depending upon success of
another
A reserved word has special meaning to the shell and cannot
be used for another purpose. Example: if, then, else, fi
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Command Separators
The && Operator
 Causes a command to execute only if the preceding command
completes successfully (exit status of 0)
 Example: rm file4.txt && ls
The ls command will only execute if rm file4.txt
completes successfully
The || Operator
 Causes a command to execute only if the preceding command
completes unsuccessfully (exit status of 1)
 Example: rm file4.txt || pwd
The ls command will only execute if rm file4.txt
completes unsuccessfully
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Operators
Logical
 -a: and
 -o: or
 !: not
Comparison
 -eq, -ne, -lt, -gt, -le, -ge: Numerical Comparison
 =, != : String comparison
 -z: check string against null
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Decision Structures
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63
By Narinder Kaur, Asstt. Prof
if statement
The shell requires you to follow strict syntax when
implementing the if statement
The syntax of the if statement follows:
if [condition]
then
Statements
else
Statements
fi
if statements can be nested if required
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
elif clause in an if statement
The optional elif clause allows you to further test an if
statement
if [condition]
then
Statements
elif [condition]
then
Statements
else
Statements
fi
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
The case statement
Another decision structure
Use when a decision is based upon multiple inputs
Syntax:
case word in
Pattern1)
statements;;
Pattern2)
statements;;
esac
Can use * to match all patterns, ? to match a single character
or […] for a range.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Looping Structures
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
The while statement
A set of statements is executed till a condition evaluates to
true
while [ condition ]
do
statements
done
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
The while statement
Use the break command to end a loop prematurely.
Example:
count=1
while [condition]
do
statements
if [ $count –gt 3 ]
then
break
fi
statements
done
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
The until statement
A set of statements is executed till a condition does not
evaluate to true
until [ condition ]
do
statements
done
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
The for statement
The for statement syntax in the shell takes two forms:
 Word list form:
for variable in list
do
statements
done
 Arithmetic expression form:
for ((var=init; test; incr))
do
Statements
done
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Understanding functions
• Functions are self-contained blocks of code
• Functions can accept values and return a result
• Function code can be reused
• Functions reduce redundancy
• Functions provide for modular code
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Components of a function
•
General format of a function in the shell:
function function-name ()
{
list
}
•
Function name is required
•
Either the keyword “function” or the parentheses are required but can use
both
•
The left and right braces are required
•
The commands in list are any valid shell commands
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Function call
Calling a function:
 Done by using its name
Example:
function DisplayHello ()
{
echo "hello"
}
DisplayHello
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Function basics
• Functions are usually placed at the beginning of the script or
prior to the function call
• A function called prior to its definition will generate a
“command not found” error
• Functions are executed in context of the current shell (unlike
shell scripts)
• Functions can be exported to subshells using export -f
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Function parameters
• On execution the parameters become the positional parameters
• $0 remains unchanged
• Variables are shared between function and its caller
• return returns control back to the caller & the positional parameters are
restored to their original value
• Variables local to the function can be declared using local keyword
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Function : Example
function testFunc ()
{
for i in $*
do
echo $i
done
}
for i in $*
do
echo $i
done
testFunc 1 2 3
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
The return statement
• Use the return command to cause a function return an exit
status
• General format:
 return n
• Where n is the return status you specify
• This value can be checked using $?
• Typically 0 for success and non-zero for indicating cause of
failure
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Arrays
• Array is a variable containing a series of elements
• Refer to elements using an integer called a subscript
• Subscripts begin with 0
• Use an array when you have values that are similar in nature
• All the data in the array can be accessed by referencing the whole array
variable
• A single element can be accesses by using its subscript value
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Declaring an array
• Use the declare -a command to declare an array
• Can use typeset -a command too
• Example to declare an array named courses:
 declare -a courses
 declare -ia codes
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Assigning values to array elements
• After declaring an array, you need to assign values to it
• Example to set the first to elements of the courses array:
 courses[0]="MCA"
 courses[1]="B Tech."
• Another method is to assign multiple elements as in:
 courses=("MCA" "B Tech. ")
 codes=(10 20 30)
• Space acts as the element separator
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using array elements
• Syntax to refer to a single element of an array:
 ${array-name[subscript]}
• Example to display the sixth element (subscript 5) in the
courses array:
 echo ${courses[5]}
• Using array elements as rvalue:
 course=${courses[5]}
• The expression $courses[5]
 is treated as $courses followed by [5]
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Operations on an array
• Reference all of the array elements:
 echo ${array-name[*]}
or
 Example: echo ${courses[*]} or
echo ${array-name[@]}
echo ${courses[@]}
• Determine the number of array elements:
 echo ${#array-name[*]}
or
echo ${#array-name[@]}
 Example: echo ${#food[*]}
or
echo ${#food[@]}
• Determine an element’s length:
 echo ${#array-name[subscript]}
 Example: echo ${#food[2]} will give the length of 3rd element in the
array.
 Similarly echo ${#array}
the array i.e. ${#array[0]}
will give the length of the first lement in
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Getting values from the user
• It is possible to use arithmetic operations and variable name
substitution to determine a subscript number
• Reading i-th element from user:
 read -p "Enter $i the element" arr[$i]
• Displaying i-th element to user:
 echo ${arr[$i]} " "
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Operations on an array
• Search and Replace array elements
• Eg : echo ${unix[@] / mandriva / RedHat}
will replace ‘mandriva’ with ‘RadHat’. But this will not
permanently replace the array relements.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Operations on an array
• Add an element to an existing bash array :
array=(“${array[@]}” “new item”)
• Copying an array
newarray = (“${oldarray[@]}”)
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Operations on an array
• Concatenate two arrays :
array3 = (“${array1[@]}” “${array2[@]}”)
• Deleting an entire array :
unset arrayname
echo ${#array[@]}
# would give 0.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
which command
which <command>
Can be used to locate executables that are in one of the
directories contained in the PATH variable
It prints the full path of the executable that would get executed in
the command was given on the command prompt.
which -a command prints all the occurrences of the command;
not just the first one
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
grep
• Used for matching patterns
• Pattern: built using regular expressions
• Options:

-i: Ignore case

-v: Don't display matching lines

-n: Display line numbers along with lines

-c: Display count of occurrences

-l: Display list of filenames that contain the pattern that is searched in a set of
files
• Variations

egrep : Equivalent to grep -E: use Extended Regular Expressions

fgrep: Equivalent to grep -f: read patterns from file
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Basic regular expressions









A*: Look for 0 or more occurrences of A
. : Look for any one character (?)
.* : Look for any number of characters
^A : Look for A in beginning of string
A$ : Look for A at the end of string
^A$ : Look for A
[abc]: Look for a single character a, b or c
[a-d]: Look for a range
[^a-d]: Look for a character not in this range range
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Extended regular expressions
•
•
•
•
A+ : Matches one or more occurrences of A
A? : Matches zero or one occurrence of A
exp1 | exp2 : Matches exp1 or exp2
( e1 | e2 ) e3 : Matches e1e3 or e2e3
• Use egrep or grep -E to make use of Extended Regular
Expressions
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Stream Editor - sed
• A multipurpose tool
• Combines work of several filters
• Performs non-interactive operations on a data stream (file or input from a
pipeline)
• Uses instructions to work on text
• An instruction combines:
 An address for selecting lines of input
 Action to be taken on them
• General form:
• sed options 'address action' file(s)
• The 'address action' pair is enclosed in single quotes
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Addressing and actions
Addresses in sed can be specified in two ways:
 By specifying line numbers: single or range
 By specifying a pattern enclosed in / /
Actions (commands) in sed are defined by sed's internal family of command.
 Can be used for
 Substitution (s)
 Global substitution (g)
 Duplication (p)
Deletion (d)
Stop searching (q)
Append (a)
Replace lines (c)
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Text Editing : Substituting text
Command: s
Usage:
sed 's/exp1/exp2/' filename
 First occurrence of exp1 will be changed to exp2 in all the lines of text
sed 's/exp1/exp2/g' filename
 All occurrences of exp1 will be changed to exp2 in all the lines of text
sed '---s/exp1/exp2/' filename
 Change will be made in lines addressed by --sed 's/^/exp2/' filename
 Will prefix exp2 in beginning of all lines
sed 's/$/exp2/' filename
 Will prefix exp2 at the end of all lines
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Text Editing : Substituting text
• Some examples :
• $ sed ‘s/mr./Mr./g’ < file1 > file2
will replace all occurrences of mr. with Mr. and place it in file2
• $ sed ‘1,3 s/mr./Mr./’ file
will make changes only in lines from 1 to 3
• $ sed ‘5,$ s/yahoo/google/’ file
will replace yahoo with google in lines from 5 till end
• Echo “hello how are you” | sed ‘s/h/H/g’
will give ouptput as : Hello How are you
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Line Addressing : Example
‘q’ option is used to stop searching after a particular line.
sed '3q' <filename': Prints and quits after 3rd line
sed -n '3p' <filename : Prints the third line
Addressing rules:
 Num : Single line
 Num1, Num2: Range of lines
 $: Last line
 ! for negation: Used before command
Use -e for combining multiple commands
sed -n -e '1,2p' -e '5,6p' <filename>
Will print lines from 1 to 2 and 5 to 6
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Appending and duplicating lines
• For appending lines
sed ‘/pattern/ a\line to append’ filename
• e.g.
• sed ‘3 a\From here the 4th line begins’ file
The above command will append a line after 3rd line.
• For duplicating lines
sed ‘/^$/p’ file
will duplicate all empty lines.
sed ‘1,5 p’ file
will duplicate lines from 1 to 5
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Text replacement
‘c’ option is used to replace lines in a file
sed ‘/pattern/ c\new line’ file
Examples:
sed ‘1 c\This is a new line’ filename
will replace the first line with new line
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Text Editing : Deleting lines
Command: d
Can be used with line / context addressing
sed '/pattern/d' filename
sed '1,3d' filename
Will remove lines from 1 to 3
Sed ‘1, /^ $/d’ file
Will remove header of a file until the first blank line.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
The awk
• An advanced filter
• Available in Linux as gawk (GNU awk)
• Named after its authors:
 Aho, Weiberger and Kernigan
• Important Features
 Operates at field level, i.e. Can easily access, transform and format individual
fields
 Can work on extended regular expressions (ERE)
 Has C-like programming contructs
 Supports user defined variables
 Allows usage of arrays and functions
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Simple filtering using awk
• Like sed, awk can be used to filter out lines (records) on basis of
 Line number; or
 Pattern
• Syntax
• awk options 'selection_criteria {action} files(s)
• Selecting on basis of pattern (fixed or regular expressions)
• awk '/pattern/ {print}' filename
• Selecting on basis of line number
• awk 'NR==n1 {print}' filename
• awk 'NR==n1 {print}' filename // prints range of lines
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Processing individual fields
• awk identifies fields using special variables $1..$n
• $0 signifies the entire record
• These variables are different from the varibles as used by shell to
idntify command line paramerters
• awk 'NR==2 { print $3, $4 }' file
• Specify field separator using -F
• awk -F “:” 'NR==2 { print $3, $4 }' file
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Generating formatted output
awk uses C-like format specifiers; as used in the printf function to
generate formatted output.
Syntax:
awk '{ printf “%5d %10s”, $2, $5 }' filename
Output
 The 2nd field as a 5 column wide number
 The 5th field as a 10 column wide string
The output generated by individual print commands can be redirected
to different files / commands enclosed in double quotes
awk '{ printf “%5d %10s”, $2, $5 > “f1” }' filename
awk '{ printf “%5d %10s”, $2, $5 | “wc” }' filename
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Arithmetic Operators
Computation can be generated on numeric fields using the basic
arithmetic operators: + , - , * , / , %
These operators can be used for both
 Providing selection criteria
 Generating Output
The operations for incrementing / decrementing (pre as well as post)
are also available.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
User defined variables
User defined variables in awk
 Are created automatically without declaration
 Are initialized to 0 to NULL depending upon context.
Can be used in statements for performing arithmetic, comparison or
output operations
As awk commands become more and more complex; the -f option can
be used to fetch an awk program from a file:
awk -f <awk command file name> <input file name>
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Built-in Variables
The built-in variable available with awk are listed as follows:
NR: Cumulative number of lines read
FS: Input field separator
OFS: Output field separator
NF: Number of fields in current line
FILENAME: Current input file
ARGC: Number of arguments in command line
ARGV: List of arguments
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
BEGIN and END sections
Two special sections available with awk can be used for initialization
and cleanup respectively.
One can use BEGIN for
 giving initial values to variables
 Printing headers; etc
Important because the commands given in {} are executed once for
each line of input
Syntax:
awk 'BEGIN{--} {--} END {--}' filename
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using functions
Awk provides various functions for performing string and arithmetic
operations
The parameters are passed using C-like syntax
Special case: parenthesis can be omitted when no parameters are
being passed
Common functions:
 int (x)
 sqrt (x)
 Length / length (x)
 substr (str, start, len)
 index (s1, s2): look for s2 in s1
 split (str, arr, ch): split str and store into arr, use ch as
delimiter
 system(“cmd”)
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Few Additional Commands









tr flags set1 set2
mail
diff, cmp, comm
cut
uniq
split
sort
head, tail
join
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using tr
•The tr command allows you to translate or delete characters
•Reads from Standard Input and write to Standard Output
– “-t” : translates characters of its input that is in first set to the corresponding
character in second set
– “-d” : deletes those characters of its input that appear in the first set.
– “-s” : replaces multiple consecutive occurrences of characters that are in set
one with a single occurrence
•Symbols that can be used are:
–Escape sequences, ranges, character sequences, character classes
–Character Classes: alnum, alpha, digit, lower, upper...
–Example: tr [:lower:] [:upper:] to translate lower case letters to
uppercase
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using mail
•Can be used to send / receive mails.
•Sending mail : mail <username>
•Reading Mail : mail
Prints one line header of each message
•Interactive Commands
# to read a particular mail
+, - to move back & forth
d to delete; u undelete
r to reply
q to quit
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using diff
Gives a formatted output after comparing
 Two files
Gives commands that can be applied to convert one
file to other
 A file and a directory
Compares the given file with a file in the given
directory having the same name.
 Two directories
Compares corresponding files in both directories
“-r” for recursive comparison
Diff <file1> <file2>
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using cmp
• Compares files
• Silent if files are same
• If they are different: reports the byte and line number at which first
difference occurs.
• Options
 “-l” Print byte number & differing byte values for each difference
 “-s” Silent mode, gives no output just returns status
 “-i n:m” skip first n bytes of file 1 and m bytes of file 2
 -”n” Compare at most n bytes
 Can specify offsets for both files if required.
 Cmp <options> <file1> <file2>
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using comm
Compare sorted files
Three sets of outputs
 Lines unique to first file
 Lines unique to second file
 Lines in both files
 “-n” to suppress nth column of output.
comm <file1> <file2>
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using cut
Extract sections of a file based upon a delimiter
Default delimiter: TAB
Options
 “-f” specify field list
 “-c” specify column list
 “-d” specify delimiter
 “-s” suppress lines that don't contain the delimiter
Example
cut -f1,4 -d”:” -s datafile
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using uniq
Extract unique / duplicate lines from a sorted file
Options
 “-i” : ignore case
 “-f” : ignore n fields while comparing
 “-s” : ignore n characters while comparing
 “-c” : print repeat count
 “-d” : print only duplicate entries
 “-u” : print only unique entries
Example
uniq -f2 -cd datafile
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using split
Split a file and generate sub-files
Options
 “-a” : specify suffix length (default 2)
 “-b” : specify bytes per output file (b:512, k:kilo, m:mega)
 “-l” : specify lines per output file
 Can give a prefix for files to be generated as the last option
Example
split -b k <filename> <prefix>
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using sort
Generate sorted output after reading a set of files
Options
 “-b” : ignore leading blanks
 “-f” : fold (ignore) case
 “-n” : numeric sort
 “-r” : reverse sort
 “-c” : check whether input is sorted
 “-k” : specify sort keys
 “-t” : specify separator
 “-o” : specify output file
Example
sort -k4,4 -k3 datafile -o outputfile
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using head
Extract top N bytes / characters from a set of files.
Options:
 “-c N” : First N bytes
 “-n N” : First N lines
 “-q” : Quiet mode – Don't print file name headers
 “-v” : Verbose mode – Print file name headers
Example:
head -n 5 datafile
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using tail
Extract top N bytes / characters from a set of files.
Options:
 “-c N” : First N bytes
 “-n N” : First N lines
If N is written as +3 => extract all after 3rd line / byte
If N is written as -3 => extract last 3 lines / bytes
 “-q” : Quiet mode – Don't print file name headers
 “-v” : Verbose mode – Print file name headers
Example:
tail -n 5 datafile
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using join
• Perform join operation on two files treating each line as a record and each
column as a field.
• A white space separates fields by default.
• If join fields are not specified then the operation is performed using first
field of each file.
• An output record (line) is generated for each pair of input records (lines)
with identical join fields.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Using join

Options:
“-1 N -2 M” : perform join using Nth field of first file and Mth field of
second file.
“-a SIDE” : print unpairable lines from SIDE
“-v SIDE : same as “a”; suppresses joined lines
“-e TEXT” : replace empty fields with TEXT
“-i” : ignore case
“-t CHAR” : specify field separator
“-o” : specify output format

Example:







join -1 2 -2 1 f1 f2 -o 1.2,1.3,2.3
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Short Question
Q.1: Write a shell script which renames all .txt files as .text
files.
Q.2: Explain different type of shells in Linux. Write a shell script
to generate Fibonacci series?
Q.3:Write a shell script which takes a name as parameter and
returns the PID (s) of the processes with that name.
Q.4: What are shell programming functions. Write a function
that adds two numbers.
Q.5: Explain the usage of the test command for checking the
type of a file.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
Long Questions
• Write a shell script which takes a name as parameter and
returns the PID(s) of processes with that name.
• Write a shell program to evaluates an arithmetic expression
like: ./calculator 10 + 20 ,where 10 is op1 and + is operator
and 20 is op2. Using command line argument and operator
may be (+,-,*,/ )
• Write a shell script for copying files. where the source file
exists and has read permission . Check for it else display
error message. Also check the target file should not exist in
the directory
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof
References:
• Sumitabha Das, “Unix Concepts and Applications”,
TMH, 4th Ed., 2009.
• Arnold Robbins, “Linux Programming by Examples
The Fundamentals”, Pearson Education, 2nd Ed.,
2008.
© Bharati Vidyapeeth’s Institute of Computer Applications and Management, New Delhi-63.
By Narinder Kaur, Asstt. Prof