Transcript Presentation Title - Information Technology Services

Linux Intermediate
ITS Research Computing Center
C. D. Poon, Ph.D.
Email: [email protected]
Outline
 Linux Command Category
 Stdout/Stdin/Stderr, Pipe and Redirection,
Wildcards
 Linux Command Review
 Break
 Tips and Tricks
 Conclusion
 Exercise
its.unc.edu
2
Linux Command Category
its.unc.edu
3
Linux Command Category
 Communication
ssh scp
 File/Directory Management
cat cd chmod cp ln ls mkdir more less
mv pwd dirs rm head tail wc
 Comparisons
diff
its.unc.edu
4
Linux Command Category
Cont’d
 Searching
grep find locate
 Archiving
compress uncompress gzip gunzip zcat
tar
 Text Processing
cut paste sort sed awk
its.unc.edu
5
Linux Command Category
Cont’d
 System Status
chgrp chown date df du env who w
uptime
 Miscellaneous
bc cal clear man
its.unc.edu
6
Stdout/Stdin/Stderr
Pipe and Redirection
Wildcards
its.unc.edu
7
stdout stdin stderr
 Output from commands
• usually written to the screen
• referred to as standard output (stdout)
 Input for commands
• usually come from the keyboard (if no arguments are
given
• referred to as standard input (stdin)
 Error messages from processes
• usually written to the screen
• referred to as standard error (stderr)
its.unc.edu
8
Pipe and Redirection
 Pipe (|): stdout of one command to stdin of
another command
 Output Redirection (>): stdout of a
command to a file
 Output Appending (>>): stdout of a
command appending to a file
 Input Redirection (<): stdin of a command
 Error Redirection (& in tcsh/csh, 2> in
ksh/bash): stderr of a command
its.unc.edu
9
Wildcards
 Multiple filenames can be specified using special
pattern-matching characters. The rules are:
• ‘?’ matches any single character in that position in
the filename
• ‘*’ matches zero or more characters in the filename.
• ‘[…]’ Characters enclosed in square brackets match
any name that has one of those characters in that
position
 Note that the UNIX shell performs these expansions
before the command is executed.
its.unc.edu
10
Linux Command Review
its.unc.edu
11
ssh
 Log on to remote machine
 Examples
ssh [email protected]
ssh emerald.isis.unc.edu –l cdpoon
ssh topsail
ssh –X cedar.isis.unc.edu
ssh –Y tarheelgrid.unc.edu
its.unc.edu
12
ssh using SecureCRT
in Windows
 Using ssh, login to Emerald, hostname
emerald.isis.unc.edu
 To start ssh using SecureCRT in Windows,
do the following.
• Start -> Programs -> Remote Services -> SecureCRT
• Click the Quick Connect icon at the top.
• Hostname: emerald.isis.unc.edu
• Login with your ONYEN and password
its.unc.edu
13
scp
 Copy files and directories to and from
remote computers
 Examples
scp file1 topsail.isis.unc.edu:/ifs1/home/cdpoon/.
scp zircon.its.unc.edu:/home/cdpoon/file2 .
scp –r dir1 emerald.isis.unc.edu:/netscr/cdpoon/.
scp –r topsail.isis.unc.edu:/ifs1/scr/cdpoon/dir2 dir3
scp emerald:/netscr/cdpoon/f topsail:/ifs1/scr/cdpoon/.
its.unc.edu
14
cat
 Read one or more files and print output to
stdout
 Examples
cat
cat
cat
cat
file1
file1 file2 file3 > file_all
file4 >> file_all
Append file4 to file_all
> file5
Create file at stdin, end with EOF (^D normally, use “stty –a” to find out)
cat > file6 << STOP
Create file at stdin, end with STOP
Here Document
its.unc.edu
15
cd
 Change directory, built-in shell command
 Examples
cd /afs/isis/home/c/d/cdpoon
cd ../../
cd ..
Change directory to 2 levels up
cd ~
cd
cd –
Change directory to Home
its.unc.edu
Change directory to 1 level up
Change directory to Home
Change to previous directory
16
chmod
 Change the access mode of one or more
files
 Examples
chmod u+x file1
chmod go-w file2
chmod u=rwx, g=rx, o=x file3
chmod 751 file3
Same as above, 7=rwx, 5=rx, 1=x
chmod =r file4
chmod 444 file4
Same as above, 4=r, 2=w, 1=x
its.unc.edu
17
cp
 Copy a file/dir to another file/dir
 Examples
cp file1 file2
Copy to the same directory and change filename
cp file1 ../dir/file2
cp file1 ../dir/.
Copy to different directory and change filename
cp –r dir1 dir2
cp –r dir1 new_dir/dir2
Copy directory recursively
cp –p file3 file4
its.unc.edu
Keep the same filename
Copy directory recursively to another directory
Preserve the modification time and permission modes
18
ln
 Create links for file/dir and allow them to
be accessed by different names
 Examples
ln file1 file2
Hard link for file
ln dir1 dir2
Hard link not allowed for directory
ln –s dir1 dir2
ln –s file3 file4
ln –s dir/file5 file6
Symbolic/Soft link for directory, dir2 -> dir1
its.unc.edu
Symbolic/Soft link, file4 -> file3
Symbolic/Soft link, file6 -> dir/file5
19
ls
 List all files and directories in the current
directory
 Examples
ls
ls
ls
ls
ls
–a
–l
–lh
–F
List files/directories starting with “.” too
Long listing
List file sizes in human readable format
Flag filenames by appending / to directories, * to executables files,
and @ to symbolic links
its.unc.edu
20
mkdir
 Create one of more directories
 Examples
mkdir dir1
mkdir –p dir1/dir2/dir3
Create intervening parent directories if they don’t exist
Same as mkdir dir1; cd dir1; mkdir dir2; cd dir2; mkdir dir3; cd ../../
its.unc.edu
21
more
 Display files on a terminal, one screen at a
time
 Examples
more file1
Hit space bar for another page, q to quit
more –d file2
Display the prompt “Press space to continue, ‘q’ to quit
more –c file3
Page through the file by clearing each window instead of
scrolling
its.unc.edu
22
less
 Works like “more” but allows backward
and forward movement
 Examples
less file1
Hit space bar for another page, q to quit
Hit b to scroll backward one window
Hit /pattern to highlight “pattern” in the text
Hit Return to scroll one line at a time
its.unc.edu
23
mv
 Move files and directories within the same
machine and/or rename them
 Examples
mv file1 dir1/file1
Move file1 to dir1, Same as mv file1 dir1/
mv file3 file4
Rename file3 to file4
mv dir1 dir2
Rename directory dir1 to dir2
mv dir3 dir4/dir5/dir6
Rename directory dir3 to dir6 and move to
dir4/dir5 directory
its.unc.edu
24
pwd dirs
 Print the full pathname of the current
directory
 Examples
pwd
dirs
C shell built-in command, works like “pwd”
dirs –l
Print working directory in long listing
its.unc.edu
25
rm
 Delete one or more files and directories
 Delete empty directory with “rmdir”
 Examples
rm file1
rm file*
Remove all files with filename starting as “file”
rm –I file*
Prompt for y (remove the file) or n (do not remove the file)
rm –r dir1
Delete directory “dir1” and its content
its.unc.edu
26
head tail
 Print first/last few lines of one or more
files
 Examples
head file1
Print the first 10 lines of file “file1”
head –n100 file2
Print the first 100 lines of file “file2”
tail file*
Print the last 10 lines of files with filename
starting as “file”
tail –f file3
Print the last 10 lines of file “file3” and follow file
as it grows
its.unc.edu
27
wc
 Print a character, word, and line count for
files
 Examples
wc –c file1
Print character count for file “file1”
wc –l file2
Print line count for file “file2”
wc –w file3
Print word count for file “file3”
its.unc.edu
28
diff
 Report lines that differ between file1 and
file2
 with file1 text flagged by < and file2 by >
 Examples
diff file1 file2
its.unc.edu
Show difference between file1 and file2
29
grep
 Search for lines that match a regular
expression
 Examples
grep abc file1
Print line(s) in file “file1” with “abc”
grep –i abc file2
Print line(s) in file “file2” with “abc” ignoring
uppercase and lowercase distinctions
its.unc.edu
30
find
 Find particular groups of files
 Examples
find . –name temp
Find file named “temp” in current directory
find /etc –name ‘rc*’
Find file(s) in /etc directory with name
starting with “rc”
find /usr/share/man –type d –name ‘man*’
Find directories in /usr/share/man with name starting with “man”
its.unc.edu
31
locate
 Find files with matching pattern in
database prepared by updatedb
 Database needed to be updated
 Examples
locate which
Find files named with pattern “which” in the OS
locate –c which
Count number of files named with pattern “which” in
the OS
locate –i which
Find files named with pattern “which” in the OS
ignoring case distinctions
its.unc.edu
32
compress uncompress
 Reduce or expand the size of one or more
files using adaptive Lempel-Ziv coding
 Use uncompress to expand data
 Examples
compress file1
Reduce the size of file1 and create new file
named file1.Z
compress –f file2
Force to reduce the size of file2 and create new
file named file2.Z
uncompress file3.Z
Expand file3.Z and restore file3
its.unc.edu
33
gzip gunzip
 Reduce or expand the size of one or more
files using Lempel-Ziv coding (LZ77)
 Use gunzip to expand data
 Examples
gzip file1
Reduce the size of file1 and create new file
named file1.gz
gzip –f file2
Force to reduce the size of file2 and create new
file named file2.gz
gunzip file3.gz
Expand file3.gz and restore file3
its.unc.edu
34
zcat
 Expand the size of one or more files
created by compress or gunzip
 List file contents to stdout without
deleting the .Z or .gz file
 Examples
zcat file1.Z
Expand file1.Z and list the content of file1 in stdout
zcat file2.gz
Expand file2.gz and list the content of file2 in stdout
its.unc.edu
35
tar
 Archive files and directories
 Create a single file with extension .tar
 Examples
tar –cvf file123.tar file1 file2 file3
Create archive file named file123.tar in verbose mode
with contents, file1, file2, and file3
tar –xvf file123.tar
Expand file123.tar in verbose mode and generate the
original files and directories back
its.unc.edu
36
cut
 Remove sections from each line of files
 Examples
cut –d: -f1,5 /etc/passwd
Use field delimiter “:” to locate fields 1 and 5 from file /etc/passwd
to extract usernames and real names
cut –k4 file1
Take column 4 out from each line of file1 and display in stdout
its.unc.edu
37
paste
 Write to stdout consisting of sequentially
corresponding lines of each given file
 Examples
$ cat file1
$paste file1 file2
1
1
a
2
2
b
$ cat file2
c
a
b
c
its.unc.edu
38
sort
 Sort lines of text files
 Examples
sort –fd file1
Alphabetize lines (-d) in file1 and ignore lower and upper cases (-f)
sort –t: -k3 -n /etc/passwd
Take column 3 of file /etc/passwd separated by “:” and sort in arithmetic order
its.unc.edu
39
sed
 Edit one or more files without user
interaction using stream editor
 Examples
sed s/xx/yy/g file1
Substitude all occurrences of “xx” in file1 with “yy” and display on stdout
sed /abc/d file3
Delete all lines containing “abc” in file3
sed /efg/!d file4
Delete all lines not containing “efg” in file4
sed /BEGIN/,/END/s/XYZ/xyz/g file5
Substitute “XYZ” on lines between BEGIN and END with “xyz” in file5
its.unc.edu
40
awk
 Process files by pattern-matching
 Examples
awk –F: ‘{print $1}’ /etc/passwd
Extract the 1st field separated by “:” in /etc/passwd and print to stdout
awk ‘/abcde/’ file1
Print all lines containing “abcde” in file1
awk ‘/xyz/{++i}; END{print i}’ file2
Find pattern “xyz” in file2 and count the number
awk ‘length <= 1’ file3
Display lines in file3 with only 1 or no character
its.unc.edu
41
chgrp
 Change the group ownership of one or
more files or directories
 Examples
chgrp employee file1
Change group ownership to “employee” for file “file1”
chgrp –R student dir1
Change group ownership to “student” for directory “dir1” including
subdirectories recursively
its.unc.edu
42
chown
 Change the ownership of one or more files
or directories
 Examples
chown employee file1
Change ownership to “employee” for file “file1”
chown –R student dir1
Change ownership to “student” for directory “dir1” including
subdirectories recursively
its.unc.edu
43
date
 Print the current date and time in certain
format
 Set the current date and time
 Examples
date
Print the current date and time
date +%D
Print the current date and time in mm/dd/yy format
date 1201160108
Set the current date and time to Dec 01 4:01pm 2008
date –d fri
Show the date of the coming Friday
its.unc.edu
44
df
 Report the number of used and free disk
block on all mounted file systems
 Examples
df
Print used and free disk block on all mounted file system
df -k
Print used and free disk block in kilobyte
its.unc.edu
45
du
 Print disk usage of directories and its
subdirectories
 Examples
du dir1
Print disk usage in kilobyte of directory “dir1”
du –-block-size=1M dir2
Print disk usage in megabyte of directory “dir2”
its.unc.edu
46
env
 Display the current environment variables
or set new values
 Examples
env
Display all of the current environment variables
its.unc.edu
47
who
 Display information about the current status of
the system
 Examples
who
Display the names of users currently logged in to the system
who –b
Report information about the last reboot
who am I
Print the username of the invoking user
its.unc.edu
48
w
 Print summaries of system usage, currently
logged-in users, and what they are doing
 Examples
w
Print summaries of system usage, currently logged-in users
w –s
Display in short form
its.unc.edu
49
uptime
 Print the current time, amount of time
logged in, and the system load averages
 Examples
uptime
Print a one line display of the current time, how long the system has been
running, how many users are currently logged on, and the system load averages for
the past 1, 5, 15 minutes
its.unc.edu
50
bc
 Interactively perform arbitrary-precision
arithmetic or convert numbers from one
base to another, type “quit” to exit
 Examples
bc
1+2
5*6/7
ibase=8
20
16
ibase=10
its.unc.edu
Invoke bc
Evaluate an addition
Evaluate a multiplication and division
Change to octal input
Evaluate this octal number
Output decimal value
Change back to decimal input
51
cal
 Print calendar of a month or all
months in a year
 Examples
cal
Print calendar of the current month
cal 12 2008
cal 2008
cal -3
Print calendar of December 2008
its.unc.edu
Print calendar of all months in 2008
Display previous/current/next months
52
clear
 Clear the terminal display and have the
prompt locate at the top of the terminal
window
 Examples
clear
its.unc.edu
Clean up the current terminal display
53
man
 Display information from the online
reference manuals
 Examples
man man
Display the manual for the command “man”
man –k link compile
Display commands related to linking and compiling
using a keyword search
its.unc.edu
54
Break
its.unc.edu
55
Tips and Tricks
its.unc.edu
56
Tips and Tricks #1
 Show files changed on a certain date in all
directories
 Example
ls –l * | grep ‘Sep 26’
Show long listing of file(s) modified on Sep 26
ls –lt * | grep ‘Dec 18’ | awk ‘{print $9}’
Show only the filename(s) of file(s) modifed on Dec 18
its.unc.edu
57
Tips and Tricks #2
 Sort files and directories from smallest to
biggest or the other way around
 Example
du –k –s * | sort –n
Sort files and directories from smallest to biggest
du –ks * | sort –nr
Sort files and directories from biggest to smallest
its.unc.edu
58
Tips and Tricks #3
 Change timestamp of a file
 Example
touch file1
If file “file1” does not exist, create it, if it does, change the timestamp of it
touch –t 200801011200 file2
Change the time stamp of file “file2” to 1/1/2008 12:00
its.unc.edu
59
Tips and Tricks #4
 Find out what is using memory
 Example
ps –ely | awk ‘{print $8,$13}’ | sort –k1 –nr | more
its.unc.edu
60
Tips and Tricks #5
 Remove the content of a file without
eliminating it
 Example
cat /dev/null > file1
its.unc.edu
61
Tips and Tricks #6
 Backup selective files in a directory
 Example
ls –a > backup.filelist
Create a file list
vi backup.filelist
Adjust file “backup.filelist” to leave only filenames of the files to be backup
tar –cvf archive.tar `cat backup.filelist`
Create tar archive “archive.tar”, use backtics in the “cat” command
its.unc.edu
62
Tips and Tricks #7
 Get screen shots
 Example
xwd –out screen_shot.wd
Invoke X utility “xwd”, click on a window to save the image as “screen_shot.wd”
display screen_shot.wd
Use ImageMagick command “display” to view the image “screen_shot.wd”
Right click on the mouse to bring up menu, select “Save” to save the image to
other formats, such as jpg.
its.unc.edu
63
Tips and Tricks #8
 Sleep for certain time, then pop up a
message
 Example
(sleep 60; xmessage –near One Minute Has Gone By) &
its.unc.edu
64
Tips and Tricks #9
 Count number of lines in a file
 Example
cat /etc/passwd > temp; cat temp | wc –l; rm temp
wc –l /etc/passwd
its.unc.edu
65
Tips and Tricks #10
 Create gzipped tar archive for some files in
a directory
 Example
find . –name ‘*.txt’ | tar –c –T - | gzip > a.tar.gz
find . –name ‘*.txt’ | tar –c –files-from=- | gzip > a.tar.gz
find . –name ‘*.txt’ | tar –cz –T - -f a.tar.gz
its.unc.edu
66
Tips and Tricks #11
 Find name and version of Linux
distribution, obtain kernel level
 Example
uname -a
head –n1 /etc/issue
its.unc.edu
67
Tips and Tricks #12
 Show system last reboot
 Example
last reboot | head –n1
its.unc.edu
68
Tips and Tricks #13
 Combine multiple text files into a single
file
 Example
cat file1 file2 file3 > file123
cat file1 file2 file3 >> old_file
its.unc.edu
69
Tips and Tricks #14
 Create man page in pdf format
 Example
man –t man | ps2pdf - > man.pdf
acroread man.pdf
its.unc.edu
70
Tips and Tricks #15
 Remove empty line(s) from a text file
 Example
awk ‘NF>0’ < file.txt
Print out the line(s) if the number of fields (NF) in a line in file
“file.txt” is greater than zero
awk ‘NF>0’ < file.txt > new_file.txt
Write out the line(s) to file “new_file.txt if the number of fields (NF)
in a line in file “file.txt” is greater than zero
its.unc.edu
71
Conclusion
its.unc.edu
72
Conclusion
 Many ways to do a certain thing
 Unlimited possibilities to combine commands
with |, >, <, and >>
 Even more powerful to put commands in
shell script
 Slightly different commands in different
Linux distributions
 Emphasized in System V, different in BSD
its.unc.edu
73
Exercise
its.unc.edu
74
Exercise





cd
Change to home directory
cat /etc/passwd > temp
Save the file /etc/passwd into a new file called “temp”
more temp | wc –l
Find out the number of lines in file “temp”
awk ‘/home/’ temp
Show line(s) with the word “home”
awk ‘/nologin/{++i}; END{print i}’ temp
Count number of word “nologin” in file “temp”

sed s/no/yes/g temp > temp1

awk ‘/nologin/{++i}; END{print i}’ temp1
Replace all “no” with “yes” in “temp” and save in
“temp1”
Count number of work “nologin” in file “temp1”

awk ‘/yeslogin/{++i}; END{print i}’ temp1
Count number of work “yeslogin” in file “temp1”



grep –i daemon temp
Display line(s) with word “daemon” regardless of case
more temp | grep –i daemon
Display line(s) with word “daemon” regardless of case
sort –t: -k3 –n temp
Sort numerically over the 3rd column in file “temp”
separated by “:”

rm –i temp*
Delete the file(s) with filename starting with “temp”
interactively
its.unc.edu
75