Transcript Presentation Title - Information Technology Services

Linux Intermediate
ITS Research Computing Center
C. D. Poon, Ph.D.
Email: [email protected]
Class Material
 Point web browser to
http://its.unc.edu/Research
 Click on “Training” on the left column
 Click on “ITS Research Computing Training
Presentations”
 Click on “Linux Intermediate – Commands,
Tips and Tricks”
its.unc.edu
2
Outline
 Linux Command Category
 Stdout/Stdin/Stderr, Pipe and Redirection,
Wildcards
 Linux Command Review
 Break
 Tips and Tricks
 Conclusion
 Question and Exercise
its.unc.edu
3
Linux Command Category
its.unc.edu
4
Linux Command Category
 Communication
ssh scp
 File/Directory Management
cat cd chmod cp ln ls mkdir
more less mv pwd dirs rm head
tail wc
 Comparisons
diff
its.unc.edu
5
Linux Command Category
Cont’d
 Searching
grep find locate
 Archiving
compress uncompress gzip
gunzip
zcat tar
 Text Processing
cut paste sort sed awk
uniq
its.unc.edu
6
Linux Command Category
Cont’d
 System Status
chgrp chown date df du env
who w uptime
 Miscellaneous
bc cal clear man
its.unc.edu
7
Stdout/Stdin/Stderr
Pipe and Redirection
Wildcards
its.unc.edu
8
stdout stdin stderr
 Output from commands
• usually written to the screen
• referred to as standard output (stdout)
 Input for commands
• usually come from the keyboard (if no arguments are
given
• referred to as standard input (stdin)
 Error messages from processes
• usually written to the screen
• referred to as standard error (stderr)
its.unc.edu
9
Pipe and Redirection
 Pipe (|): stdout of one command to stdin
of another command
 Output Redirection (>): stdout of a
command to a file
 Output Appending (>>): stdout of a
command appending to a file
 Input Redirection (<): stdin of a command
from a file
its.unc.edu
10
Stderr Redirection
For tcsh
&> filename
For bash
2>&1 filename
its.unc.edu
11
Wildcards
 Multiple filenames can be specified using special
pattern-matching characters. The rules are:
• ‘?’ matches any single character in that position in
the filename
• ‘*’ matches zero or more characters in the filename.
• ‘[…]’ Characters enclosed in square brackets match
any name that has one of those characters in that
position
 Note that the UNIX shell performs these expansions
before the command is executed.
its.unc.edu
12
Linux Command Review
its.unc.edu
13
ssh
Log on to remote machine from Linux
ssh [email protected]
ssh killdevil.unc.edu –l cdpoon
ssh killdevil
ssh –X kure.unc.edu
ssh –Y kure.unc.edu
its.unc.edu
14
ssh using SecureCRT
in Windows
Using ssh, login to killdevil.unc.edu
To start ssh using SecureCRT in Windows,
do the following.
• Start -> Programs -> Remote Services -> SecureCRT
• Click the Quick Connect icon at the top.
• Hostname: killdevil.unc.edu
• Login with your ONYEN and password
its.unc.edu
15
scp
Copy files and directories to and from remote
computers
scp file1 killdevil.unc.edu:/nas02/home/cdpoon/.
scp zircon.its.unc.edu:/home/cdpoon/file2 .
scp –r dir1 killdevil.unc.edu:/netscr/cdpoon/.
scp –r killdevil.unc.edu:/netscr/cdpoon/dir2 dir3
scp kure:/netscr/cdpoon/f killdevil:/nas02/home/cdpoon/.
its.unc.edu
16
cat
Read one or more files and print the on stdout
cat file1
cat file1 file2 file3 > file_all
cat file4 >> file_all
Append file4 to file_all
cat > file5
Create file at stdin, end with EOF (cntl-d normally, use “stty –a” to find out)
cat > file6 << STOP
Create file at stdin, end with STOP
its.unc.edu
17
cd
Change directory, build-in shell command
cd /nas02/home/c/d/cdpoon
cd ../../
Change directory to 2 levels up
cd ..
Change directory to 1 level up
cd ~
cd
cd –
its.unc.edu
Change directory to Home
Change directory to Home
Change to previous directory
18
chmod
Change the access mode of one or more files
chmod u+x file1
chmod go-w file2
chmod u=rwx, g=rx, o=x file3
chmod 751 file3
Same as above, 7=rwx, 5=rx, 1=x
chmod =r file4
chmod 444 file4
its.unc.edu
Same as above, 4=r, 2=w, 1=x
19
cp
Copy a file/dir to another file/dir
cp file1 file2
Copy to the same directory and change filename
cp file1 ../dir/file2
Copy to different directory and change filename
cp file1 ../dir/.
Keep the same filename
cp –r dir1 dir2
Copy directory recursively
cp –r dir1 new_dir/dir2
Copy directory recursively to another directory
cp –p file3 file4
its.unc.edu
Preserve the modification time and permission modes
20
ln
Create links for file/dir and allow them to be
accessed by different names
ln file1 file2
Hard link for file
ln dir1 dir2
Hard link not allowed for directory
ln –s dir1 dir2
Soft link for directory, dir2 -> dir1
ln –s file3 file4
ln –s dir/file5 file6
its.unc.edu
Soft link, file4 -> file3
Soft link, file6 -> dir/file5
21
ls
List all files and directories in the current
directory
ls
ls –a
List files/directories starting with “.” too
ls –l
Long listing
ls –lh
List file sizes in human readable format and long list format
ls –F
Flag filenames by appending / to directories, * to executables files,
and @ to symbolic links
its.unc.edu
22
mkdir
Create one of more directories
mkdir dir1
mkdir –p dir1/dir2/dir3
Create intervening parent directories if they don’t exist
Same as mkdir dir1; cd dir1; mkdir dir2; cd dir2; mkdir dir3; cd ../../
its.unc.edu
23
more
Display files on a terminal, one screen at a
time
more file1
Hit space bar for another page, q to quit
more –d file2
Display the prompt “Press space to continue, ‘q’ to quit
more –c file3
Page through the file by clearing each window instead of
scrolling
its.unc.edu
24
less
Works like “more” but allows backward and
forward movement
less file1
Hit space bar for another page, q to quit
Hit b to scroll backward one window
Hit /pattern to highlight “pattern” in the text
Hit Return to scroll one line at a time
its.unc.edu
25
mv
Move files and directories within the same
machine and/or rename them
mv file1 dir1/file1
Move file1 to dir1, Same as mv file1 dir1/
mv file3 file4
Rename file3 to file4
mv dir1 dir2
Rename directory dir1 to dir2
mv dir3 dir4/dir5/dir6
Rename directory dir3 to dir6 and move to
dir4/dir5 directory
its.unc.edu
26
pwd dirs
Print the full pathname of the current
directory
pwd
dirs
C shell and bash shell built-in command, works
like “pwd”
dirs –l
its.unc.edu
Print working directory in long listing
27
rm
Delete one or more files and directories
Delete empty directory with “rmdir”
rm file1
rm file*
Remove all files with filename starting as “file”
rm –i file*
Prompt for y (remove the file) or n (do not remove the file)
rm –r dir1
Delete directory “dir1” and its content
its.unc.edu
28
head tail
Print first/last few lines of one or more files
head file1
Print the first 10 lines of file “file1”
head –n100 file2
Print the first 100 lines of file “file2”
tail file*
Print the last 10 lines of files with filename
starting as “file”
tail –f file3
its.unc.edu
Print the last 10 lines of file “file3” and follow file
as it grows
29
wc
Print a character, word, and line count for
files
wc –c file1
Print character count for file “file1”
wc –l file2
Print line count for file “file2”
wc –w file3
Print word count for file “file3”
its.unc.edu
30
diff
Report lines that differ between file1 and
file2, with file1 text flagged by < and file2
by >
diff file1 file2
its.unc.edu
Show difference between file1 and file2
31
grep
Search for lines that match a regular
expression
grep abc file1
Print line(s) in file “file1” with “abc”
grep –i abc file2
Print line(s) in file “file2” with “abc” ignoring uppercase and lowercase
distinctions
its.unc.edu
32
find
Find particular groups of files
find . –name temp
Find file named “temp” in current directory
find /etc –name ‘rc*’
Find file(s) in /etc directory with name starting
with “rc”
find /usr/share/man –type d –name ‘man*’
Find directories in /usr/share/man with name starting with “man”
its.unc.edu
33
locate
Find files with matching pattern in database
prepared by updatedb, Database needed to
be updated daily
locate which
Find files named with pattern “which” in the OS
locate –c which
Count number of files named with pattern
“which” in the OS
locate –i which
Find files named with pattern “which” in the OS
ignoring case distinctions
its.unc.edu
34
compress uncompress
Reduce or expand the size of one or more
files using adaptive Lempel-Ziv coding
Use uncompress to expand data
compress file1
compress –f file2
create
file2.Z
uncompress file3.Z
its.unc.edu
Reduce the size of file1 and create new
file named file1.Z
Force to reduce the size of file2 and
new file named
Expand file3.Z and restore file3
35
gzip gunzip
Reduce or expand the size of one or more
files using Lempel-Ziv coding (LZ77)
Use gunzip to expand data
gzip file1
Reduce the size of file1 and create new file
named file1.gz
gzip –f file2
Force to reduce the size of file2 and create new
file named file2.gz
gunzip file3.gz
Expand file3.gz and restore file3
its.unc.edu
36
zcat
Expand the size of one or more files created
by compress or gunzip
List file contents to stdout without deleting
the .Z or .gz file
zcat file1.Z
Expand file1.Z and list the content of file1 in stdout
zcat file2.gz
Expand file2.gz and list the content of file2 in stdout
its.unc.edu
37
tar
Archive files and directories
Create a single file with extension .tar
tar –cvf file123.tar file1 file2 file3
Create archive file named file123.tar in verbose mode
with contents, file1, file2, and file3
tar –xvf file123.tar
Expand file123.tar in verbose mode and generate the
original files and directories back
its.unc.edu
38
cut
Remove sections from each line of files
cut –d: -f1,5 /etc/passwd
Use field delimiter “:” to locate fields 1 and 5 from file /etc/passwd
to extract usernames and real names
cut –c4 file1
Take character 4 out from each line of file1 and display in stdout
its.unc.edu
39
paste
Merge lines of files
$ cat file1
1
$ paste file1 file2
2
1
a
2
b
$ cat file2
a
b
c
its.unc.edu
c
$ paste –s file1 file2
1
2
a
b
c
40
sort
Sort lines of text files
sort –fd file1
Alphabetize lines (-d) in file1 and ignore lower and upper cases (-f)
sort –t: -k3 -n /etc/passwd
Take column 3 of file /etc/passwd separated by “:” and sort in arithmetic order
its.unc.edu
41
sed
Edit one or more files without user interaction
using stream editor
sed s/xx/yy/g file1
Substitude all occurrences of “xx” in file1 with “yy” and display on stdout
sed /abc/d file1
Delete all lines containing “abc” in file1
sed /BEGIN/,/END/s/abc/123/g file1
Substitute “XYZ” on lines between BEGIN and END with “xyz” in file1
its.unc.edu
42
awk
Process files by pattern-matching
awk –F: ‘{print $1}’ /etc/passwd
Extract the 1st field separated by “:” in /etc/passwd and print to stdout
awk ‘/abcde/’ file1
Print all lines containing “abcde” in file1
awk ‘/xyz/{++i}; END{print i}’ file2
Find pattern “xyz” in file2 and count the number
awk ‘length <= 1’ file3
Display lines in file3 with only 1 or no character
its.unc.edu
43
uniq
Report or omit repeated lines
uniq file1
Filter adjacent matching lines from file named file1 , writing to stdout
uniq –c file1
Prefix lines by the number of occurrences from file named file1
its.unc.edu
44
chgrp
Change the group ownership of one or more
files or directories
chgrp employee file1
Change group ownership to “employee” for file “file1”
chgrp –R student dir1
Change group ownership to “student” for directory “dir1” including
subdirectories recursively
its.unc.edu
45
chown
Change the ownership of one or more files or
directories
chown employee file1
Change ownership to “employee” for file “file1”
chown –R student dir1
Change ownership to “student” for directory “dir1” including
subdirectories recursively
its.unc.edu
46
date
Print the current date and time in certain
format
Set the current date and time
date
Print the current date and time
date +%D
Print the current date and time in mm/dd/yy format
date 1201160108
Set the current date and time to Dec 01 4:01pm 2008
date –d fri
Show the date of the coming Friday
its.unc.edu
47
df
Report the number of used and free
disk block on all mounted file systems
df
Print used and free disk block on all mounted file system
df -h
Print used and free disk block in human readable format
df -k
Print used and free disk block in kb
its.unc.edu
48
du
Print disk usage of directories and its
subdirectories
du dir1
Print disk usage in kilobyte of directory “dir1”
du –-block-size=1M dir2
Print disk usage in megabyte of directory “dir2”
du –hs dir3
format of
its.unc.edu
Print summarized disk usage in human-readable
directory “dir3”
49
env
Display the current environment variables or
set new values
env
its.unc.edu
Display all of the current environment variables
50
who
Display information about the current status
of the system
who
Display the names of users currently logged in to the system
who –b
Report information about the last reboot
who am I
Print the username of the invoking user
its.unc.edu
51
w
Print summaries of system usage, currently
logged-in users, and what they are doing
w
w –s
its.unc.edu
Print summaries of system usage, currently logged-in users
Display in short form
52
uptime
Print the current time, amount of time
logged in, and the system load
averages
uptime
its.unc.edu
Print a one line display of the current time, how long the system has
been running, how many users are currently logged on, and
the system load averages for the past 1, 5, 15 minutes
53
bc
Interactively perform arbitrary-precision
arithmetic or convert numbers from one
base to another, type “quit” to exit
bc
Invoke bc
1+2
Evaluate an addition
5*6/7
Evaluate a multiplication and division
ibase=8
Change to octal input
20
Evaluate this octal number
16
Output decimal value
ibase=10
Change back to decimal input
its.unc.edu
54
cal
Print calendar of a month or all months in a
year
cal
Print calendar of the current month
cal 2 2009
Print calendar of February 2009
cal 2009
Print calendar of all months in 2009
cal -3
Display previous/current/next months
its.unc.edu
55
clear
Clear the terminal display and have the
prompt locate at the top of the terminal
window
clear
its.unc.edu
Clean up the current terminal display
56
man
Display information from the online reference
manuals
man man
Display the manual for the command “man”
man –k link compile
Display commands related to linking and compiling
using a keyword search
its.unc.edu
57
Break
its.unc.edu
58
Tips and Tricks
its.unc.edu
59
Tips and Tricks #1
Show files changed on a certain date in all
directories
ls –l * | grep ‘Sep 26’
Show long listing of file(s) modified on Sep 26
ls –lt * | grep ‘Dec 18’ | awk ‘{print $9}’
Show only the filename(s) of file(s) modifed on Dec 18
its.unc.edu
60
Tips and Tricks #2
Sort files and directories from smallest to
biggest or the other way around
du –k –s * | sort –n
Sort files and directories from smallest to biggest
du –ks * | sort –nr
Sort files and directories from biggest to smallest
its.unc.edu
61
Tips and Tricks #3
Change timestamp of a file
touch file1
If file “file1” does not exist, create it, if it does, change the timestamp of it
touch –t 200902111200 file2
Change the time stamp of file “file2” to 2/11/2009 12:00
its.unc.edu
62
Tips and Tricks #4
Find out what is using memory
ps –ely | awk ‘{print $8,$13}’ | sort –k1 –nr | more
its.unc.edu
63
Tips and Tricks #5
Remove the content of a file without
eliminating it
cat /dev/null > file1
its.unc.edu
64
Tips and Tricks #6
Backup selective files in a directory
ls –a > backup.filelist
Create a file list
vi backup.filelist
Adjust file “backup.filelist” to leave only filenames of the files to be
backup
tar –cvf archive.tar `cat backup.filelist`
Create tar archive “archive.tar”, use backtics in the “cat” command
its.unc.edu
65
Tips and Tricks #7
Get screen shots
xwd –out screen_shot.wd
Invoke X utility “xwd”, click on a window to save the image as
“screen_shot.wd”
display screen_shot.wd
Use ImageMagick command “display” to view the image
“screen_shot.wd”
Right click on the mouse to bring up menu, select “Save” to save
the image to other formats, such as jpg.
its.unc.edu
66
Tips and Tricks #8
Sleep for 5 minutes, then pop up a message
“Wake Up”
(sleep 300; xmessage –near Wake Up) &
its.unc.edu
67
Tips and Tricks #9
Count number of lines in a file
cat /etc/passwd > temp; cat temp | wc –l; rm temp
wc –l /etc/passwd
its.unc.edu
68
Tips and Tricks #10
Create gzipped tar archive for some files in a
directory
find . –name ‘*.txt’ | tar –c –T - | gzip > a.tar.gz
find . –name ‘*.txt’ | tar –cz –T - -f a.tar.gz
its.unc.edu
69
Tips and Tricks #11
Find name and version of Linux distribution,
obtain kernel level
uname -a
head –n1 /etc/issue
its.unc.edu
70
Tips and Tricks #12
Show system last reboot
last reboot | head –n1
who -b
its.unc.edu
71
Tips and Tricks #13
Combine multiple text files into a single file
cat file1 file2 file3 > file123
cat file1 file2 file3 >> old_file
cat `find . –name ‘*.out’` > file.all.out
its.unc.edu
72
Tips and Tricks #14
Create man page in pdf format
man –t man | ps2pdf - > man.pdf
acroread man.pdf
its.unc.edu
73
Tips and Tricks #15
Remove empty line(s) from a text file
awk ‘NF>0’ < file.txt
Print out the line(s) if the number of fields (NF) in a line in file
“file.txt” is greater than zero
awk ‘NF>0’ < file.txt > new_file.txt
Write out the line(s) to file “new_file.txt if the number of fields (NF)
in a line in file “file.txt” is greater than zero
its.unc.edu
74
Conclusion
its.unc.edu
75
Conclusion
 Many ways to do a certain thing
 Unlimited possibilities to combine commands
with |, >, <, and >>
 Even more powerful to put commands in
shell script
 Slightly different commands in different
Linux distributions
 Emphasized in System V, different in BSD
its.unc.edu
76
Questions ?
Exercise
its.unc.edu
77