Transcript UNIX & LINUX Fundamentals

Introduction to UNIX
UNIX & LINUX Fundamentals
1
Purpose of This Lecture
●
●
●
●
●
Fundamentals of using UNIX and UNIX-like
systems
History of UNIX & Linux
Basic system commands
Data Management
Constructing basic shell scripts
2
History of UNIX
●
"...the number of UNIX installations has grown to 10, with
more expected..."
–
●
Dennis Ritchie and Ken Thompson, June 1972
"... When BTL withdrew from the project, they needed to
rewrite an operating system (OS) in order to play space
war on another smaller machine (a DEC PDP-7
[Programmed Data Processor] with 4K memory for user
programs). The result was a system which a punning
colleague called UNICS (UNiplexed Information and
Computing Service)--an 'emasculated Multics'; no one
recalls whose idea the change to UNIX was"
3
Early Movers and Shakers
●
●
Dennis Ritchie & Ken Thompson
PDP11
4
And then there was C…
●
in 1972, Ritchie rewrote B and called the new
language C; Thompson created the pipe--a
"Write programs that do one thing and do it
well. Write programs to work together. Write
programs that handle text streams, because that
is a universal interface."
5
Bringing UNIX to the desktop
●
Unix was very expensive
Microsoft DOS was the mainstream OS
MINIX, tried but was not a full port
●
An Open Source solution was needed!
●
●
6
Linux 0.02 – October 5, 1991
●
●
“Do you pine for the nice days of minix-1.1, when men
were men and wrote their own device drivers?
Are you without a nice project and just dying to cut your
teeth on a OS you can try to modify for your
needs? Are you finding it frustrating when everything
works on minix? No more all-nighters to get a nifty
program working? Then this post might be just for you :-)”
- Linus Benedict Torvalds
"I still maintain the point that designing a monolithic
kernel in 1991 is a fundamental error. Be thankful you are
not my student. You would not get a high grade for such a
design :-)"
(Andrew Tanenbaum to Linus Torvalds)
7
1990’s Movers and Shakers
●
●
Richard Stallman, father of the GNU Project
Linus Torvalds
8
Why UNIX/LINUX is Still Used
●
●
●
●
●
●
30+ years of development
Many academic, scientific, and system tools
Open Source
System Stability
Lightweight
Easy Development
9
The Basics
●
Command-Line
–
–
●
Interaction with UNIX & LINUX is based on
entering commands to a text terminal
Oftentimes there are no ‘warnings’ with commands,
no ‘undo’
The Shell
–
–
The user environment that enables interaction with
the kernel, or lower-system OS.
Windows Explorer would be a shell for Microsoft
Windows.
10
Common Shells
●
●
●
●
●
●
sh – the original UNIX shell; still located in /bin/sh
Bash is a Unix shell written for the GNU Project and is
installed on most Linux systems.
csh – C Shell, modeled after the C programming language
used by UNIX systems
tcsh – C Shell with modern improvements such as
filename completion
echo $SHELL – displays what shell your account is using
chsh – change your shell
11
Before we go further…
●
●
●
Read the Manual.
–
man command
–
man [section] command
–
man –k keyword (search all manuals based on keyword)
Most commands have a built-in UNIX manual, even the
man command!
Commands without manuals have help too, with –h, -help, or /? option.
12
The Manual
●
The manual pages are divided into eight sections
depending on type of command.
–
1 commands and applications
–
2 system calls
–
3 C library functions
–
4 special files
–
5 file formats
–
6 games
–
7 misc.
–
8 system administration utilities
13
Conventions for this lecture
●
●
This lecture is too short to give you all of the
options, so look at the manual for specific
syntax for commands.
Commands will be in bold, options will be in
italics.
–
●
command -arguments
Output will be shown in its own bordered table
14
Command Conventions
●
●
In help files and manuals, commands will have
required input and option input
cp [OPTION] source destination
–
●
Optional arguments are in brackets, required
arguments are not.
cp –R or cp - -recursive
–
Short options ‘-’, long options ‘- -’
15
Who’s Path is it Anyway?
●
UNIX treats the directory structure as a
hierarchy of individual paths
/
dev
home
usr
bhecker
bin
16
Directories
●
What is a working directory?
–
●
●
The directory your shell is currently associated with.
At anytime in the system your login is associated
with a directory
pwd – view the path of your working directory
ls – view your working directory
17
Finding your home
●
●
●
Each user has a home directory which can be found with:
–
cd
–
cd ~bhecker
–
cd $HOME
The tilde character ‘~’ will tell the shell to auto-complete
the path statement for the cd command
$HOME refers to an environment variable which contains
the path for home.
18
More file commands
●
cd directory
- change your current working directory to the new path
●
ls –a
– Show hidden files
–
●
mv
–
●
moves one file to another
cp
–
●
Hidden files are files that begin with a period in the filename ‘.’
copies files or directories
rm
–
–
–
remove files & directories
rm –rf
● remove everything with no warnings
rm –rf *
● most dangerous command you can run!
19
Recursive Directories
●
Often times a manual will refer to ‘recursive’
actions on directories. This means to perform an
action on the given directory and recursively to
all subdirectories.
–
cp –R source destination
●
Copy recursively all directories under source to
destination
20
Relative vs. Absolute Path
●
●
Commands expect you to give them a path to a file. Most commands
will let you provide a file with a relative path, or a path relative to your
working directory.
–
../directory - the ‘..’ refers to looking at our previous directory first
–
./executable - ‘.’ says this directory, or our working directory
Absolute, or Full paths are complete. An easy way to know if a path is
absolute is does it contain the ‘/’ character at the beginning?
–
/home/user/directory/executable - a full path to file executable
21
Poking around in $home
●
How much space do I have?
–
Quota
●
●
command to see all quotas for your directories are, if
any.
How much space am I taking up?
–
du
●
–
command to find out how much space a folder or
directory uses.
df
●
display space information for the entire system
22
Helpful Hints on Space
●
●
Almost all commands that deal with file space
will display information in Kilobytes, or Bytes.
Nobody finds this useful.
Many commands will support a ‘-h’ option for
“Human Readable” formatting.
–
ls –lh
●
displays the working directory files with a long listing
format, using “human readable” notation for space
23
Representing Space
●
●
●
●
●
●
●
Bit – either a 1 or 0
Byte – 8 bits “0000 1111”, or “x0F”, or 16
KB, Kilobyte - 1024 Bytes
MB, Megabyte – 1024KB
GB, Gigabyte – 1024MB
TB, Terabyte – 1024 GB
PB, Petabyte – 1024 TB
24
Permissions
●
●
The *NIX systems are multi-user environments where
many users run programs and share data. Files and
directories have three levels of permissions: World,
Group, and User.
The types of permissions a file can contain are:
Read permissions
Write Permissions
Execute Permissions
r
w
x
25
Permissions Cont.
●
●
File permissions are arranged in three groups of
three characters.
In this example the owner can read & write a
file, while others have read access
User (owner)
Group
Others (everyone else)
rw-
r--
r-26
Changing Permissions
●
●
chmod – change permissions on a file or
directory
chown – change file ownership to another user
–
Both options support ‘-R’ for recursion.
27
All About Me
●
●
Every user id corresponds to a unique user or system process
–
whoami – returns the use rid of the current user
–
passwd – change password
What is my group? – support!
lslogin2$ ls -l ~eturner
total 391632
drwx------ 2 eturner support
-rwx------ 1 eturner support
drwx------ 5 eturner support
4096 Sep 12 15:41 anothertest
8487 Jun 7 13:12 a.out
4096 Apr 7 2006 automake-test
28
What everyone else is up to
●
●
top – show a detailed, refreshed, description of running
processes on a system.
uptime – show the system load and how long the system
has been up.
–
‘load’ is a number based on utility of the cpu’s of the system. A
load of 1 indicates full load for one cpu.
lslogin2$ uptime
15:43:52 up 3 days, 20:45, 154 users, load average: 2.04, 2.00, 2.36
29
Working With Programs
●
Commands or programs on the system are
identified by their filename and by a process ID
which is a unique identifier.
–
ps
●
–
kill pid
●
–
–
●
Display process information on the system
Terminates the process id
^c (control+c) terminates the running program
^d (control+d) terminates your session.
Only you and the superuser (root) has
permissions to kill processes you own.
30
Advanced Program Options
●
Oftentimes we must run a command in the
background with the ampersand ‘&’ character
–
command –options &
●
●
Runs command in background, prompt returns
immediately
Match zero or more characters wildcard ‘*’
–
cp * destination
●
–
Copy everything to destination
This option can get you into trouble if misused
31
Editing Files
●
●
emacs vs vi
–
Among the largest ‘nerd battle’ in history. emacs relies heavily on
key-cords (multiple key strokes), while vi is mode based. (editor
mode vs command mode)
–
vi users tend to enter and exit the editor repeatedly, and use the
Unix shell for complex tasks, whereas emacs users usually remain
within the editor and use emacs itself for complex tasks
pico (nano)
–
editor originally used for the email client pine, simple no-frills
editor which resembles notepad for windows.
32
Input and Output
●
Programs and commands can contain an input and output.
These are called ‘streams’. UNIX programming is
oftentimes stream based.
–
Programs also have an error output. We will see later how to catch
the error output.
●
STDIN – ‘standard input,’ or input from the keyboard
●
SDTOUT – ‘standard output,’ or output to the screen
●
STDERR – ‘standard error,’ error output which is sent to
the screen.
33
File Redirection
●
Oftentimes we want to save output (stdout) from a
program to a file. This can be done with the
‘redirection’ operator.
–
●
myprogram > myfile – using the ‘>’ operator we redirect
the output from myprogram to file myfile
Similarly, we can append the output to a file instead
of rewriting it with a double ‘>>’
–
myprogram >> myfile – using the ‘>’ operator we
append the output from myprogram to file myfile
34
Input Redirection
●
Input can also be given to a command from a
file instead of typing it to the screen, which
would be impractical.
–
cat programinput > mycommand - This command
series starts with the command ‘cat’ which prints a
file to a screen. programinput is printed to stdout,
which is redirected to a command mycommand.
35
Redirecting stderr
●
Performing a normal redirection will not redirect
sdterr. In Bash, this can be accomplished with
‘2>’
–
●
command 2> file1
Or, one can merge stderr to stdout (most
popular) with ‘2>&1’
–
command > file 2>&1
36
Pipes
●
●
Using a pipe operator ‘|’ commands can be
linked together. The pipe will link the standard
output from one command to the standard input
of another.
Very helpful for searching files
37
Searching
●
A large majority of activity on UNIX systems involve
searching for files and information.
–
find – utility to find files
–
grep – the best utility ever written for UNIX, searches for
patterns inside files and will return the line, if found
lslogin2$ find ./ -name output-evansapp-n0.out534086
./test/run_0/output-evansapp-n0.out534086
lslogin2$ cat test/run_0/output-evansapp-n0.out534086 | grep CPU
CPU time :
0.03 sec.
38
Packing Files
●
●
When creating backups of files, or transferring
to other hosts, files must be packed into larger
files. This is needed for ease of manipulation,
transfer speeds, and file management.
tar – create or extract a packed file. tar stands
for ‘tape archive’.
39
Compressing Files
●
●
●
Compressing files can gain file space at the
expense of cpu time to compress and
decompress files.
Compression works well for text files, but not as
well for binary files with random data such as
float values.
Compression algorithms & commands
–
gzip, gunzip, bzip2, bunzip2
40
Using tar to create compressed files
●
Tar will create compressed files for you
–
tar –czvf mytarfile.tar.gz directory
●
–
creates a compressed file named mytarfile.tar.gz
containing all of the files in the directory directory
tar –xzvf mytarfile.tar.gz
●
uncompresses all directories and files inside the file
mytarfile.tar.gz into the working directory
41
Connecting to Another Machine
●
Secure Shell vs. Restricted Shell
–
●
ssh is an encrypted remote login program that is ‘secure’ to trust
across non secure networks.
ssh userid@hostname
42
Copying Files to Remote Hosts
●
copy local file lfile to rfile on remote machine
rsys
–
scp lfile rsys:rfile
●
-p preserves modification time, access time and mode
from original
–
●
scp -p lfile rsys:rfile
copy rfile from remote machine rsys to local file
lfile
–
scp -p rsys:rfile lfile
43
Running Commands on a Remote Host
●
●
Commands can be executed on a remote host
with ssh
ssh userid@hostname “ls”
–
Run ls on remote host hostname
44
My Environment
●
●
View all system variables by the command
‘env’
Depending on shell, startup commands can be
managed with the files .profile for bash and
.cshrc with c shell
45
Basic Shell Scripts
●
●
Many times it is helpful to create a ‘script’ of commands
to run instead of typing them in individually. Scripts can
be made to aid in post-processing, system administration,
and automate menial tasks
#!/bin/bash
–
●
First statement inside a script, will list which shell to run this script
in
# - says what will follow is a comment and not to execute
46
Variables
●
●
●
By convention system variables are capitalized
–
HOME – location of the home directory
–
OLDPWD – location of the previous working directory
–
PATH – locations to look inside for executable files
Setting system variables differs by shell. bash uses
export, csh uses setenv
User defined variables in scripts are lower-case by
convention
–
myvariable=10
●
–
sets myvariable to 10
echo $myvariable
●
prints myvariable
47
Conditionals
if condition
then
condition is zero (true - 0) execute all
commands up to else statement
else
if condition is not true then execute all
commands up to fi
fi
48
Performing Loops
●
Loops are statements that are repeated until the conditions are met.
for { variable name } in { list }
do
execute one for each item in the list until the list is not
(And repeat all statement between do and done)
done
finished
for i in 1 2 3 4 5
do
echo "Welcome $i times“
done
49
Putting it Together
#!/bin/bash
#my first script
#scp replacement
remotefile=labresults
myserver=saguaro.fulton.asu.edu
mylsinfo=`ssh $myserver ls $remotefile 2>&1`
ismissing=`echo $mylsinfo | grep ERROR`
if [ "$ismissing" ]
then
echo "$remotefile not found! Exiting!"
else
ssh -n "cat < $remotefile" > $localfile
Fi
50
More…
●
mylsinfo=`ssh $myserver ls $remotefile 2>&1`
–
●
`Backticks` are used to place output from a
command into a variable
if [ "$ismissing" ]
–
Is $ismissing set (has a value)? If so then the
expression is true, otherwise false
51
References
The textbook for this course:
●
Graham Glass & King Ables UNIX for
Programmers and Users
52