Transcript Introduction to the Linux Command Line for High-Performance Computing Dr. Charles J Antonelli LSAIT ARS October, 2014

Introduction to the
Linux Command Line for
High-Performance Computing
Dr. Charles J Antonelli
LSAIT ARS
October, 2014
Roadmap
Linux overview
The command shell
Navigating the filesystem
Basic commands & wildcarding
Shell redirection & pipelining
Editing text files
Permissions
Processes
cja 2014
2
10/14
Poll
How many here to learn the command
line
For HPC, specifically Flux, work?
For other computing cluster work?
For desktops/other Linux/Unix/Mac OS X?
Other?
cja 2014
3
10/14
Course Text
William E Shotts, Jr.,
“The Linux Command Line: A Complete Introduction,”
No Starch Press, January 2012.
Download Creative Commons Licensed
version
athttp://downloads.sourceforge.net/project/linuxcomma
nd/TLCL/13.07/TLCL-13.07.pdf.
cja 2014
4
10/14
Linux Overview
cja 2014
5
10/14
A compute node
48 GB RAM
12 Intel cores
Local disk
Network
cja 2014
6
10/14
cja 2014
7
http://linuxforums.org.uk/index.php?action=media;sa=item;in=262
10/14
The command shell
cja 2014
8
10/14
The command line
A basic way of interacting with a Linux system
Execute commands
Create files and directories
Edit file content
Access the web
Copy files to and from other hosts
Run HPC jobs
… do things you can’t do from the conventional pointand-click Graphical User Interface (GUI)
cja 2014
9
10/14
Why command line?
1.
Linux was designed for the command line
2. You can create new Linux commands using the command
line, without programming
3. Many systems provide only the command line, or poorly
support a GUI interface
•
Such as most HPC systems
4. Many things can be accomplished only through the
command line
•
Much systems administration & troubleshooting
5. You want to be cool
cja 2014
10
10/14
The command shell
The command shell is an application that reads
command lines from the keyboard and passes them to
the Linux operating system to be executed
When you login to a remote Linux system using a tool
like ssh, you will automatically be connected to a shell
On your desktop, laptop, or tablet, you may have to
find and execute a terminal emulator application to
bring up a shell in a window
cja 2014
11
10/14
The command shell
cja 2014
12
http://www.livefirelabs.com/208_global/images/unix-operating-system.gif
10/14
Types of shells
Standard Linux shell (used in this course)
bash
“Bourne-again” shell
Other shells
sh
csh
tcsh
Bourne shell, original AT&T Unix shell
“C shell”, common on older Unix systems
“Enhanced C shell”, better version of csh
What shell am I using?
echo $SHELL
cja 2014
13
10/14
Connecting via ssh
Terminal emulators
Linux and Mac OS X
Start Terminal
Use ssh command
Windows
U-M PuTTY/WinSCP (U-M Blue Disc)
https://www.itcs.umich.edu/bluedisc/
PuTTY
http://www.chiark.greenend.org.uk/~sgtatham/putty/
SSH Secure Shell (deprecated)
cja 2014
14
10/14
Lab 0
Task: Start a local shell on Mac OS X
1.
Login to Mac OS X with your uniqname and Kerberos password
2.
If there is a Terminal icon in the Dock , double-click it
Otherwise:
a.
b.
Bring up a Finder window:
Click on the desktop, type Command-N
Start the Terminal Application:
In the Finder window, click on Applications on the left, scroll
down on the right until you find the Utilities folder, double-click
on the Utilities folder, scroll down on the right until you find the
Terminal application, double-click it
3.
This creates a Terminal window with a shell running inside it
4.
From a Terminal window, Command-N will start a new Terminal
cja 2014
15
10/14
Lab 0
cja 2014
16
10/14
The shell prompt
The “~$ “ is the shell prompt
This means the shell is waiting for you to type
something
Format can vary, usually ends with “$” , “%” or “#”
If $ or %, you have a normal shell
This shell has your privileges
If #, you have a so-called “root shell”
This shell has administrator privileges
You can do a great deal of irreversible damage
cja 2014
17
10/14
Typing into the shell
Basic input line editing commands
Backspace erases previous character
Left and right arrow move insertion point on the line
Control-U erases the line so you can start over
Enter executes the line you typed
Control-C interrupts whatever command you started and
returns you to the shell prompt (usually)
Up and down arrow will access your command history
Type “exit” without the quotes to exit the shell
Click the red "close" icon at the top of the Terminal window
to close it
cja 2014
18
10/14
Lab 1
Task: Enter some basic commands
~$ date
~$ id
~$ ps
~$ df -kh
~$ who
~$ top
cja 2014
# type Control-C to exit
19
10/14
Interlude: Flux
cja 2014
20
10/14
Flux
Flux is a university-wide shared computational discovery / high-performance
computing service.
Interdisciplinary
Provided by Advanced Research Computing at U-M (ARC)
Operated by CAEN HPC
Hardware procurement, software licensing, billing support by U-M ITS
Used across campus
Collaborative since 2010
Advanced Research Computing at U-M (ARC)
College of Engineering’s IT Group (CAEN)
Information and Technology Services
Medical School
College of Literature, Science, and the Arts
School of Information
http://arc.research.umich.edu/resources-services/flux/
cja 2014
21
10/14
The Flux cluster
…
cja 2014
22
10/14
A Flux node
48 GB RAM
12 Intel cores
Local disk
Network
cja 2014
23
10/14
Logging in to Flux
ssh flux-login.engin.umich.edu
MToken (or Software Token) required
You will be randomly connected a Flux login node
Currently flux-login1 or flux-login2
Firewalls restrict access to flux-login.
To connect successfully, either
Physically connect your ssh client platform to the U-M
campus wired or MWireless network, or
Use VPN software on your client platform, or
Use ssh to login to an ITS login node (login.itd.umich.edu),
and ssh to flux-login from there
cja 2014
24
10/14
A shell on a login node
Now you have a shell on the login node
Your shell prompt has changed to something like
[flux-login1 ~]$
Try those simple commands here!
Flux is for HPC work
The login nodes are reserved for copying data, editing and
compiling programs and scripts, and short test runs
For HPC computation, you should run on the compute nodes
A batch job will always run on the compute nodes
An interactive batch job will get you a shell on a compute node
cja 2014
25
10/14
A shell on a compute node
Getting a shell on a compute node
Enter this command on the login node:
[flux-login1 ~]$ ~cja/hpc100/getshell
After a short delay, you’ll get an interactive shell:
qsub: waiting for job 11147004.nyx.engin.umich.edu to start
qsub: job 11147004.nyx.engin.umich.edu ready
[nyx5509 ~]$
You can type commands into this shell for all of our in-class
experiments
Type “exit” to exit this shell
After two hours, your interactive shell will be killed
cja 2014
26
10/14
Navigating the filesystem
cja 2014
27
10/14
Linux Filesystem Concepts
Files are stored in a directory
Directories may contain other directories as
well as files
A hierarchy of these directories is called a
directory tree
A directory tree (a connected graph with no
cycles) has a single topmost root directory
A directory tree rooted at the system root
directory “/” is called a filesystem
cja 2014
28
10/14
cja 2014
29
http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html
10/14
Linux Filesystem Concepts
A file is accessed using its path name
Absolute path name
/dir1/dir2/…/dirn/filename
/usr/X11R6/bin
Relative path name
current-working-directory/filename
bin
Every shell maintains a notion of a current working directory
Initialized at login to your home directory
Changed via cd command
Two special directories
. refers to the current directory
.. refers to the current directory’s parent directory
cja 2014
30
10/14
Basic commands
cja 2014
31
10/14
Prerequisites
Some fundamental commands:
~$ file file
# what kind of file is file?
~$ cat file
# display contents of text file
~$ less file
# paginate text file
~$ man command
# get info about command
Exercise: figure out how to make the date command
display the date in Coordinated Universal Time (UTC)
cja 2014
32
10/14
Lab 2
Task: navigate the file system
Commands:
~$ cd
# make your home directory
the current working directory
~$ pwd
# print working directory
~$ mkdir foo
# create directory foo
~$ cd foo
# cd to the foo directory
~$ mkdir bar
# create directory bar
~$ cd ..
# cd back to the foo directory
~$ tree
# display foo’s directory tree
cja 2014
33
10/14
Navigating the filesystem
Some fundamental commands:
~$ pwd
# print working directory
~$ cd dir
# make dir the current working directory
~$ cd
# cd to your home dir
~$ cd ~cja
# cd to cja’s home dir
~$ mkdir dir
# create directory dir
~$ rmdir dir
# remove (empty) directory dir
~$ tree
# display dir tree
cja 2014
34
10/14
Listing info on files
ls – list information about files
~$ ls
# list contents of cur dir
~$ ls dir
# list contents of dir
~$ ls –l
# list details of files in cur dir
including access, owner & group,
size, and last-modified time
~$ ls –t
# list newest files first
~$ ls –R dir
# list all files in tree dir
~$ ls –lt dir
# options can be combined
cja 2014
35
10/14
Working with files
These commands manipulate files
~$ mv foo bar
# rename file foo to bar
~$ cp foo bar
# copy file foo to bar
~$ cp –r dir1 dir2
# copy dir tree dir1 to dir2
~$ cp f1 f2 dir
# copy f1 and f2 to directory dir
~$ mkdir dir
# create empty directory dir
~$ rmdir dir
# remove empty directory dir
~$ rm file
# remove file file
~$ rm –r dir
# remove directory tree dir
cja 2014
36
10/14
Lab 3
Exercise:
Create a directory named tutorial in your home
directory. In that directory, create a directory named
sample and a directory named test . Create a file
named msg in directory test that contains a copy of the
file /etc/motd .
Extra credit: Make the last-modified time of your copy
identical to that of /etc/motd. Hint: look at the options
of the copy command
cja 2014
37
10/14
Compressing and archiving
These commands compress and archive files
~$ gzip foo
# compress foo to foo.gz
~$ gunzip foo
# uncompress foo.gz to foo
~$ bzip2 foo
# better compress foo to foo.bz2
~$ bunzip2 foo
# uncompress foo.bz2 to foo
~$ tar cf foo.tar bar # archive subtree bar in file foo.tar
~$ tar xf foo.tar
# restore archive from file foo.tar
~$ tar zcf foo.tgz bar # archive and compress
~$ tar jcf foo.tjz bar # archive and compress better
cja 2014
38
10/14
Wildcards
The shell accepts wildcarded arguments
This is also called "shell globbing"
Wildcards:
?
*
[chars]
[c1-c2]
[^chars]
~$ ls foo.?
Matches a single character
Matches zero or more characters
Matches any of the chars
Matches chars ‘c1’ through ‘c2’
Matches any but the chars
# match files named foo.x, where x
is any character
~$ echo *.[cs]
# echo files that end in .c or .s
~$ mv [o-z]* save
# move files starting with o through
z to directory save
~$ echo [^A-Z]?
# ???
cja 2014
39
10/14
Shell redirection &
pipelining
cja 2014
40
10/14
Shell redirection
A Linux command can have its inputs and outputs redirected
~$ ls >foo
# put list of files in current
directory into file foo
~$ ls >>bar
# add list of files in current
directory to end of file foo
~$ sort <bar
# sort lines from file bar
~$ sort <<EOF
whiskey
bravo
tango
EOF
# sort lines entered at keyboard
~$ wc -l </etc/motd >~/mycounts
# count number of lines from file
/etc/motd and put result in file
mycounts in my home directory
cja 2014
41
10/14
More Linux commands
More useful Linux tool commands
~$ grep string
# show lines of input containing
string
~$ tail
# show last few lines of input
~$ head
# show first few lines of input
~$ sort
# sort the input
cja 2014
42
10/14
Shell pipelining
A Linux command can have its output connected to the
input of another Linux command
~$ ls | wc –l
# count files in current
directory
~$ last | grep reboot # when did we reboot?
Exercise:
How many people are running MATLAB on your compute
node?
cja 2014
43
10/14
Editing text files
cja 2014
44
10/14
Editing text files
Simple editor
nano or pico
"What you see is what you get” editor
Simple to learn if you want to get started quickly
Supported editors
vi or vim
emacs
Powerful but more complex
If you have time and inclination to become
proficient, spend time here
cja 2014
45
10/14
Text files
Watch out for source code or data files
written on Windows systems
Use these tools to analyze and convert source
files to Linux format
file
dos2unix
cja 2014
46
10/14
Permissions
cja 2014
47
10/14
File Permissions
Three permission bits, aka mode bits
Files: Read, Write, Execute
Directories: List, Modify, Search
Three user classes
User (File Owner), File Group, Other
man chmod
cja 2014
48
10/14
File Permissions, examples
-rw------- cja lsait 40 Oct 1 12:03 foo.bz2
file read and write rights for the owner, no access for
anyone else
chmod u=rw,g=r,o= file
-rw-r----- cja lsait 40 Oct 1 12:03 foo.bz2
file read and write rights for the owner, read for
members of the lsait group, no access and for others
drwxr-x--x cja lsait 4096 Oct 1 12:15 bar
list, modify, and search for the owner, list and search
for group, and execute only for others
cja 2014
49
10/14
Processes
cja 2014
50
10/14
Processes
On Linux, every program runs in a process
You can examine these processes
man ps
ps
ps ax
top
cja 2014
51
10/14
Processes
You can signal a running process
To stop it, or "kill" it
man kill
cja 2014
52
10/14
Any Questions?
Charles J. Antonelli
LSAIT Advocacy and Research Support
[email protected]
http://www.umich.edu/~cja
734 763 0607
cja 2014
53
10/14
References
1.
2.
3.
cja 2014
http://en.wikipedia.org/wiki/History_of_Linux
http://redhatlinux4u.blogspot.com/2011/11/file-hierarchy-system.html
William E Shotts, Jr., “The Linux Command Line: A Complete Introduction,” No Starch Press,
January 2012. Download Creative Commons Licensed version
athttp://downloads.sourceforge.net/project/linuxcommand/TLCL/13.07/TLCL-13.07.pdf.
54
10/14
Extra
Task: compile and execute simple programs on the Flux compute node
Copy sample code to your login directory:
cd
cp ~brockp/cac-intro-code.tar.gz .
tar -xvzf cac-intro-code.tar.gz
cd ./cac-intro-code
Examine, compile & execute helloworld.f90:
ifort -O3 -ipo -no-prec-div -xHost -o f90hello helloworld.f90
./f90hello
Examine, compile & execute helloworld.c:
icc -O3 -ipo -no-prec-div -xHost -o chello helloworld.c
./chello
Examine, compile & execute MPI parallel code:
mpicc -O3 -ipo -no-prec-div -xHost -o c_ex01 c_ex01.c
mpirun -np 2 ./c_ex01
cja 2014
55
10/14