Shell Scripting - Information Technology Services

Download Report

Transcript Shell Scripting - Information Technology Services

Shell Scripting

Shubin Liu, Ph.D.

Research Computing Center University of North Carolina at Chapel Hill

its.unc.edu

Agenda

    Introduction • • • UNIX/LINUX and Shell UNIX Commands and Utilities Basic Shell Scripting Structure Shell Programming • • • Variable Operators Logic Structures Examples of Application in Research Computing Hands-on Exercises The PPT/WORD format of this presentation is available here: http://its2.unc.edu/divisions/rc/training/scientific/ /afs/isis/depts/its/public_html/divisions/rc/training/scientific/short_courses/

2

Why Shell Scripting ?

    

Shell scripts can be used to prepare input files, job monitoring, and output processing.

Useful to create own commands.

Save lots of time on file processing.

To automate some task of day to day life.

System Administration part can be also automated.

its.unc.edu

3

Objectives & Prerequisites

 

After this workshop, you should be:

• Familiar with UNIX/LINUX, Borne Shell, shell variables/operators • Able to write simple shell scripts to illustrate programming logic • Able to write scripts for research computing purposes We assume that you have/know • • • An account on the Emerald cluster Basic knowledge of UNIX/LINUX and commands UNIX editor e.g. vi or emacs

its.unc.edu

4

History of UNIX/Linux

     Unix is a command line operating system developed around 1969 in the Bell Labs Originally written using C Unix is designed so that users can extend the functionality • • • To build new tools easily and efficiently To customize the shell and user interface. To string together a series of Unix commands to create new functionality.

• To create custom commands that do exactly what we want.

Around 1990 Linus Torvalds of Helsinki University started off a freely available academic version of Unix Linux is the Antidote to a Microsoft dominated future

its.unc.edu

5

What is UNIX/Linux ?

its.unc.edu

Simply put

  

Multi-Tasking O/S Multi-User O/S Available on a range of Computers

     SunOS IRIX HP-UX AIX Linux Sun Microsystems Silicon Graphics Hewlett Packard IBM ….

6

its.unc.edu

UNIX/LINUX Architecture

7

What is a “Shell”?

 The “Shell” is simply another program on top of the kernel which provides a basic human-OS interface. • It is a command interpreter  Built on top of the kernel  Enables users to run services provided by the UNIX OS • In its simplest form, a series of commands in a file is a shell program that saves having to retype commands to perform common tasks.

Shell  How to know what shell you use echo $SHELL user OS user

its.unc.edu

user

8

UNIX Shells

    sh Bourne Shell (Original Shell) (

Steven Bourne of AT&T

) bash Bourne Again Shell (

GNU Improved Bourne Shell

) csh C-Shell (C-like Syntax)(

Bill Joy of Univ. of California

) ksh Korn-Shell (Bourne+some C-shell)(

David Korn of AT&T

)   tcsh Turbo C-Shell (More User Friendly C-Shell).

To check shell: • $ echo $SHELL (shell is a pre-defined variable) 

its.unc.edu

To switch shell: • $ exec shellname (e.g., $ exec bash • or simply type $ bash ) You can switch from one shell to another by just typing the name of the shell. exit return you back to previous shell.

9

Which Shell to Use?

•        

sh

( Bourne shell) was considered better for programming

csh

(C-Shell ) was considered better for interactive work.

tcsh

and

korn

were improvements on c-shell and bourne shell respectively.

bash

is largely compatible with sh and also has many of the nice features of the other shells On many systems such as our LINUX clusters sh is symbolically linked to bash, /bin/sh -> /bin/bash We recommend that you use sh/bash for writing new shell scripts but learn csh/tcsh to understand existing scripts. Many, if not all, scientific applications require csh/tcsh environment (GUI, Graphics Utility Interface)

All Linux versions use the Bash shell default shell

Bash/Bourn/ksh/sh prompt: $ (Bourne Again Shell) as the All UNIX system include C shell and its predecessor Bourne shell.

Csh/tcsh prompt: % its.unc.edu

10

What is Shell Script?

A shell script is a script written for the shell

Two key ingredients

UNIX/LINUX commands

Shell programming syntax

its.unc.edu

11

A Shell Script Example

` ls *.log

| awk '{print $8}' | sed 's/.log//g' > file_list ` file_list| while read ' '# nosymmetry integral=Grid=UltraFine scf=tight rhf/6-311++g** pop=(nbo,chelpg) '> head ' ' >>head '' '' $each_file' opt pop nbo chelp aim charges ' >> head ' ' >>head ' 0 1 ' >>head ` sed '1,2d' $each_file.xyz > junk ` input= ./$each_file " .com

" cat junk echo >> ' ' >> $input $input done /bin/rm ./junk ./head ./file_list

its.unc.edu

12

UNIX/LINUX Commands

File Management and Viewing Filesystem Mangement Help,Job/Process Management Network Management System Management User Management Printing and Programming Document Preparation Miscellaneous

its.unc.edu

To understand the working of the command and possible options use ( man command) Using the GNU ( info , info command) Info System Listing a Description of a Program ( whatis command) Many option, tools have ` −−help ', a that long−style outputs usage information about the tool, including the options and arguments the tool takes. Ex:

whoami --help

13

File and Directory Management

cd

Change the current directory. With no arguments "cd" changes to the users home directory. (cd )

chmod

Change the file permissions.

Ex: chmod 751 myfile : change the file permissions to rwx for owner, rx for group and x for others ( x=1,r=4,w=2 ) Ex: chmod go=+r myfile : Add read permission for the group and others (character meanings u-user, g-group, o-other, + add permission,-remove,r-read,w-write,x-exe) Ex: chmod +s myfile - Setuid bit on the file which allows the program to run with user or group privileges of the file.

chown

Change owner.

Ex: chown : Change ownership of a file to owner1.

chgrp

Change group.

Ex: chgrp : Change group of a file to group1.

cp

Copy a file from one location to another.

Ex: cp file1 file2 : Copy file1 to file2; Ex: cp –R dir1 dir2 : Copy dir1 to dir2

its.unc.edu

14

File and Directory Management

its.unc.edu

ls

List contents of a directory.

Ex: ls, ls –l , ls –al, ls –ld, ls –R

mkdir

Make a directory.

Ex: mkdir : Makes a directory Ex

mkdir –p /www/chache/var/log

will create all the directories starting from www.

mv

Move or rename a file or directory.

Ex: mv

find

Find files (find -name -print) Ex:

find /home –name readme -print

Search for readme starting at home and output full path, “/home" = Search starting at the home directory and proceed through all its subdirectories; "-name readme" = Search for a file named readme "-print" = Output the full path to that file

locate

File locating program that uses the slocate database.

Ex: locate –u to create the database, locate to find file/directory

15

File and Directory Management

pwd

Print or list the present working directory with full path.

rm

Delete files (Remove files). (rm –rf )

rmdir

Remove a directory. The directory must be empty. (rmdir )

touch

Change file timestamps to the current time. Make the file if it doesn't exist. (touch )

whereis

Locate the ) binary and man page files for a command.

(whereis

which

Show full path of commands where given commands reside. (which )

File viewing and editing emacs

Full screen editor.

pico

Simple text editor.

vi

Editor with a command mode and text mode. Starts in command mode.

gedit

GUI Text Editor

tail

Look at the last 10 lines of a file.

Ex: tail –f ; Ex: tail -100

head

Look at the first 10 lines of a file. (head )

its.unc.edu

16

File and Directory Management

File compression, backing up and restoring compress

Compress data.

uncompress

Expand data.

cpio

Can store files on tapes. to/from archives.

gzip

- zip a file to a gz file.

gunzip

- unzip a gz file.

tar

Archives files and directories. Can store files and directories on tapes.

Ex: tar -zcvf - Archive copy groups of files. tar to uncompress –zxvf

zip

– Compresses a file to a .zip file.

unzip

– Uncompresses a file with .zip extension.

cat

View a file Ex: cat filename

cmp

Compare two files.

cut

Remove sections from each line of files.

its.unc.edu

17

File and Directory Management

its.unc.edu

diff

Show the differences between files.

Ex: diff file1 file2 : Find differences between file1 & file2.

echo

Display a line of text.

grep

List all files with the specified expression. (

grep pattern

>) Ex: ls –l |grep sidbi : List all lines with a sidbi in them.

Ex: grep " R " : Search for R with a space on each side

sleep

Delay for a specified amount of time.

sort

Sort a file alphabetically.

uniq

Remove duplicate lines from a sorted file.

wc

Count lines, words, characters in a file. (wc –c/w/l ).

sed

stream editor, extremely powerful!

awk

an extremely versatile programming language for working on files

18

Useful Commands in Scripting

 grep • • Pattern searching Example: grep ‘boo’ filename  sed • • Text editing Example: sed 's/XYZ/xyz/g' filename  awk • • Pattern scanning and processing Example: awk ‘{print $4, $7}’ filename

its.unc.edu

19

Shell Scripting

 Start vi scriptfilename.sh

with the line #!/bin/sh    All other lines starting with # are comments.

• make code readable by including comments Tell Unix that the script file is executable $ chmod u+x scriptfilename.sh

$ chmod +x scriptfilename.sh

Execute the shell-script $ ./scriptfilename.sh

its.unc.edu

20

My First Shell Script

$ vi myfirstscript.sh

#! /bin/sh its.unc.edu

# The first example of a shell script

directory=`pwd` echo Hello World!

echo The date today is `date` echo The current directory is $directory $ chmod +x myfirstscript.sh

$ ./myfirstscript.sh

Hello World!

The date today is Mon Mar 8 15:20:09 EST 2010 The current directory is /netscr/shubin/test 21

Shell Scripts

 

Text files that contain sequences of UNIX commands , created by a text editor No compiler required to run a shell script, because the UNIX shell acts as an interpreter when reading script files

 

After you create a shell script, you simply tell the OS that the file is a program that can be executed, by using the chmod command to change the files’ mode to be executable Shell programs run before it is executed less quickly than compiled programs, because the shell must interpret each UNIX command inside the executable script file its.unc.edu

22

Commenting

   Lines starting with first line where #!

# are comments except the very indicates the location of the shell that will be run to execute the script.

On any line characters following an unquoted # are considered to be comments and ignored.

Comments are used to; • • • • • • Identify who wrote it and when Identify input variables Make code easy to read Explain complex code sections Version control tracking Record modifications

its.unc.edu

23

Quote Characters

There are three different quote characters with different behaviour. These are: “ : double quote , weak quote. If a string is enclosed in “ ” the references to variables (i.e $variable ) are replaced by their values. Also back-quote and escape \ characters are treated specially.

‘ : single quote , strong quote. Everything inside single quotes are taken literally, nothing is treated as special. ` : back quote . A string enclosed as such is treated as a command and the shell attempts to execute it. If the execution is successful the primary output from the command replaces the string.

Example: echo “Today is:” `date`

its.unc.edu

24

Echo

Echo command is well appreciated when trying to debug scripts. Syntax : echo {options} string Options: -e : expand \ (back-slash ) special characters -n : do not output a new-line at the end.

String can be a “weakly quoted” or a ‘strongly quoted’ string. In the weakly quoted strings the references to variables are replaced by the value of those variables before the output.

As well as the variables some special backslash_escaped symbols are expanded during the output. If such expansions are required the –e option must be used.

its.unc.edu

25

User Input During Shell Script Execution

    As shown on the hello script input from the standard input location is done via the read command.

Example echo "Please enter three filenames:” read filea fileb filec echo “These files are used:$filea $fileb $filec” Each read statement reads an entire line. In the above example if there are less than 3 items in the response the trailing variables will be set to blank ‘ ‘.

Three items are separated by one space.

its.unc.edu

26

Hello script exercise continued…

 The following script asks the user to enter his name and displays a personalised hello. #!/bin/sh echo “Who am I talking to?” read user_name echo “Hello $user_name”  Try replacing “ with ‘ in the last line to see what happens.

its.unc.edu

27

its.unc.edu

Debugging your shell scripts

 Generous use of the echo command will help.

 Run script with the –x parameter. E.g. or sh –x ./myscript set –o xtrace before running the script.

 These options can be added to the first line of the script where the shell is defined.

e.g. #!/bin/sh -xv

28

Shell Programming

Programming features of the UNIX/LINUX shell:

Shell variables

: Your scripts often need to keep values in memory for later use. Shell variables are symbolic names that can access values stored in memory

Operators

: Shell scripts support many operators, including those for performing mathematical operations

Logic structures

logic : Shell scripts support sequential logic performing a series of commands), decision logic (for branching from one point in a script to another), looping (for repeating a command several times), and case (for logic (for choosing an action from several possible alternatives) its.unc.edu

29

Variables

 

Variables are symbolic names that represent values stored in memory Three different types of variables

• Global Variables

: Environment and configuration variables, capitalized, such as HOME, PATH, SHELL, USERNAME, and PWD.

When you login, there will be a large number of global System variables that are already defined. These can be freely referenced and used in your shell scripts. • Local Variables Within a shell script, you can create as many new variables as needed. Any variable created in this manner remains in existence only within that shell.

• Special Variables Reversed for OS, shell programming, etc. such as positional parameters $0, $1 …

its.unc.edu

30

SHELL DISPLAY HOME PATH MANPATH PS1 & PS2

its.unc.edu

USER TERM PWD

A few global (environment) variables

Current shell Used by X-Windows system to identify the display Fully qualified name of your login directory Search path for commands Search path for pages Primary and Secondary prompt strings Your login name terminal type Current working directory

31

Referencing Variables

Variable contents are accessed using ‘ $ ’: e.g. $ echo $HOME $ echo $SHELL To see a list of your environment variables : $ printenv or: $ printenv | more its.unc.edu

32

Defining Local Variables

   As in any other programming language, variables can be defined and used in shell scripts.

Unlike other programming languages, variables in Shell Scripts are not typed.

Examples : a=1234 # a is NOT an integer, a string instead b=$a+1 # will not perform arithmetic but be the string ‘ 1234+1 ’ b=`expr $a + 1 ` will perform arithmetic so b is 1235 now.

Note : +,-,/,*,**, % operators are available. b=abcde # b is string b=‘abcde’ # same as above but much safer.

b=abc def # will not work unless ‘quoted’ b=‘abc def’ # i.e. this will work. IMPORTANT NOTE: DO NOT LEAVE SPACES AROUND THE =

its.unc.edu

33

Referencing variables --curly bracket

  Having defined a variable, its contents can be referenced by the $ symbol. E.g. ${variable} or simply $variable . When ambiguity exists $variable will not work. Use ${ } the rigorous form to be on the safe side.

Example: a=‘abc’ b=${a}def # this would not have worked without the{ } as #it would try to access a variable named adef

its.unc.edu

34

Variable List/Arrary

its.unc.edu

 

To create lists (array) – round bracket $ set Y = (UNL 123 CS251) To set a list element – square bracket $ set Y[2] = HUSKER

To view a list element: $ echo $Y[2]

Example: Results: #!/bin/sh a=(1 2 3) echo ${a[*]} echo ${a[0]} 1 2 3 1 35

Positional Parameters

        When a shell script is invoked with a set of command line parameters each of these parameters are copied into special variables that can be accessed.

$0 This variable that contains the name of the script $1, $2, ….. $n 1 st , 2 nd 3 rd command line parameter $# Number of command line parameters $$ $@ process ID of the shell same as $* but as a list one at a time (see for loops later ) $?

Return code ‘exit code’ of the last command Shift command: This shell command shifts the positional parameters by one towards the beginning and drops $1 from the list. After a shift $2 becomes $1 , and so on … It is a useful command for processing the input parameters one at a time.

Example: Invoke : ./myscript one two buckle my shoe During the execution of myscrip t variables $1 $2 $3 $4 and $5 will contain the values

one, two, buckle, my, shoe

respectively.

its.unc.edu

36

vi myinputs.sh

#! /bin/sh echo Total number of inputs: $# echo First input: $1 echo Second input: $2

chmod u+x myinputs.sh

myinputs.sh HUSKER UNL CSE Total number of inputs: 3 First input: HUSKER Second input: UNL its.unc.edu

Variables

37

Shell Programming

programming features of the UNIX shell:

Shell variables

Operators

Logic structures

its.unc.edu

38

Shell Operators

The Bash/Bourne/ksh shell operators are divided into three groups: defining and evaluating operators, arithmetic operators, and redirecting and piping operators its.unc.edu

39

Defining and Evaluating

 

A shell variable take on the generalized form variable=value (except in the C shell).

$ set x=37; echo $x

37

$ unset x; echo $x x: Undefined variable.

You can set a pathname or a command to a variable or substitute to set the variable.

$ set mydir=`pwd`; echo $mydir its.unc.edu

40

Linux Commands

Pipes & Redirecting

its.unc.edu

Piping : An important early development in Unix , a way to pass the output of one tool to the input of another.

$ who | wc −l By combining these two tools, giving the wc command the output of who, you can build a new command to list the number of users currently on the system Redirecting via angle brackets

a similar principle to that of piping except that redirects work with files, not commands.

:

Redirecting input and output follows

tr '[a-z]' '[A-Z]' < $in_file > $out_file

The command must come first, the in_file is directed in by the less_than sign (<) and the out_file is pointed at by the greater_than sign (>).

41

Arithmetic Operators

expr

• • • • •

supports the following operators:

arithmetic operators: +,-,*,/,% comparison operators: <, <=, ==, !=, >=, > boolean/logical operators: &, | parentheses: (, ) precedence is the same as C, Java its.unc.edu

42

Arithmetic Operators

vi math.sh

#!/bin/sh count=5 count=`expr $count + 1 ` echo $count

 

chmod u+x math.sh

math.sh

6

its.unc.edu

43

Arithmetic Operators

vi real.sh

#!/bin/sh a=5.48

b=10.32

c=`echo “scale=2; $a + $b” |bc` echo $c

 

chmod u+x real.sh

./real.sh

15.80

its.unc.edu

44

its.unc.edu

Arithmetic operations in shell scripts

var++ ,var-- , ++var , - var + , * , / , % ** ! , ~ & , | && || post/pre increment/decrement add subtract multiply/divide, remainder power of logical/bitwise negation bitwise AND, OR logical AND, OR

45

Shell Programming

programming features of the UNIX shell:

Shell variables

Operators

Logic structures

its.unc.edu

46

Shell Logic Structures

The four basic logic structures needed for program development are:

Sequential logic: to execute commands in the order in which they appear in the program

Decision logic: to execute commands only if a certain condition is satisfied

Looping logic: to repeat a series of commands for a given number of times

Case logic: to replace “if then/else if/else” statements when making numerous comparisons its.unc.edu

47

its.unc.edu

Conditional Statements (if constructs )

The most general form of the if construct is; if command executes successfully then elif then execute command this command executes successfully execute this command and execute this command else execute default command fi

However- elif and/or else clause can be omitted.

48

Examples

SIMPLE EXAMPLE: if date | grep “Fri” then echo “It’s Friday!” fi FULL EXAMPLE: if [ “$1” == “Monday” ] then echo “The typed argument is Monday.” elif [ “$1” == “Tuesday” ] then echo “Typed argument is Tuesday” else echo “Typed argument is neither Monday nor Tuesday” fi its.unc.edu

# Note: = or == will both work in the test but == is better for readability.

49

Tests

String and numeric comparisons used with test or [[ ]] which is an alias for test and also [ ] which is another acceptable syntax       string1 = string2 String1 == string2 string1 !=string2 string -n string -z string True if strings are identical … ditto … .

True if strings are not identical Return 0 exit status (=true) if string is not null Return 0 exit status (=true) if string is not null Return 0 exit status (=true) if string is null       int1 –eq int2 int1 –ne int2 int1 –lt int2 int1 –gt int2 int1 –le int2 int1 –ge int2 Test identity Test inequality Less than Greater than Less than or equal Greater than or equal

its.unc.edu

50

Combining tests with logical operators || (or) and && (and)

Syntax: if cond1 && cond2 || cond3 … An alternative form is to use a compound statement using the –a and –o keywords, i.e. if cond1 –a cond22 –o cond3 … Where cond1,2,3 .. Are either commands returning a a value or test conditions of the form [ ] or test … Examples: if date | grep “Fri” && `date +’%H’` -gt 17 then echo “It’s Friday, it’s home time!!!” fi

its.unc.edu

if [ “$a” –lt 0 –o “$a” –gt 100 ] # note the spaces around ] and [ then echo “ limits exceeded” fi

51

File enquiry operations

-d file -f file -s file -r file -w file -x file -o file -e file -z file Test if file is a directory Test if file is not a directory Test if the file has non zero length Test if the file is readable Test if the file is writable Test if the file is executable Test if the file is owned by the user Test if the file exists Test if the file has zero length All these conditions return true if satisfied and false otherwise.

its.unc.edu

52

Decision Logic

A simple example #!/bin/sh if [ “$#” -ne 2 ] then echo $0 needs two parameters!

echo You are inputting $# parameters.

else par1=$1 par2=$2 fi echo $par1 echo $par2 its.unc.edu

53

Decision Logic

Another example: #! /bin/sh # number is positive, zero or negative echo – e "enter a number:\c" read number if [ “$number” -lt 0 ] then echo "negative" elif [ “$number” -eq 0 ] then echo zero else echo positive fi its.unc.edu

54

Loops

Loop is a block of code that is repeated a number of times.

The repeating is performed either a pre determined number of times determined by a list of items in the loop count ( for loops ) or until a particular condition is satisfied ( while and until loops ) To provide flexibility to the loop constructs there are also two statements namely break and continue are provided.

its.unc.edu

55

for loops

Syntax: for

arg

do done in

list command(s)

...

Where the value of the variable

arg

is set to the values provided in the list one at a time and the block of statements executed. This is repeated until the list is exhausted. Example:

for i in 3 2 5 7 do done echo " $i times 5 is $(( $i * 5 )) "

its.unc.edu

56

The while Loop

A different pattern for looping is created using the while statement

The while statement best illustrates how to set up a loop to test repeatedly for a matching condition

The while loop tests an expression in a manner similar to the if statement

As long as the statement inside the brackets is true, the statements inside the do and done statements repeat its.unc.edu

57

while loops

its.unc.edu

Syntax: while this_command_execute_successfully do this command and this command done EXAMPLE:

while test "$i" -gt 0 # can also be while [ $i > 0 ] do i=`expr $i - 1` done

58

Looping Logic

Example: #!/bin/sh for person do in

Bob Susan Joe Gerry

echo Hello $person done Output: Hello Bob Hello Susan Hello Joe Hello Gerry

Adding integers from 1 to 10 #!/bin/sh i=1 sum=0 while [ “$ i ” -le 10 ] do echo Adding $i into the sum.

sum=`expr $sum + $i ` i=`expr $i + 1 ` done echo The sum is $sum.

its.unc.edu

59

until loops

The syntax and usage is almost identical to the while loops.

Except that the block is executed until the test condition is satisfied, which is the opposite of the effect of test condition in while loops. Note: You can think of until as equivalent to not_while Syntax: until test do commands ….

done

its.unc.edu

60

Switch/Case Logic

The switch logic structure simplifies the selection of a match when you have a list of choices

It allows your program to perform one of many actions, depending upon the value of a variable its.unc.edu

61

Case statements

The case structure compares a string ‘usually contained in a variable’ to one or more patterns and executes a block of code associated with the matching pattern. Matching tests start with the first pattern and the subsequent patterns are tested only if no match is not found so far.

case argument in pattern 1) execute this command and this and this;; pattern 2) execute this command and this and this;;

its.unc.edu

esac

62

Functions

 Functions are a way of grouping together commands so that they can later be executed via a single reference to their name. If the same set of instructions have to be repeated in more than one part of the code, this will save a lot of coding and also reduce possibility of typing errors.

SYNTAX: functionname() { block of commands }

#!/bin/sh sum() { x=`expr $1 + $2` echo $x } sum 5 3 echo "The sum of 4 and 7 is `sum 4 7`"

its.unc.edu

63

Take-Home Message

     

Shell script is a high-level language that must be converted into a low-level (machine) language by UNIX Shell before the computer can execute it UNIX shell scripts, created with the vi or other text editor, contain two key ingredients: a selection of UNIX commands glued together by Shell programming syntax UNIX/Linux shells are derived from the UNIX Bourne , Korn , and C/TCSH shells UNIX keeps three types of variables:

Configuration; environmental; local The shell supports numerous operators, including many for performing arithmetic operations The logic structures supported by the shell are sequential , decision , looping , and case its.unc.edu

64

To Script or Not to Script

 Pros • • • • • File processing Glue together compelling, customized testing utilities Create powerful, tailor-made manufacturing tools Cross-platform support Custom testing and debugging  Cons • • Performance slowdown Accurate scientific computing

its.unc.edu

65

Shell Scripting Examples

   

Input file preparation Job submission Job monitoring Results processing

its.unc.edu

66

Input file preparation

#!/bin/sh `ls -l *.log| awk '{print $8}' |sed 's/.log//g' > file_list` cat file_list|while read each_file do babel -ig03 $each_file".log" -oxyz $each_file".xyz“ echo '# nosymmetry integral=Grid=UltraFine scf=tight rhf/6-311++g** pop=(nbo,chelpg)'>head echo ' ' >>head echo ''$each_file' opt pop nbo chelp aim charges ' >> head echo ' ' >>head echo '0 1 ' >>head `sed '1,2d' $each_file.xyz >junk` input=./$each_file".com" cat head > $input cat junk >> $input echo ' ' >> $input done /bin/rm ./junk ./head ./file_list

its.unc.edu

67

its.unc.edu

LSF Job Submission

$ vi submission.sh

#!/bin/sh -f #BSUB -q week #BSUB -n 4 #BSUB -o output #BSUB -J job_type #BSUB -R “RH5 span[ptile=4]” #BSUB -a mpichp4 mpirun.lsf ./executable.exe

exit $chmod +x submission.sh

$bsub < submission.sh

68

Results Processing

its.unc.edu

#!/bin/sh `ls -l *.out| awk '{print $8}'|sed 's/.out//g' > file_list` cat file_list|while read each_file do file1=./$each_file".out" Ts=`grep 'Kinetic energy =' $file1 |tail -n 1|awk '{print $4}' ` Tw=`grep 'Total Steric Energy:' $file1 |tail -n 1|awk '{print $4}' ` TsVne=`grep 'One electron energy =' $file1 |tail -n 1|awk '{print $5}' ` Vnn=`grep 'Nuclear repulsion energy' $file1 |tail -n 1|awk '{print $5}' ` J=`grep 'Coulomb energy =' $file1 |tail -n 1|awk '{print $4}' ` Ex=`grep 'Exchange energy =' $file1 |tail -n 1|awk '{print $4}' ` Ec=`grep 'Correlation energy =' $file1 |tail -n 1|awk '{print $4}' ` Etot=`grep 'Total DFT energy =' $file1 |tail -n 1|awk '{print $5}' ` HOMO=`grep 'Vector' $file1 | grep 'Occ=2.00'|tail -n 1|cut -c35-47|sed 's/D/E/g' ` orb=`grep 'Vector' $file1 | grep 'Occ=2.00'|tail -n 1|awk '{print $2}' ` orb=`expr $orb + 1 ` LUMO=`grep 'Vector' $file1 |grep 'Occ=0.00'|grep ' '$orb' ' |tail -n 1|cut -c35-47|sed 's/D/E/g' echo $each_file $Etot $Ts $Tw $TsVne $J $Vnn $Ex $Ec $HOMO $LUMO $steric >>out done /bin/rm file_list

69

its.unc.edu

Reference Books

      

Class Shell Scripting

http://oreilly.com/catalog/9780596005955/

LINUX Shell Scripting With Bash

http://ebooks.ebookmall.com/title/linux-shell-scripting-with-bash-burtch ebooks.htm

Shell Script in C Shell http://www.grymoire.com/Unix/CshTop10.txt

Linux Shell Scripting Tutorial

http://www.freeos.com/guides/lsst/

Bash Shell Programming in Linux

http://www.arachnoid.com/linux/shell_programming.html

Advanced Bash-Scripting Guide

http://tldp.org/LDP/abs/html/

Unix Shell Programming

http://ebooks.ebookmall.com/title/unix-shell-programming-kochan-wood ebooks.htm

70

Questions & Comments

Please direct comments/questions about research computing to

E-mail: [email protected]

Please direct comments/questions pertaining to this presentation to

E-Mail: [email protected]

The PPT file of this presentation is available here: http://its2.unc.edu/divisions/rc/training/scientific/short_courses/Shell_Scripting.ppt

its.unc.edu

Hands-on Exercises

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

The simplest Hello World shell script – Echo command Summation of two integers – If block Summation of two real numbers – bc (basic calculator) command Script to find out the biggest number in 3 numbers – If –elif block Operation (summation, subtraction, multiplication and division) of two numbers – Switch Script to reverse a given number – While block A more complicated greeting shell script Sort the given five numbers in ascending order (using array) – Do loop and array Calculating average of given numbers on command line arguments – Do loop Calculating factorial of a given number – While block An application in research computing – Combining all above Optional : Write own shell scripts for your own purposes if time permits

its.unc.edu

The PPT/WORD format of this presentation is available here: http://its2.unc.edu/divisions/rc/training/scientific/ /afs/isis/depts/its/public_html/divisions/rc/training/scientific/short_courses/

72