Transcript Programming Fundamentals

CCPR Computing Services
Workshop 1:
Programming Basics, Unix, Remote Computing
October 13, 2004
Part 1: Programming Basics



Motivation
Before you start coding
Programming Conventions





Documentation
Names, comments
Directory Structure
Basic Constructs
Miscellaneous (debugging, cross-checking
results)
Motivation





Facilitate research
Save time
Cleaner code
Easily share programs
Basic Concepts
MUCH better programming
Programming Conventions

What are conventions?


Who cares?




Examples
Readability of code
Organization
Transferring code to others
Apply conventions consistently to:



variable names and function names
comments
directory structure
Before you start coding…



THINK
WRITE down the problem
WRITE down the algorithm in English (not code)






Modularity
Comments
Create test (if reasonable)
TRANSLATE one section to code
TEST the SECTION thoroughly
Translate/Test next section, etc.
Documentation - File Header

File header includes:
 Name
 Project
 Project location
 Date
 Version of software
 Purpose
 Inputs
 Outputs
 Special Instructions
#Laura Piersol ([email protected])
#HRS project
#/u/socio/laurapiersol/HRS/
#October 11, 2004
#Python version 2.4
#Stata version 8
#Purpose: Create and merge two datasets in Stata,
# then convert data to SAS
#Input programs:
# HRS/staprog/H2002.do,
# HRS/staprog/x2002.do,
# HRS/staprog/mergeFiles.do
#Output:
# HRS/stalog/H2002.log,
# HRS/stalog/x2002.log,
# HRS/stalog/mergeFiles.log
# HRS/stadata/Hx2002.dta
# HRS/sasdata/Hx2002.sas
#Special instructions: Check log files for errors
# check for duplicates upon new data release
Naming Conventions




Not a detail!
Good names clarify your code
Portray meaning/purpose
Adopt a convention and BE CONSISTENT
Naming Conventions, cont.


Use language standard (if it exists)
If no standard, pick one and BE CONSISTENT
Functions:
Scalar variables:
String variables:
Global variables:

getStats, calcBetas, showResults
scPi, scGravity, scWorkHours
stName, stCareer
_Country, _Nbhd
Be aware of language-specific rules

Max length, underscore, case, reserved words
Naming Conventions, cont.

Differentiating log files:




LogWt vs. var1
AgeLt30 vs. x
Procedure that cleans missing values of Age:


MergeHH.sas, MergeHH.do
MergeHHsas.log, MergeHHsta.log
Meaningful variable names:


Programs
Log files
fixMissingAge
Matrix multiplication X transpose times X

matXX
Commenting Code
Good code is SELF-COMMENTING


Naming conventions, structure, header explain 95%
Comments explain




PURPOSE, not every detail
TRICKS
(good) reasons for unusual coding
Comments DO NOT



fix sloppy code
translate syntax
Commenting Code - Stata example
SAMPLE 1
program def function1
foreach v of varlist _all {
local x = lower("`v'")
if `"`v'"' != `"`x'"' {
rename `v' `=lower("`v'")'
}
}
end
SAMPLE 2
*Convert names in dataset to lowercase.
program def lowerVarNames
foreach v of varlist _all {
local LowName = lower("`v'")
*If variable is already lowercase,
*rename statement throws error.
if `"`v'"' != `"`LowName'"' {
rename `v' `=lower("`v'")'
}
}
end
No conventions, comments, structure
Comments: succinct and not overdone
Names:
lowerVarNames
-action word for program
-distinct use of case
LowName
-descriptive
-distinct use of case
v
-looping variable and short scope
-non-descriptive, but does not detract
from meaning!
Structure- indentations, parentheses lined up!
Directory Structure




A project consists of
many different types of
files
Use folders to
SEPARATE files in a
logical way
Be consistent across
projects if possible
ATTIC folder for older
versions
HOME
PROJECT NAME
DATA
RESULTS
LOG
PROGRAMS
ATTIC
Miscellaneous Tips





BACKUP! Weekly zip file stored externally
README.txt file to describe folder
BE ORGANIZED
CROSS-VERIFY results
Something not working?

Remember the computer is following YOUR
directions… go back to your code
Programming Constructs


Tools to simplify and clarify your coding
Available in virtually all languages
Constructs - Looping



Repeat section of code
START value, INCREMENT, STOP value
Example
convert uppercase to lowercase for each variable
in a dataset
Constructs – Looping Examples
C for loop: Start with x=1, Increment = x+1, Stop when x==10
for(x=1; x<10; x++) {
…code…
}
PERL while loop: Start with count= 1, Increment= count+1, Stop when count==11
$count=1;
while ($count<11) {
print "$count\n";
$count++;
}
STATA foreach loop: Start = first variable in varlist, Increment = next variable in
varlist, Stop =last variable in varlist
foreach v of varlist _all {
local LowName = lower("`v'")
if `"`v'"' != `"`LowName'"' {
rename `v' `=lower("`v'")'
}
}
Constructs - If/then/else

Execute section of code if condition is true:
if condition then
{execute this code if condition true}
end

Execute one of two sections of code:
if condition then
{execute this code if condition true}
else
{execute this code if condition false}
end
Constructs - Elseif/case

Elseif - Execute one of many sections of code:
if condition1 then
{execute this code if condition1 true}
elseif condition2 then
{execute this code if condition2 true}
else
{execute this code if condition1, condition2, condition3 are all false}
end

Case- same idea, different name
case condition1 then
{execute this code if condition1 true}
case condition2 then
{execute this code if condition2 true}
etc.
Constructs - And, or, xor
AND OR XOR -
BOTH conditions must be true results in True
AT LEAST ONE condition must be true results in True
EXACTLY ONE condition must be true for statement to be true
1
AND
1
True
1
AND
0
False
0
AND
0
False
1
OR
1
True
1
OR
0
True
0
OR
0
False
1
XOR
1
False
1
XOR
0
True
0
XOR
0
False
Constructs - Break


Stop execution of program
Examples:


Debugging. If particular error occurs then break.
Parameters in function call are nonsensical. Print
error and break.
Constructs - keywords




Looping - for, foreach, do, while
If statements – if, then, else, case
And/Or/Xor – logical, and, or, xor, &, |
Break – exit, break
PART 2: Unix






Motivation
Basic Commands
Job submission and management
Pipes
Unix Shell
Script files
Unix



Motivation
A quick history
Unix variants


(AIX, Solaris, FreeBSD, Linux)
Where?



Nicco (SSC’s server)
CCPR’s linux cluster (coming soon)
CCPR’s data server (coming soon)
Unix – Basic Commands
Getting Help
man command
list help for command (man if short for
manual)
man -k command
keyword search for command
whatis command
give a brief description of command
apropos keyword
list commands with keyword in the NAME
section their man page
Unix – Basic Commands
File Management
ls
list files (options –l for long, -a for all)
mv filename1 filename2
rename filename1 with the name filename2
cp filename1 filename2
make a copy of filename1 and call it filename2
rm filename
delete filename
more filename
print contents of filename1 to the screen
cat filename
print contents of filename to the screen
cat filename1 >>
filename2
append contents of filename1 to the file filename2
cat part1 part2 >>
bothparts
appends the contents of file part1 and file part to to the
file bothparts
head filename
show first 10 lines to screen
tail filename
show last 10 lines to screen
Unix – Basic Commands
Directory Management
pwd
show current directory
mkdir dirname
create new directory dirname
rmdir dirname
remove directory dirname
cd dirname
Change to directory dirname
cd
change to your home directory
cd ..
move one directory up
cd ../..
move two directories up
~
home directory
cd ~/scripts
change to the scripts folder in your home directory
Unix – Basic Commands
Using Previous Commands
!!
repeat last command
!v
repeat the last command that started with v
!-2
repeat second to last command
arrow up/down
scroll through list of previous commands
history
list history of commands used
Unix – Basic Commands
Other Useful Unix Tools
*
wildcard (matches any number of characters)
?
wildcard (matches single character)
grep word filename
list lines of filename containing word
diff filename1
filename2
shows differences between filename1 and filename2
wc filename
counts number of lines, words, and characters in
filename
sort < filename
sorts the lines of filename
who
lists users currently on the system
cal
displays current month's calendar
date
displays date
Unix – Basic Commands
Disk Usage
du –s
Total kilobytes used in current directory
du –a
Same as above, but more detail
ls –l
Gives individual file sizes in bytes
Editing Files in Unix

Vi Editor and Emacs




Neither are user-friendly for starters
Look at CCPR internet when you start
Best way to learn is to start editing a document
Once you get used to them, they’re easy and fast to
use.
Being nice



Always run your jobs “nicely”
Prevents interfering with other users
Precede command with “nice +19” (no quotes)
user@nicco%nice +19 stata –b jobfile.do
Job Submission in Unix

Interactive
user@nicco%stata

Foreground jobs
user@nicco%nice +19 stata –b do jobfile.do
user@nicco%nice +19 sas jobfile.sas

Background jobs – &
user@nicco%nice +19 stata –b do jobfile.do &
user@nicco%nice +19 sas jobfile.sas &

Background jobs with logoff – nohup
user@nicco%nohup nice +19 stata –b do jobfile.do &
user@nicco%nohup nice +19 sas jobfile.sas &
Job Management
Ctrl-c
Cancel a foreground job
Ctrl-z
Suspend a job in the foreground
bg
Move a suspended foreground job to the background
ps –u
List information for processes you own (under current shell)
ps –ux
Lists information for process owned by you and others
ps –aux
Lists information for all processes (including root, bin, etc.)
ps –aux | more
Display output of ps –aux one page at a time
ps –aux | grep PID
List lines from command ps –aux containing PID
kill PID
Kills (cancels) process number PID
top
Lists 15 processes using the most cpu processing power
q
Stops command top
Pipes

“|” redirects
command 1 output
command 2 input
Command
Output from Input to
Result
top | grep piersol
top
grep piersol
Jobs in top 15 containing piersol
ps -aux | more
Ps –aux
More
Jobs listed one page at a time
ls | wc -l
ls
Wc –l
Count number of files and
directories in current directory

Extends to more than 2 commands
Unix Shell

What’s a Unix Shell?


What’s a shell script?


The Unix shell is the program that provides the
interface between the user and the kernel
A list of commands put into a file that can be
interpreted by the Unix Shell.
What are scripting languages?


Generally easier to code but less efficient
Shell scripts, Perl, Python
Remote Computing
Windows to Whitney
Remote Login
Remote Desktop
Connection
>= XP included*
< XP download*
Web-based
Remote Desktop
Connection*
File Transfer
Map a Drive via
Windows Explorer
*See http://www.ccpr.ucla.edu/asp/compserv.asp, XP users get latest version
Remote Computing
Windows to Unix
Remote Login
SSH Secure
Shell Client*
File transfer
SSH Secure
File Transfer Client*
* http://computing.sscnet.ucla.edu/training/tutorial_SSH.htm
** http://computing.sscnet.ucla.edu/training/tutorial_samba.htm
Map a Drive via
Windows Explorer –
Need Samba account**
Finally…

Questions and Feedback