Transcript FAST High-level Overview - University of Pennsylvania

Administering PennGroups
Administering PennGroups
Chris Hyzer
ISC/ASTT
August 1, 2011
7/15/2015
ISC
1
Administering PennGroups
Administering PennGroups

Install
–
–
–
–




DDL
API
UI
WS
Source control
Upgrade
Maintenance
Roadmap
7/15/2015
ISC
2
Administering PennGroups
Demo server

Internet2 has a Grouper Demo Server
 Address is: https://grouperdemo.internet2.edu/
 Host various versions of Grouper
 Show features (e.g. permissions, external users,
syncing between groupers)
 Allow users or potential users to kick the tires (not for
production obviously)
Administering PennGroups
Demo server for this training





9 accounts setup
/home/test1, /home/test2, etc
Each participant gets a temporary account and pass
(ask Chris)
The pass will work for SSH, Apache (UI, WS), mysql
Each account has a mysql database, and user with
access to that database
Administering PennGroups
Demo server training tomcats

Except for development, you should only have one
application per Tomcat
– One application wont take all memory and kill tomcat
– You can restart after deploys without affecting other apps

Note: for development, you can use either basic
authentication
– Or link cosign/shib tomcat on test cluster to dev tomcat
Administering PennGroups
Demo server training tomcats (continued)

Install dir:
[test6@i2midev1 tomcat]$ pwd
/home/test6/tomcat
[test6@i2midev1 tomcat]$ ls
conf logs temp webapps work

TOMCAT_HOME, CATALINA_HOME elsewhere
[test6@i2midev1 tomcat]$ more \
/etc/init.d/tomcat_test6
…
export TOMCAT_HOME="/opt/tomcat6base"
export CATALINA_HOME="/opt/tomcat6base"
Administering PennGroups
Demo server training tomcats (continued)

Note you have your own Java symlink (for ps)
[test6@i2midev1 tomcat]$ more /etc/init.d/tomcat_test6
…
export JAVA_HOME="/opt/javas/java_test6"

Start/stop tomcat, lets run these commands (for your user)
[test6@i2midev1 ~]$ tomcat start
[test6@i2midev1 ~]$ ps -ef | grep java_test6
[test6@i2midev1 ~]$ netstat -pan | grep 5231
(note the port numbers)
[test6@i2midev1 ~]$ jstack 5231 | less
[test6@i2midev1 ~]$ tomcat stop
[test6@i2midev1 ~]$ ps -ef | grep java_test6
Administering PennGroups
Demo server training tomcats (continued)
To set this up:
$ sudo useradd -g i2mi test1
$ sudo passwd test1
create user in DB, database assign permissions with sql yog
mysql> create database grouper;
mysql> create user 'grouper'@'localhost' identified by
'somesecret';
mysql> grant all on grouper.* to 'grouper'@'localhost';
$ sudo htpasswd /etc/httpd/conf.d/users.pass test1
$ sudo htpasswd /etc/httpd/conf.d/users.pass test1a
$ cp -R /opt/tomcats/tomcat6bullet /home/test6/tomcat
ports start at 9010, edit the ~/tomcat/conf/server.xml, set the three ports:
http: 90x0, jk: 90x1, shutdown: 90x2
[root@i2midev1 init.d]# cp tomcat_k tomcat_test1
[root@i2midev1 init.d]# chkconfig --add tomcat_test1
[appadmin@i2midev1 javas]$ ln -s ../java6 java_test1
[test1@i2midev1 test1]$ mkdir bin
[test1@i2midev1 bin]$ ln -s /etc/init.d/tomcat_test1
tomcat

Administering PennGroups
Demo server training tomcats (continued)

Put this in /etc/profile
JAVA_HOME=/opt/java6
export JAVA_HOME
ANT_HOME=/opt/ant
export ANT_HOME
#note: maven3 is needed for grouper 2.0
M2_HOME=/opt/maven
export M2_HOME
M2=$M2_HOME/bin
export M2
PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$M2:$PATH:$HOME/bin
export PATH
Administering PennGroups
Demo server training tomcats (continued)

Put this in /var/www/html
<b>test1</b>
<ul>
<li><a href="test1_grouper/">Grouper UI</a></li>
<li><a href="test1_grouperWs/">Grouper WS</a></li>
</ul>

Put this in /etc/httpd/conf.d/proxy_ajp.conf
ProxyPass /test1_grouper/ ajp://localhost:9011/test1_grouper/
ProxyPass /test1_grouperWs/ ajp://localhost:9011/test1_grouperWs/

Bounce apache
[root@i2midev1 init.d]# /sbin/service httpd
configtest
Syntax OK
[root@i2midev1 init.d]# /sbin/service httpd
graceful
[root@i2midev1 init.d]#
Administering PennGroups
Install Grouper API (First step in training)

Download the API (1.6.3, latest stable)
$ mkdir 1.6.3
$ cd 1.6.3
$ pwd
/home/test1/1.6.3
$ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.apiBinary-1.6.3.tar.gz
$ tar xzvf grouper.apiBinary-1.6.3.tar.gz
$ cd grouper.apiBinary-1.6.3/conf/
$ emacs grouper.properties
Administering PennGroups
Install Grouper API (continued)

Note, Powerpoint might mess-up chars (e.g. dashes)…
might need to type them in if there is a problem, instead of
copy/paste 

Note, a backslash: \ means the next line is the same line 
Note, all commands are in the "notes" section of the
presentation, copy/paste from there

Administering PennGroups
Emacs cheatsheet
Note: Feel free to use your favorite editor obviously
$ emacs somefile.whatever
Note: might need to
Save:
Exit:
Find:
Find (wrap):
Find backwards:
Stop command:
Replace:
Cut rest of line:
Paste:
Put in background:
BASH get back:
do CTRL-backspace instead of backspace
CTRL-x CTRL-s
CTRL-x CTRL-c
CTRL-s
CTRL-s, put in criteria, CTRL-s
CTRL-r
CTRL-g
ESC-x query-replace (then “y” to replace each)
CTRL-k
CTRL-y
CTRL-z
fg
Administering PennGroups
Configure grouper.properties
$ emacs grouper.properties
groups.wheel.use = true
configuration.autocreate.system.groups = true
configuration.autocreate.group.name.0 = etc:webServiceClientUsers
configuration.autocreate.group.description.0 = users allowed in WS
configuration.autocreate.group.subjects.0 = GrouperSystem,testX,testXa
configuration.autocreate.group.name.1 = etc:sysadmingroup
configuration.autocreate.group.description.1 = sys admin users
configuration.autocreate.group.subjects.1 = testX
grouperIncludeExclude.use = true
grouperIncludeExclude.requireGroups.use = true
Administering PennGroups
Configure
grouper.hibernate.properties
$ emacs grouper.hibernate.properties
hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
hibernate.connection.driver_class = com.mysql.jdbc.Driver
hibernate.connection.url = jdbc:mysql://localhost:3306/testX
hibernate.connection.username = testX
hibernate.connection.password = *******************
Administering PennGroups
Configure grouper-loader.properties
$ emacs grouper-loader.properties
loader.autoadd.typesAttributes = true
Administering PennGroups
Init DB and startup GSH
$ mysql -utest1 -p test1
mysql> show tables;
Empty set (0.00 sec)
mysql> exit
Bye
$ cd /home/testX/1.6.3/grouper.apiBinary-1.6.3/bin/
$ ./gsh.sh -registry
<Shows menu>
$ ./gsh.sh -registry -check -runscript
$ ./gsh.sh -registry -check
$ mysql -utestX -p testX
mysql> show tables;
76 rows in set (0.00 sec)
mysql> exit
Bye
Administering PennGroups
Quickstart and subjects
$ wget -O quickstart.xml http://anonsvn.internet2.edu/cgi-\
bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\
builder/quickstart.xml?view=co
$ wget -O subjects.sql http://anonsvn.internet2.edu/cgi-\
bin/viewvc.cgi/i2mi/tags/GROUPER_1_6_3/grouper-qs-\
builder/subjects.sql?view=co
$ ./gsh.sh -registry -runsqlfile subjects.sql
$ ./gsh.sh -xmlimportold GrouperSystem quickstart.xml
Administering PennGroups
MySQL GUI
Optional (if you have SQLYog free gui, or whatever)
Port forward over SSH, remote port 3306, local port whatever (3302?)
Connect to localhost 3302
In either case, lets open a GSH window, a mysql window, and a linux
window
Administering PennGroups
Add your users
$ ./gsh.sh
(note: testX was automatically inserted and into wheel and WS)
gsh 0% grouperSession = GrouperSession.startRootSession();
gsh 1% addSubject("testX", "person", "John Smith");
gsh 2% addSubject("testXa", "person", "Johna Smitha");
gsh 3% addMember("etc:webServiceClientUsers", "testXa");
gsh 4% exit
Review other GSH commands
Note: GrouperSession is a ThreadLocal ActAs for the API
Administering PennGroups
Add your users (continued)
$ mysql -utestX -p testX
INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\
VALUES ('testX', 'loginid', 'testX', 'testX');
INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\
VALUES ('testX', 'name', 'John Smith', 'john smith');
INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\
VALUES ('testX', 'description', 'John Smith', 'john smith');
INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\
VALUES ('testXa', 'loginid', 'testXa', 'testXa');
INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\
VALUES ('testXa', 'name', 'Johna Smitha', 'johna smitha');
INSERT INTO subjectattribute (subjectId, NAME, VALUE, searchValue)\
VALUES ('testXa', 'description', 'Johna Smitha', 'johna smitha');
COMMIT;
exit;
Administering PennGroups
Get the UI
$ cd ~/1.6.3
$ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ui-\
1.6.3.tar.gz
$ tar xzvf grouper.ui-1.6.3.tar.gz
$ cd grouper.ui-1.6.3/
$ cp build.properties.template build.properties
$ emacs build.properties
grouper.folder=../grouper.apiBinary-1.6.3
should.copy.context.xml.to.metainf=false
webapp.name=testX_grouper
default.webapp.folder=/home/testX/tomcat/webapps/${webapp.name}
Administering PennGroups
Get the UI (continued)
$ emacs webapp/WEB-INF/web.core.xml
From the bottom, remove security-contraint,login-config,security-role
Note, if you are running on tomcat locally, leave that in, and edit
tomcat-users.xml
$ ant default
$ tomcat restart
Go to: https://grouperdemo.internet2.edu and click on your UI:
https://grouperdemo.internet2.edu/testX_grouper/
Administering PennGroups
UI authentication
In this case, I added this to the httpd.conf or include:
<LocationMatch ^/test.*>
AuthType Basic
AuthName "By Invitation Only"
AuthUserFile /etc/httpd/conf.d/users.pass
Require valid-user
</LocationMatch>
Bounce apache:
[root@i2midev1 conf.d]# /sbin/service httpd configtest
Syntax OK
[root@i2midev1 conf.d]# /sbin/service httpd graceful
Authentication is pluggable, can write a servlet filter to put in
remote user
Works with Shib, CAS, Cosign, Basic (web server), Basic (app server),
any web server plugin, out of the box
Note: if UI is internal users, but using external users via invites,
might map multiple URLs to multiple authn schemes (like Penn or
demo server)
Administering PennGroups
Create objects




Act as admin in upper right.
Create a root folder named "test". (system and friendly)
Inside that folder (aka stem), create a group called
"testGroup", but uncheck allow all to "read" and "view".
(system and friendly name is testGroup)
Add some members to testGroup: babl, babr, babu
Administering PennGroups
Get the WS
$ cd ~/1.6.3
$ wget http://www.internet2.edu/grouper/release/1.6.3/grouper.ws-\
1.6.3.tar.gz
$ tar xzvf grouper.ws-1.6.3.tar.gz
$ cd grouper.ws-1.6.3/grouper-ws
$ emacs build.properties
grouper.dir=../../grouper.apiBinary-1.6.3
webapp.name=testX_grouperWs
$ emacs conf/grouper-ws.properties
ws.client.user.group.name = etc:webServiceClientUsers
$ emacs webapp/WEB-INF/web.xml
From the bottom, remove security-contraint,login-config,security-role
Note, if you are running on tomcat locally, leave that in, and edit
tomcat-users.xml
Administering PennGroups
Get the WS (continued)
$ ant dist
$ cp -R build/dist/testX_grouperWs ~/tomcat/webapps
$ tomcat restart
Go to: https://grouperdemo.internet2.edu and click on your WS:
https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest
Should get error  though there is a URL to get members… 
Administering PennGroups
WS authentication
In this case, it’s the same as UI:
<LocationMatch ^/test.*>
AuthType Basic
AuthName "By Invitation Only"
AuthUserFile /etc/httpd/conf.d/users.pass
Require valid-user
</LocationMatch>
Bounce apache:
[root@i2midev1 conf.d]# /sbin/service httpd configtest
Syntax OK
[root@i2midev1 conf.d]# /sbin/service httpd graceful
Authentication is pluggable, or can write a servlet filter to put in remote
user
Works with Kerberos, SOAP WS-security, Basic (web server), Basic (app
server), any web server plugin, out of the box
Administering PennGroups
Loader example
Lets
make a table with subjectIds in it
mysql> CREATE OR REPLACE VIEW loader_employee AS
\
(SELECT subjectId AS subject_id FROM testX.subject\
WHERE subjectId LIKE 'b%');
Make
a folder for community under root
Add a group called employee under folder "community"
Edit the employee group, select type "grouperLoader" (make
sure acting as admin in upper right of UI)
Administering PennGroups
Loader example (continued)
Edit attributes on the group:
grouperLoaderDbName: grouper
NOTE:
configure other DB connections in grouperloader.properties
NOTE: every minute just for testing…
grouperLoaderQuartzCron: 0 * * * * ?
grouperLoaderQuery: select subject_id subject_id from \
loader_employee
grouperLoaderScheduleType: CRON
grouperLoaderType: SQL_SIMPLE
Administering PennGroups
Loader example (continued)
Run
manually just to see it work:
gsh 0% grouperSession = GrouperSession.startRootSession();
gsh 1% loaderGroup = GroupFinder.findByName(grouperSession, \
"community:employee");
gsh 2% loaderRunOneJob(loaderGroup);
loader ran successfully, inserted 12 memberships, deleted 0
memberships, total membership count: 12
gsh 3%
Administering PennGroups
Loader example (continued)
Run continuously:
% ./gsh.sh -loader
Change the view:
mysql> CREATE OR REPLACE VIEW loader_employee AS
(SELECT subjectId AS subject_id FROM SUBJECT
\
\
WHERE subjectId LIKE 'b%o' or subjectId like 'el%');
At
the top of the minute, check the memberships, and there
should be some deleted, and some added
Administering PennGroups
Loader details at Penn
In
the fastGrouperProdDaemon web application, we run the
loader jobs in a FAST daemon
All loader jobs are based on views, to ease maintenance
Then we don’t have a command line application to monitor
etc.
The grouper_loader_log table has an entry for every daemon
run
Generally the only problem we have is Warehouse jobs for
people without PennId’s, the people can’t be found, and it’s an
error
–The warehouse will assign a fake pennId which starts with 0
–Change the loader view to have where clause PENN_ID not like '0%'
Administering PennGroups
Loader include/exclude example
Create
a group
Read/update should not be granted to everyone
Use addIncludeExclude type
Look
in folder, there will be 5 groups created with that type.
Open the system of record, and lets make that the loader
group
Create this view in the DB:
mysql> CREATE OR REPLACE VIEW loader_student AS \
(SELECT subjectId AS subject_id FROM SUBJECT WHERE \
subjectId LIKE 'fi%');
Administering PennGroups
Loader include/exclude example (continued)
Never
edit the loader group, unless you expect it to get
overwritten
Add fico to the excludes group
Add bapo to the includes group
Look at the overall group
Generally the privileges are:
Assign READ on all to admins
Assign UPDATE on include/exclude groups to admins
Assign READ to service principal of app for overall group or
other people who need to use the group
Administering PennGroups
Customize UI text
% cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources
% mkdir custom
% cd custom
% touch media.properties
% emacs nav.properties
find.browse.here=Current folder is:
% emacs init.properties
default.module=grouper
default.locale=en_US
% cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes/resources
% emacs init.properties
default.module=custom
default.locale=en_US
Bounce tomcat: % tomcat restart
Administering PennGroups
Customize UI text (continued)
Should
see:
Administering PennGroups
Customize lite UI for an application
% cd ~/tomcat/webapps/testX_grouper/WEB-INF/classes
% mkdir membershipLiteName
% cd membershipLiteName
% touch testName.properties
% cd ~/tomcat/webapps/testX_grouper/WEB-\
INF/classes/resources
% emacs custom/nav.properties
Add line:
membershipLiteName.testName.simpleMembershipUpdate.updateTitle = PTO\
admins
Bounce tomcat: % tomcat restart
https://grouperdemo.internet2.edu/testX_grouper/grouper\
Ui/appHtml/grouper.html?operation=SimpleMembership\
Update.init&groupName=apps:pto:ptoAdmins_systemOfRecord\
&membershipLiteName=testName
Administering PennGroups
Customize lite UI for an application (continued)
Administering PennGroups
Get the Grouper Client
$ cd ~/1.6.3
$ wget \
http://www.internet2.edu/grouper/release/1.6.3/ \
grouper.clientBinary-1.6.3.tar.gz
$ tar xzvf grouper.clientBinary-1.6.3.tar.gz
$ cd grouper.clientBinary-1.6.3
$ emacs grouper.client.properties
grouperClient.webService.url =
\
https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest
grouperClient.webService.login = testX
grouperClient.webService.password = **************
Administering PennGroups
Get the Grouper Client (continued)
Get usage:
$ java -jar grouperClient.jar
$ java -jar grouperClient.jar --operation=getMembersWs \
--groupNames=test:testGroup
Customize the output:
$ java -jar grouperClient.jar --operation=getMembersWs \
--groupNames=test:testGroup \
--outputTemplate='${wsSubject.id}$newline$'
Administering PennGroups
Try from your local machine (win, mac, etc)
NOTE: you need java6+
> cd c:\temp
(or translate for mac or whatever)
Download and unzip:
http://www.internet2.edu/grouper/release/1.6.3/grouper.clientBinary
-1.6.3.tar.gz
> cd grouper.clientBinary-1.6.3
Edit grouper.client.properties
grouperClient.webService.url =
\
https://grouperdemo.internet2.edu/testX_grouperWs/servicesRest
grouperClient.webService.login = testX
grouperClient.webService.password = **************
Get usage:
> java -jar grouperClient.jar
> java -jar grouperClient.jar --operation=getMembersWs \
--groupNames=test:testGroup
Administering PennGroups
Grouper deployment control at Penn





See document
The Grouper team hopes to have a maven version of
this some time soon
Everything in Penn’s CVS, external encrypted
passwords, separated out filters, customizations
Localdev, Dev, Test, Prod managed for custom
application, WS, UI, client
Checkout pennGrouper project from CVS and look at
customizations
Administering PennGroups
Grouper upgrades at Penn

See document of 1.6 to 1.7 upgrade
 I believe we skipped 1.5, though we generally
upgrade to new versions when available since we
need features
 Generally we want everything to be up as much as
possible in readonly mode
 Switch to readonly mode for WS/UI, stop updates on
LDAP
 Get counts of important groups for sanity test
 Upgrade the DB (generally most time consuming
part, if have to edit millions of rows)
 Upgrade the WS/UI, test everything
 Confirm counts of important groups
Administering PennGroups
Grouper maintenance at Penn






Process forms from DA, add kerberos principals
Add new loader jobs
Add new folders and delegate to new clients
Consult on design of how applications use Grouper
Look at errors emailed from Grouper log4j
Run GSH scripts
Administering PennGroups
Grouper sample GSH script


See this documentation
E.g. will need to do this to delete old course groups
Administering PennGroups
Daily report


Email sent everyday to give a pulse of Grouper
Need to deal with unresolvable subjects
Administering PennGroups
Monitoring


grouperWs has a status servlet hooked up to nagios
/grouperWs/status?diagnosticType=all
Administering PennGroups
Roadmap

Hopefully uses for central permissions
– E.g. warehouse permissions
– E.g. PennCommunity Direct permissions

Always available read-only web services
 Shibboleth entitlement group membership integration
 PennCommunity Direct getPerson WS secure
attributes
Administering PennGroups
Questions?
7/15/2015
ISC
50