Transcript sakora-csv - Confluence

Simple SIS Service
(sakora-csv)
Dan McCallum – Unicon, Kim Eke – UNC,
Patty Wolfe - Unicon
Sakai Conference – Denver, Jun 17, 2010
© Copyright Unicon, Inc., 2006. This work is the intellectual property of Unicon, Inc. Permission is granted for this material to be shared for
non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that
the copying is by permission of Unicon, Inc. To disseminate otherwise or to republish requires written permission from Unicon, Inc.
1. Concepts
2. sakora-csv at UNC
3. Usage
4. Gotchas
5. Other Options
6. File Formats
Concepts
SSSS Data Flow
Sakai
Course
Management
APIs
SIS
C
S
V
CM
Entities
Sakai
DB
Upload Tool
WWW
line
line
Scheduled
Job
C
S
V
Saka
i
FS
C
S
V
Sakai CM Entities
CM-to-Site Mappings
WS Setup
Tool
CM APIs
GroupProvider
CM
Entities
Provider
IDs
Users+Roles
Site
Site Groups/Sections
RoleResolvers
<util:map id="org.sakaiproject.coursemanagement.GroupProviderConfiguration">
<entry key="siteRoleResolutionOrder">
<list>
<value>Instructor</value>
<value>Teaching Assistant</value>
<value>Student</value>
</list>
</entry>
<entry key="officialInstructorToSiteRole" value="Instructor"/>
<entry key="enrollmentStatusToSiteRole">
<map>
<entry key="enrolled" value="Student"/>
<entry key="wait" value="Student"/>
</map>
</entry>
<entry key="sectionRoleToSiteRole">
<map>
<entry key="I" value="Instructor"/>
<entry key="GSI" value="Teaching Assistant"/>
<entry key="S" value="Student"/>
</map>
</entry>
<!– (to be continued…) -->
</util:map>
RoleResolvers
<util:map
id="org.sakaiproject.coursemanagement.GroupProviderConfigurat
ion">
<!– (continued…) -->
<entry key="courseOfferingRoleToSiteRole">
<map>
<entry key="CourseAdmin" value="Instructor"/>
<entry key="I" value="Instructor"/>
</map>
</entry>
<entry key="courseSetRoleToSiteRole">
<map>
<entry key="DeptAdmin" value="Instructor"/>
</map>
</entry>
</util:map>
sakora-csv at UNC
Sakai Growth
sakora-csv at UNC
• Sakai section awareness killer feature relative
to Bb
• Still feeding files to Bb
• Sites provisioned by request – would like to
remove search/request features in WS Setup
• Coordinator and Tech Support memberships
added at Course Offering level
Usage
Snapshots
• Send all “current” records every time
• (previous file) – (latest file) = deletions
• Applies to:
– sessions.csv
– people.csv
– sectionMembership.csv
– courseMembership.csv
• (Currently) no way to ignore omitted
memberships
Upload
curl -F "username=sakora-csv-sub-admin" -F "password=f00b4r" \
-F "sessions=@latest-upload-batch/sessions.csv" \
-F "courseSections=@latest-upload-batch/courseSections.csv" \
-F "courseMembership=@latest-upload-batch/courseMembership.csv" \
-F "courseOfferings=@latest-upload-batch/courseOfferings.csv" \
-F "courses=@latest-upload-batch/courses.csv" \
-F "courseSets=@latest-upload-batch/courseSets.csv" \
-F "people=@latest-upload-batch/people.csv" \
-F "sectionMembership=@latest-upload-batch/sectionMembership.csv" \
-F "runJob=true" \
https://$SAKAI_DOMAIN/sakai-sakora-csv-tool/csv
User Credentials
• Use HTTPS, obviously
• Any super user (site.upd in !admin)
• Batch will always be processed as admin
Directory Polling
• Schedule “SIS CSV Data Loader” job
• Configure (need to do this anyway):
[email protected]
• Currently no way to detect partial uploads
Monitoring
• No UI (patches welcome!)
– Tail catalina.out
– Watch SAKORA_LOG
• Remember processing is async w/r/t upload
• Most logging is at DEBUG level
log.config.count=n
log.config.n=DEBUG.net.unicon.sakora
Monitoring
• {upload.dir}/*-finished.csv
– Batch completed
w/o fatal error
• {upload.dir}/*-failed.csv
– Batch aborted w/
fatal error
• Fatal errors…
– DB, file system exceptions
– Shutdown
– Some file syntax errors
• Batches never retried
Site Membership Syncing
• Pre-2.6 needed SAK-14259
• Not typically needed any longer b/c Site Info
refreshes site membership (SAK-14336)
What About LDAP?
• You’re fine
• Don’t send people.csv if using LDAP for authN
and attributes
• User IDs in people.csv and referencing files
need to be EIDs
– Usually net IDs
Gotchas
Where Did My Terms Go?
• To show up in WS Setup, a term must be:
– “current”
– Within roster.available.weeks.before.term.start
• sessions.csv
is the authority for “currentness”
• …even for terms never sent in that file!
I Don’t See My Rosters in Tool X
• Kernel and WS Setup patches
• 2.7
– SAK-18636
• 2.6.2
– SAK-18636
– SAK-17790
– KNL-320
• 2.6.x
– SAK-18636
– KNL-320
I Can’t Remove Instructors!
• Creating user automatically added to new
Site
• Trumps “provided” membership pre-2.7
• SAK-18536
• KNL-403
Instructors Don’t Show up in Roster Tool
• It’s a bug – official instructors aren’t saved
properly
• Actually… it’s the only reason KNL-403 works
with sakora-csv at all
My Section Names Have Commas
• Escape all fields if you can
• Use quotes to escape quotes
EID, Title, Description, Category, parentEID, esEID, coEID
“FOO101.SP10.001”,”Intro to “”Foo”””,”T/T, Intro to
“”Foo”””,“Lect”,””,””,”FOO101.SP10”
“BAR101.SP10.001”,”Intro to “”Bar”””,”M/W/F, Intro to
“”Bar”””,“Lect”,””,””,”BAR101.SP10”
My Files are Getting Huge!
• (Currently) no way to ignore omitted
memberships
• Make sure you have indexes in {sakora-csvsrc}/sakora-csv-impl/impl/src/sql/
• Can truncate SAKORA_LOG (but not other
SAKORA_* tables)
Sakai is Ignoring the First Line in my Files!
• Set
[email protected]=f
alse
• sakora-csv doesn’t actually use header rows
as metadata even if hasHeader=true
I Sent an Empty File and Nothing Happened
• E.g…. I just want to delete all records of type
X
• sakora-csv always ignores empty/missing
files.
• Trying to avoid accidental data deletion
• Allows for partial batch uploads
• Sometimes helpful. Sometimes annoying.
How do I Configure X in WS Setup?
• wsetup.skipCourseSectionSelection
• sitemanage.cm.courseset.categories
How do I Configure X in WS Setup?
• titleEditableSiteType
• wsetup.disable.joinable
• roster.available.weeks.before.term.start
How do I Configure X in WS Setup?
• SectionFieldProvider
CourseSet X Doesn’t Show Up In WS Setup
Drill-Down Search
• Only top-level CourseSets appear in pulldown
• Use parentEID field in courseSets.csv reluctantly
Other Options
CM Synchronizer
• A fine option
• Modified versions used at UDel, URI,
probably others
• Undocumented file format
• Code intended as a demo
• Reads entire document into memory
• XML – might simplify your life, might create
strife
SAKORA “Proper”
• IMS LIS reference implementation
• Appealing if you’ve already purchased
PeopleSoft SAIP
• Standardized outcome reporting (but no
Sakai impl yet)
• Event-based
• Less widely deployed than sakora-csv
Others
• Home-grown solutions abound
• Check sakai-dev
– http://bit.ly/c6mEqI
File Formats
File Formats
• Detailed field definitions in Confluence
– http://bit.ly/djyG3o
• Note attached template files
Questions?
[email protected]
[email protected]
[email protected]
http://www.surveymonkey.com/s/sakai10
http://bit.ly/djyG3o