Transcript Experiences Developing Tools for Scientific Communities in the Apache Software Foundation: Beyond Open Source Marlon Pierce Indiana University February 14, 2012

Experiences Developing Tools for
Scientific Communities in the
Apache Software Foundation:
Beyond Open Source
Marlon Pierce
Indiana University
February 14, 2012
IU Science Gateway Group Overview
• IU Science Gateway Group members
– Marlon Pierce: Group Lead
– Suresh Marru: Principal Software Architect
– Raminder Singh, Chathura Herath, Yu Ma, Lahiru Gunathilake:
Senior team members
– Five interns and research assistants
• NSF SDCI funding of Open Gateway Computing
Environments project
– TACC (M. Dahan), SDSC (N. Wilkins-Diehr), SDSU (M. Thomas),
NCSA (S. Pamidighantam), UIUC (S. Wang), Purdue (C. Song),
UTHSCSA (E. Brookes)
• We participate in two Apache incubators
– Apache Rave: http://incubator.apache.org/rave/
– Apache Airavata: http://www.airavata.org
Key Problems for Science Gateway, CI
Software
• Reusability
– Reuse or write own gateway software?
• Sustainability
– Small teams
– Successful gateways become infrastructure
• Hard to fund
– Cyberinfrastructure Software Sustainability and Reusability:
Report from an NSF-funded workshop
• https://scholarworks.iu.edu/dspace/handle/2022/6701
• Governance
–
–
–
–
How are design decisions made?
Who decides if the software is suitable for release?
How do you handle contributions?
How do you add people to the development and project
management teams?
Proposed Solution: Open Community
Software
• More than SourceForge, GitHub, Google Code,
etc
– Those provide excellent Web tools to help
developers.
• Here we are concerned with community
building.
– Diverse community of developers means
reusability and sustainability
– But diverse communities require governance
Some Open Model Examples in CI
• NSF-funded CDIGS project
– http://confluence.globus.org/display/CDIGS/CDIGS+Home+Page
• HUBzero Consortium, Sakai Foundation
– Institutional level organizations
• Eclipse Parallel Tools Platform
– Jay Alameda, NCSA: NSF SI2 funding to develop HPC tools
workbench
– http://www.eclipse.org/ptp/
• Enzo Project
– Excellent talk at TG11 by Prof. Brian O’Shea on their open
community efforts
– http://enzo-project.org/
• Apache Software Foundation
– OODT and TIKKA Data Management Projects at NASA JPL
Two Apache Software
Foundation Case Studies
Apache Rave and Airavata Incubators
Apache Airavata
• Science Gateway software framework to
– Compose, manage, execute, and monitor
computational workflows on Grids and Clouds
– Web service abstractions to legacy command linedriven scientific applications
– Modular software framework to use as individual
components or as an integrated solution.
• More Information
– Airavata Web Site: airavata.org
– Developer Mailing Lists: [email protected]
Apache Airavata High Level Overview
Example Workflow: Nuclear Physics
Courtesy of collaboration with Prof. James Vary and team, Iowa State
Apache Rave
• Open Community Software for Enterprise
Social Networking, Shareable Web
Components, and Science Gateways
• Founding members:
•
•
•
•
Mitre Software
SURFnet
Hippo Software
Indiana University
• More information
• Project Website: http://incubator.apache.org/rave/
• Mailing List: [email protected]
1
Gadget Dashboard
View
Gadget Store
View
Extending Rave for Science Gateways
• Rave is designed to be extended.
– Good design (interfaces, easily pluggable
implementations) and code organization are required.
– It helps to have a diverse, distributed developer
community
• How can you work on it if we can’t work on it?
• Rave is also packaged so that you can extend it
without touching the source tree.
• GCE11 paper presented 3 case studies for Science
Gateways
The Apache Way and Science
Gateways
Why Apache for Gateway Software?
• Apache Software Foundation is a neutral playing field
– 501(c)(3) non-profit organization.
– Designed to encourage competitors to collaborate on
foundational software.
– Includes a legal cell for legal issues.
• Foundation itself is sustainable
– Incorporated in 1999
– Multiple sponsors (Yahoo, Microsoft, Google, AMD, Facebook,
IBM, …)
• Proven governance models
– Projects are run by Program Management Committees.
– New projects must go through incubation.
• Provides the social infrastructure for building communities.
• Opportunities to collaborate with other Apache projects
outside the usual CI world.
The Apache Way
• Projects start as incubators with 1 champion and several mentors.
– Making good choices is very important
• Graduation ultimately is judged by the Apache community.
– +1/-1 votes on the incubator list
• Good, open engineering practices required
– DEV mailing list design discussions, issue tracking
– Jira contributions
– Important decisions are voted on
• Properly packaged code
–
–
–
–
Build out of the box
Releases are signed
Licenses, disclaimers, notices, change logs, etc.
Releases are voted
• Developer diversity
– Three or more unconnected developers
– Price is giving up sole ownership, replace with meritocracy
Apache and Science Gateways
• Apache rewards projects for cross-pollination.
– Connecting with complementary Apache projects
strengthens both sides.
– New requirements, new development methods
• Apache methods foster sustainability
– Building communities of developers, not just users
– Key merit criterion
• Apache methods provide governance
– Incubators learn best practices from mentors
– Releases are peer-reviewed
Apache Contributions Aren’t Just
Software
• Apache committers and PMC members aren’t just
code writers.
• Successful communities also include
–
–
–
–
Important users
Project evangelists
Content providers: documentation, tutorials
Testers, requirements providers, and constructive
complainers
• Using Jira and mailing lists
– Anything else that needs doing.
How To Get Involved
• Join the DEV mailing lists.
• Grab the software and start complaining.
• Post Jira tickets
– Add your patches to Jira if you want to solve a
problem.
– Frequent patch submission is the best way to get
voted in as a committer.