Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst Department of Computer Science The University of Wisconsin – Eau Claire Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

A Little Background …

• “Grew Up” (Graduate School) in Computer Architecture – Efficient, fast, low-power processors (2 ISCA, 1 MICRO papers, +) • “Dabbled” in supercomputing – (Becoming a bigger part of my job now…) • Became a CS educator • Now – Trying to make sense of how these are all coming together Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Moore’s Dividend

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Where Did All The Performance Go?

Software Development, c. 1950-2005

“What Andy giveth, Bill taketh away.” Parallelism is Everywhere – So How Do We Make It Accessible?

Figure by Jim Larus [CACM 09] Daniel Ernst The University of Wisconsin-Eau Claire

Can Multi-Core Supplant Moore’s “Dividend”?

(for individual programs) • Double number of cores instead of speed • NO, at least not without major innovation – Sequential code base will not scale without lots of help – Lack of parallel algorithms (and/or poor knowledge of them) – Difficult (or at least different) programming challenges – Fewer abstractions currently available Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Parallel Algorithms

“In the context of sequential algorithms, it is standard practice to design more complex algorithms that outperform simpler ones (for example, by implementing a balanced tree instead of a list). For non-blocking algorithms, however, implementing more complex data structures has been prohibitively difficult.” [Herlihy, Luchangco, Moir, Scherer, PODC 2003]

(Discussing a concurrent red-black tree)

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Sequential to Parallel – A History

• Existing code is sequential • We tried parallel compilers in the ‘80s-’90s – Can get good code for a few specific cases, but not generally • We tried to get the hardware to extract parallelism in 90’s-00’s – Dynamic mechanisms cannot find more than 2–4x parallelism • The issue is with how programmers fundamentally think about problems – Not language specific • If we’re training programmers to think only in sequence, how well will they be prepared for dealing with explicit parallelism?

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Fundamental Take-aways

• Explicit parallelism is important, and is getting more so – Being able to identify it will be an increasingly marketable skill • Understanding the underlying architectures is also becoming more important – Necessary to have at least basic understanding to extract reasonable amounts of parallelism – Hybrid machines will make this critical if you want good performance • ESPECIALLY before the tools catch up (but still after) Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Programming Futures

• There is a lack of consensus on the parallel programming model – Data parallelism (Vector) – Unstructured (thread) parallelism • (Edward Lee – “ The Problem with Threads ”) • Speculative threading?

– Message passing • Lots of ongoing research with not much resolved • Moral: Everyone in the CS field needs to be

prepared to learn

about new parallel programming methods and tools.

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

State of the Game

Educators This Talk Industry

This (slightly modified) graph shamelessly stolen from Jim Larus Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

An Opportunistic Approach

• Typically, he/she who develops functionality to use fully use the

hardware capabilities “wins”

– Google, facebook, intel, Microsoft, etc. all took advantage of technological inflection points • So, if we don’t know what to teach yet, how do we prepare our students to be leaders?

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Some Context

• UW – Eau Claire “Founded in 1916, UW-Eau Claire is one of the Midwest’s top public universities. UW-Eau Claire provides students with small classes led by talented professors, cutting-edge programs, hands-on research and life-enhancing study abroad opportunities.

Motto: Excellence. Our measure, our motto, our goal. • What that actually means: – Approx. 11k students, only 500 are grad students – Focus is undergraduate engagement • Teaching excellence • Undergraduate research Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

UW – Eau Claire Computer Science

• ABET Accredited comprehensive CS major • 6 faculty (+1 vacancy), 1 part-time academic staff • Approximately 40-50 graduates per year in 3 majors: – Comprehensive CS – Software Engineering – Computer Engineering (CS degree with CE emphasis) • We’re filthy rich (sort of) • Constituencies / Industrial Partners / Landing Spots – Healthcare, insurance (Marshfield Clinic, NW Mutual) – Some systems development (Cray/IBM) Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

UW – Eau Claire Computer Science

• Focus is on keeping an innovative curriculum – A majority of our scholarship is within this scope – SIGCSE-related activity is large • total of 31 papers and 7 panels presented in SIGCSE sponsored conferences and journals since 2002 – The department has recently been nominated three times as the University of Wisconsin Teaching Department of the Year (2001, 2002, 2003) • Curricular Basics: – Java early, C/C++ later, plus a full PL course – Full P&H Comp Org.

– Late: strong software engineering coursework along with robust systems classes (OS/Networks)

Not for the casual student

Parallelism is Everywhere – Daniel Ernst So How Do We Make It Accessible?

The University of Wisconsin-Eau Claire

Parallelism in the Curriculum – Current

• Many (most?) schools treat parallel computing as an independent topic, addressed in the senior year as an elective.

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Curricular Goals at UW-EC

Students should gain: – a generic understanding of computational concurrency and parallelism.

• see parallelism in problems • be able to exploit it in a reasonable way – exposure to parallelism as an orthogonal method with respect to other standard curricular items.

• Don’t just teach parallelism by itself – integrate it in with other elements as simply another tool in the programming box.

• Teaching it in context can lower time overhead, as well – confidence in using parallelism as a tool.

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Principles for Implementation

• Simplicity – Keep to easy (deterministic) and natural concepts • Ubiquity – More than a “1-shot” issue • Accessibility – Don’t make the learning curve steeper than it has to be Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Simplicity

Keeping Parallelism Simple

• First, tweak the underlying concept of a computer – A “collection of brains” • The concepts behind basic parallelism are simple – Sharing work • Different tasks • Split up data • Especially if they’re given a more natural context Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Simplicity

Natural Parallelism/Concurrency

• We have very natural allies in the educational graphics/animation toolsets – Alice, Scratch, etc.

• These tools make basic concurrency natural and largely deterministic – easier to debug!

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Simplicity

Another Natural Example – Swarm Intelligence

• “Creative” Monte Carlo solution to TSP – Instantiate multiple agents to probabilistically discover the shortest trip – “Threads as Entities” • Each agent (“ant”) operates in its own thread – ants leave pheromone trails along arcs to influence later ants • Arc information needs to be synchronized • Orthogonal topics: – Monte Carlo – heuristics Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Simplicity

Other “Clean” Parallel Examples

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Ubiquity

Parallelism Everywhere



Parallelism the Norm

• Unlike the past, sequential code written today is likely to only see very small (if any) gains from hardware upgrades. – As the number of cores increases, non-parallel performance will become relatively more unacceptable in practice.

• Exploiting parallelism needs to be a first-class citizen – (or at least second-class) – not an item studied once and then not used again.

– Ubiquitous Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Ubiquity

An Integrated Model at UW-EC

• CS1 –

Decomposition and Assignment

– Introduce methods for recognizing obvious parallelism – Practice with simple, natural, and integrated examples • CS2 –

Basic Orchestration

– Continued practice with obvious parallelism – Introduce communication • Algorithms –

Parallelism Limits

– Extend algorithm analysis to parallel algorithms – Introduce more complicated communication patterns • Operating Systems –

Synchronization Constructs

– “The usual topics “ Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Ubiquity

After a Reality Check…

• CS2 –

Decomposition and Assignment

– Introduce methods for recognizing obvious parallelism – Practice with simple, natural, and integrated examples • CS2/Algorithms –

Basic Orchestration

– Continued practice with obvious parallelism – Introduce communication • Algorithms/OS –

Parallelism Limits

– Extend algorithm analysis to parallel algorithms – Introduce more complicated communication patterns • Operating Systems –

Synchronization Constructs

– “The usual topics “ Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Ubiquity

CS2 Possibilities – Image Mosaic

• “Nifty Assignment” (Pattis, SIGCSE 2005) – create an image out of tiles of smaller images – tiles are independent (unless you want it to look good!) • Basic parallelism is easy, but good quality needs synchronization – Payoff!

• Can vary synchronization difficulty • Orthogonal topics: – searching/sorting/matching Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Ubiquity

The Role of Architecture

• Understanding how a program “lays out” on a machine.

– What is it capable of?

– What are the bottlenecks?

• My take thus far: – Decreased emphasis on assembly programming – Slightly decreased emphasis on microarchitecture • (Depending on program needs) – Increased emphasis on system-level view (interaction between processor and memory) Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Accessibility

Keeping Parallelism Accessible

• For each individual curriculum, the choice of language/platform has a significant effect on accessibility.

• At UW-EC, we are largely a “Java shop”, with some C/C++ – To keep learning curve gentle, we try to operate within that framework – Java.util.concurrent / OpenMP • Taking the time to introduce a new platform eats far too much time and can lead to projection of learning difficulties* – (and they’re just going to change, anyways)

The concepts are more important than the platform

Parallelism is Everywhere – So How Do We Make It Accessible?

*YMMV Daniel Ernst The University of Wisconsin-Eau Claire

Accessibility

The Delicate Balance

While we want to have student learning to be as platform-agnostic as possible… …they also need to learn real skills for the workforce… …and they need to practice with the concepts somehow!

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Current Challenges

1. General Workload – – Especially given the ubiquity argument, there’s a lot of material that needs to be created/re-worked in new contexts How do we motivate instructors?

2. Faculty Readiness – – At least anecdotally, the percentage of faculty who feel comfortable teaching parallel computing is painfully small How can we help with faculty accessibility?

3. Future Uncertainty – Given that we’re not quite sure what future programming models will look like – what topics should we be teaching?

Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Current Challenges (More?!?)

4. Availability of Tools/Language Support for Parallelism – • • • • There are very few language/toolset combinations that collectively fulfill the major criteria for curricular languages: Language used in industry (or related to one) Supports OO Good support for “easy” parallelism Not syntactic suicide Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

The Reality of Implementation

• If change is really going to be cross-curriculum, it requires that there is significant buy-in from every faculty/staff member that teaches any course that includes the new material.

• This is a serious professional development challenge Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

EAPF – Addressing The Challenges

Educational Alliance for a Parallel Future

• Combined effort of Industry: – Intel, nVidia, AMD, Microsoft, etc.

• and Academia – Members range from research (UIUC/Berkeley) to supercomputing experts to small college educators • Goals: – Increase awareness – Encourage and support innovation in training and tools • Keep an eye out for more… Parallelism is Everywhere – So How Do We Make It Accessible?

Daniel Ernst The University of Wisconsin-Eau Claire

Parallelism is Everywhere – So How Do We Make It Accessible?

Thank You!

Q & A

Daniel Ernst The University of Wisconsin-Eau Claire