Transcript here - Typesafe

Whitepages Practical
Experience Converting from
Ruby to Reactive
John Nestor and Dragos Manolescu
{jnestor,dmanolescu}@whitepages.com
August 13, 2014
Disclaimer
The following material is subjective and represents the views of
the presenters.
We hope our views and experience may be of use to others
making the transitive from Ruby to Scala, but your mileage may
vary.
Whitepages Typesafe Case Study
• WhitePages Rebuilds Core Parts of Application Stack with
Scala and Akka to Improve Scaling.
• http://j.mp/wpscala
Outline
•Background
•Technology
•People
Background
About Whitepages
• Top web and mobile site for finding phones, people and locations.
• 50M unique users per month.
• 35M search queries per day.
• 70 engineers, mostly in Seattle.
Why replace Ruby?
Problems with Ruby legacy systems:
•
High latencies.
•
High resource consumption: memory and processors.
•
Poor tolerance of upstream degradation:
– Limited concurrency.
– Worker starvation when blocking on slow upstream.
– Poor connection management and healing.
Why replace Ruby with Scala?
•
Elegant blend of Functional and OO paradigms.
•
Static type system:
•
•
•
Type inference avoids the excess boilerplate of Java.
•
Compiler catches many errors.
Runs on the JVM:
–
Fast.
–
Near seamless interop with JVM libraries.
–
Reasonably mature tooling support.
Actor-based concurrency framework: Akka.
Whitepages Reactive Services
•
Service-oriented-architecture where communication is Thrift or
Json over HTTP.
•
Both latency and throughput are very important.
•
We require excellent logging and monitoring.
•
Agility of development, testing, build and deployment provide
leverage and competitive advantage.
Results
Performance
Service
p50 ms
p99 ms
Throughput RPS/core
DirSvc Scala
25
300
80
DirSvc Ruby
140
1200
7
Status of Reactive Scala Services
• We have been using Scala for a little more than 1.5 years.
•
We started with 1 development team, we now have 4.
•
We started with a single back end service. We now have 4 services in
production and another 6 in development.
•
Our first project had 6 Scala developers, we now have over 20 Scala
developers.
•
We changed not only language but also tools (more automation) and
processes (more agile).
Future Reactive Work
•
Transition additional Ruby back-end services to Scala.
•
Exploring new technologies:
•
Spark and Mesos.
•
A messaging system based on Kafka.
•
A graph store based on Titan/Cassandra.
•
A deployment and lifecycle-management system based on
Docker that uses remote actors.
Technology
Technology Selection
• Scala
• GitHub Enterprise
• Akka
• IntelliJ
• Spray
• Jenkins (for builds)
• SBT – with custom plugins
• Artifactory
• Typesafe Config
• Yourkit
• Scala Test
• SCoverage
• Coda Hale Metrics
• Scrooge (for thrift, from
Twitter)
• Persist Json
Whitepages Service Framework
•
Configuration (based on Typesafe Config, with multi-level overrides).
•
Monitoring (central actor that uses Coda Hale metrics and publishes to Graphite, Nagios
and Riemann).
•
Logging (custom system with central actor, Json log content and real time streaming to
external analytics).
•
Admin REST API.
•
REST Thrift/JSON Server (based on Spray).
•
Clients (Spray, JDBC, Redis, SOLR, Riak,… -- advanced error detection/ recovery).
Technology Successes
•
Scala concise syntax. – Higher developer productivity.
•
Use of async future based code. – Great performance.
•
Immutable collections and functional programming. – Fewer bugs.
•
Strong type checking. – Fewer bugs and more maintainable code (But meta-programming becomes
more difficult).
•
Concurrency. – No race conditions or deadlock on complex concurrent system.
•
Spray. – Great performance and an async API.
•
SBT. – Able to easily customize to our needs (after steep learning curve).
•
IntelliJ IDEA. – Scala support is good and keeps getting better.
•
Typesafe Developer support contract. – Fast response, great answers to complex questions. (We
should have gotten it earlier!)
Technology Problems -1
• Poor documentation:
•
•
SBT is the poster child.
•
We still often have to read source code of Scala and Akka libraries.
Stable APIs:
•
Typesafe has made great strides here.
•
Other components are still troublesome.
•
We lost many weeks as a result of breaking changes in Scala Test.
•
We currently can’t compile a 2.11 version of Kafka because the Zinc Scala
compiler is not backward compatible.
Technology Problems -2
•
Performance analysis of concurrent constructs:
•
We went with the Typesafe Console.
•
It never fully worked and then was abandoned.
•
There are some new tools, but we have not had time to evaluate.
Technology Problems -3
•
Ecosystem:
•
Not as mature as the Java ecosystem and often missing Scala specific
features. For example, Scala Mock does not support named
parameters.
•
Some needed components were missing: until recently code coverage.
•
In some cases there were too many options: over 10 Json libraries.
•
Lots of Scala GitHub projects. Quality is variable; many become
abandoned.
Technology Problems -4
•
Debugging:
•
Debuggers work on the Java byte code, that for some Scala constructs
is quite convoluted.
•
Debugging chains of futures is much harder than debugging nested
calls. Some kind of tracing is desperately needed.
•
Actors have lots of power but can be exceptionally hard to debug. We
need both better approaches to building debug-able actors and better
debugging tools.
Technology Problems -5
•
Language and Library Issues:
•
Type erasure is a major flaw in the Scala language. It causes lots of
developer confusion and errors. The workarounds are awkward at best.
•
The lack of type checking on Actor constructors is an odd hole in the
type checking and a real annoyance.
•
Some Scala code that looks simple and intuitive creates major
difficulties when a developer needs to understand what is really going
on. Implicits are one of many problems here.
Technology Overall
•
The problems mentioned above create real day to day difficulties.
•
In many cases Typesafe has been actively working to incrementally correct
some of these problems.
•
Since we started using Scala, the number of problems has significantly
decreased and we expect that trend to continue.
•
Overall, Scala/Akka is a great technology for building reactive
systems and its benefits far outweigh the problems.
People
Levels of Scala Expertise
•
Basic OO features. Easy transition from most other languages including
Java and Ruby.
•
Adding functional. Ruby has functional features that are similar. But
knowing when best to use functional versus OO requires experience.
•
Adding concurrency. Via futures, promises, and actors. A whole new
world for Ruby programmers that requires a new way of thinking.
•
API design. Not all developers need to design reusable APIs, but those
that do need a higher level of experience and feature mastery to do it well.
Building Scala Development Teams
•
The are still not a large number of experienced Scala developers.
•
Those out there are in considerable demand.
•
We have followed a mixed strategy of both recruiting Scala developers and
also training existing Ruby developers to become Scala developers.
•
Having an experienced Scala developer greatly accelerated the training of
Ruby developers.
Note: Using Scala, Akka and other advanced technologies makes Whitepages
very attractive to some of the very best developers.
Recruiting: Finding Developers
• Personal contacts. The best way to find great developers.
• Meet-ups and Events
• John Nestor is the organizer for the Seattle Scala meet-up.
•
Whitepages hosts the Scala meet-up (and several others)
•
Whitepages sponsors many other technology conferences and events.
•
Job posts: local Seattle and internet Scala focused.
•
College recruiting events. UW and other top CS schools.
Recruiting: Attracting Developers
•
Strong support from the Whitepages CTO and CEO and other management for
Scala and other advanced technologies.
•
We have a great office (recently completely redone)
•
We provide developers with the best workstations (top-end MacBook pros).
•
We have competitive benefits, excellent work-life balance and a collaborative
supportive culture.
Training: External Resources
•
Books. Odersky: Programming in Scala, and others
•
Coursera. Functional programming, reactive programming.
•
Typesafe training classes. Both Seattle public and in-house.
•
Typesafe developer support contract and consulting. Particularly valuable if
you don’t have in-house Scala expertise.
•
Conferences. We sent people to Scala Days in NY and Berlin.
Training: Internal Resources
•
•
Whitepages developers are in a single office and come in most days.
•
This permits a high degree of personal interactions.
•
Those interactions enable rapid learning and high code quality.
This would be much harder with multiple offices and/or widely distributed
teams.
Training: Internal Approaches
•
•
Expert mentoring.
•
Provide quick answers to questions.
•
Detailed reviews and feedback for Ruby developers initial Scala code.
Code reviews
•
Improve code quality.
•
Help those learning Scala.
•
Develop a shared culture and style.
•
We use a mixture of group and pair code reviews.
•
Electronic means like code review tools and pull requests were less effective.
Training: Success and Failure
•
Not everyone made a successful transition from Ruby to Scala.
•
Next slide give some positive and negative indicators…
Training: What to Look For
• Desire to learn new technologies.
• Desire to learn new ways to think.
• Prior functional language experience.
• Likes to work closely with others.
Scala Features to Avoid
•
Scala is a very large language.
•
New developers can easily get lost on the rich feature set. Often there are
several different ways of doing something, and they don’t always know how
to choose.
– We also provide the following list of features for new developers to
avoid (or to fully understand before using).
– We encourage asking more experienced developers to provide that
understanding.
– These features are also flags for discussion in code reviews.
Scala Features to Avoid
• Nulls
• Reflection and macros
• Return, break, continue
• asInstanceOf
• Mutable collections
• var, while, until
• Top level objects with mutable
state
• Generics
• Lazy vals
• Implicits
• Scala enumerations
• Actors
• ? (the ask pattern)
• Await
• onComplete (unless body
enclosed in try)
We Are
*
Hiring !
*Smart people (not just technologies) are key to our success
Thank you!
Questions?