Transcript BUILDING STABLE SELENIUM TESTS ON A FOUNDATION OF JELLO DAN CUELLAR LEAD SOFTWARE ENGINEER ZOOSK – TEST TEAM.

BUILDING STABLE
SELENIUM TESTS ON A
FOUNDATION OF JELLO
DAN CUELLAR
LEAD SOFTWARE ENGINEER
ZOOSK – TEST TEAM
A BIT ABOUT
•
Zoosk is a romantic social network with tens of millions of users that
operates in 25 languages and in over 70 countries.
•
Products include:
•
•
•
•
•
•
•
web site
mobile web site
Facebook app
iPhone app
iPad app
Android app
Adobe Air app
•
The development pace is rapid, features are churned out in around 2
weeks time (including testing)
•
Product management is always running experiments and adjusting
plans based on data from instrumentation
•
Developer to quality assurance ratio hovers somewhere around 10:1
AGENDA
• So, What Do You Mean by Jello?
• How Good Tests Go Bad
• Automation Philosophy
• Self Testing Automation
SO WHAT DO YOU
MEAN BY JELL-O®
Jell-O [jel-oh]
Site wide A/B tests where ½ of users will see a complete site
redesign and the other ½ will use the site as it was before.
Facebook app users can also split between different A and B
designs at times.
Each A & B may have its own JS, CSS, and business logic to go
along with it.
Not to mention that different versions of the API were being used
by the web, Facebook app, and mobile apps.
All the while, different features were being enabled and disabled
for different subsets of the A/B test groups
HOW GOOD TESTS
GO BAD
WHAT CAN GO WRONG?
• General
• Not enough information is retrieved to reproduce the failure
• Elements
• The type can change (e.g. dropdown becomes a checkbox)
• An element can change location
• An element can have multiple ways of identifying itself depending
on an A/B test
• Business Logic
• Logic can change for one platform
• Logic can change for some users and not others (A/B test)
• Logic can change for all platforms
HOW DO WE COMBAT THESE
PROBLEMS?
• General
• Not enough information is retrieved to reproduce the failure
•
•
Separate test actions from controller code, add abundant
logging, and unify the patterns by which common
operations are accomplished (e.g. selecting from
dropdown) Don’t access selenium directly in any scripts.
Create a wrapper that logs everything and performs basic
checks. (e.g. is the choice available on the dropdown)
If libraries are used then you can log just about everything
possible every time without having to re-paste all the
logging code
HOW DO WE COMBAT THESE
PROBLEMS? (CONT.)
• Elements
• The type can change
•
Abstracting away all uses of the item into a library
• An element can change location
•
•
Centralize all hooks into the UI, you can change it once
and it will be fixed for all
Using reflection, you can verify that all of your hooks are
functional
• An element can have multiple ways of identifying itself
depending on an A/B test
•
ID’s are great, but you can also use an XPath Or
expression in your centralized hook
HOW DO WE COMBAT THESE
PROBLEMS? (CONT.)
• Business Logic
• Logic can change for one platform
•
Update the library for that platform
• Logic can change for some users and not others (A/B test)
•
Detect the A/B test in the library
• Logic can change for all platforms
•
Generic Libraries or test scripts which coordinate platform
specific libraries are a great place to address this
AUTOMATION
PHILOSOPHY
IN ONE SLIDE
• Separate logical actions from calls to controller (e.g.
Selenium) code into libraries and interfaces
• Centralize your site model and logic, so that all
automation runs through one code path
• Write code that can test the test automation.
Investigating test script failures can be costly, but if
you test code can test itself you will be able to pick
out certain kinds of problems easily.
• Write platform-agnostic test scripts when possible
HOOKS
• means by which test automation can identify
pieces of the user interface
• to reduce redundancy and lower the cost of test
repair, it’s important to centralize your hooks into
a single location
• If a hook changes, it only needs to be updated
in a single location
CONTROLLERS
• pieces of code that control a component or platform
• perform low-level actions to manipulate a component
• Web Controller Action Examples
• click
• go to URL
• gathering cookies
• iPhone Controller Action Examples
• tap
• pinch
• rotate
LIBRARIES
• methods (groups of calls to controllers) that perform
product-level actions using a controller on a particular
platform
• contains methods that orchestrate controllers into higher
level actions
• Web Example
• A login method in a library would check if the user is
signed in by looking at a cookie first.
• If it then determined that the user was not logged in,
it would then fill out the login form and submit it and
validate the proper cookie is set.
INTERFACES
• used to remove the platform context from libraries
• each platform supports common tasks like
authentication, messaging, searching, and more
• implementing an interface allows for tests to be written
in a platform-agnostic fashion.
• Example
• A user deletes their Zoosk account
• Write a test that deletes an account, then verifies on
each platform using a sign in method in the interface
to valid authentication fails on each platform.
TEST SCRIPTS
• coordinate interface calls (or library calls if the test is
platform specific.)
• test a particular product scenario and report back if the
behavior is as expected.
• when tests call the interface, the business logic of the
test can be maintained in a single place, minimizing the
amount of code changes that need to be made in test
automation when business logic changes.
CONSOLIDATING AND ABSTRACTING
PRODUCT LEVEL ACTIONS
• Libraries handle almost all calls to the controller
• Product Level Actions (e.g. Sign In To The Site) are
stored in libraries, so that there exists one good
copy that everyone can use
• The libraries implement generic interfaces so the
platform can be abstracted away
• Test scripts orchestrate interface, library, and
controller calls into scenario test cases
• Test script calls to interface > calls to library > calls
to controller
SEPARATION OF CONTROL
FROM SCRIPTING
An extra layer is great to log information and perform additional actions that
Selenium can’t assume it needs to do
We use a layer between that wraps logging, error handling, and some other things
on top of most WebDriver commands
Script
• Log.Comment(“Validating Test Was Successful”);
• bool fAuthenticated = Interface.IsSignedIn();
• Log.Verify(true, fAuthenticated, “Validating Sign In Was Successful”);
• return WebLib.IsSignedIn();
Interface
Library
• Log.Info(“Checking that a user is signed in”);
• var c = WebDriverWrapper.GetCookies();
• return c.Contains(authCookie);
• Log.Info(“Getting Cookies from WebDriver);
• Log.Verbose(Driver.Manage.Cookies.ToString());
WebDriver • return Driver.Manage().Cookies;
LIBRARY EXAMPLE
[C#]
namespace Automation.Libraries
{
public SeleniumActionLibrary WebSignUpLib : ISignUpLib
{
public void SignIn(string email, string password)
{
if (IsSignedIn())
return;
if (!Driver.Url.contains(“login.php”);
Driver.Navigate().GoToUrl(“https://www.zoosk.com/login.php”);
Hooks.Web.LoginPage.EmailField.type(email);
Hooks.Web.LoginPage.PasswordField.type(password);
Hooks.Web.LoginPage.SubmitButton.click();
WaitForElementToNotExist(Hooks.Web.LoginPage.Spinner);
}
}
}
LIBRARY INTERFACE
[C#]
namespace Automation.Libraries
{
interface ISignUpLib
{
void SignIn(string email, string password);
bool IsSignedIn(string email, string password);
void SignOut();
}
}
PLATFORM AGNOSTIC
AUTOMATION SCRIPTS
• Many tests that need to be run are logically identical
across all platforms.
• Deactivate an account, attempt to sign in, expect failure
• Send a message, verify it shows up in sent items
• Take down some web service, verify an error is produced
• Implemented these tests at an abstract level allows you to
have one test for all platforms
• When the logic changes, the logic only exists one place in
your automation code
AN ABSTRACTED TEST
[C#]
namespace Automation.Tests.PlatformAgnostic
{
public static class SignInWithDeactivatedAccountTest: Test
{
public static void Run()
{
var account = APIUtility.Accounts.CreateTestAccount();
APIUtility.Accounts.Deactivate(account);
client.SignUpInterface.SignIn(account.email, acount.password);
Log.VerifyFalse(!client.IsSignedIn());
}
}
}
SELF-TESTING
AUTOMATION
CENTRALIZING YOUR SITE
MODEL FOR SELF -TESTING
•
All hooks (means by which we retrieve WebElements) are stored
in a single class, namespace, or something similar
•
Elements are stored lazily; the means to retrieve them are stored,
but the actual WebElement is not retrieved until run-time. We use
classes that wrap calls to By or methods which will return a
WebElements
•
Hooks are grouped relevantly and each group has private self-test
setup methods which will navigate the client (e.g. selenium) to
where the hooks exist
•
Reflection is used to iterate through all of these groups and run
the private self-test setup methods and then validate the
functionality of the elements
•
Annotations (Java) or Attributes (C#) are used to exclude
elements from the tests
ABOUT INTROSPECTION
& REFLECTION
Introspection (Java) and Reflection (C#) is the process by which a
computer program can observe, do type introspection, and modify
its own structure and behavior at runtime.
In the example to follow we store information about how to use
various hooks (references to pieces of user interface) in the code
so that later at runtime we can use the information to determine
how to test our library of hooks.
CENTRALIZED HOOKS
[C#]
namespace Automation.Hooks.Web
{
public static class LoginPage
{
public static WebHook EmailField = By.Id(“email”);
public static WebHook PasswordField = By.Id(“password”);
public static WebHook SubmitButton = By.Id(“submit”);
[OmitFromSelfTest]
public static WebHook Spinner = By.Xpath(“//div[@class=’wait’]”);
private static void _StartSelfTest()
{
_SelfTestSeleniumInterface.Open(“http://www.zoosk.com”)
}
}
}
SIMPLE SELF-TEST
[PseudoCode]
// iterate in depth-first-search order
foreach(class c in Automation.Hooks.Web)
{
if c.hasMethod(“StartSelfTest”)
c.executeMethod(“StartSelfTest”);
foreach(WebHook h in c)
Log.Verify(IsElementPresent(h));
if c.hasMethod(“StopSelfTest”)
c.executeMethod(“StopSelfTest”);
}
MORE ADVANCED TECHNIQUES
• Subclass WebHook to things like TextBoxHook and have the
test check for reading and writing from the text field using
Selenium
• Do the same for Links, Images, etc. and write custom test
methods
• Randomize the order of which elements are tested to check for
robustness
• Add random data to the tests, e.g. type a random string into
the text box.
• Write custom test methods for structures that require it
THE END