My blog has been moved to http://beili.wordpress.com/
Thanks.
Wednesday, June 24, 2009
Monday, June 22, 2009
Test Automation Layered Architecture
Abstract
In test automation, code involved in testing is not only test logic, but also a
bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing
and etc. Test logic can be buried in these unrelated code which has nothing to do
with test logic itself, making test code hard to maintain, complex to reason about.
In this post, the layered architecture of test automation is presented to solve
this problem. In this layered architecture, test automation application is divided
into three layers: (1) test cases, focusing on the test logic of the application,
(2) domain and service layer, encapsulating http request, browser driving, result
parsing logics, providing interface for test cases layer, (3) system under test,
layer 2 will operate directly on this layer.
Problem
In test automation, code involved in testing is not only test logic, but also a
bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing
libraries driving and etc. For example, to test a web service which carries out
operations like search by different keywords and return an xml containing certain
information (like customer information), the test automation code must (1) assemble
a URL based on operation under test, (2) send out http request with some http libraries,
(3) interpret the response sent back from web server, parse the xml, (4) compare
results returned to expected results.
In some test automation code, all these url concatenation, html/xml parsing, xpath
expression and test logic code gets written directly together, usually in one class
or one method. The test logic is with the string operation, XPath navigation code.
This form is easy to pick up and it's intuitive at the very beginning.
But this form has its problems:
- Test logic is hard to reason about and modify. When test logic is embedded into
a big amount of other unrelated code, it's difficult to see what is tested in the
code. To add new test cases, one inevitably has to reread the supporting code and
find out where is the best point to add new code. Test logic becomes hard to reason
about too. - Tests become fragile. Since test logic and supporting code like html parsing are
mixed together, one small chance in the 'contract' between system under test and
test automation code, can breaks the test automation. For example, if UI changes,
like moving an input element to different div, or changing an ID of some UI element,
all test code operating this part of UI is affected. - Maintenance cost is high. Since for a particular part of system, there are generally
several test cases. A big part of each test case is similar, for example, they may
all have to (1) assemble a URL based on operation under test, (2) send out http
request with some http libraries, (3) interpret the response sent back from web
server, parse the xml, (4) compare results returned to expected results. Since these
code are duplicated in all test cases, if any thing changes, you have to search
and modify all of them accordingly (and may be several times).
Solution
The domain of software development had experienced the same thing, and developed
a solution, that is 'Layered Architecture'. Basically, the value of the layered
architecture, to quote Domain-Driven Design, is
The value of layers is that each specializes in a particular aspect of a computer
program. This specialization allows more cohesive designs of each aspect, and it
makes these designs much easier to interpret. Of course, it is vital to choose layers
that isolate the most important cohesive design aspects.
In the domain of test automation, though focus on different thing, the fundamental
problem is the same, thus, similar solution can be applied:
Test Cases Layer | All (and only) test logic resides here. Test logic can be expressed concisely, with the help of the layer below. Test cases for different stories, scenarios, and corner cases rely on the same piece of code in the layer below, the only difference is in parameters or test data representing different cases. |
|---|---|
Domain & Service Layer | This layer will encapsulate operations to system under test, like url concatenation, response xml/html parsing, rich-client GUI/browser driving and etc. It will present the system under test in an domain language, rather in terms of xpath, sql or html. |
System Under Test Layer |
Example
Say we are testing a restful web service. With this web service, you can search
for some customer information, with telephone numbers as keyword.
To call this web service, the get http request in the following format should be
sent out:
http://{endpoint_name}/{version}/subscribers?telephoneNumber={telephoneNumber}
Then the piped data will be returned. In this piped string, subscriber's name, phone
number, address and other information is contained.
1|00001|Success|user1|1234|205 FIRST STREET|N|1||||||404|||FIRST|ST|SW|CA|AB||||937203|8|APARTMENT
Test cases for this service are (1) Search with a phone number which has a exact
match, (2) Search with a phone number which has several exact matches, (3) Search
with partial phone number.... The number of test cases is only limited by the imagination
of QA.
For each test case, the process is essentially the same: (1) assemble a URL containing
the telephone number keyword, (2) send http get request with http library, (3) parse
the piped data, (4) compare the data received with expected value. To avoid the
problems mentioned before, we apply the layered architecture:
Test Cases Layer
The implementation of this layer is test framework related. In this example, we
are using NBehave (but not in a traditional way, please refer to the post 'Fix NBehave').
[Story]
public class SearchCustomerbyTelephoneNumberStory: TestBase
{
[Scenario]
public void Search_with_a_phone_number_which_has_a_exact_match()
{
story.WithScenario("Search with a phone number which has a exact match")
.Given("")
.When(SEARCH_WITH, "04319820701",
SEARCH_WITH_ACTION)
.Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "73120205504",
ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION)
.Given("")
.When(SEARCH_WITH, "30419820808")
.Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "12666056628");
...
}
[Scenario]
public void Search_with_partial_phone_number()
{
story.WithScenario("Search with partial phone number")
.Given("")
.When(SEARCH_WITH, "7032")
.Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "12120000504")
.And(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "74123400504")
.And(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "23120022504")
...
}
[Scenario]
public void Search_with_a_phone_number_which_has_several_exact_matches() {...}
[Scenario]
public void Search_with_non_existent_phone_numbers() {...}
[Scenario]
public void Search_with_invalid_phone_number_values() {...}
...
...
}
Variables with 'ACTION' suffix are lambda expressions. They make this piece of code
alive.
SEARCH_WITH_ACTION is for sending request to web service and parsing pipled data
returned. The code for CustomerService and Subscriber is in domain & service layer,
because these code are common supporting code for a variety of test cases.
SEARCH_WITH_ACTION = (phoneNumber) =>
{
actualSubscriber = customerService.SearchWithTelephoneNumber(phoneNumber);
};
ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION is for verifying the data
ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION = (accountNumber) =>
{
//Get subscriber information from test data fixture
Subscriber expectedSubscriber = SubscriberFixture.Get(accountNumber)
//Verify each property of subscriber object returned from web service.
CustomAssert.AreEqual(expectedSubscriber, actualSubscriber);
};
Domain & Service Layer
Class CustomerService named after the real name of this web service. In requirement
document, daily interpersonal conversation, architecture map and code, this web
service is referred to by the same name. With this unified domain term, the ambiguity
is eliminated. For a more complete introduction, please refer to the post 'Domain
Based Testing'
public class CustomerService
{
public Subscriber SearchWithTelephoneNumber(string telephoneNumber)
{
string url =
string.Format(
"{0}/{1}/subscribers?telephoneNumber={2}",
endpoint, version, telephoneNumber);
//Send http request to web service, parse the xml returned,
//populate the subscriber object and etc.
return GetResponse(url);
}
...
}
Class Subscriber models, well, the subscriber. Comparing to the piped string format,
this tangible format is easier to catch (or do you like referring to telephone number
as pipedData[101]?).
public class Subscriber
{
public string AccountNumber { get; set; }
public string FirstName { get; set; }
public string Surname { get; set; }
public string TelephoneNumber { get; set; }
...
}
With this domain model, data verification can be carried out based on object. For
example, you can verify the first name is 'Bei'
Assert.AreEqual("Bei", subscriber.FirstName);
Or the phone number starts with '010'
Assert.IsTrue(subscriber.TelephoneNumber.StartsWith("010"));
How this solve the problem?
- Problem 'Test logic is hard to reason and modify'. Since we have a separated layer,
focusing only on test logic, making use of supporting code from the layer below,
test cases can be expressed in a way similar to English language, thus, difficulties
in reading, reasoning and modifying test code depend more on the coder's English
skill, rather than the code itself. - Problem 'Tests become fragile'. Since now we have a domain & service layer, isolating
the test cases from the real system under the test, any changes in system, can only
propagate to this newly added layer. We change the code in this layer accordingly,
and tests cases depending on this layer can still be running. - Problem 'Maintenance cost is high'. Thanks to the encapsulation in the domain &
service layer, duplicated code is removed from test cases. You have to modify only
one piece of code. And since the services and domain models are a modelling the
system under test, they are easier to understand, thus not hard to change.
Frequently Asked Questions
Q: This solution seems complex, do I have to use this?
A: It depends on the size and complexity of the system under test. If the size of
the system is very small, and business logic is simple enough, this way is over-killing.
In this situation, even test automation is a waste of time. Since if it only takes
a couple of minutes to manually test the system, when bother automating tests? For
moderate system, mixing test and supporting code can be working. If the business
logic is complex, I prefer this way.
Q: This architecture requires an investment before real tests can be started, is
that wasteful?
A: This is just another way of organizing code, even if test code isn't
organized in this way, the same piece of code like url concatenation, xml/html response
parsing, result verification must be written anyway. With this architecture, you
just have to break the code into different classes/methods. Plus, you don't actually
have to fully implement these layers all at once. These layers are scenario/test
cases driven. You implement if you need it right now.
Q: Designing this requires substantial object-oriented experience, not all QA can
do this.
A: I would say test automation is not only QA's responsibility. Other team members,
including developers should contribute to that too. And problems mentioned above
should be solved anyway, otherwise automated tests might be in trouble later.
Monday, June 8, 2009
Fix NBehave
We are using NBehave as a Behaviour-Driven Development framework in our project
for test automation. At first we were doing according to the example provided in
the NBehave source code. During the way, we found some features of NBeahve didn't
really fit our need. Thus some solution / workaround are developed. This post is
mainly presenting the solution.
NBehave example - Smoke and mirrors
The example provided by NBehave is pretty straightforward:
- Tests for multiple stories are put in one class, mark as some Theme.
- All tests for one story is put in one method, with story name as method name.
- Each story is divided into multiple scenarios.
- In each scenario, there is multiple give / when / then, as an example.
Example code is like this:
[Theme("Account transfers and deposits")]
public class AccountSpecs
{
[Story]
public void Transfer_to_cash_account()
{
Account savings = null;
Account cash = null;
var transferStory = new Story("Transfer to cash account");
transferStory
.AsA("savings account holder")
.IWant("to transfer money from my savings account")
.SoThat("I can get cash easily from an ATM");
transferStory
.WithScenario("Savings account is in credit")
.Given("my savings account balance is $balance", 100, accountBalance => { savings = new Account(accountBalance); })
.And("my cash account balance is $balance", 10, accountBalance => { cash = new Account(accountBalance); })
.When("I transfer $amount to cash account", 20, transferAmount => savings.TransferTo(cash, transferAmount))
.Then("my savings account balance should be $balance", 80, expectedBalance => savings.Balance.ShouldEqual(expectedBalance))
.And("my cash account balance should be $balance", 30, expectedBalance => cash.Balance.ShouldEqual(expectedBalance))
.Given("my savings account balance is 400")
.And("my cash account balance is 100")
.When("I transfer 100 to cash account")
.Then("my savings account balance should be 300")
.And("my cash account balance should be 200")
.Given("my savings account balance is 500")
.And("my cash account balance is 20")
.When("I transfer 30 to cash account")
.Then("my savings account balance should be 470")
.And("my cash account balance should be 50");
transferStory
.WithScenario("Savings account is overdrawn")
.Given("my savings account balance is -20")
.And("my cash account balance is 10")
.When("I transfer 20 to cash account")
.Then("my savings account balance should be -20")
.And("my cash account balance should be 10");
}
[Story]
public void Withdraw_from_savings_account_pending()
{
var transferStory = new Story("Withdraw from savings account");
transferStory
.AsA("savings account holder")
.IWant("to withdraw money from my savings account")
.SoThat("I can pay for things with cash");
transferStory
.WithScenario("Savings account is in credit")
.Pending("ability to withdraw from accounts")
.Given("my savings account balance is 400")
.When("I withdraw $amount from my savings account", 100)
.Then("my savings account balance should be 300");
}
}
This example does a good job for spiking purpose. Anyone pick up this framework,
write sample code according to this example, and then get a pretty test result report.
But in the battle with a real enterprise project, the limitation of NBehave and
this example emerges.
Problems
First, The tests for the whole story halt, when the first error is encountered.
For example, in the test method 'Transfer_to_cash_account', if there is an assertion
failure in the first example (given / when / then structure), the rest of them won't
get executed. This can cause a big problem, since the first error (bug caught) is
the road blocker for the following tests. Ideally, by executing tests, we can get
all errors caught, rather than the first one. In the implementation level, this
is caused by exception thrown by NUnit.Framework.Assert class, since whenever there
is an assertion failure, NUnit throws an Exception. Because of this exception, the
code below, in the same method, is skipped.
Second, the file becomes too lengthy if multiple stories are jammed into one class,
causing maintenanceproblems. In this approach, multiple related stories are put
in one single class, and in each story, there is multiple scenario, covering all
possible situation might be encountered in that story. Thus, the source code in
one single method can easily grows to over 100 lines, and depends on the number
of stories within one 'Theme', the whole class containing all related stories can
easily get over several hundreds to one thousand lines (basically, the more complete
your test cases are, the messier your code will be). One has to scroll up and down,
do searching, add bookmark to the source code to be able to get to one specific
point. This is the easist way to slow the automation work down. Ideally, the code
should be simple and neat, intent revealing, rather than jammed together, burying
the small piece of code where are working on in a chunk of other unrelated code.
Third, when writing new tests, one has to execute all existing test code for that
story, wasting time waiting for the newly added code to be executed (and the newly
added code is of course, the last one get executed). Ideally, one can pick up one
specific scenario test code and execute it easily. This is essential in automating
phrase, since whenever new code is added, we want to try this new code to see if
it works as expected. Again, the more complete your cases are for one story, the
more test code you have. If anytime you add a piece of new code, you have to execute
all existing tests for that story, that would be a time killer.
Fourth, there is no built-in runner for IDE like Visual Studio. One has to run it
in a separate process, like NBehave-Console.exe or NBehave task for NAnt/MSBuild.
Since NBehave is relatively new, at least comparing NUnit, the supporting tools
are not complete, running code when developing is time-consuming. Ideally, one should
be able to run the specific test within IDE, with a single-click.
Solution
The solution for these problems were actually evolved by refactoring. Here
I tend to put the final result, since the whole process is lengthy.
Let's see the current way of organizing NBehave code:
Test for story 'Transfer to cash account'
[TestFixture]
[Theme("Account transfers and deposits")]
public class TransferToCashAccount : TestBase
{
private Account savings;
private Account cash;
private Story transferStory;
public override void StorySetUp()
{
transferStory = new Story("Transfer to cash account");
transferStory
.AsA("savings account holder")
.IWant("to transfer money from my savings account")
.SoThat("I can get cash easily from an ATM");
}
[Test]
public void SavingsAccountIsInCredit()
{
transferStory
.WithScenario("Savings account is in credit")
.Given("my savings account balance is $balance", 100, accountBalance => { savings = new Account(accountBalance); })
.And("my cash account balance is $balance", 10, accountBalance => { cash = new Account(accountBalance); })
.When("I transfer $amount to cash account", 20, transferAmount => savings.TransferTo(cash, transferAmount))
.Then("my savings account balance should be $balance", 80, expectedBalance => Assert.AreEqual(expectedBalance, savings.Balance))
.And("my cash account balance should be $balance", 30, expectedBalance => Assert.AreEqual(expectedBalance, cash.Balance))
.Given("my savings account balance is 400")
.And("my cash account balance is 100")
.When("I transfer 100 to cash account")
.Then("my savings account balance should be 300")
.And("my cash account balance should be 200")
.Given("my savings account balance is 500")
.And("my cash account balance is 20")
.When("I transfer 30 to cash account")
.Then("my savings account balance should be 470")
.And("my cash account balance should be 50");
}
[Test]
public void SavingsAccountIsOverdrawn()
{
transferStory
.WithScenario("Savings account is overdrawn")
.Given("my savings account balance is -20")
.And("my cash account balance is 10")
.When("I transfer 20 to cash account")
.Then("my savings account balance should be -20")
.And("my cash account balance should be 10");
}
}
Test for story 'Withdraw from savings account'
[Theme("Account transfers and deposits"), TestFixture]
public class WithdrawFromSavingsAccount : TestBase
{
public override void StorySetUp()
{
...
}
[Test]
public void SavingsAccountIsInCredit()
{
...
}
}
The code for TestBase:
public class TestBase
{
private void RunScenario(MethodInfo methodInfo)
{
Console.WriteLine(methodInfo.Name);
try
{
methodInfo.Invoke(this, null);
}
catch { }
}
private bool HasAttribute(MethodInfo methodInfo, Type attribute)
{
object[] attributes = methodInfo.GetCustomAttributes(true);
foreach (object o in attributes)
{
if (o.GetType().Equals(attribute))
{
return true;
}
}
return false;
}
[Story]
public void Runner()
{
StorySetUp();
MethodInfo[] methods = GetType().GetMethods();
foreach (MethodInfo m in methods)
{
if (HasAttribute(m, typeof(TestAttribute)))
{
RunScenario(m);
}
}
StoryTearDown();
}
[TestFixtureSetUp]
public virtual void StorySetUp() { }
[TestFixtureTearDown]
public virtual void StoryTearDown() { }
[SetUp]
public virtual void ScenarioSetUp() { }
[TearDown]
public virtual void ScenarioTearDown() { }
}
The basic structure is:
- Each story is in its own class, with story name as class name.
- Each scenario is in one single method, with scenario description as method name.
[Test]attribute is attached to each method. - Stories related (previously in one single class) are all mark with the Theme attribute
with the same description. [TestFixture]attribute is applied to each story class.- All story class inherits
TestBase. - All story level code are put into method
StorySetUp
How this solves problems stated before?
"Problem one: The tests for the whole story halt, when the first error is encountered"
Answer: Now scenario is in its own method, and a try ... catch() block
is placed in the RunScenario method, surrounding each scenario. If
any assertion exception is thrown, the following tests for other scenarios won't
be skipped. Say if there is an error in 'Savings account is in credit' scenario,
other scenarios like 'Savings account is overdrawn' will still get executed. This
part is hidden in the TestBase class, and is transparent to test writers.
"Problem two: the file becomes too lengthy if multiple stories are jammed into one
class."
Answer: Now each story is in its own class and each scenario is in its own method.
By splitting the giant 'Theme' class, the nuber of lines if each class drops dramatically.
By browsing in Solution Explorer (well, it is even simpler when resharper is installed),
one can navigate to specific story. Since we only work on one story at a time, putting
code for one story in one class, we can simply edit that specific class, without
potentially modifying code for other stories accidently.
"Problem three: when writing new tests, one has to execute all existing test code
for that story"
Answer: Since now each scenario is in its own method marked with [Test]
attribute, one can always specific the scenario code currently working on and execute.
"Problem four: there is no built-in runner"
Answer: Since now each scenario is in its own method, and marked with [Test]
attribute, if Resharper or TestDriven.NET is installed, one can execute one scenario
/ one story / all tests in one directly / project / solution, in exactly the same
way as executing NUnit tests in Visual Studio.
What's inside that TestBase?
For people who just want to write test case, there is nothing inside. Just inherits,
and use it. Forget about this class, nothing important.
For people who want to change it, the only thing that is important is the Runner
method marked with [Story] attribute. NBehave will search for methods
marked with [Story] attribute, and execute them. When this class is
inherited, this method will be also inherited by the subclass, thus every Runner
method in subclasses will be executed. During the executing, this method will use
Reflection API to get the list of methods marked as [Test], and execute.
Friday, May 22, 2009
Domain Based Testing (1) - An Example
When doing test automation for end to end functional test, each scenarios of every story should be included in the
test suite. As the number of stories grows, the complexity of test automation code increases accordingly, thus the
maintainability of test code becomes more important. In this post, a way of organizing code is shown for handling
this problem. We start from an example, gradually refactor code toward a more ‘domain based testing’ style, and then
explain its benefits.
Domain based testing is simply a way of writing tests in terms of domain concept, rather than implementation
details.
Let’s start from an example.
Say we are doing testing automation for Google search engine. At the first step, we want to navigate to the main page
(www.google.com). Using Watir in Ruby, code is as below:
ie = Watir::IE.new
ie.goto "http://www.google.com"
Second, input search keyword, and click ‘Google Search’ button:
ie.text_field(:name, "q").set "Watir"
ie.button(:name, "btnG").click
Then, extract data from entries returned:
title_of_first_entry = ie.cell(:xpath, "XPATH_FOR_TITLE").text
description_of_first_entry = ie.cell(:xpath, "XPATH_FOR_DESC").text
url_of_first_entry = ie.cell(:xpath, "XPATH_FOR_URL").text
title_of_second_entry = ie.cell(:xpath, "XPATH_FOR_TITLE").text
description_of_second_entry = ie.cell(:xpath, "XPATH_FOR_DESC").text
url_of_second_entry = ie.cell(:xpath, "XPATH_FOR_URL").text
…
Finally, assertions can be carried out based on these values:
title_of_first_entry.should == "Watir - Overview"
description_of_first_entry.should == "Watir is an open-source library for automating web browsers."
url_of_first_entry.should == "wtr.rubyforge.org"
…
Now we are going to verify that the ‘site:’ keyword works. (With this keyword, search will be carried out only in
site specified.). To do that, we basically have to open the main page again, type in keywords and click the search
button, then extract the values of each entry and do verification. But should we copy & paste the code? Definitely
not, duplicated code is root of all evils. We can extract a method and reuse:
def search(keywords)
ie.goto http://www.google.com
ie.text_field(:name, "q").set keywords
ie.button(:name, "btnG").click
end
Now we can simply search by just one function call:
search("site:www.amazon.com Watir")
Isn’t this much simpler and readable?
Then how to deal with the code extract data from returned html? Of course, extract a method again:
def extract_data()
title_of_nth_entry = ie.cell(:xpath, "XPATH_FOR_TITLE").text
description_of_nth_entry = ie.cell(:xpath, "XPATH_FOR_DESC").text
url_source_of_nth_entry = ie.cell(:xpath, "XPATH_FOR_URL").text
…
end
Now we can simply call two methods:
search("site:www.amazon.com Watir")
extract_data()
and do assertion against these data.
But, it is still verbose, since every time you search something and do verification, the exactly same two function
calls will be made. To deal with this, we can simply merge them together, calling extract_data inside of the search
function. So now, all we have to do is calling:
search("site:www.amazon.com Watir")Done.
But can it be even simpler? Yes, of course. I won’t ask if the answer is no.
Did you notice a group of ‘three variables’? We have variable title, variable description and variable url for the
first search result entry, second entry, third entry…… Is there any relationship between these three variables? Yes,
they all belong to one result entry. Based on the search keywords, search engine simply returns a list matching
result entries. Thus, we should model this out loudly!
class ResultEntry
attr_accessor :title
attr_accessor :description
attr_accessor :url
end
Now, the search function can return an array of ResultEntry objects, and assertions can be carried out on these
objects:
result_entries = search("site:wtr.rubyforge.org Watir")
result_entries[0].title.should == "Watir - Overview"
result_entries[0].description.should == "Watir is an open-source library for automating web browsers."
result_entries[0].url.should == "wtr.rubyforge.org"
This is domain based testing!
Comparing this to the first style:
ie.goto "http://www.google.com"
ie.text_field(:name, "q").set "Watir"
ie.button(:name, "btnG").click
title_of_first_entry = ie.cell(:xpath, "XPATH_FOR_TITLE").text
description_of_first_entry = ie.cell(:xpath, "XPATH_FOR_DESC").text
url_of_first_entry = ie.cell(:xpath, "XPATH_FOR_URL").text
title_of_second_entry = ie.cell(:xpath, "XPATH_FOR_TITLE").text
description_of_second_entry = ie.cell(:xpath, "XPATH_FOR_DESC").text
url_of_second_entry = ie.cell(:xpath, "XPATH_FOR_URL").text
…
title_of_first_entry.should == "Watir - Overview"
description_of_first_entry.should == "Watir is an open-source library for automating web browsers."
url_of_first_entry.should == "wtr.rubyforge.org"
…
Which one is better in terms of readability and maintainability? The result is evident.
Subscribe to:
Posts (Atom)