Abstract
In test automation, code involved in testing is not only test logic, but also a
bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing
and etc. Test logic can be buried in these unrelated code which has nothing to do
with test logic itself, making test code hard to maintain, complex to reason about.
In this post, the layered architecture of test automation is presented to solve
this problem. In this layered architecture, test automation application is divided
into three layers: (1) test cases, focusing on the test logic of the application,
(2) domain and service layer, encapsulating http request, browser driving, result
parsing logics, providing interface for test cases layer, (3) system under test,
layer 2 will operate directly on this layer.
Problem
In test automation, code involved in testing is not only test logic, but also a
bunch of other supporting code, like url concatenation, html/xml parsing, UI accessing
libraries driving and etc. For example, to test a web service which carries out
operations like search by different keywords and return an xml containing certain
information (like customer information), the test automation code must (1) assemble
a URL based on operation under test, (2) send out http request with some http libraries,
(3) interpret the response sent back from web server, parse the xml, (4) compare
results returned to expected results.
In some test automation code, all these url concatenation, html/xml parsing, xpath
expression and test logic code gets written directly together, usually in one class
or one method. The test logic is with the string operation, XPath navigation code.
This form is easy to pick up and it's intuitive at the very beginning.
But this form has its problems:
- Test logic is hard to reason about and modify. When test logic is embedded into
a big amount of other unrelated code, it's difficult to see what is tested in the
code. To add new test cases, one inevitably has to reread the supporting code and
find out where is the best point to add new code. Test logic becomes hard to reason
about too. - Tests become fragile. Since test logic and supporting code like html parsing are
mixed together, one small chance in the 'contract' between system under test and
test automation code, can breaks the test automation. For example, if UI changes,
like moving an input element to different div, or changing an ID of some UI element,
all test code operating this part of UI is affected. - Maintenance cost is high. Since for a particular part of system, there are generally
several test cases. A big part of each test case is similar, for example, they may
all have to (1) assemble a URL based on operation under test, (2) send out http
request with some http libraries, (3) interpret the response sent back from web
server, parse the xml, (4) compare results returned to expected results. Since these
code are duplicated in all test cases, if any thing changes, you have to search
and modify all of them accordingly (and may be several times).
Solution
The domain of software development had experienced the same thing, and developed
a solution, that is 'Layered Architecture'. Basically, the value of the layered
architecture, to quote Domain-Driven Design, is
The value of layers is that each specializes in a particular aspect of a computer
program. This specialization allows more cohesive designs of each aspect, and it
makes these designs much easier to interpret. Of course, it is vital to choose layers
that isolate the most important cohesive design aspects.
In the domain of test automation, though focus on different thing, the fundamental
problem is the same, thus, similar solution can be applied:
Test Cases Layer | All (and only) test logic resides here. Test logic can be expressed concisely, with the help of the layer below. Test cases for different stories, scenarios, and corner cases rely on the same piece of code in the layer below, the only difference is in parameters or test data representing different cases. |
|---|---|
Domain & Service Layer | This layer will encapsulate operations to system under test, like url concatenation, response xml/html parsing, rich-client GUI/browser driving and etc. It will present the system under test in an domain language, rather in terms of xpath, sql or html. |
System Under Test Layer |
Example
Say we are testing a restful web service. With this web service, you can search
for some customer information, with telephone numbers as keyword.
To call this web service, the get http request in the following format should be
sent out:
http://{endpoint_name}/{version}/subscribers?telephoneNumber={telephoneNumber}
Then the piped data will be returned. In this piped string, subscriber's name, phone
number, address and other information is contained.
1|00001|Success|user1|1234|205 FIRST STREET|N|1||||||404|||FIRST|ST|SW|CA|AB||||937203|8|APARTMENT
Test cases for this service are (1) Search with a phone number which has a exact
match, (2) Search with a phone number which has several exact matches, (3) Search
with partial phone number.... The number of test cases is only limited by the imagination
of QA.
For each test case, the process is essentially the same: (1) assemble a URL containing
the telephone number keyword, (2) send http get request with http library, (3) parse
the piped data, (4) compare the data received with expected value. To avoid the
problems mentioned before, we apply the layered architecture:
Test Cases Layer
The implementation of this layer is test framework related. In this example, we
are using NBehave (but not in a traditional way, please refer to the post 'Fix NBehave').
[Story]
public class SearchCustomerbyTelephoneNumberStory: TestBase
{
[Scenario]
public void Search_with_a_phone_number_which_has_a_exact_match()
{
story.WithScenario("Search with a phone number which has a exact match")
.Given("")
.When(SEARCH_WITH, "04319820701",
SEARCH_WITH_ACTION)
.Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "73120205504",
ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION)
.Given("")
.When(SEARCH_WITH, "30419820808")
.Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "12666056628");
...
}
[Scenario]
public void Search_with_partial_phone_number()
{
story.WithScenario("Search with partial phone number")
.Given("")
.When(SEARCH_WITH, "7032")
.Then(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "12120000504")
.And(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "74123400504")
.And(ACCOUNT_INFORMATION_SHOULD_BE_RETURNED, "23120022504")
...
}
[Scenario]
public void Search_with_a_phone_number_which_has_several_exact_matches() {...}
[Scenario]
public void Search_with_non_existent_phone_numbers() {...}
[Scenario]
public void Search_with_invalid_phone_number_values() {...}
...
...
}
Variables with 'ACTION' suffix are lambda expressions. They make this piece of code
alive.
SEARCH_WITH_ACTION is for sending request to web service and parsing pipled data
returned. The code for CustomerService and Subscriber is in domain & service layer,
because these code are common supporting code for a variety of test cases.
SEARCH_WITH_ACTION = (phoneNumber) =>
{
actualSubscriber = customerService.SearchWithTelephoneNumber(phoneNumber);
};
ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION is for verifying the data
ACCOUNT_INFORMATION_SHOULD_BE_RETURNED_ACTION = (accountNumber) =>
{
//Get subscriber information from test data fixture
Subscriber expectedSubscriber = SubscriberFixture.Get(accountNumber)
//Verify each property of subscriber object returned from web service.
CustomAssert.AreEqual(expectedSubscriber, actualSubscriber);
};
Domain & Service Layer
Class CustomerService named after the real name of this web service. In requirement
document, daily interpersonal conversation, architecture map and code, this web
service is referred to by the same name. With this unified domain term, the ambiguity
is eliminated. For a more complete introduction, please refer to the post 'Domain
Based Testing'
public class CustomerService
{
public Subscriber SearchWithTelephoneNumber(string telephoneNumber)
{
string url =
string.Format(
"{0}/{1}/subscribers?telephoneNumber={2}",
endpoint, version, telephoneNumber);
//Send http request to web service, parse the xml returned,
//populate the subscriber object and etc.
return GetResponse(url);
}
...
}
Class Subscriber models, well, the subscriber. Comparing to the piped string format,
this tangible format is easier to catch (or do you like referring to telephone number
as pipedData[101]?).
public class Subscriber
{
public string AccountNumber { get; set; }
public string FirstName { get; set; }
public string Surname { get; set; }
public string TelephoneNumber { get; set; }
...
}
With this domain model, data verification can be carried out based on object. For
example, you can verify the first name is 'Bei'
Assert.AreEqual("Bei", subscriber.FirstName);
Or the phone number starts with '010'
Assert.IsTrue(subscriber.TelephoneNumber.StartsWith("010"));
How this solve the problem?
- Problem 'Test logic is hard to reason and modify'. Since we have a separated layer,
focusing only on test logic, making use of supporting code from the layer below,
test cases can be expressed in a way similar to English language, thus, difficulties
in reading, reasoning and modifying test code depend more on the coder's English
skill, rather than the code itself. - Problem 'Tests become fragile'. Since now we have a domain & service layer, isolating
the test cases from the real system under the test, any changes in system, can only
propagate to this newly added layer. We change the code in this layer accordingly,
and tests cases depending on this layer can still be running. - Problem 'Maintenance cost is high'. Thanks to the encapsulation in the domain &
service layer, duplicated code is removed from test cases. You have to modify only
one piece of code. And since the services and domain models are a modelling the
system under test, they are easier to understand, thus not hard to change.
Frequently Asked Questions
Q: This solution seems complex, do I have to use this?
A: It depends on the size and complexity of the system under test. If the size of
the system is very small, and business logic is simple enough, this way is over-killing.
In this situation, even test automation is a waste of time. Since if it only takes
a couple of minutes to manually test the system, when bother automating tests? For
moderate system, mixing test and supporting code can be working. If the business
logic is complex, I prefer this way.
Q: This architecture requires an investment before real tests can be started, is
that wasteful?
A: This is just another way of organizing code, even if test code isn't
organized in this way, the same piece of code like url concatenation, xml/html response
parsing, result verification must be written anyway. With this architecture, you
just have to break the code into different classes/methods. Plus, you don't actually
have to fully implement these layers all at once. These layers are scenario/test
cases driven. You implement if you need it right now.
Q: Designing this requires substantial object-oriented experience, not all QA can
do this.
A: I would say test automation is not only QA's responsibility. Other team members,
including developers should contribute to that too. And problems mentioned above
should be solved anyway, otherwise automated tests might be in trouble later.
No comments:
Post a Comment