We are using NBehave as a Behaviour-Driven Development framework in our project
for test automation. At first we were doing according to the example provided in
the NBehave source code. During the way, we found some features of NBeahve didn't
really fit our need. Thus some solution / workaround are developed. This post is
mainly presenting the solution.
NBehave example - Smoke and mirrors
The example provided by NBehave is pretty straightforward:
- Tests for multiple stories are put in one class, mark as some Theme.
- All tests for one story is put in one method, with story name as method name.
- Each story is divided into multiple scenarios.
- In each scenario, there is multiple give / when / then, as an example.
Example code is like this:
[Theme("Account transfers and deposits")]
public class AccountSpecs
{
[Story]
public void Transfer_to_cash_account()
{
Account savings = null;
Account cash = null;
var transferStory = new Story("Transfer to cash account");
transferStory
.AsA("savings account holder")
.IWant("to transfer money from my savings account")
.SoThat("I can get cash easily from an ATM");
transferStory
.WithScenario("Savings account is in credit")
.Given("my savings account balance is $balance", 100, accountBalance => { savings = new Account(accountBalance); })
.And("my cash account balance is $balance", 10, accountBalance => { cash = new Account(accountBalance); })
.When("I transfer $amount to cash account", 20, transferAmount => savings.TransferTo(cash, transferAmount))
.Then("my savings account balance should be $balance", 80, expectedBalance => savings.Balance.ShouldEqual(expectedBalance))
.And("my cash account balance should be $balance", 30, expectedBalance => cash.Balance.ShouldEqual(expectedBalance))
.Given("my savings account balance is 400")
.And("my cash account balance is 100")
.When("I transfer 100 to cash account")
.Then("my savings account balance should be 300")
.And("my cash account balance should be 200")
.Given("my savings account balance is 500")
.And("my cash account balance is 20")
.When("I transfer 30 to cash account")
.Then("my savings account balance should be 470")
.And("my cash account balance should be 50");
transferStory
.WithScenario("Savings account is overdrawn")
.Given("my savings account balance is -20")
.And("my cash account balance is 10")
.When("I transfer 20 to cash account")
.Then("my savings account balance should be -20")
.And("my cash account balance should be 10");
}
[Story]
public void Withdraw_from_savings_account_pending()
{
var transferStory = new Story("Withdraw from savings account");
transferStory
.AsA("savings account holder")
.IWant("to withdraw money from my savings account")
.SoThat("I can pay for things with cash");
transferStory
.WithScenario("Savings account is in credit")
.Pending("ability to withdraw from accounts")
.Given("my savings account balance is 400")
.When("I withdraw $amount from my savings account", 100)
.Then("my savings account balance should be 300");
}
}
This example does a good job for spiking purpose. Anyone pick up this framework,
write sample code according to this example, and then get a pretty test result report.
But in the battle with a real enterprise project, the limitation of NBehave and
this example emerges.
Problems
First, The tests for the whole story halt, when the first error is encountered.
For example, in the test method 'Transfer_to_cash_account', if there is an assertion
failure in the first example (given / when / then structure), the rest of them won't
get executed. This can cause a big problem, since the first error (bug caught) is
the road blocker for the following tests. Ideally, by executing tests, we can get
all errors caught, rather than the first one. In the implementation level, this
is caused by exception thrown by NUnit.Framework.Assert class, since whenever there
is an assertion failure, NUnit throws an Exception. Because of this exception, the
code below, in the same method, is skipped.
Second, the file becomes too lengthy if multiple stories are jammed into one class,
causing maintenanceproblems. In this approach, multiple related stories are put
in one single class, and in each story, there is multiple scenario, covering all
possible situation might be encountered in that story. Thus, the source code in
one single method can easily grows to over 100 lines, and depends on the number
of stories within one 'Theme', the whole class containing all related stories can
easily get over several hundreds to one thousand lines (basically, the more complete
your test cases are, the messier your code will be). One has to scroll up and down,
do searching, add bookmark to the source code to be able to get to one specific
point. This is the easist way to slow the automation work down. Ideally, the code
should be simple and neat, intent revealing, rather than jammed together, burying
the small piece of code where are working on in a chunk of other unrelated code.
Third, when writing new tests, one has to execute all existing test code for that
story, wasting time waiting for the newly added code to be executed (and the newly
added code is of course, the last one get executed). Ideally, one can pick up one
specific scenario test code and execute it easily. This is essential in automating
phrase, since whenever new code is added, we want to try this new code to see if
it works as expected. Again, the more complete your cases are for one story, the
more test code you have. If anytime you add a piece of new code, you have to execute
all existing tests for that story, that would be a time killer.
Fourth, there is no built-in runner for IDE like Visual Studio. One has to run it
in a separate process, like NBehave-Console.exe or NBehave task for NAnt/MSBuild.
Since NBehave is relatively new, at least comparing NUnit, the supporting tools
are not complete, running code when developing is time-consuming. Ideally, one should
be able to run the specific test within IDE, with a single-click.
Solution
The solution for these problems were actually evolved by refactoring. Here
I tend to put the final result, since the whole process is lengthy.
Let's see the current way of organizing NBehave code:
Test for story 'Transfer to cash account'
[TestFixture]
[Theme("Account transfers and deposits")]
public class TransferToCashAccount : TestBase
{
private Account savings;
private Account cash;
private Story transferStory;
public override void StorySetUp()
{
transferStory = new Story("Transfer to cash account");
transferStory
.AsA("savings account holder")
.IWant("to transfer money from my savings account")
.SoThat("I can get cash easily from an ATM");
}
[Test]
public void SavingsAccountIsInCredit()
{
transferStory
.WithScenario("Savings account is in credit")
.Given("my savings account balance is $balance", 100, accountBalance => { savings = new Account(accountBalance); })
.And("my cash account balance is $balance", 10, accountBalance => { cash = new Account(accountBalance); })
.When("I transfer $amount to cash account", 20, transferAmount => savings.TransferTo(cash, transferAmount))
.Then("my savings account balance should be $balance", 80, expectedBalance => Assert.AreEqual(expectedBalance, savings.Balance))
.And("my cash account balance should be $balance", 30, expectedBalance => Assert.AreEqual(expectedBalance, cash.Balance))
.Given("my savings account balance is 400")
.And("my cash account balance is 100")
.When("I transfer 100 to cash account")
.Then("my savings account balance should be 300")
.And("my cash account balance should be 200")
.Given("my savings account balance is 500")
.And("my cash account balance is 20")
.When("I transfer 30 to cash account")
.Then("my savings account balance should be 470")
.And("my cash account balance should be 50");
}
[Test]
public void SavingsAccountIsOverdrawn()
{
transferStory
.WithScenario("Savings account is overdrawn")
.Given("my savings account balance is -20")
.And("my cash account balance is 10")
.When("I transfer 20 to cash account")
.Then("my savings account balance should be -20")
.And("my cash account balance should be 10");
}
}
Test for story 'Withdraw from savings account'
[Theme("Account transfers and deposits"), TestFixture]
public class WithdrawFromSavingsAccount : TestBase
{
public override void StorySetUp()
{
...
}
[Test]
public void SavingsAccountIsInCredit()
{
...
}
}
The code for TestBase:
public class TestBase
{
private void RunScenario(MethodInfo methodInfo)
{
Console.WriteLine(methodInfo.Name);
try
{
methodInfo.Invoke(this, null);
}
catch { }
}
private bool HasAttribute(MethodInfo methodInfo, Type attribute)
{
object[] attributes = methodInfo.GetCustomAttributes(true);
foreach (object o in attributes)
{
if (o.GetType().Equals(attribute))
{
return true;
}
}
return false;
}
[Story]
public void Runner()
{
StorySetUp();
MethodInfo[] methods = GetType().GetMethods();
foreach (MethodInfo m in methods)
{
if (HasAttribute(m, typeof(TestAttribute)))
{
RunScenario(m);
}
}
StoryTearDown();
}
[TestFixtureSetUp]
public virtual void StorySetUp() { }
[TestFixtureTearDown]
public virtual void StoryTearDown() { }
[SetUp]
public virtual void ScenarioSetUp() { }
[TearDown]
public virtual void ScenarioTearDown() { }
}
The basic structure is:
- Each story is in its own class, with story name as class name.
- Each scenario is in one single method, with scenario description as method name.
[Test]attribute is attached to each method. - Stories related (previously in one single class) are all mark with the Theme attribute
with the same description. [TestFixture]attribute is applied to each story class.- All story class inherits
TestBase. - All story level code are put into method
StorySetUp
How this solves problems stated before?
"Problem one: The tests for the whole story halt, when the first error is encountered"
Answer: Now scenario is in its own method, and a try ... catch() block
is placed in the RunScenario method, surrounding each scenario. If
any assertion exception is thrown, the following tests for other scenarios won't
be skipped. Say if there is an error in 'Savings account is in credit' scenario,
other scenarios like 'Savings account is overdrawn' will still get executed. This
part is hidden in the TestBase class, and is transparent to test writers.
"Problem two: the file becomes too lengthy if multiple stories are jammed into one
class."
Answer: Now each story is in its own class and each scenario is in its own method.
By splitting the giant 'Theme' class, the nuber of lines if each class drops dramatically.
By browsing in Solution Explorer (well, it is even simpler when resharper is installed),
one can navigate to specific story. Since we only work on one story at a time, putting
code for one story in one class, we can simply edit that specific class, without
potentially modifying code for other stories accidently.
"Problem three: when writing new tests, one has to execute all existing test code
for that story"
Answer: Since now each scenario is in its own method marked with [Test]
attribute, one can always specific the scenario code currently working on and execute.
"Problem four: there is no built-in runner"
Answer: Since now each scenario is in its own method, and marked with [Test]
attribute, if Resharper or TestDriven.NET is installed, one can execute one scenario
/ one story / all tests in one directly / project / solution, in exactly the same
way as executing NUnit tests in Visual Studio.
What's inside that TestBase?
For people who just want to write test case, there is nothing inside. Just inherits,
and use it. Forget about this class, nothing important.
For people who want to change it, the only thing that is important is the Runner
method marked with [Story] attribute. NBehave will search for methods
marked with [Story] attribute, and execute them. When this class is
inherited, this method will be also inherited by the subclass, thus every Runner
method in subclasses will be executed. During the executing, this method will use
Reflection API to get the list of methods marked as [Test], and execute.
No comments:
Post a Comment