Floehopper

Ruby Acceptance Testing

2010-08-19T21:23:00+00:00

In his recent article You’re Cuking It Wrong, Jonas Nicklas compares the following scenario which was submitted in a Cucumber issue :-

Scenario: Adding a subpage
  Given I am logged in
  Given a microsite with a Home page
  When I click the Add Subpage button
  And I fill in "Gallery" for "Title" within "#document_form_container"
  And I press "Ok" within ".ui-dialog-buttonpane"
  Then I should see /Gallery/ within "#documents"

With an improved version which he has written :-

Scenario: Adding a subpage
  Given I am logged in
  Given a microsite with a home page
  When I press "Add subpage"
  And I fill in "Title" with "Gallery"
  And I press "Ok"
  Then I should see a document called "Gallery"

Levels of Abstraction

Jonas says :-

there’s one crucial difference: the first feature is code, the second isn’t

Although I see what Jonas is getting at, neither of them are actually Ruby code – both are written in Cucumber’s Gherkin DSL. The key difference is really the level of abstraction – the first version has a test that is less abstracted from the implementation than the second version. As Jonas correctly points out, the test in the second version is consequently more readable and is less coupled to the underlying implementation.

Readability

Jonas also says :-

The argument against cucumber that’s often presented is that as a programmer, plain text is unnecessary, because we can all read code.

However, it’s a mistake to think that making a test readable necessarily means that it has to be written in a natural language. We could equally well write something like this :-

scenario "adding a subpage" {
  given {
    i_am_logged_in
    a_microsite_with_a_home_page
  }
  when {
    i_press "Add subpage"
    i_fill_in "Title",:with=>"Gallery"
    i_press "Ok"
  }
  then {
    i_should_see_a_document_called "Gallery"
  }
}

While I would agree that my version isn’t as pretty as Jonas’ version, I’d suggest it’s almost as readable. The difference being this is all valid Ruby code. I once saw someone (Nat Pryce or Steve Freeman, I think) do a neat trick where they changed the syntax highlighting in an editor to make some characters invisible and as a result made the code more readable. We could probably do something similar to make braces, underscores, etc invisible, and end up with something as good as Jonas’ version :-

scenario "adding a subpage"
  given
    i am logged in
    a microsite with a home page

  when 
    i press "Add subpage"
    i fill in "Title"  with  "Gallery"
    i press "Ok"

  then 
    i should see a document called "Gallery"

Overhead

It seems obvious to me that Cucumber adds an overhead in writing tests. It requires me to use regular expressions to convert natural language into code. While I recognise that the overhead is often small, it does still exist and I’ve definitely spent time debugging this aspect of tests. Given that I think it’s possible to write nicely abstracted readable tests in Ruby, it’s not obvious to me that the overhead of using Cucumber is worthwhile.

I’m going to try and find the time to look at frameworks like Coulda and Steak which seem to be more in line with my favoured approach.

Update

I thought it might be useful to include a quote from and a link to James Shore’s article on The Problems of Acceptance Testing

My experience with Fit and other agile acceptance testing tools is that they cost more than they’re worth. There’s a lot of value in getting concrete examples from real customers and business experts; not so much value in using “natural language” tools like Fit and similar.

Converting S5 slides to a single PDF on Mac OSX

2010-01-25T20:33:49+00:00

I used S5 in combination with Webby for my Ruby & Cocoa presentation at RubyManor. I hate PowerPoint with a passion and haven’t done enough presentations to be bothered to learn KeyNote. So I was much happier generating my slides from a plain text file marked up in Textile.

However, like Paul Battley, I found that S5 doesn’t work so well when someone wants to take those slides and put them in a video. Anyway, I’ve managed to convert my S5 slides into a single PDF using a combination of webkit2png and a custom Automator workflow :-

Ruby Script using `webkit2png`

#!/usr/bin/env ruby

NUMBER_OF_SLIDES = 94
SEQUENCE_FORMAT = "%0#{Math.log10(NUMBER_OF_SLIDES).to_i}d"
HOST = "jamesmead.local"
PATH = "/presentations/ruby-and-cocoa-ruby-manor-2009-12-14/"
BASE_URL = "http://#{HOST}#{PATH}"

(0...NUMBER_OF_SLIDES).each do |index|
  anchor = "#slide#{index}"
  url = "#{BASE_URL}#{anchor}"
  sequence = SEQUENCE_FORMAT % index
  puts %x[webkit2png --fullsize --filename=slide-#{sequence} #{url}]
  raise unless $?.success?
end

Custom Automator Workflow

Mocha on Bacon

2010-01-19T10:53:00+00:00

Some time ago, I removed Bacon integration from Mocha after deciding to only maintain integration with Test::Unit and MiniTest which are both Ruby standard libraries.

Happily Eloy Duran has just re-incarnated the Mocha adapter for Bacon. He’s updated the code a bit and re-written the tests. Thanks, Eloy.

Here’s an example of what you can do with Bacon and Mocha :-

require File.expand_path('../spec_helper', __FILE__)

describe "Bacon specs using Mocha, with a mock" do

  extend MochaBaconHelper

  it "passes when all expectations were fulfilled" do
    mockee = mock()
    lambda do
      mockee.expects(:blah)
      mockee.blah
    end.should.be satisfied
  end

end

Ruby scripting in the Webby "environment"

2010-01-16T17:04:00+00:00

As I mentioned in my previous post, I’m in the process of converting my blog from being a Rails app using Typo to being a static site generated by Webby.

I’ve found myself wanting to be able to write Ruby scripts with access to the Webby “environment” e.g. the @pages instance variable.

I couldn’t find any documentation about this, so with a bit of delving into the source code and a bit of experimentation, I came up with the following code. I thought I’d share it here in case it was of use to anyone else. And if anyone knows a better way of doing it, please let me know.

Note that I think this script has to run at the top level of the project.

require 'webby'

# load Sitefile, setup load path
main = Webby::Apps::Main.new
main.init([])
main.app.init 'webby'
main.app.load_rakefile

# load helpers
include BlogHelper
include UrlHelper

# load pages, layouts, etc.
builder = Webby::Builder.new
builder.load_files
@pages = Webby::Resources.pages

Ruby wrapper for the Delicious v2 API using the OAuth gem

2010-01-14T13:56:00+00:00

I’ve got a bit sidetracked recently while trying to convert my blog from using Typo to a static site using Webby. Typo provides a facility to tag your blog posts. This allows visitors to find all the articles tagged e.g. mocha.

Generating these tag pages becomes more awkward with the Webby-based site which is generated at deploy-time. I decided I liked the approach that Chris Roos took which was to tag his blog posts in del.icio.us so that visitors can use a del.icio.us URL to find all the articles with a particular tag

Initially I used WWW::Delicious which worked well enough with my main del.icio.us account. But then I decided I didn’t want to clutter up this account with bookmarks to all my blog posts, so I created a new account. Unfortunately since this is a new account, I have to use the v2 API with OAuth if I want to programmatically add bookmarks :-

To access data from accounts created using a Yahoo! ID, use the same API’s as below, but change the path to /v2, and make HTTP requests using OAuth as provided by the Yahoo! Developer Network.

I couldn’t find any examples of anyone doing this using Ruby, so I decided to have a go using the OAuth gem. It was a bit of a painful process due to a lack of decent documentation, but I got there in the end. I thought I’d share the code in case it saves anyone else a bit of time.

The code probably isn’t very robust since it’s lacking tests and it currently only offers the ability to add bookmarks which was all I needed. Probably what I should do next is integrate this code with something like WWW:Delicious to offer access to both v1 and v2 APIs, but since I can now do what I wanted, this is unlikely to happen in the near future.

In order to get the example code to work, you’ll need to login to a Yahoo! ID based del.icio.us account and go to your My Projects page :-

There you need to add a project and grant it read/write access to del.icio.us :-

Granting access to your project will generate a Consumer Key and a Consumer Secret. These values should be used to define the API_KEY and SHARED_SECRET constants respectively. The example.rb code does this by requiring a constants.rb file which you could add.

Here’s the example code which creates a single bookmark. Note that the first time this is run, it will open a browser and require you to login to obtain an oauth_verifier code :-

You will need to copy and paste this code into the terminal window :-

All the relevant tokens are saved to YAML files, so that subsequent calls to the API will work without user interaction. However, you should note that an access token expires after 1 hour. It ought to be possible to refresh this token without user interaction, but I haven’t been able to get that working yet, so in this case you’ll probably need to delete request_token.yml and access_token.yml and re-run the script.

$LOAD_PATH << File.join(File.dirname(__FILE__), 'oauth-extensions')
$LOAD_PATH << File.join(File.dirname(__FILE__), 'delicious')

require 'delicious/api'
require 'constants'

api = Delicious::API.new(API_KEY, SHARED_SECRET)
api.posts_add!(
  :url => 'http://www.google.com/',
  :description => 'Testing 1 2 3',
  :extended => 'Blah blah blah',
  :tags => 'testing google blah'
)

Finally, here’s the bookmark successfully added to my account :-

Ruby and Cocoa talk at Ruby Manor

2009-12-14T11:59:46+00:00

Here are the slides from my recent talk at Ruby Manor 2 on “Ruby and Cocoa”.

And here’s the code for my demo of acceptance testing the MacRuby HotCocoa Calculator example app using the OSX Accessibility API.

ActiveRecord Model Class Name Clash

2009-11-02T16:21:44+00:00

I’ve just spent way too long trying to work out why the validation callbacks were getting called twice on one of my ActiveRecord models. I thought I’d write this up in case it saves someone else some pain.

In the end, I narrowed the problem down to the name of the model class which is Sync. I should’ve thought of this sooner, but it turns out there is a Ruby standard library class called Sync. I haven’t delved any further into why validation was being called twice. As far as I’m concerned, once you re-open an existing Ruby class like this, it makes sense that all bets are off.

However, I am a bit confused why the original Sync model class definition didn’t give a “superclass mismatch” TypeError, which is what happens if I do :-

  script/runner "class Sync < ActiveRecord::Base; end"

It must be a load order issue, since if I put the class definition in a file in app/models, I don’t see the TypeError. It would’ve saved a lot of time if my Sync model class definition had failed fast in this way. I might see if there’s a way to make that happen.

Mocha NoMethodError / Load order since version 0.9.6

2009-09-18T11:35:40+00:00

There have been a number of load order issues since 0.9.6 was released for which I apologise.

The key message is that since 0.9.6, you must ensure that your test framework (Test::Unit or MiniTest) is loaded before Mocha is loaded, otherwise Mocha will not monkey-patch the test framework and calls to the Mocha API will result in NoMethodError.

I hope to write this up in more detail soon, but in the meantime here are a couple of useful links :-

Please contact us on the mailing list if you have any questions.

Mocha Release 0.9.8

2009-09-18T11:17:44+00:00

Release Notes

Fixed bug NameError raised when using Mocha as a Rails plug-in. Since 0.9.6 the Rails plugin has been broken. See bug report for details. You will need to explicitly load Mocha after the test framework has been loaded, e.g. by adding “require ‘mocha’” at the bottom of test/test_helper.rb.
Make Mocha::ParameterMatchers#regexp_matches, #includes, #has_value, #has_key more robust. Thanks to Sander Hartlage.
Allow passing a block to Mocha::Configuration methods to only change configuration for the duration of the block. Thanks to Dan Manges.
Fixed bug doc generation fails in 0.9.7 gem.
Remove rdoc template incorporating google analytics from source control. The file just needs to exist locally and be ignored by source control. This should stop the warning showing up on e.g. RunCodeRun build results.

Mocha Test Spies

2009-09-14T16:58:56+00:00

Introduction

Today I finally got round to looking at Joe Ferris’ Test Spies fork of Mocha. Joe explained how Test Spies work and why he thinks they are useful in Spy vs spy on the ThoughtBot blog. I apologise to Joe for not getting round to this sooner.

I have a number of hesitations about merging these changes into Mocha as they stand. My thinking hasn’t really crystallized, but in these situations I often find it’s useful to force myself to try and explain myself in writing. Hopefully it’ll also generate some feedback.

I’d like to thank Joe for taking the time to fork Mocha and add this functionality. I’m particularly grateful that he’s taken the trouble to write a bunch of high-level acceptance tests.

Concept

In this section I’ve tried to write up some comments on Joe’s blog post under roughly the same headings :-

Why would I let spies in my code?

Joe correctly points out that setting up expectations before exercising the object under test is a bit of a departure from the traditional Four-Phase Test regime. I can see how some people might see this as counter-intuitive, but it doesn’t seem particularly so to me. Also the idea of Expected Behaviour Specification (i.e. setting up expectations before the Exercise phase) has a couple of advantages :-

Using Test Spies, you have to (a) set up a stub so the mock object knows what value to return and (b) assert that the stubbed method was invoked the requisite number of times with the appropriate parameters. Whereas using Expected Behaviour Specification, these are combined into setting up an expectation on the mock object. I think this results in terser code with less duplication.
Using Expected Behaviour Specification means that the test can fail fast if an unexpected invocation is made against a mock object. Failing fast is generally a good idea, because you are more likely to be able to identify the root cause of the problem.

Sharing stubs in a context

describe PostsController, "on GET show" do
  before(:each) do
    @post = stub('a post', :to_param => '1')
    Post.stubs(:find => @post)

    get :show, :id => @post.to_param
  end

  it { should render_template(:show) }

  it "should find and assign the given post" do
    Post.should have_received(:find).with(@post.to_param)
    should assign_to(:post).with(@post)
  end
end

I’m going to struggle a bit when commenting on this code example for a couple of reasons :-

I dislike the use of setup methods (before in this RSpec example). I prefer to do my setup in-line in the test to make the test as independent and as explicit as possible. I particularly dislike exercising the object under test within the setup method. To me this seems like a more fundamental breakage of the Four-Phase Test concept.
In this example, I would not want to assert that Post.find has been called. Nat Pryce (one of the authors of jMock) has a rule of thumb that says: “stub queries and expect commands”. In this example, the call to Post.find is clearly a query, so I would just stub it. I think about it like this: (a) stubbing Post.find is setting up the environment and making the instance of Post available; (b) the assertion then checks that the instance of Post is assigned to the @post instance variable; (c) the combination of (a) and (b) is enough to tell me that the Post.find method has been called – where else would the controller have got the instance of Post from?

So I’m not convinced I would need Test Spies for this example.

Sharing stubs between tests

module PostHelpers
  def stub_post(post_attrs = {})
    post_attrs = {
      :to_param => '1',
      :published? => true,
      :title => 'a title'
    }.update(post_attrs)
    stub('a post', post_attrs)
  end

  def stub_post!(post_attrs = {})
    returning stub_post do |post|
      Post.stubs(:find => post)
    end
  end
end

describe PostsController do
  include PostHelpers
  it "should show a published post on GET show" do
    post = stub_post!
    get :show, :id => post.to_param
    Post.should have_received(:find).with(post.to_param)
    post.should have_received(:published?)
    should render_template(:show)
    should assign_to(:post).with(post)
  end
end

describe "/posts/show" do
  include PostHelpers
  it "should display a post" do
    assigns[:post] = stub_post
    render '/posts/show'
    template.should have_tag('h1', assigns[:post].title)
  end
end

In this example, as above, I would stub the call to Post.find rather than expect it.
I’d also prefer to use a real Post instance rather than a stub object. I’d use some kind of Test Data Builder like factory_girl to remove any duplication rather than methods like those in the PostHelpers module. While these ActiveRecord objects aren’t strictly Value Objects, I can’t help but think that this quote from Nat Pryce is relevant :-

Some languages might allow you to stub a value, but what’s the point? It is easier and less error prone to use the real type. If it’s not easier to use the real type then, I think, you should investigate why. What can you do to make the value type easier to use?

Steve Freeman makes a similar point in Test Smell: Everything is mocked.

Reusable matchers and assertions

# post.class.should have_received(:find).with(post.to_param)
should find(post)
# Post.should have_received(:new).with(post_attrs)
should build(Post, post_attrs)
# Post.should have_received(:new).with(post_attrs)
# post.should have_received(:save)
should build_and_save(Post, post_attrs)

I’m not convinced that extracting the assertion part of an expectation is sensible, because it’ll always need to be used in conjunction with the stub part of the same expectation.

Syntax

I think the suggested syntax extension is inconsistent with the existing Mocha API and a little clunky. See James Rosen’s comment on the mailing list. Looking at the following examples given in the documentation :-

  assert_received(mock, :to_s)
  assert_received(Radio, :new) {|expect| expect.with(1041) }
  assert_received(radio, :volume) {|expect| expect.with(11).twice }

I think I’d prefer to see something like :-

  assert_that mock.received(:to_s)
  assert_that Radio.received(:new).with(1041)
  assert_that radio.received(:volume).with(11).twice

Or something like :-

  mock.assert_received(:to_s)
  Radio.assert_received(:new).with(1041)
  radio.assert_received(:volume).with(11).twice

Note that I haven’t completely thought through the implementation implications of these suggestions i.e. whether they are even possible!

Implementation

The code looks good and, since Joe has written acceptance tests, there shouldn’t be too much difficulty in doing a bit of refactoring. I like the idea of introducing a model for an Invocation , but I think the Invocation instances should be stored by the relevant Mock instance rather than the Mockery. Similarly I’d push more of the logic in the HaveReceived class into relevant objects. And I’d like to avoid having to make Expectation#invocation_count writeable.

One slight concern is what happens if someone mixes up Expected Behaviour Specification expectations with Test Spies stubs and assertions? I’d probably at least want some acceptance tests to convince me that there are no complications on this front.

Conclusion

I’m sure that some people find the changes useful, but I’m not yet convinced I want to merge the Test Spies changes into Mocha. Please feel free to try and convince me, either here in the comments or in the mailing list thread.