iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson

In the previous article iOS: Setting Up Acceptance Testing I introduced Frank. The next and last step will be to run Cucumber and Frank headless from Jenkins/Hudson.

Frank is still new, and it uses apple script to remote control XCode to run the “frankified” app in the simulator. That’s fine for running the cucumber scenarios manually but it is not so good for running them from Jenkins/Hudson.

To run the cucumber tests from Jenkins/Hudson I need it to run headless, i.e. it should not open any window while running the tests. I don’t want Jenkins/Hudson to interfere with whatever I’m doing right know. It should run invisible in the background.

Luckily it is possible to run an iOS app headless simply by running the app directly like this:

<path to app>/MyApp.app/MyApp -RegisterForSystemEvents

By default I would use a cucumber step like the following to run my app from cucumber:

Given /^the app has just started$/ do
  launch_app_in_simulator
end

with launch_app_in_simulator beeing the frank function that uses apple script  to run the app by controlling XCode.

I have changed that to:

Given /^the app has just started$/ do
  launch_app_headless
end

with launch_app_headless looking like this:

def launch_app_headless
  @apppid = fork do
    exec(APP, "-RegisterForSystemEvents")
  end

  wait_for_frank_to_come_up
end

APP is a string variable that contains the path to the frankified app. It is currently set hardcoded in my cucumber Before block:

require 'fileutils'

##
## adjust app, sdk dir and app dir
##
APP    = "build/Debug-iphonesimulator/Readme Frankified.app/Readme Frankified"
APPDIR = "build/Readme.build/Debug-iphonesimulator/Readme Frankified.build"
SDKDIR = "/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.2.sdk"

USERDIR = "#{APPDIR}/iPhone Simulator User Dir"
PREFDIR = "#{USERDIR}/Library/Preferences"

ACCESIBILITY_PLIST   = "com.apple.Accessibility.plist"
ACCESIBILITY_CONTENT = <<PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ApplicationAccessibilityEnabled</key>
 <true/>
</dict>
</plist>
PLIST

Before do
  # check that pwd contains the "build" dir as we are creating
  # items relative to it.
  Dir["build"].length.should == 1

  # make sure we do start with a clean environment
  FileUtils.remove_dir("#{USERDIR}",true)

  pwd     = "#{Dir.pwd}"
  prefdir = "#{pwd}/#{PREFDIR}"
  FileUtils.mkdir_p prefdir

  File.open("#{PREFDIR}/#{ACCESIBILITY_PLIST}", 'w') do |f|
    f <<ACCESIBILITY_CONTENT
  end

  ENV['SDKROOT']               = "#{SDKDIR}"
  ENV['DYLD_ROOT_PATH']        = "#{SDKDIR}"
  ENV['IPHONE_SIMULATOR_ROOT'] = "#{SDKDIR}"
  ENV['TEMP_FILES_DIR']        = "#{pwd}/#{APPDIR}"
  ENV['CFFIXED_USER_HOME']     = "#{pwd}/#{USERDIR}"
end

The accessibility stuff is required to create a configuration file for the app that enables accessibility. Without it Frank will not find the accessibilityLabels which are used to identify the ui elements.

Now I’m able to run the cucumber test for my app headless in Jenkins/Hudson. I created a second build configuration for the acceptance tests. It uses nearly the same setup as for unit testing. I just have added an additional “Execute Shell” step after building the frankified app that runs cucumber:

cd trunk
cucumber -f junit --o ../cucumber --tags ~@ignore features.frank

Using the -f junit parameter cucmber will create an junit xml file so Jenkins/Hudson can display the test results in the same ways as the unit tests. features.frank is the folder were I keep my cucumber scenarios.

Code coverage is set up the same way as for the unit tests (see iOS: Running Unit Tests with Code Coverage in Jenkins/Hudson). Now here is also the last complication. Frank currently doesn’t shut down the app and the result is that the app doesn’t write the coverage information.

To fix this I simply added a super simple exit command (only calling exit(0)) to Franks embedded http server and a ruby function to call it. I have forked the Frank git repository here http://github.com/hauner/Frank.

My cucumber After block now shuts down the app so it does write the code coverage info:

After do
  frankly_exit
end

with franky_exit beeing:

def frankly_exit
  get_to_uispec_server('exit')
  # calling exit in the app will not return any response
  # so we simply catch the error caused by exiting.
  rescue EOFError
end

Now the app will shut down with a clean exit, write the code coverage information and the code coverage step in Jenkins/Hudson will properly display the code coverage for my acceptance tests.

Done, finally :-)

Summary: I have reached my goal to have a test environment to create unit tests (using Google Toolbox for Mac,  OCMock & OCHamcrest) and acceptance tests (using Cucumber & Frank) of an iOS  app. Both test categories are running in Jenkins/Hudson as two separate builds and  both builds provide code coverage information. It took a little bit of work, but it is running now.

That is all I have to say regarding testing an iOS app for now! :-)

This is the 6th article of iOS: Setting Up a Test Environment is a Mess.

23 thoughts on “iOS: Running Cucumber/Frank with Code Coverage in Jenkins/Hudson”

  1. Frank Tribbulations of rvm « Tech Blogs

  2. The junit output has nothing to do with the code coverage output. That are two different files… The coverage information should be in a file called coverage.xml (or whatever you called it).

    Reply

  3. Can you post a sample junit output file? Mine looks like below. I dont see any code coverage information like I could using coverstory. Please advise.

    Reply

  4. Hi Hauner,

    Nice series of blog posts. +1.

    I have followed your steps and i have everything setup perfectly.
    Coverage for both my unit as acceptance tests.

    I have only 1 issue. My unit test results for the cucumber features are not properly shown in Hudson and when i look into the TEST-*.xml files, the test suite/class is not set properly.
    This way i cannot check which tests fails or not.

    Did/are you encounter(ing) the same issue?
    How did you solve it?

    My cucumber command is as following:

    ‘cucumber -f junit –o ProjectDir/features’

    Thanks

    Reply

    • Hi,
      did you set the path were junit looks for the test output?

      that’ is how I have configured it:
      I have a separate jenkins build for my cucumber features, the execute shell command is:

      cd trunk
      cucumber -f junit --o ../cucumber --tags ~@ignore features.frank

      and the path for the post build action of junit is
      cucumber/*.xml

      Reply

  6. Yes, if running the app from XCode all is fine.
    The problem seems to be that the app tries to run as an iphone app when running headless. Unfortunately I have no idea how I can change this. Is there any parameter or environment variable which I can set to get the app started as an iPad app?

    hauner :

    Uh, I fear not.. I don’t use a SplitView…
    Is the “Device” stuff set to iPad?
    Or maybe it is this issue:
    http://groups.google.com/group/google-toolbox-for-mac/tree/browse_frm/thread/eb04668093d509/cf7ed468a1e37237?_done=%2Fgroup%2Fgoogle-toolbox-for-mac%2Fbrowse_frm%2Fthread%2Feb04668093d509%2Fcf7ed468a1e37237%3Ftvc%3D1%26&tvc=1

    Reply

    • When I looked for a way to tell the Simulator to run in iPad mode, I did not find anything.

      The current Frank version uses an additional tool to run the simulator that can switch the simulator to iPad mode. That’s not headless but it works.

      Personally I have dropped the headless stuff for now, because I had some issue with “button clicks” that did not work in headless mode…

      Reply

  7. hi,
    Unfortunately I cannot run the App headless.
    I’m getting this error message:
    “dyld: Library not loaded: /System/Library/Frameworks/MessageUI.framework/MessageUI”
    Can anyone please help?

    Reply

  8. Hi

    Do you know how to deal with text boxes with frank? I click on a UITabBarButton and the native edit box appears. Somehow, I cannot make it appear. What do I do?

    Reply

  9. iOS: Setting Up Acceptance Testing « Software Noise

  10. Thank for the fix of the plist xml file; that really helped a lot!

    Another quick question: You mention that you hardcode the APP string variable in a cucumber file. Which cucumber source file do you set the APP string variable?

    Reply

    • Thanks for noticing that it was broken :)

      I have created a before.rb below features/support that contains the “Before” code for cucumber. That’s the code with the broken and now fixed plist code (see article). At the beginning I define the APP & APPDIR variables. In my case the apps names is “Readme” so the frankified app is called “Readme Frankified”. The paths are relative to the main folder of the XCode project so they start with “build” subfolder.

      Reply

  11. What exactly does the following line do? I get a Ruby error ” syntax error, unexpected ‘<' (SyntaxError)" with the following line:

    ACCESIBILITY_CONTENT = <

    ApplicationAccessibilityEnabled

    PLIST

    Reply

    • Hi RBD,

      that code is incomplete. It is creating a plist xml file enabling the accessibility feature that’s required by frank.

      I didn’t notice the xml was dropped when I pasted that code into wordpress.

      I’ve fixed the code now, Thanks!

      Reply

  12. I’m just experimenting the frank framework with cucumber, and when I executed my script, I have the following error. I’m wondering am I missing some preference settings required on my macbook?

    /Users/testProject/trunk/src/cucumber/frank_helper.rb:144:in `block in make_http_request’
    /usr/local/lib/ruby/1.9.1/net/http.rb:627:in `start’
    /Users/testProject/trunk/src/cucumber/frank_helper.rb:143:in `make_http_request’
    /Users/testProject/trunk/src/cucumber/frank_helper.rb:125:in `post_to_uispec_server’
    /Users/testProject/trunk/src/cucumber/frank_helper.rb:60:in `frankly_map’

    Reply

  13. iOS: Setting Up a Test Environment is a Mess « Software Noise

Leave a comment