iPhone Unit Testing Explained – Part 1

 

Here’s the first part of a multi-part iPhone Unit Testing Series. (Updated 3/31/12 with Xcode 4 testing)

For the second part of iPhone Unit Testing Explain – Part II

How comfortable are you on a bike without a helmet? Writing code without tests is like riding a bike without a helmet. You might feel free and indestructible for now, but one day you’ll fall and it’s going to hurt.

I can’t start a new project without source control, it’s one of my requirements. Personally, I feel very uncomfortable and exposed without something to track my code changes. In reality, I hardly need to revert changes, but the knowledge that I can go back to something else enables me to make bigger and more confident changes.

Testing should also be a requirement and it doesn’t get enough attention in the classroom or on code projects. Part of the problem is that there is a lack of information and the high learning curve. Most tutorials provide the bare minimum and don’t provide real world examples. There are two major hurdles that you’ll need to overcome.

1. How do I use this testing framework?

Getting started with some new technology is always a daunting task. The only way you’re going to learn is if you teach yourself. Make sure to free yourself from distractions and get a cup of coffee so you can think clearly. When there are lots of unit testing frameworks and you need to evaluate what your requirements are going to be. For beginners it’s not always clear and you’re going to have to sample the available options.

2. How do I write testable code?

Most computer science courses don’t explain how to write testable code, they focus on the output matching. Students assume that if my code displays X and I’m trying to display X, then the code must be correct. Testable code can be hard to master, but it’s worth the effort. There’s two important aspects of unit testing that I’ve discovered.

a. A unit test, or a set of unit tests, validates that the function you just wrote works like you think it works. It takes microseconds for the computer to tell you if something works or not, and it takes you seconds or minutes to validate if it works. Your time is valuable, let computers do the grunt work of testing and you’ll have more time to write code.

b. Unit tests force design testing. High-level designs don’t translate into direct code without usability issues. As you write and test code you will find that a function should take X parameters instead of Y parameters, or that the function name doesn’t match the functionality it provides. Take the time to fix your design and you’ll have code that’s easier to use and better documented. The sooner you fix design issues, the more time you’ll have to work on features.

I still haven’t answered how to write testable code. Getting started is a matter of baby steps; take two steps at a time, and soon you’ll be walking.

a. Isolate the basic functions that have clear input and output. Write a test that provides a function with good input and bad input. Think about what additional functions you’ll need to prove that the function you’re testing works as expected. For example, in order to test a setter function, you’ll need a getter function. As you write more tests, refactor common tasks into helper functions, so that you can spend more time testing functionality instead of writing boilerplate test code.

b. Make sure you document each function as you test it, if you don’t write comments now, it’s never going to happen. Documentation is best when the details of the function are fresh in your mind and you can explain the edge cases.

Not everything is easy to test, but don’t be discouraged, since the tests you write will begin to validate that your code works as documented. You’re making progress and it’s going to get easier as you write more tests. Another benefit is that you’ll have higher quality code that you can rely on into the future.

iPhone Testing

Over the past weekend I set out to integrate a unit test framework into my current project. I ended up testing three different frameworks: OCUnit, Google Toolbox for Mac (GTM) iPhone Unit Testing, and GHUnit. I will provide a brief overview and pros and cons to each framework and my final conclusion.

OCUnit (SenTesting)

Pros:  Xcode integration (super easy), acceptance testing during builds, test debugging in Xcode 4 (Updated 3/31/12 for Xcode 4 improvements)
  • Xcode 4+ has built in support for unit testing using the OCUnit testing (SenTesting) framework. It’s easy to create a new test case, since there are Xcode test/target templates.
  • Xcode can display test failures like syntax errors with the code bubbles during the build/run stages, which can facilitate acceptance testing. These errors will link to the file and unit test that is failing, which is very helpful in terms of usability.

OCTest Unit Test Failure Code Bubble

Cons:  logic tests vs. application tests, device/simulator test limitations
  • There are far too many steps required to create the unit tests in Xcode 3.2.4. I’m not sure why more of the process isn’t automated, maybe in Xcode 4? To provide test coverage I needed to create 3 additional test targets each with specific settings
  • Writing a test for something basic (addition) is trivial using OCUnit, but testing code (memory management) that may crash isn’t. You need to follow a special set of instructions to get unit testing debugging work, otherwise you’ll be scratching your head. (Link 1) (Link 2) (Link 3)
  • Apple created their own categories of tests: logic tests and application tests, but the distinction isn’t clear. Logic tests are meant to test code that doesn’t use any UI code, while Application testsare designed for testing UI related code. This sounds fine, but when you try to create unit tests, the code you’re testing becomes very tightly coupled to the tests you can write.
    • For example, logic tests are not run in an application, so code that would provide a documents or bundle directory for saving/loading may not work without refactoring the code.
  • The biggest limitation is that logic tests only run using the iPhone Simulator and application tests only run on the device. Logic tests are not run when you build for the actual device, so you have to constantly switch between simulator/device to get acceptance test coverage. Running application tests only on the device is very slow, compared to running it in the iPhone Simulator.

OCTest Application Testing on the Device

Google Toolbox for Mac (GTM):  iPhone Unit Testing

Pros:  Xcode integration, acceptance testing during builds, easy setup
  • GTM iPhone unit testing provides code bubbles with test failures, like OCUnit, when building the target for the iPhone Simulator. The failure indicators are better than OCUnit, since it includes the error icon on the line of failure in GTM. However, the code bubbles do not appear when building for the device. The code bubbles can be clicked on to take you to the unit test that is failing. You can perform acceptance testing when building for the iPhone Simulator.

GTM iPhone Unit Testing Test Failure Code Bubbles

  • Compared to OCTest, GTM iPhone Unit Testing is a breeze to setup. You can write unit tests that target the UI and logic of your code within the same class.

Cons:  Documentation, output
  • GTM iPhone unit testing was straightforward to setup using the google code guide. The documentation is a little sparse, but it provides enough to get going. I think pictures would help explain the process and help break up all the text. I found the following visual tutorial semi-helpful, but there’s no definitive source.
  • When building a test and running the script during the build phase there is a lot of extraneous output that can be overwhelming. Some of the output is visible in the previous code bubble screenshot and the screenshot below. The output is not formatted as nicely as the OCTest output.

GTM iPhone Unit Testing Log Output from a Test Run

GHUnit

Pros:  Device/simulator testing, easy debugging, easy setup, iPhone Test Result GUI, command line interface
  • GHUnit has the easiest setup process of the three unit testing frameworks. It’s based on GTM and it is capable of running OCUnit, GTM, and GHUnit test cases. The documentation is pretty good and there is a nice visual setup guide.
  • GHUnit is easy to debug any test case without any additional setup. Place a breakpoint and run the test application in the simulator or on the device.
  • GHUnit provides a way to run tests on the command line, if you want to integrate with continuous integration tools. (Hudson)
  • The major selling point is that it provides a graphics user interface to display the results of each test, including test duration, if performance is important. The interface allows the user to switch between all tests and failed tests in a unit test application. Additionally, it drill into a test failure to display the unit test failure message and the stack trace. After using GHUnit, GTM Unit Testing leaves a lot to be desired beyond log output for visual feedback.
    • The GUI enables a pleasant workflow where you can set breakpoints and re-run tests to try and isolate bugs in the test code. The workflow replaces the need to perform acceptance testing during the build process, because it is so much more interactive.
    • On iPhone 4 it will resume on the All/Failed tests tab that you were last running/debugging. Note:  There is a minor issue described below in the cons section.

GHUnit Testing All Test Cases on the Device

GHUnit Test Case Failures

GHUnit Test Case Failure Message

Cons:  No acceptance testing, Documentation, minor GUI issues, Setup Time
  • There is no way to run tests as part of the build process using GHUnit. It’s a paradigm shift from the OCUnit and other similar unit testing frameworks that enable testing during the build process. However, you can use a continuous integration framework to run unit testing as you check in code to a central repository.

  • Documentation is good, but it could be better. The behaviors of the GUI are not clearly defined. Finding the GHUnitIOS.Framework was not clear in the documentation. I updated an article on github that addressed the documentation issue.
  • There are some minor GUI issues, if you use the GUNIT_AUTORUN environment variable. If you switch to the Failed tests tab on the GUI, it will only run the failed tests. Running the subset is good when you’re fixing an issue, but it can be confusing if you’re adding a new test. The new test won’t run until you switch back to the Alltests tab. Setting a breakpoint on the new test, will also do nothing, since it doesn’t run.
    • I’d like to see an additional button on the Failed tests tab that says “Run All Tests” and on the All tests tab a “Run Failed Tests” button. I think the additional buttons would make it clear that run doesn’t always do the same thing.

Only the Test Failures are Rerun on the Failed Tests Tab

Conclusion

Update: 3/31/12 With Xcode 4, creating and debugging unit tests in Xcode is so easy, it’s not worth the initial effort to integrate GHUnit, unless you need visual feedback on the device.

I would recommend using the built in Xcode 4 unit testing bundles and then add GHUnit after the fact, when you need visual feedback. GHUnit is compatible with the default testing available in Xcode 4.

The testing interface in GHUnit is the easiest to see test results and it doesn’t require as much log output digging. I like the workflow I have when using GHUnit.

  1. Switch to the GHUnit Test Target
  2. Write unit test for new function
  3. Test new function (Command-R) and see it fails (Optional)
  4. Stub and implement new function
  5. Test implementation (Command-R). Repeat 4 until the test passes
  6. Repeat for next function
  7. Switch back to the Application Test Target

Updated: (3/31/12) With the release of Xcode 4 and more integrated testing, using OCUnit/SenTest is a lot easier to get started. I will be providing a walk through on setting up Xcode 4 and working with bundle resources in the next part.

(Part II – working with unit tests in Xcode 4)

35 Comments

  1. Many thanks… This article has saved me a couple days doing the same evaluation. I owe you a beer. Please publish the second part !

  2. @Frank You’re welcome! Let me know if there’s anything in particular that you think I should focus on. Future testing topics that I’m interested in exploring are application level testing, interface testing, mock objects.

    Judging by my current workload/Thanksgiving, it’ll be up in the next two weeks. I’ve been integrating the ImageCache with my project http://www.artworkevolution.com/ Testing the ImageCache exposed a lot of things that I can cover. I’ve created most of the basic test cases, but the next step is to look into error testing. I need to validate that the ImageCache works when inputs aren’t expected.

  3. Great article! I’ve been trying to incorporate unit testing into one of my projects, and this helped me decide.

  4. @Damian You’re welcome, if you have any other suggestions for future unit testing topics, let me know. I plan on writing a followup in the next couple of weeks or sooner.

  5. Thanks for the article! Very helpful, especially the link to the step-by-step installation guide with screenshots.

    I noticed that GHUnit 4.27 is out, and there seems to be a change to the GHUnit header names — no longer GHUnitIOS.h; seems to be GHUnit.h now. I say seems to be, as I cannot find any release notes to confirm this. A bit disturbing … I went with 4.26 just to minimize chaos and having to fiddle with the instructions.

    I’m very interested in your next article!

  6. You’re welcome! Thanks for the note, let me know if there’s anything specific you’re interested in reading about.

    I plan on publishing two more articles about setting up GHUnit for Objective-C and Google Test for C++ code on the iPhone/Mac in the near future.

  7. Hi,

    As you seem to use unit testing regularly, maybe you help me.

    I have an application that is displaying and doing lot of animations on images.

    So the main part of my code is about graphical element layout and memory managment.

    Is there an easy way to test this kind of code ?

  8. @Cedric Do you have functions that can be tested?

    For instance, I have a function to create a position for an image on my iPhone screen and one to position it off screen. I use the function to slide an image from a point off screen to a random point on the screen. I can write tests that verify that if I create points using the function that it never violates the goal of the function. Make sure you define the goals of all your functions ahead of time, so that you can pick out the areas to test against.

    1. With memory management, you want to verify that when the allocation fails, you get an expected return value, such as nil, rather than garbage.
    2. You need to try to break down the functionality and separate it from the user interface if possible, so that you can test it in a stand-alone fashion. I create a separate class to do work, and the view controller’s create an object and call the functions that are available in the separate class. This allows me to separate logic from interface.

  9. Hi Paul,

    I found your tutorial helpful. When are you planning to release your next one?

    Bhavin

  10. @Crick At the moment I’ve been swamped with updates for Artwork Evolution. http://www.ArtworkEvolution.com/ Hopefully within the next 1-3 months as things settle down. I would like to blog more often.

  11. ok.Thanks

  12. Thanks so much for this article! Can I ask, have you tried UISpec? http://code.google.com/p/uispec/
    It provides a way to automate the actual iPhone UI and interact with views.

    I’m wondering if you think it would be a good idea to use a combination of GHUnit for unit tests, and UISpec for integration tests?

  13. You’re welcome! I’m sorry I haven’t finished my followup articles to the initial post.

    I have not used UISpec, I was looking at some of the things available in the Google Toolbox, but it really didn’t make sense for me to focus on UI testing, since I keep changing the interface design. The more tests you write the more code you have to maintain, so I’m trying to limit the amount at the moment.

    I think UI testing can be beneficial once you have a UI defined and don’t plan on changing from it, but I think focusing on unit testing can be more beneficial if your a single developer. You have to have some balance between testing and development.

  14. Thanks a lot for this useful article ;)

  15. You’re welcome!

  16. Nice work Paul! I have GHUnit installed and successfully running the example tests. However, I am now lost on where to go next. This is my first attempt at TDD so I am looking for guidance (tutorials) on how to implement TDD and GHUnit into my application. Do you have any suggestions?

  17. I like to work on components that can be separated from the GUI and tested in isolation. Look for anything that you know what should be input and output and then test those conditions.

    I don’t have any good resources on hand at the moment, and I need to get back to the series soon.

    1. Setup a GHUunit test file for your new source file
    2. Write a method stub in the source/header file
    3. Implement a test that makes sure it works. It should fail on your first test of GHUnit
    4. Implement the logic for the stubbed method.
    5. Re-run the test and verify that it fixed the test and there are no errors.
    6. Write the documentation for the method and make sure to address any memory management ownership or requirements of the function/method.
    7. Do these steps for each method you want to add to your class
    8. Wrap up the tested class in your GUI class and now you have a set of tests to run that make sure the logic is sound.

    -Paul

  18. Rutger van Dijk

    May 22, 2011 at 6:26 am

    Hello Paul,

    I think I’m mixing two concepts: code coverage and test coverage.

    I’ve been trying to setup my Xcode project to generate coverage for my unit tests with SenTesting, but I get the feeling that I only see ‘code coverage’ with CoverStory.

    Basically I want to see which percentage of my test code (unit tests) hits the actual ‘production’ code.

    Can you help me out setting up my Xcode project?

    With regards,

    Rutger

  19. CoverStory is probably your best bet for getting some idea of how much code you have covered. Beyond that I’m not sure how to see the percentage of production code that gets used. That would probably require more work than it’s worth, unless you have a team of people.

    I think an estimate of the code coverage with CoverStory should give you a big picture of how your testing coverage looks on all of the code. At the end of the day, testing isn’t everything.

  20. Rutger van Dijk

    May 22, 2011 at 2:25 pm

    Ok, but then I must have setup my Xcode project the wrong way:

    When running my app, in CoverStory I only see the code which was hit during the run. So when don’t touch a button or click an option, no code is hit and shows up red in CoverStory.

    There seems to be no relation with the unit tests I created. I did create a separate target for that and running that target shows which test passed or failed.

    The ‘production’ target has no links to the ‘unit test’ target.

    Is that the way to go ?

  21. Hello Paul,

    Thanks a lot for this useful article

    Can you help me for continuous integration (CI) of GHUnit with Hudson please

  22. I have not had time to integrated with Hudson yet. I don’t have any pointers except to take a look at the documentation. http://hudson-ci.org/ and Gabriel’s documentation on integrating ghunit http://gabriel.github.com/gh-unit/_hudson.html

  23. I haven’t used CoverStory, so I can’t really help with the details of the project setup. It was on my list of things to try, but I’ve been swamped with working on my Apps and my masters project.

  24. Hello Paul,
    Thanks a lot for this useful links.

  25. You’re welcome.

  26. Urgent!!!!!!!

    hello Paul;

    I try to configure Xcode4, GHUnit, and Jenkins
    by following the link below:

    http://www.youtube.com/watch?v=6ycxFcIPhQg

    but when I run the “make test” I get this result:

    ////
    ** BUILD FAILED **

    The following build commands failed:
    Tests:
    PhaseScriptExecution “Run Script” “/Users/chafikasait/My Firest TDD Project/My Firest TDD Project/build/My Firest TDD Project.build/Debug-iphonesimulator/Tests.build/Script-29E4AEBA13957D8A00D0234A.sh”
    (1 failure)

    make: *** [test] Error 1
    /////
    can you just tell me something about it??????

  27. @anji

    I haven’t used Jenkins before, so I don’t have any recommendations on that error. Try contacting the author of the youtube video you linked or restart from scratch and see if that helps.

  28. hello Paul;

    Thanks for your response, so I correct by following this link:
    http://cmdz.org/kbase/cli/xcode-select.html

    But I have another question??
    in which file the Buil Settings Target parameters are stored?? In pbxproj file????

    anji

  29. Hi;

    actually it is the right file

    but can I modify build settings and targets without using XCode UI

  30. @anji I only edit Xcode build settings with Xcode. It’s a lot easier to manage for my projects. Everything should be stored in the pbxproj file, but it’s not a safe way to go about editing Xcode projects, unless something is broken. Always have a backup in case you destroy something.

  31. @Paul

    What is the tool you use for formatting your code in Xcode4? I have try Uncrustify but it doesn’t works.

    Thanks

  32. @anji I just format the code by hand when typing it, or if I need to fix a large block I use Xcode4 to format the code. Goto Editor -> Structure -> Re-indent

    You can also select multiple lines of code and use the hotkeys Command + ] or Command + [ to change the indentation for a block of code.

  33. Really nice tutorial, when are you releasing second one? i have an application but don’t know how to write test cases for that, could you please take any method and write a test case for that so we can have learn to write test cases for our apps.

    Thanks

  34. I wish I had more time, it’s been a long time since I started that post and too many things have eaten my time. I’m working on a new App and then once that’s in a good place I will get back to my tutorial.

    Here’s what I’ve been working on: http://blog.artworkevolution.com/ios-apps/

Leave a Reply

Your email address will not be published.

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

© 2014 Paul Solt

Theme by Anders NorenUp ↑