Here's the first part of a multi-part iPhone Unit Testing Series. (Updated 3/31/12 with Xcode 4 testing)
For the second part of iPhone Unit Testing Explain - Part II
How comfortable are you on a bike without a helmet? Writing code without tests is like riding a bike without a helmet. You might feel free and indestructible for now, but one day you'll fall and it's going to hurt.
I can't start a new project without source control, it's one of my requirements. Personally, I feel very uncomfortable and exposed without something to track my code changes. In reality, I hardly need to revert changes, but the knowledge that I can go back to something else enables me to make bigger and more confident changes.
Testing should also be a requirement and it doesn't get enough attention in the classroom or on code projects. Part of the problem is that there is a lack of information and the high learning curve. Most tutorials provide the bare minimum and don't provide real world examples. There are two major hurdles that you'll need to overcome.
1. How do I use this testing framework?
Getting started with some new technology is always a daunting task. The only way you're going to learn is if you teach yourself. Make sure to free yourself from distractions and get a cup of coffee so you can think clearly. When there are lots of unit testing frameworks and you need to evaluate what your requirements are going to be. For beginners it's not always clear and you're going to have to sample the available options.
[ad#Link Banner]
2. How do I write testable code?
Most computer science courses don't explain how to write testable code, they focus on the output matching. Students assume that if my code displays X and I'm trying to display X, then the code must be correct. Testable code can be hard to master, but it's worth the effort. There's two important aspects of unit testing that I've discovered.
a. A unit test, or a set of unit tests, validates that the function you just wrote works like you think it works. It takes microseconds for the computer to tell you if something works or not, and it takes you seconds or minutes to validate if it works. Your time is valuable, let computers do the grunt work of testing and you'll have more time to write code.
b. Unit tests force design testing. High-level designs don't translate into direct code without usability issues. As you write and test code you will find that a function should take X parameters instead of Y parameters, or that the function name doesn't match the functionality it provides. Take the time to fix your design and you'll have code that's easier to use and better documented. The sooner you fix design issues, the more time you'll have to work on features.
I still haven't answered how to write testable code. Getting started is a matter of baby steps; take two steps at a time, and soon you'll be walking.
a. Isolate the basic functions that have clear input and output. Write a test that provides a function with good input and bad input. Think about what additional functions you'll need to prove that the function you're testing works as expected. For example, in order to test a setter function, you'll need a getter function. As you write more tests, refactor common tasks into helper functions, so that you can spend more time testing functionality instead of writing boilerplate test code.
b. Make sure you document each function as you test it, if you don't write comments now, it's never going to happen. Documentation is best when the details of the function are fresh in your mind and you can explain the edge cases.
Not everything is easy to test, but don't be discouraged, since the tests you write will begin to validate that your code works as documented. You're making progress and it's going to get easier as you write more tests. Another benefit is that you'll have higher quality code that you can rely on into the future.
[ad#Large Box]
iPhone Testing
Over the past weekend I set out to integrate a unit test framework into my current project. I ended up testing three different frameworks: OCUnit, Google Toolbox for Mac (GTM) iPhone Unit Testing, and GHUnit. I will provide a brief overview and pros and cons to each framework and my final conclusion.
OCUnit (SenTesting)
Pros: Xcode integration (super easy), acceptance testing during builds, test debugging in Xcode 4 (Updated 3/31/12 for Xcode 4 improvements)
- Xcode 4+ has built in support for unit testing using the OCUnit testing (SenTesting) framework. It's easy to create a new test case, since there are Xcode test/target templates.
- Xcode can display test failures like syntax errors with the code bubbles during the build/run stages, which can facilitate acceptance testing. These errors will link to the file and unit test that is failing, which is very helpful in terms of usability.
[caption id="attachment_854" align="aligncenter" width="555"]
OCTest Unit Test Failure Code Bubble[/caption]
Cons: logic tests vs. application tests, device/simulator test limitations
There are far too many steps required to create the unit tests in Xcode 3.2.4. I'm not sure why more of the process isn't automated, maybe in Xcode 4? To provide test coverage I needed to create 3 additional test targets each with specific settings. Writing a test for something basic (addition) is trivial using OCUnit, but testing code (memory management) that may crash isn't. You need to follow a special set of instructions to get unit testing debugging work, otherwise you'll be scratching your head. (Link 1) (Link 2) (Link 3)- Apple created their own categories of tests: logic tests and application tests, but the distinction isn't clear. Logic tests are meant to test code that doesn't use any UI code, while Application testsare designed for testing UI related code. This sounds fine, but when you try to create unit tests, the code you're testing becomes very tightly coupled to the tests you can write.
- For example, logic tests are not run in an application, so code that would provide a documents or bundle directory for saving/loading may not work without refactoring the code.
- The biggest limitation is that logic tests only run using the iPhone Simulator and application tests only run on the device. Logic tests are not run when you build for the actual device, so you have to constantly switch between simulator/device to get acceptance test coverage. Running application tests only on the device is very slow, compared to running it in the iPhone Simulator.
[caption id="attachment_856" align="aligncenter" width="528"]
OCTest Application Testing on the Device[/caption]
Pros: Xcode integration, acceptance testing during builds, easy setup
- GTM iPhone unit testing provides code bubbles with test failures, like OCUnit, when building the target for the iPhone Simulator. The failure indicators are better than OCUnit, since it includes the error icon on the line of failure in GTM. However, the code bubbles do not appear when building for the device. The code bubbles can be clicked on to take you to the unit test that is failing. You can perform acceptance testing when building for the iPhone Simulator.
[caption id="attachment_850" align="aligncenter" width="540"]
GTM iPhone Unit Testing Test Failure Code Bubbles[/caption]
- Compared to OCTest, GTM iPhone Unit Testing is a breeze to setup. You can write unit tests that target the UI and logic of your code within the same class.
[ad#Link Banner]
Cons: Documentation, output
- GTM iPhone unit testing was straightforward to setup using the google code guide. The documentation is a little sparse, but it provides enough to get going. I think pictures would help explain the process and help break up all the text. I found the following visual tutorial semi-helpful, but there's no definitive source.
- When building a test and running the script during the build phase there is a lot of extraneous output that can be overwhelming. Some of the output is visible in the previous code bubble screenshot and the screenshot below. The output is not formatted as nicely as the OCTest output.
[caption id="attachment_852" align="aligncenter" width="542"]
GTM iPhone Unit Testing Log Output from a Test Run[/caption]
Pros: Device/simulator testing, easy debugging, easy setup, iPhone Test Result GUI, command line interface
- GHUnit has the easiest setup process of the three unit testing frameworks. It's based on GTM and it is capable of running OCUnit, GTM, and GHUnit test cases. The documentation is pretty good and there is a nice visual setup guide.
- GHUnit is easy to debug any test case without any additional setup. Place a breakpoint and run the test application in the simulator or on the device.
- GHUnit provides a way to run tests on the command line, if you want to integrate with continuous integration tools. (Hudson)
- The major selling point is that it provides a graphics user interface to display the results of each test, including test duration, if performance is important. The interface allows the user to switch between all tests and failed tests in a unit test application. Additionally, it drill into a test failure to display the unit test failure message and the stack trace. After using GHUnit, GTM Unit Testing leaves a lot to be desired beyond log output for visual feedback.
- The GUI enables a pleasant workflow where you can set breakpoints and re-run tests to try and isolate bugs in the test code. The workflow replaces the need to perform acceptance testing during the build process, because it is so much more interactive.
- On iPhone 4 it will resume on the All/Failed tests tab that you were last running/debugging. Note: There is a minor issue described below in the cons section.
[caption id="attachment_857" align="aligncenter" width="248"]
GHUnit Testing All Test Cases on the Device[/caption]
[caption id="attachment_858" align="aligncenter" width="248"]
GHUnit Test Case Failures[/caption]
[caption id="attachment_859" align="aligncenter" width="248"]
GHUnit Test Case Failure Message[/caption]
Cons: No acceptance testing, Documentation, minor GUI issues, Setup Time
- There is no way to run tests as part of the build process using GHUnit. It's a paradigm shift from the OCUnit and other similar unit testing frameworks that enable testing during the build process. However, you can use a continuous integration framework to run unit testing as you check in code to a central repository.
[ad#Link Banner]
- Documentation is good, but it could be better. The behaviors of the GUI are not clearly defined. Finding the GHUnitIOS.Framework was not clear in the documentation. I updated an article on github that addressed the documentation issue.
- There are some minor GUI issues, if you use the GUNIT_AUTORUN environment variable. If you switch to the Failed tests tab on the GUI, it will only run the failed tests. Running the subset is good when you're fixing an issue, but it can be confusing if you're adding a new test. The new test won't run until you switch back to the Alltests tab. Setting a breakpoint on the new test, will also do nothing, since it doesn't run.
- I'd like to see an additional button on the Failed tests tab that says "Run All Tests" and on the All tests tab a "Run Failed Tests" button. I think the additional buttons would make it clear that run doesn't always do the same thing.
[caption id="attachment_861" align="aligncenter" width="248"]
Only the Test Failures are Rerun on the Failed Tests Tab[/caption]
Conclusion
Update: 3/31/12 With Xcode 4, creating and debugging unit tests in Xcode is so easy, it's not worth the initial effort to integrate GHUnit, unless you need visual feedback on the device.
I would recommend using the built in Xcode 4 unit testing bundles and then add GHUnit after the fact, when you need visual feedback. GHUnit is compatible with the default testing available in Xcode 4.
The testing interface in GHUnit is the easiest to see test results and it doesn't require as much log output digging. I like the workflow I have when using GHUnit.
- Switch to the GHUnit Test Target
- Write unit test for new function
- Test new function (Command-R) and see it fails (Optional)
- Stub and implement new function
- Test implementation (Command-R). Repeat 4 until the test passes
- Repeat for next function
- Switch back to the Application Test Target
Updated: (3/31/12) With the release of Xcode 4 and more integrated testing, using OCUnit/SenTest is a lot easier to get started. I will be providing a walk through on setting up Xcode 4 and working with bundle resources in the next part.
(Part II - working with unit tests in Xcode 4)
[ad#Large Box]
