iPhone Unit Testing Explained - Part II

Xcode 4 has drastically improved iPhone unit testing and Mac unit testing from my previous post, iPhone Unit Testing Explained - Part I Creating the unit testing target is easy and you can start writing test code in under 5 minutes.

The biggest hassle in testing is setting up the project correctly, and Xcode 4 makes it simple. If you read Part I, I pushed for GHUnit because of the GUI interface, but now Xcode's built in testing is enough to get you started. If you need a GUI, add GHUnit later, but start writing your tests today, since they're compatible with GHUnit when you decide to integrate with it.

Testing is important to start from the beginning or you will never have the motivation to write the tests unless your boss demand

Getting Started

To start writing unit tests you have two options, either create a new project with unit tests or add unit tests to an existing project.

New Project with Unit Tests

Create a new project and make sure the checkbox is enabled for unit tests.

 

New Xcode 4 Project with Unit Tests

Add Unit Tests Target to Existing Projects

Add a unit test target to your project by clicking on your Project (top left) -> Add Target (bottom middle) -> iOS -> Other -> Unit Testing Bundle.

 

Add Unit Tests to Existing Xcode Project

(Optional) Share the Target and Testing Scheme

If you add a unit test target, you'll most likely want to share your testing scheme with your team over version control (git, svn, etc.) Otherwise you're teammates will have to setup for themselves.

Goto Editor -> Manage Schemes -> Click Shared next to the Unit Test

 

Share Xcode Schemes with Teammates

Adding Resources

When you want to test code or import resources like images or data files you'll need to tell the testing target about the resources. There are two ways, you can do it when you first add the resource to the project or you can do it by editing the Build Phases for the unit test target.

 

Adding New Resources to the Unit Test Target

Click on File -> Add Files to "TestProject" -> Click on the checkbox on unit test target and Copy items

 

Copy Items and add to Unit Test Target in Xcode 4

Adding Existing Resources to the Unit Test Target

Click on your project "TestProject" -> Build Phases -> Expand one of three (Compile Sources, Link Binary With Libraries, or Copy Bundle Resources)

 

Build Phases for unit test in Xcode 4

Resource Paths are Different!

Many assumptions that your bundle is the main bundle will cause problems when testing. (Especially when adding tests to existing code) Look at the difference in bundles, the main bundle isn't what you'd expect in a unit test.

NSString *mainBundlePath = [[NSBundle mainBundle] resourcePath];

NSString *directBundlePath = [[NSBundle bundleForClass:[self class]] resourcePath];

NSLog(@"Main Bundle Path: %@", mainBundlePath);

NSLog(@"Direct Path: %@", directBundlePath);

NSString *mainBundleResourcePath = [[NSBundle mainBundlepathForResource:@"Frame.png" ofType:nil];

NSString *directBundleResourcePath = [[NSBundle bundleForClass:[self class]] pathForResource:@"Frame.png" ofType:nil];

NSLog(@"Main Bundle Path: %@", mainBundleResourcePath);

NSLog(@"Direct Path: %@", directBundleResourcePath);

Output:

Main Bundle Path: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.1.sdk/Developer/usr/bin

Direct Path: /Users/paulsolt/Library/Developer/Xcode/DerivedData/PhotoTable-dqueeqsjkjdthcbkrdzcvwifesvl/Build/Products/Debug-iphonesimulator/Unit Tests.octest

Main Bundle Path: (null)

Direct Path: /Users/paulsolt/Library/Developer/Xcode/DerivedData/PhotoTable-dqueeqsjkjdthcbkrdzcvwifesvl/Build/Products/Debug-iphonesimulator/Unit Tests.octest/Frame.png

Problem: My Unit test has a nil image, data file, etc. Why?

The unit test doesn't use the same bundle for resources that you're accustomed to when running an app. Therefore, the resource we're trying to load cannot be found.  You'll need to make changes to the code to support testing external resources (images, data files, etc). For example the following code

- (UIImage *)resizeFrameForImage:(NSString *)theImageName {
UIImage *image = [UIImage imageNamed:theImageName];
// ... do magical resize and return
return image;
}

Solution 1: Change the function parameters

Functions like these are semi-black boxes that aren't ideal for testing. You want access to all your inputs/outputs, especially if we're working with any kind of file resource. To fix it, just pass in the resource from the unit test, rather than having the function load it from a NSString object.

- (UIImage *)resizeFrameForImage:(UIImage *)theImage {
// ... do magical resize and return
return theImage;
}

Solution 2: Change the resource loading inside the function

If you need to load the resource in the function, you can alternatively change the way it is loaded. You need to stop using UIImage's imageNamed: method and switch to imageWithContentsOfFile: This way you can pass in the resource with the correct path, however it'll change logic elsewhere in your app.

- (UIImage *)resizeFrameForImage:(NSString *)theImagePath {
UIImage *image = [UIImage imageWithContentsOfFile:theImagePath];
// ... do magical resize and return
return image;
}

Solution 3: Load resources using the bundle for the current class

- (UIImage *)resizeFrameForImage:(NSString *)theImageName {
// Note: There are several ways you can write it, but make sure you include
//  the extension or you'll have trouble finding the resource
// 1. NSString *imagePath = [[NSBundle bundleForClass:[self class]] pathForResource:@"Image.png" ofType:nil];
// 2. NSString *imagePath = [[NSBundle bundleForClass:[self class]] pathForResource:@"Image" ofType:@"png"];
NSString *imagePath = [[NSBundle bundleForClass:[self class]] pathForResource:theImageName ofType:nil];
UIImage *image = [UIImage imageWithContentsOfFile:imagePath];
// ... do magical resize and return
return image;
}

My First Test

Code completion in Xcode will make writing tests easy. To test different things you'll use the following macros:

  • STAssertNotNil(Object, Description);
  • STAssertEquals(Value1, Value2, Description);
  • STAssertEqualObjects(Object1, Object2, Description);

Example: MyTest.m

@implementation TestImagePrintHelper
- (void)setUp
{
[super setUp];
// Set-up code here.
}
- (void)tearDown
{
// Tear-down code here.
[super tearDown];
}
- (void)testName {
NSString *testFirstName = @"Paul";
STAssertEqualObjects([person firstName], testFirstName, @"The name does not match");
}
@end

The STAssertEqualObjects macro will invoke the object's isEqual method, make sure you write one. See the section below. If you used the STAssertEquals it will test for primitive/pointer equality, not object equality.

Testable Code

Writing testable code requires that you add some additional functions that might feel optional before you decided to start testing.

1. Create a isEqual method for your class.

Most of the time you'll want to compare if the object is the correct object. This always requires that you write an isEqual method, otherwise you'll be using the NSObject isEqual test and it'll compare address pointers for the objects.

Example: Person.m

- (BOOL)isEqual:(id)other {
if (other == self) { // self equality, compare address pointers
return YES;
}
if (!other || ![other isKindOfClass:[self class]]) { 
// test not nil and is same type of class
return NO;
}
return [self isEqualToPerson: other]; // call our isEqual method for Person objects
}
- (BOOL)isEqualToPerson:(Person *) other {
BOOL value = NO;
if (self == other) { // test for self equality
value = YES;
 } elseif([[selffirstName] isEqualToString:[other firstName]] &&
[self age] == [other age]) {
// Add any other tests for instance variables (ivars) that need to be compared
value = YES;
}
return value;
}

2. Create a description method for your class.

This is what will output on the command line, rather than the objects memory address. It can be also called when you decide to print the value in a tests

Example: Person.m

- (NSString *)description {
return [NSStringstringWithFormat:@"Person Name: %@ Age: %d", [self firstName], [self age]];
}

Example: Test using the description method, and we'll see the first name and age printed like it was formatted in our description method.

STAssertEqualObjects([personfirstName], testFirstName, @"The name does not match %@", person);

Further reading in Apple's Unit Testing Guide is available. Now you have the basics for unit testing. The next part will provide an example project using resources and providing unit tests.

(Part III: Coming Soon)

iPhone Unit Testing Explained - Part 1

 

Here's the first part of a multi-part iPhone Unit Testing Series. (Updated 3/31/12 with Xcode 4 testing)

For the second part of iPhone Unit Testing Explain - Part II

How comfortable are you on a bike without a helmet? Writing code without tests is like riding a bike without a helmet. You might feel free and indestructible for now, but one day you'll fall and it's going to hurt.

I can't start a new project without source control, it's one of my requirements. Personally, I feel very uncomfortable and exposed without something to track my code changes. In reality, I hardly need to revert changes, but the knowledge that I can go back to something else enables me to make bigger and more confident changes.

Testing should also be a requirement and it doesn't get enough attention in the classroom or on code projects. Part of the problem is that there is a lack of information and the high learning curve. Most tutorials provide the bare minimum and don't provide real world examples. There are two major hurdles that you'll need to overcome.

1. How do I use this testing framework?

Getting started with some new technology is always a daunting task. The only way you're going to learn is if you teach yourself. Make sure to free yourself from distractions and get a cup of coffee so you can think clearly. When there are lots of unit testing frameworks and you need to evaluate what your requirements are going to be. For beginners it's not always clear and you're going to have to sample the available options.

[ad#Link Banner]

2. How do I write testable code?

Most computer science courses don't explain how to write testable code, they focus on the output matching. Students assume that if my code displays X and I'm trying to display X, then the code must be correct. Testable code can be hard to master, but it's worth the effort. There's two important aspects of unit testing that I've discovered.

a. A unit test, or a set of unit tests, validates that the function you just wrote works like you think it works. It takes microseconds for the computer to tell you if something works or not, and it takes you seconds or minutes to validate if it works. Your time is valuable, let computers do the grunt work of testing and you'll have more time to write code.

b. Unit tests force design testing. High-level designs don't translate into direct code without usability issues. As you write and test code you will find that a function should take X parameters instead of Y parameters, or that the function name doesn't match the functionality it provides. Take the time to fix your design and you'll have code that's easier to use and better documented. The sooner you fix design issues, the more time you'll have to work on features.

I still haven't answered how to write testable code. Getting started is a matter of baby steps; take two steps at a time, and soon you'll be walking.

a. Isolate the basic functions that have clear input and output. Write a test that provides a function with good input and bad input. Think about what additional functions you'll need to prove that the function you're testing works as expected. For example, in order to test a setter function, you'll need a getter function. As you write more tests, refactor common tasks into helper functions, so that you can spend more time testing functionality instead of writing boilerplate test code.

b. Make sure you document each function as you test it, if you don't write comments now, it's never going to happen. Documentation is best when the details of the function are fresh in your mind and you can explain the edge cases.

Not everything is easy to test, but don't be discouraged, since the tests you write will begin to validate that your code works as documented. You're making progress and it's going to get easier as you write more tests. Another benefit is that you'll have higher quality code that you can rely on into the future.

[ad#Large Box]

iPhone Testing

Over the past weekend I set out to integrate a unit test framework into my current project. I ended up testing three different frameworks: OCUnit, Google Toolbox for Mac (GTM) iPhone Unit Testing, and GHUnit. I will provide a brief overview and pros and cons to each framework and my final conclusion.

OCUnit (SenTesting)

Pros:  Xcode integration (super easy), acceptance testing during builds, test debugging in Xcode 4 (Updated 3/31/12 for Xcode 4 improvements)
  • Xcode 4+ has built in support for unit testing using the OCUnit testing (SenTesting) framework. It's easy to create a new test case, since there are Xcode test/target templates.
  • Xcode can display test failures like syntax errors with the code bubbles during the build/run stages, which can facilitate acceptance testing. These errors will link to the file and unit test that is failing, which is very helpful in terms of usability.

[caption id="attachment_854" align="aligncenter" width="555"] OCTest Unit Test Failure Code Bubble[/caption]

Cons:  logic tests vs. application tests, device/simulator test limitations
  • There are far too many steps required to create the unit tests in Xcode 3.2.4. I'm not sure why more of the process isn't automated, maybe in Xcode 4? To provide test coverage I needed to create 3 additional test targets each with specific settings
  • Writing a test for something basic (addition) is trivial using OCUnit, but testing code (memory management) that may crash isn't. You need to follow a special set of instructions to get unit testing debugging work, otherwise you'll be scratching your head. (Link 1) (Link 2) (Link 3)
  • Apple created their own categories of tests: logic tests and application tests, but the distinction isn't clear. Logic tests are meant to test code that doesn't use any UI code, while Application testsare designed for testing UI related code. This sounds fine, but when you try to create unit tests, the code you're testing becomes very tightly coupled to the tests you can write.
    • For example, logic tests are not run in an application, so code that would provide a documents or bundle directory for saving/loading may not work without refactoring the code.
  • The biggest limitation is that logic tests only run using the iPhone Simulator and application tests only run on the device. Logic tests are not run when you build for the actual device, so you have to constantly switch between simulator/device to get acceptance test coverage. Running application tests only on the device is very slow, compared to running it in the iPhone Simulator.

[caption id="attachment_856" align="aligncenter" width="528"] OCTest Application Testing on the Device[/caption]

Google Toolbox for Mac (GTM):  iPhone Unit Testing

Pros:  Xcode integration, acceptance testing during builds, easy setup
  • GTM iPhone unit testing provides code bubbles with test failures, like OCUnit, when building the target for the iPhone Simulator. The failure indicators are better than OCUnit, since it includes the error icon on the line of failure in GTM. However, the code bubbles do not appear when building for the device. The code bubbles can be clicked on to take you to the unit test that is failing. You can perform acceptance testing when building for the iPhone Simulator.

[caption id="attachment_850" align="aligncenter" width="540"] GTM iPhone Unit Testing Test Failure Code Bubbles[/caption]

  • Compared to OCTest, GTM iPhone Unit Testing is a breeze to setup. You can write unit tests that target the UI and logic of your code within the same class.

[ad#Link Banner]

Cons:  Documentation, output
  • GTM iPhone unit testing was straightforward to setup using the google code guide. The documentation is a little sparse, but it provides enough to get going. I think pictures would help explain the process and help break up all the text. I found the following visual tutorial semi-helpful, but there's no definitive source.
  • When building a test and running the script during the build phase there is a lot of extraneous output that can be overwhelming. Some of the output is visible in the previous code bubble screenshot and the screenshot below. The output is not formatted as nicely as the OCTest output.

[caption id="attachment_852" align="aligncenter" width="542"] GTM iPhone Unit Testing Log Output from a Test Run[/caption]

GHUnit

Pros:  Device/simulator testing, easy debugging, easy setup, iPhone Test Result GUI, command line interface
  • GHUnit has the easiest setup process of the three unit testing frameworks. It's based on GTM and it is capable of running OCUnit, GTM, and GHUnit test cases. The documentation is pretty good and there is a nice visual setup guide.
  • GHUnit is easy to debug any test case without any additional setup. Place a breakpoint and run the test application in the simulator or on the device.
  • GHUnit provides a way to run tests on the command line, if you want to integrate with continuous integration tools. (Hudson)
  • The major selling point is that it provides a graphics user interface to display the results of each test, including test duration, if performance is important. The interface allows the user to switch between all tests and failed tests in a unit test application. Additionally, it drill into a test failure to display the unit test failure message and the stack trace. After using GHUnit, GTM Unit Testing leaves a lot to be desired beyond log output for visual feedback.
    • The GUI enables a pleasant workflow where you can set breakpoints and re-run tests to try and isolate bugs in the test code. The workflow replaces the need to perform acceptance testing during the build process, because it is so much more interactive.
    • On iPhone 4 it will resume on the All/Failed tests tab that you were last running/debugging. Note:  There is a minor issue described below in the cons section.

[caption id="attachment_857" align="aligncenter" width="248"] GHUnit Testing All Test Cases on the Device[/caption]

[caption id="attachment_858" align="aligncenter" width="248"] GHUnit Test Case Failures[/caption]

[caption id="attachment_859" align="aligncenter" width="248"] GHUnit Test Case Failure Message[/caption]

Cons:  No acceptance testing, Documentation, minor GUI issues, Setup Time
  • There is no way to run tests as part of the build process using GHUnit. It's a paradigm shift from the OCUnit and other similar unit testing frameworks that enable testing during the build process. However, you can use a continuous integration framework to run unit testing as you check in code to a central repository.

[ad#Link Banner]

  • Documentation is good, but it could be better. The behaviors of the GUI are not clearly defined. Finding the GHUnitIOS.Framework was not clear in the documentation. I updated an article on github that addressed the documentation issue.
  • There are some minor GUI issues, if you use the GUNIT_AUTORUN environment variable. If you switch to the Failed tests tab on the GUI, it will only run the failed tests. Running the subset is good when you're fixing an issue, but it can be confusing if you're adding a new test. The new test won't run until you switch back to the Alltests tab. Setting a breakpoint on the new test, will also do nothing, since it doesn't run.
    • I'd like to see an additional button on the Failed tests tab that says "Run All Tests" and on the All tests tab a "Run Failed Tests" button. I think the additional buttons would make it clear that run doesn't always do the same thing.

[caption id="attachment_861" align="aligncenter" width="248"] Only the Test Failures are Rerun on the Failed Tests Tab[/caption]

Conclusion

Update: 3/31/12 With Xcode 4, creating and debugging unit tests in Xcode is so easy, it's not worth the initial effort to integrate GHUnit, unless you need visual feedback on the device.

I would recommend using the built in Xcode 4 unit testing bundles and then add GHUnit after the fact, when you need visual feedback. GHUnit is compatible with the default testing available in Xcode 4.

The testing interface in GHUnit is the easiest to see test results and it doesn't require as much log output digging. I like the workflow I have when using GHUnit.

  1. Switch to the GHUnit Test Target
  2. Write unit test for new function
  3. Test new function (Command-R) and see it fails (Optional)
  4. Stub and implement new function
  5. Test implementation (Command-R). Repeat 4 until the test passes
  6. Repeat for next function
  7. Switch back to the Application Test Target

Updated: (3/31/12) With the release of Xcode 4 and more integrated testing, using OCUnit/SenTest is a lot easier to get started. I will be providing a walk through on setting up Xcode 4 and working with bundle resources in the next part.

(Part II - working with unit tests in Xcode 4)

[ad#Large Box]