I needed to get ImageMagick installed this morning to play with an iOS testing tool called Zucchini that I’ll write about separately. My new MacBook Air is running OS X Lion (never ran Snow Leopard), and Xcode 4.3. I’ve found installing native tools like this has been pretty painful on Lion from the default compiler changes (like installing rvm), so I wasn’t surprised ImageMagick didn’t install correctly first time. First I tried to brew install imagemagick. While it succeeded in installing the library, right at the end there was the following error:
ln: ImageMagick: Permission denied
Error: The linking step did not complete successfully
The formula built, but is not symlinked into /usr/local
You can try again using `brew link imagemagick'
There were similar errors talking about little-cms, libtiff and jpeg as well. I thought, I’ll ignore that, maybe it’ll work anyway and went ahead and tried to use imagemagick. When I tried to use the tool though, I saw errors like dyld: Library not loaded: /usr/local/lib/liblcms.1.0.19.dylib.
A bit of googling took me to a thread on the homebrew issues on github. I didn’t actually need to uninstall/reinstall like some of the comments there suggest, I just needed to link the libraries giving the errors into /usr/local like this:
sudo brew link little-cms
sudo brew link imagemagick
sudo brew link libtiff
sudo brew link jpeg
Maybe it’s a better solution to allow myself write access to /usr/local, and then the brew install would have linked automatically. I’ve just ended up with such a mess of libraries in /usr/local on old machines, I’m trying to keep my new little machine as clean as possible.
A common question I get asked is: how to I run my OCUnit tests from the command line? I’ve answered this a number of times, so here is a quick brain dump so I can just point to this post each time. I say OCUnit, but since Kiwi is just a wrapper above OCUnit, these instructions should also work for Kiwi tests.
Xcodebuild
OCUnit tests are run from a shell script during the build for that target, which is usually called something like MyAppTests. If you look at the Build Phases for that target, you’ll see a little RunScript phase that actually runs the tests:
# Run the unit tests in this test bundle.
"${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests"
We can then use the xcodebuild command line utility to run this same build from the command line. On the command line (or from a Makefile if you are so inclined), try building and running the tests:
This may or may not work, depending on whether or not you have Logic or Application tests. What?
Logic & Application Unit Tests
I’m not sure why Apple bothers to differentiate these tests. The basic difference is that Application tests need to be run inside of a UIKit environment, while Logic tests don’t. I’d prefer if the build system just detected whether it needed UIKit and decided for you. There’s a longer explanation in Apple’s unit testing guide. I often want to write unit tests for my UIViewController classes and any other UI logic, so I always tick the Application checkbox when adding unit tests to an Xcode project. If you’re making a static library with no UI then you might want to select Logic.
Anyway, if your project has Logic tests, the above xcodebuild command should run your unit tests fine and you will see the test output in the Terminal window. However, if they are Application tests, the command silently succeeds without actually running your tests. If you look near the bottom of the output, you’ll see this in the output:
/Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests:95:
warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set).
So the tests didn’t even run, but the command succeeds, the build stays green and everyone’s happy… but the tests aren’t running.
Running From The Command Line
I usually point people to a post from the guys at Long Weekend for how to get these Application tests running from the command line. Here’s quick summary:
Open up /Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests in your favourite editor and check out the code around line 95. You’ll see the culprit line:
Warning ${LINENO} "Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)."
THe TEST_HOST is only set when running the tests through Xcode, not from the command line. If the host isn’t set, the tests don’t run. We want them to run in this case. We could replace that line with the following:
This command will actually boot up your app in the iOS simulator and run the tests in that UI environment.
I don’t really feel comfortable hacking scripts in my Developer directory, so I’d rather copy this script into my project and run it from there. Also, editing the actual installed tools like this means that you have to do it on each and every machine running the tests. Even once you get it running on your machine, you’ll need to log into your build machine and make the same hack there. Try making a copy of the same shell script Apple uses into your project directory:
cd your-project
mkdir scripts
cp /Developer/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests scripts/
Now edit that local copy of your shell script to replace line 95 to actually run the tests in the simulator environment. The MyAppTests target is still pointing to the system developer script. In Xcode, open up the Build Phases for your test target and change the Run Script to point to your local script:
# Run the unit tests in this test bundle.
"${SRCROOT}/../scripts/RunPlatformUnitTests"
Run the tests in the IDE to make sure you didn’t screw anything up. Now quit the simulator, and run them from the command line with xcodebuild again. You should see the test output printed and a few statements at the bottom saying that the tests succeeded. Your tests do pass, right?
/Developer/Tools/RunPlatformUnitTests.include:334: note: Passed tests for architecture 'i386' (GC OFF)
/Developer/Tools/RunPlatformUnitTests.include:345: note: Completed tests for architectures 'i386'
The reason I said to close the simulator before running the tests from the command line is that the tests don’t seem to like running when the iOS Simulator is already doing something. Try it. Open the simulator, and then run the tests. If your system is anything like mine, you get an error like this:
** BUILD FAILED **
The following build commands failed:
PhaseScriptExecution "Run Script" path-to-the-script-in-derived-data
If I’m running them locally, I see this, quit the simulator and try again. That’s a bit of a pain, and it certainly wont work on the CI server. To solve that, in my Makefile, I just use a little Applescript to close the simulator before running the tests:
Now I can use the command make test to run the tests from the command line both on my machine and the CI server without having to change the development environment.
I wrote a post a little while ago about transferring my Ruby dev environments to use rvm to organise and separate Ruby envinroments and assosiated gemsets. I just got a new MacBook Air which I’m setting up at the moment, so I’m running through the set up process again. Here are my notes.
There are a few rvm haters out there who find it to be over-engineered and trying to do too much. If that’s you, there is another, more lightweight tool called rbenv that you can use and if you really like separate gemsets, someone has written rbenv-gemset to go with it. I have a handful of projects already set up with rvm, so I’m going to keep using it for now.
Installing rvm itself is pretty easy. The rvm homepage gives a quick install command:
That works fine. Now it’s time to install some rubies - at the moment I have projects using 1.8.7 and 1.9.2, and I’d like to use 1.9.3 going forward. I used the following command and got an error:
rvm install 1.8.7
...
ERROR: There has been an error while running configure. Halting the installation.
It turns out this is a common issue in OS X Lion because gcc is just a sym link to LLVM, and the whole installation gets a bit confused (that’s the technical term for it). I read on Matt Polito’s blog that you can simply point the CC environment variable straight to GCC 4.2 like this:
CC=/usr/bin/gcc-4.2 rvm install <ruby version>
That didn’t work for me. I’m using a brand new machine with Xcode 4.2 installed. It turns out this solution worked for Xcode 4.1 but 4.2 does not install the right gcc to use. One solution I saw was to roll back to Xcode 4.1 and then install Xcode 4.2 again. That seemed like it would take a while, so I used the standalone OS X GCC installer from https://github.com/kennethreitz/osx-gcc-installer. After that, installing different Rubies worked fine. I’m hoping that installing gcc like this doesn’t mess up my Xcode 4.2. So far so good, the few Xcode projects I’ve tried still compile and run fine.
Edit: I think next time I will try the approach mentioned by Tony Arnold in the comments below, using the --with-gcc=clang option to rvm, something like this:
I wrote a recent post on enabling accessibility for iOS applications, which ended with a snippet of code for automatically enabling accessibility on the iOS simulator. This is essential if you want to have your tests running on a continuous integration server, since the accessibility inspector is off by default.
I hadn’t yet found a solution for enabling accessibility on the device. We need accessibility turned on so that we can access the UIAccessibility values that we use in automated functional tests in one of the common automated testing tools. My solution up until now has been to turn VoiceOver on, which is a real pain. Since I sometimes use screenshot-based regression tests, VoiceOver breaks the tests since it visually highlights the selected item.
Cedric Luthi commented on my previous post that it may be possible to modify his code to also enable accessibility on the device. I wasn’t sure how that would work with the app sandbox, and whether it was an application or system preference setting. Last night I gave it a try, and amazingly it worked.
I wrote a simple application based on the master-detail iPhone template in Xcode 4, and wrote a quick KIF test that checked a label on the master screen, pushed through to the detail screen and checked a label there also. It’s a gimmick test, but enough that it will fail if accessibility is not enabled because none of the labels will be available to the test.
After following the standard KIF set up instructions, I write the following test:
-(void)initializeScenarios;{KIFTestScenario*loadScreen=[KIFTestScenarioscenarioWithDescription:@"Test app loads up on correct screen"];[loadScreenaddStep:[KIFTestStepstepToWaitForViewWithAccessibilityLabel:@"Master"]];[selfaddScenario:loadScreen];KIFTestScenario*navigateToDetails=[KIFTestScenarioscenarioWithDescription:@"Test can navigate"];[navigateToDetailsaddStep:[KIFTestStepstepToTapViewWithAccessibilityLabel:@"Detail"]];[navigateToDetailsaddStep:[KIFTestStepstepToWaitForViewWithAccessibilityLabel:@"Detail view content goes here"]];[selfaddScenario:navigateToDetails];}
That works fine on the simulator, whether you have explicitly enabled accessibility or not now that the maintainers of KIF have merged in my pull request. That code only runs for the simulator, and does nothing if IPHONE_SIMULATOR_ROOT is not available in the [[NSProcessInfo processInfo] environment]. The KIF test will only run on the device (at least for me) if I turned VoiceOver on, which is a pain and not possible to automate.
I modified the code so that on the device, it does not try to prepend the simulator root when running on the device, and points to @"/System/Library/PrivateFrameworks/AppSupport.framework/AppSupport" directly. Amazingly, the KIF test then worked and all accessibility values were available to the tests. The updated code to enable accessibility programmatically on the device or the simulator looks like this:
I’m a little bit unsure of the security of this, and whether you actually want your tests messing with your phones system settings on the device - but I usually have separate test devices to my personal phone, and the KIF code is never part of the actual production app that you submit to Apple, so I’m comfortable with it for now. I’m hoping this combined with my recent work with fruitstrap could get us all the way to functional tests running on a physical device in CI.
You can find this code in my fork of KIF, and I’m hoping after a bit of testing it can be merged into the main KIF repo, so I sent them this pull request.
One of the reasons I run most of my tests in the simulator is that it is easy to install and run applications automatically from the command line with tools like ios-sim. Being able to run your tests from the command line make s it simple to set up continuous integration. I’ve used an Applescript to drive iTunes back in Xcode 3 days but it just felt wrong.
The most recent functional testing tool I’ve been playing with is Apple’s own UIAutomation, which runs within the Instruments app. I’ll write more about that separately, but let’s just say I don’t think it’s an ideal testing tool, especially if TDD and CI are important to you. Some details are in my comparison of a few functional testing tools. The instruments command line tool does not seem to install the app on the device before running tests, which means you still need to use Xcode to manually install the app. Enter fruitstrap.
Fruitstrap
Fruitstrap is a command line tool that uses the private MobileDevice API to install an iOS application on a physical device over USB. It’s pretty easy to get set up.
git clone git://github.com/ghughes/fruitstrap.git
cd fruitstrap
make fruitstrap
Fruitstrap comes with a demo applicaiton, which you can compile and install on a device using make install. I actually had a few issues getting the demo app to work, but I did get it working for actual sample applications. You now have the fruitstrap command compiled and ready to go - if you want to access the command from anywhere you probably want to add it to you path, or sym link it to /usr/local/bin or however you like to tinker with your machine.
Building from the command line
I made a little sample application to play around with fruitstrap and scripting on my github, in a project called fruitstrap-demo. It’s just a simple Single View Application with a couple of labels so you know it’s the right app. To get the repository:
Just to make sure everything is working, open up the project in Xcode and build and run to your device. If that doesn’t work, fruitstrap isn’t going to help you much. The xcodebuild command allows us to build our iOS apps from the command line fairly easily. The command I used was:
Remember to use the iphoneos so that it builds your app for the device. Note, I originally tried this with the old target settings for xcodebuild, but it turned out I needed to use schemes for reasons explained below. The app will be built to build/Debug-iphoneos/fruitstrap-demo.app.
Try out fruitstrap
Now we have an application build on the command line, let’s make sure fruitstrap works for our app. Make sure to remove your sample app from the device beforehand so you know it’s working. Then use the xcodebuild command above to compile the app so it’s ready to go, and make sure you know the full path to fruitstrap, or you’ve put fruitstrap on your path.
You should see a bunch of output and progress information finishing with the magic [100%] Installed package build/Debug-iphoneos/fruitstrap-demo.app, and in a few moments, the app appears in Springboard on your phone. That’s pretty cool - I’ve been trying to find a solution for installing on the device like that for a long time!
If you have more than one device plugged in (which is usually the case on a mobile continuous integration server), you’ll need to also specify the device id.
Scripting the fruitstrap installation
My next goal is to write a little shell script that we can integrate into a build phase in Xcode, so that we don’t have to hard code the path build/Debug-iphoneos into out shell script. I immediately reached for the Build Phases tab of the fruitstrap-demp target to optionally run the fruitstrap install code. However, these shell scripts seem to get called before the code signing is run, in which case, installing to the device will fail.
I found out from this stackoverflow thread that you can run pre and post scripts for a scheme. This allows us to hook up a shell script to run fruitstrap after the code signing.
There is only one scheme in the sample project, so select Edit Scheme…, and select the Build action from the list on the left hand side. There are no actions at the moment, so press the + button and add a Run Script Action. Since we need to know where our target has been compiled, make sure that the provide build settings from option is set to fruitstrap-demo, as show in the following screenshot.
The actual script code is shown below. It only runs if the FRUITSTRAP_CLI environment variable is set, since most of the time we don’t want Xcode to be using this third party tool to install on the device. We only need it to run when running from the command line as part of our continuous integration build. It seems the scheme scripts do not get run in the same working directory as you run xcodebuild, so our script makes sure to change to SRCROOT before running fruitstrap.
# Do nothing unless we are running from the command line
if [ "$FRUITSTRAP_CLI" = "" ]; then
exit 0
fi
echo "******************"
echo "Installing app to device using fruitstrap..."
echo "Workspace location: $SRCROOT"
echo "Install location: $TARGET_BUILD_DIR/$FULL_PRODUCT_NAME"
echo "******************"
cd $SRCROOT
fruitstrap $TARGET_BUILD_DIR/$FULL_PRODUCT_NAME
echo "******************"
Check that when you run the xcodebuild example above that it does not run fruitstrap, since we don’t want it to in that normal operation. Now, try building the scheme with the environment variable set and check that it does in fact build and install to the device.
Done. One command to build and install the app on the device.
More about fruitstrap
There is more information about fruitstrap on its github page.
One extra feature fruitstrap has is to be able to launch the application and attach a debugger, by using the -d option. I’ve had mixed success with this feature, it doesn’t always work for me. I’m not sure how much use it is to me anyway if the point of this is running in CI.
Now we can build the application and install it on the device from the command line. From here, the next step is to hook it up to the instruments command line interface. Massive thanks to Richie for letting me know about fruitstrap.
Ideally, I would like to be able to boot the app on the device without being hooked into the debugger. I’m not sure if this is possible, I certainly haven’t got it working with fruitstrap yet - and the hairy C code isn’t making me want to jump in and try just yet. What we have now is enough to get UIAutomation up and running, since instruments will boot the app when the tests start. However, I’d prefer to use Frank or KIF in which case I need to find a way to boot onto the device.
I always include unit tests by default in any project that is more than a demo, but up until recently I have always used GHUnit. Mark Makdad wrote a piece earlier in the year comparing GHUnit and OCUnit. That post combined with Kiwi’s Rspec-style wrappers around OCUnit has made me consider using Apple’s built in unit testing again.
I worked through the post from Two Bit Labs to add a unit test target to an existing project, which all worked fine. Xcode automatically added a new scheme for my new unit test target that only had the Test build action hooked up. This isn’t exactly what I wanted. Ideally, I want to run my unit tests without changing the current scheme, and just press Cmd + U or select Product > Test from the menu.
Adding the test action to an existing scheme
When I select my target, called Development, that option is greyed out. I have a number of targets set up for this project, so perhaps for simple projects Xcode sorts it out for you… but here is what I had to do in order to get my unit tests hooked up to my main Development target’s scheme.
Select your main target’s scheme in the little scheme selector in the Xcode toolbar, and choose Edit Scheme… from the drop down menu. When you select the Test build action on the left, you’ll notice no tests appear in the list. Tap the + button at the bottom edge of the table, which should show a list of available test bundles (I had a whole lot to choose from because my project has the unfortunate privilege of still including Three20). Choose the unit test bundle you just created you just created and press Add.
You should see the test bundle appear in the table, as shown in the image above. Tap on the Build action for this scheme, and you will notice that there are now two targets lists: our application and our unit tests. The tests are only linked up to the Test build action by default, which should look similar to the image below.
Running your unit tests
Now that your unit tests are hooked up to your test build action, select your main target’s scheme. The menu item Product > Test should now be active. Select Test, and Xcode will build your application and run your tests. When creating the test bundle, Apple’s default test case includes a failure, so if the tests run, you will see that failure.
Just to double check you’ve got all your target dependencies set up correctly, try a full clean (hold Option while selecting Product > Clean) and then press Cmd + U. This should trigger a full build and run the unit tests and show your test failure again.
If you see an error like ld: file not found:<path to app>/Debug-iphonesimulator/Development.app/Development, it’s likely your test target does not automatically build your main application under test, as explained in the original article: select your project in the navigator, select your unit test target, open the Build Phases tab and add your app target as a dependency. Ending up with something like the image shown.
Now… do I use this approach and start convert all my unit tests slowly from GHUnit to OCUnit? I think I might leave existing projects on GHUnit, and just use OCUnit for new tests.
You only have to look at github to see the explosion of new open source iOS frameworks. Incorporating third party libraries into your app is still a pain. I spent today fighting with static libraries and frameworks for the DCIntrospect library, and I think I won, so that’s got to be work writing about.
Until recently, my default use case was to actually drag in the source files from a third party library and compile it into my application. For libraries that I have forked and plan to modify or extend, this would still be my preferred option. Some third party libraries are stable and reliable, and I do not wish to change the source code at all. For these tools, I would prefer to just drag in a static library or framework and get on with my own app.
I’ve tried my hand at static libraries before, and it definitely wasn’t plain sailing. A twitter conversation between two local iOS devs, Pat and Oliver prompted me to have another go at getting these libraries working. Pat’s own tool, DCIntrospect is a perfect example. It’s a great tool for introspecting and testing strange UI layout behaviour in your app, and it’s a library that just works (famous last words, I bet you find a bug in it now)… but credit to Pat, I’ve used it on a few apps, and never touched the source code. DCIntrospect seems like a good case for bundling everything up in a library.
Static Library
I created an Xcode 4 workspace for DCIntrospect, which initially just contained the existing DCIntrospectDemo. The demo included the source code directly, so I removed the references to those files and created a new static library project that included the existing DCIntrospect source files. That basically compiled straight away, but to make it useful, I had to tweak a few build settings.
The target name is DCIntrospectStaticLib, but I wanted the output to be libDCIntrospect.a, which can be done by changing the Product Name build setting. I also wanted to group the header files with the compiled binary, so I set the Public Headers Folder Path to be $(PRODUCT_NAME) so that it would match whatever I entered for the product name.
To link to this static library, you simply have to drag it into your project. Xcode sorts out adjusting the Library Search Paths automatically when you add the library. However, if you want to import headers and make calls to the library, you will need to manually set the Header Search Paths. For DCIntrospectDemo, the search path is "$(SRCROOT)/../Products/lib" and then use #import <DCIntrospect/DCIntrospect.h>. That’s easy enough to do, but it is one more barrier to entry for people using the library. A static framework could make it a single step.
Static Framework
I followed the steps described in Diney Bomfim’s updated post on building a universal framework for iPhone, and it all went pretty smoothly. He describes the different build settings you have to adjust to get it all to work.
Diney’s instructions also include a sample shell script for compiling a universal static library using the lipo tool. I adjusted the script slightly, so that it would build both a universal static library as well as the static framework. I haven’t had much success running DCIntrospect on the device, so I’m not sure if that part is working or not.
The joy of using the static framework in this way is that installing DCIntrospect is now a single drag and drop of a framework into your source code. To get it working at this point, you would still need to add #import <DCIntrospect/DCIntrospect.h> into your app delegate and after you have loaded the window, call [[DCIntrospect sharedIntrospector] start].
Automatic Loading of DCIntrospect
My next task is to get DCIntrospect to automatically load if it is linked to your target. I had a play around with this a few months ago with the Frank iOS testing library, with some success but never pursued the final solution, but Pete Hodgson obviously got sick of waiting for me and did it himself.
The solution is pretty neat. You basically hook into when a class in the library is being loaded, and set yourself up to listen for the UIApplicationDidBecomeActiveNotification notification, which gets called after the app delegate - (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions gets called. When that happens, we start DCIntrospect. I’ve wrapped the call to load DCIntrospect in a check that we are running in the simulator, so it will never be called on the app we submit to Apple. Initially I tried to do a check using #ifdef DEBUG, but that wont work with the static precompiled library approach, since those flags are only valid when the library is initially compiled, not at run time.
#import "DCIntrospect.h"@interfaceDCIntrospectLoader : NSObject@end@implementationDCIntrospectLoader// This is called after application:didFinishLaunchingWithOptions:// so the statusBarOrientation should be reported correctly+(void)applicationDidBecomeActive:(NSNotification*)notification{NSString*simulatorRoot=[[[NSProcessInfoprocessInfo]environment]objectForKey:@"IPHONE_SIMULATOR_ROOT"];if(simulatorRoot){NSLog(@"Running in simulator, loading DCIntrospect");[[DCIntrospectsharedIntrospector]start];}}+(void)load{NSLog(@"Injecting DCIntrospect loader");[[NSNotificationCenterdefaultCenter]addObserver:[selfclass]selector:@selector(applicationDidBecomeActive:)name:@"UIApplicationDidBecomeActiveNotification"object:nil];}@end
With the auto loading class above now part of DCIntrospect, installation is literally a single drag and drop of the framework. It will automatically load as needed, as long as we are running in the simulator. I usually have a separate target for App Store submission without any test libraries linked into it just in case, since they bloat the binary and probably call some private API magic at some point.
DCIntrospect Demo Project
To use either the static library or the framework in the two demo project targets, the only change I had to make was to add the -ObjC setting to Other Linker Flags. For the static library, you still need to set a header path but only if you want to make direct calls to DCIntrospect, since it now loads automatically.
Since I had both the DCIntrospect and DCIntrospectDemo projects in the same workspace, XCode 4 had the smarts to find the implicit dependency, so when I edit the library source code, it’s automatically rebuilt. However, I set up the project to use the BundleProducts target to create the universal distributable binaries, which did not get picked up as a dependency.
You can create explicit dependencies between targets in different projects by first creating a project dependency. That is, dragging the dependent project into the demo project. Once that is done, when editing the Target Dependencies in the demo target’s Build Phases, you will be able to explicitly select the dependency. In this case, it is the BundleProducts target.
Unless you plan to edit the source code and recompile the binary often, the chances are you wont need to set up the project dependencies like this.
Issues with static libraries and frameworks
DCIntrospect is a fairly simple framework to generate. Larger libraries are likely to come across two issues, neither of which affect DCIntrospect.
categories from static libraries don’t seem to load properly
resources is static libraries or frameworks need to be linked into the target using them separately
For the categories issue, I know if two solutions. First, you can add -all_load to the Other Linker Flags section of the target using the static library. We’ve already added the -ObjC flag in there, so it’s not really adding much of a burden. The other option is to use Karl Stenerud’s MAKE_CATEGORIES_LOADABLE as used in Frank.
For the resources issue, you can generate a resource bundle along with your static library or framework. When linking the library to your target, you can also drag in the resources bundle as a dependency and it should work. Karl has a neat way of making this even easier in his iOS Univeral Framework project in github. He creates an embedded framework, which is essentially a single container for both the static framework and the resources bundle. This extra container makes it easy to drag in a single .embeddedframework file into your project and have everything link up correctly.
Final Thoughts
This is probably the first time I have fought with static libraries and frameworks and felt like I came out on top, and DCIntrospect seems like a good case for using it. That said, there are lots of libraries I use where I prefer being able to see the source code, and potentially edit the implementation. For those, git submodules and adding the source directly is still my preferred option. The single drag-and-drop solution here does make using a new library pretty easy though.
I’ve been playing around with testing frameworks on iOS for over a year now. There are quite a few out there, all with communities building around them, but I think there are currently a few that stand out:
Frank
KIF
UIAutomation
Other tools that might be worth looking at, but I haven’t used and wont comment on are: NativeDriver, LessPainful, iCuke and UISpec.
I recently watched a recording of Pete Hodgson (@beingagile), the primary maintainer of Frank in which he gave a quick summary of the other testing tools, and when it might be appropriate to use them. I made a few notes, and added some of my own thoughts. There is a video of the talk Pete gave at Pivotal Labs.
Frank
Frank has been described as Selenium for native iOS apps. It’s a tool written mainly by some fellow Thoughtworkers that uses a combination of Cucumber and JSON commands that are sent to a server running inside your native application, and leverages UISpec to run the commands. Frank used to require changes to your production code, but that’s got a lot easier with the new static library approach.
I’ve used Frank quite a bit, and I like it. I have a lot more I could say about it, but I’ll leave that for another time. You can find out more about Frank at:
Clean CSS-like selector syntax, allowing for fairly tolerant tests
Active community discussing and extending the library
Driven by Cucumber (if you’re a cuke fan)
Includes Symbiote, a live introspection tool
Command line and CI come pretty much for free
The bad parts of Frank
Difficult, but not impossible, to run on the device
Separation between tests/native code over HTTP can confuse the cause of failure
Decoupling test/native code into separate processes makes it a bit slower (Pete makes the good point that most of the slowness in your iOS tests are likely to be waiting for animations and transitions, not the HTTP communication)
Very little support for gestures (but hopefully that’s coming soon)
When using Frank might be appropriate
Your team has experience with web application testing tools (Selenium, Cucumber)
You prefer a BDD style of development, with CI support
Your app is not driven by complex gestures
You have a goal of having cross platform tests across Android, iOS and mobile web
KIF
KIF is an Objective C based framework written at Square. It’s a nice solid implementation from what I’ve seen, with active development going on. Since it’s Objective C, you can call out to anything else in your app as well. It’s a brand new test framework, so it’s not going to immediately integrate with other tools (like JUnit-style XML output and that kind of thing). I haven’t KIF a lot yet, so I can’t comment too much but I think it shows a lot of promise. You can find out more about KIF at:
Your tests are in Objective C. Everything in one language, easier for pure iOS devs to pick up
Active community and good support
Pretty reasonable support for gestures
Command line and CI
The bad parts of KIF
Your tests are in Objective C. Non-devs will find it hard to read, doesn’t make a great executable spec)
Tricky to integrate with back end stubs because it’s all running in-process
Not stand alone
When using KIF might be appropriate
Primarily a developer driven team
Developers have stronger Objective C skills than Ruby/Cucumber/Javascript
Don’t need business folk to read or write test specs
Don’t want to deal with the whacky regex from Cucumber
Don’t have really complex back end interactions to stub out
Don’t have cross platform requirements
UIAutomation
UIAutomation is Apple’s own solution for automated testing. It runs tests written in Javascript through the Instruments application that comes with the developer tools. It sounds like a no-brainer to use Apple’s solution for building iOS apps, but I’ve found it a real pain to deal with in practice. UIAutomation is pretty good for actually driving the UI, but not great for organising and running a test suite, especially from CI. My impression is that Apple QA staff must use UIAutomation for test scripts with profiling instruments attached and actually sit there and watch, it doesn’t seem to be purpose built for fully automated testing.
At this point I would like to link to a lot of useful documentation about UIAutomation, but in practice I have found Apple’s documentation to be either minimal or non-existent, and very little online discussion past the “boot up an app and tap a button in a single javascript file”.
The good parts of UIAutomation
Apple’s own tool. Doing things the way Apple wants is generally a good idea (and as Pete mentions, Apple’s not going to go bust or quit iOS any time soon, so the chances are it will be supported)
More closely linked to the device, I primarily run UIAutomation tests on the device, not in the simulator
Good support for gestures (pinch zooming and swipes) and rotation
The bad parts of UIAutomation
Not built with CI in mind. The command line integration is pretty bad
Can’t integrate with other tools very well
It’s Apple’s tool, and it’s not open source. You can’t jump in and fix the bits that are missing
Runs within Instruments, seems to be aimed at regression testing not TDD and aimed at QAs not devs
When using UIAutomation might be appropriate
You have a separation between development and QA on your team
You prefer regression test suites over a test-first approach
You don’t really care about CI
You prefer manual QA and you just want to speed that up a bit
Conclusion
So far I’ve used Frank and UIAutomation on fairly large projects, and I’m very keen to try KIF. Ideally what I would like is the Frank architecture using Cucumber to drive your tests, but using KIF’s implementation on the native side which is a lot more solid than UISpec. Frank would give nice clean readable test features, and integration with other tools through cucumber as well as the concise selector syntax. KIF would give gesture support and a much cleaner implementation of those features.
I’m looking to write up a few posts about using accessibility for testing native iOS applications. Here is the first one, dealing with enabling accessibility for your apps in the simulator and on the device.
Why use accessibility
Firstly, the iPhone and iPad are setting a new standard for usability by impaired users. That’s a great thing, and I think we should be making a bigger effort to support these features. Apple provides assistants like VoiceOver that use the building UIAccessibility framework. If you’d like to find out more about accessibility for the sake of accessibility, Matt Gemmell has a great post on the topic
A great side effect of making an app accessible to assistive devices is that the app also becomes easier to test in an automated fashion. This post is not about how and why to use accessibility to test your app (although that is a valid topic that I will write a separate post on), it a quick guide to turning accessibility on. Generally, the properties of the UIAccessibilty framework are only available to third party testing tools like Frank and KIF if you have accessibility enabled for the application, either in the simulator or on the device.
Enabling accessibility in the Simulator
During development, we need to enable accessibility for both OS X and the iOS Simulator. Under Mac OS X, open up the System Preferences and open the Universal Access pane at the top right. In the pane that opens, check the box for Enable Access for Assistive Devices. The OS will persist this setting from now on.
Load the iOS Simulator and open the Settings application. Enable accessibility using the switch under General > Accessibility. If you see a little coloured box appear, you have successfully enabled accessibility in the simulator. The setting is stored in an underlying plist file under that iOS version of the simulator, so you will need to enable this setting for for both iOS 4 and 5 separately, but that will effect all of the device types (iPhone, iPhone Retina and iPad) for that OS version. The iOS Simulator will keep accessibility enabled as long as you don’t clear out its settings. If you’re anything like me, you have Reset Content and Settings mapped to a keyboard shortcut, you’ll regularly need to navigate in and re-enable accessibility.
The small coloured box that appears is called the accessibility inspector. It shows a small summary of what is available from the UIAccessibility framework for iOS. There are two main types of information shown in the inspector: notifications and properties. Notifications are fired when the UI changes. To be honest, I’ve not played around with firing accessibility notifications much at all. Notifications could be a potential solution for tests that need to ‘wait’ until a screen transition is finished before continuing rather than busy-waiting or just a plain old sleep (you know it’s not a real test suite unless there are a couple of sleep calls in there!).
The properties show aspects of the selected UI element. With the accessibility inspector expanded, tap around in some of Apple’s built in applications to see the UIAccessibility properties. With the accessibility inspector expanded like this, the first touch even brings up the accessibility - which is great if that’s the only way to use the app, but can get in the way if you’re not used to it. If you collapse the inspector using the little cross-button, touch interaction returns to normal. In the image above I’ve collapsed the accessibility inspector and dragged it to the side, since the properties are available to our tests anyway, so it’s easiest to get it as far off the screen as possible.
Enabling accessibility on the device
Usually, to get access to the accessibility framework on an actual iOS device, you need to enable VoiceOver. If you do your testing with Apple’s sanctioned UIAutomation Instrument, it seems to be able to hook in automatically without you having to specifically enable those features. Although with iOS 5, I’ve found that to not always be the case.
VoiceOver is pretty easy to enable in the Settings app under General > Accessibility > VoiceOver. Once this is enabled, the device acts in a similar way to when you have the accessibility inspector visible in the simulator. A pleasant computerised voice now describes your every gesture, and more importantly, activates the accessibility framework for all applications, including the one you want to test. Your first tap will select a UI element and read the available information about it. Double-tapping actually executes the action for a button. People using VoiceOver as a means to navigate the OS are likely to drag their finger on the screen to get a better idea of where items are relative to each other, so single-finger scrolling is also disabled. You can scroll by dragging with three fingers.
Navigate some well known apps on the phone, and see how Apple’s own applications integrate with VoiceOver. To actually design an accessible application, you’ll be wanting to spend a whole lot of time using VoiceOver yourself to get an idea of what information is useful and necessary. At some stage, you will pick up a test device with VoiceOver on, so it’s good to know at least how to get into the settings and turn accessibility off in order to operate manually.
If you are only enabling VoiceOver for testing purposes, the changed gestures and audio instructions can seem to get in the way of you just using the device. If you regularly switch accessibility on and off, using iTunes is going to be much more convenient. Before the latest version of iTunes, enabling accessibility this way required the device to be plugged in via USB. I was pleasantly surprised to discover that this VoiceOver can now be toggle on and off over wifi.
Enabling accessibility programmatically
Note: these steps apply to the iOS Simulator only. If someone knows how to enable accessibility programmatically on the device, I would love to know
When you are running tests in the simulator, it’s likely that you use the Reset Content and Settings… menu item frequently. If you forget to reenable accessibility after this, tests will fail because UI information will not be available to the tests. We need a programmatic way to turn accessibility on. I’m not sure how to do this on the device, but in the simulator, this is just a matter of setting a flag in one of the underlying plist files. It’s possible to call plist editors from the command line to do this, but then the plist file is separate for each iOS version supported by the simulator. I find it’s easier to call from the Objective C code at run time, since it it possible to obtain the root for that version of the simulator.
I first tried to use this code in init for a test framework class, and it didn’t seem to work. Moving the code to be called from load solved the problem. Since this code actually changes a plist file on the file system, perhaps it needs to be executed early before the UI loads so that the rest of the system acts as if accessibility is enabled. Not only does this approach ensure accessibility is always enabled in the simulator, it doesn’t bring up the inspector over the top of the UI.
Special thanks to Cedric Luthi, who originally wrote the code to enable accessibility programmatically for DCIntrospect in this commit, and to an awesome colleague of mine, Sadat Rahman (@sadatrahman), for bringing that code to my attention.
My Wordpress blog was a complete failure in terms of content and effort on my part. I blame part of this on the barrier of having to use Wordpress, which I find bloated and slow. I want to write blogs the same way I write code: in my text editor using git to keep track of changes, and deploying with a git push. I’m hoping most of my posts will involve a lot of code snippets, so being able to edit the posts in a proper programmer’s text editor, in plain text, using Markdown is ideal.
I tried to get jekyll set up, but being a bit of a noob when it comes to stylesheets as well as Ruby, I just couldn’t find the time to get the blog set up properly. This morning I read about Matt Gemmell’s blog moving to Octopress, which seemed the best of both worlds. The simple ongoing use and management that jekyll provides, and with some additional scripts to set it all up for me.
I’ve gone and set it all up this afternoon, now to see if that means I actually have the time to write up posts!
