Speed of deployment has always been an important metric used to judge how often and efficiently an app release can be. As we’ve mentioned in our previous post iOS Continuous Deployment , here at Hotwire, we spend a lot of time figuring out ways to reduce release regression time. Two pockets of opportunities are – reduce the test writing/execution time and another is to reduce test framework maintenance time. In the past, it has been a challenge to keep our functional testing framework robust enough with every iOS release considering third party tools(in our case, Appium) often lag behind on new iOS support. Another hurdle was the maintenance of all the various continuous integration Jenkins pipelines and reporting mechanisms. An obvious solution to both the problems is to reduce dependency on third party tools to gain more control and robustness. So at the brink of Xcode 7 UI Tests announcement at WWDC ’15 and Apple’s unconditional support to Swift, we started out functional testing in UI Testing with Xcode 7. Even with its few quirks, it was indeed a low barrier of start to get going. In this blog post, we will discuss how we adopted Xcode UI Testing for a more integrated and faster approach to BDD testing.
UI Recording feature introduced as a part of UI Testing framework is a true game changer. Not only did it help the user to write tests faster, but it also offers a quick way to debug Accessibility element values and behavior on the app. Of course it isn’t flawless though. It has been noticed that not all interactions, like for eg long tap, get recorded reliably. But as a guiding tool to start writing a test, UI Recording proves to still save us a lot of time. Printing the accessibility hierarchy is a good way to “see what the framework sees.” You can use this and the Accessibility Inspector to debug querying and selecting elements like this
Additionally, we decided to handle our environment variables at the setup() method to re-run the same tests with different flavors, driven by schemes. For example :
Speed and performance
The same testsuite that took 11 minutes to run with Appium, only took 1 minute to run on Xcode UI Tests. That’s 91% faster! Also, it only took us a couple of weeks to get our sanity tests ported over, which is pretty fast considering Swift was a new language for all QA Automation engineers on the team as well. However, the need for a BDD framework emerged so that we could write reusable and readable tests from the end user/customer perspective.
BDD is Behavior Driven Test Development that drives the notion that tests can be written in plain English steps (in a Given/When/Then syntax) to be managed by both Product and Engineering Teams to define user stories. Since UI Tests are written in Swift, we resorted to an open source framework called XCTest Gherkin. Installing it as a Cocoa pod, we leverage it to create a modularized, functional and well integrated test framework in UI Testing with XCode. Include the following in Podfile and do a ‘pod install‘
A sample test feature with XCTest Gherkin syntax looks like:
For Step definition, we need to extend StepDefiner which is XCTest Gherkin method and override defineSteps() to add our step mapping to screen object class methods. These steps match regular expressions and return capture groups “matches”
As noticed, this looks very familiar to Cucumber feature files. Also, XCTest Gherkin offers a way to port native files written in Cucumber parsed to Swift files but since tests are only generated at runtime, making it hard to debug, we decided to skip this.
Continuous Integration and Reporting
An equivalent to Jenkins is OS X Server driven Bots. Bots perform continuous integration testing that corresponds to Schemes. Each application scheme can be configured to run specific tests, with their own environment variables and essentially any other configuration required. We used this mechanism to mock tagging of our tests to run on bots (or skip tests) like this:
Bots can run on various simulators and iOS versions in parallel, giving visibility to version/device specific issues. There is also pre and post build scripts that can be leveraged to send HipChat messages, email committers when build breaks, test status or artifact handling.
Reporting is integrated in XCode, which is highly useful in stack traversing and debugging for both QA and Developers. A bonus – Snapshots at test failure.
XCode UI testing has proven be more than a mere winner for iOS so far. As a viable functional testing approach, UI Testing achieved:
- Behavior driven tests (BDD) – With the help of XCTest Gherkin
- Faster speed of execution and writing new tests – 91% faster test execution for us ; UI Recording to aid faster test writing
- Reliable Continuous integration of these tests on each commit – XCode Bots
- Integrated reporting – XCode integrated with screen snapshots at test failure