Hotwire Tech Blog

Scribes from Hotwire Engineering

Goal

Speed of deployment has always been an important metric used to judge how often and efficiently an app release can be. As we’ve mentioned in our previous post iOS Continuous Deployment , here at Hotwire, we spend a lot of time figuring out ways to reduce release regression time. Two pockets of opportunities are – reduce the test writing/execution time and another is to reduce test framework maintenance time. In the past, it has been a challenge to keep our functional testing framework robust enough with every iOS release considering third party tools(in our case, Appium) often lag behind on new iOS support. Another hurdle was the maintenance of all the various continuous integration Jenkins pipelines and reporting mechanisms. An obvious solution to both the problems is to reduce dependency on third party tools to gain more control and robustness. So at the brink of Xcode 7 UI Tests announcement at WWDC ’15 and Apple’s unconditional support to Swift, we started out functional testing in UI Testing with Xcode 7. Even with its few quirks, it was indeed a low barrier of start to get going. In this blog post, we will discuss how we adopted Xcode UI Testing for a more integrated and faster approach to BDD testing.

Writing Tests

UI Recording feature introduced as a part of UI Testing framework is a true game changer. Not only did it help the user to write tests faster, but it also offers a quick way to debug Accessibility element values and behavior on the app. Of course it isn’t flawless though. It has been noticed that not all interactions, like for eg long tap, get recorded reliably. But as a guiding tool to start writing a test, UI Recording proves to still save us a lot of time. Printing the accessibility hierarchy is a good way to “see what the framework sees.” You can use this and the Accessibility Inspector to debug querying and selecting elements like this

print(app.debugDescription)

Additionally, we decided to handle our environment variables at the setup() method to re-run the same tests with different flavors, driven by schemes. For example :

let app = XCUIApplication()
app.launchEnvironment = ["EnvironmentVariable": "true"]

Speed and performance

The same testsuite that took 11 minutes to run with Appium, only took 1 minute to run on Xcode UI Tests. That’s 91% faster! Also, it only took us a couple of weeks to get our sanity tests ported over, which is pretty fast considering Swift was a new language for all QA Automation engineers on the team as well. However, the need for a BDD framework emerged so that we could write reusable and readable tests from the end user/customer perspective.

BDD Framework

BDD is Behavior Driven Test Development that drives the notion that tests can be written in plain English steps (in a Given/When/Then syntax) to be managed by both Product and Engineering Teams to define user stories. Since UI Tests are written in Swift, we resorted to an open source framework called XCTest Gherkin. Installing it as a Cocoa pod, we leverage it to create a modularized, functional and well integrated test framework in UI Testing with XCode. Include the following in Podfile and do a ‘pod install

pod “XCTest-Gherkin”

A sample test feature with XCTest Gherkin syntax looks like:

Feature file
func testHotelBookingPath() {
    Given ("I select Hotel")
    And ("I am searching for a hotel near Miami")
    And ("I select 1st hotel from Standard results")
}

 

For Step definition, we need to extend StepDefiner which is XCTest Gherkin method and override defineSteps() to add our step mapping to screen object class methods.  These steps match regular expressions and return capture groups “matches

Step Definition file
class HotelSteps: StepDefiner {
  override func defineSteps() {
     step("I select (Hotel|Car)") { (matches: [String]) in
         if (matches.first == "Hotel"){
             HomeScreen().goToHotel()
             }
             else if (matches.first == "Car"){
                 HomeScreen().goToCar()
             }
      }
   }
}

 

As noticed, this looks very familiar to Cucumber feature files. Also, XCTest Gherkin offers a way to port native files written in Cucumber parsed to Swift files but since tests are only generated at runtime, making it hard to debug, we decided to skip this.

Continuous Integration and Reporting

An equivalent to Jenkins is OS X Server driven Bots. Bots perform continuous integration testing that corresponds to Schemes. Each application scheme can be configured to run specific tests, with their own environment variables and essentially any other configuration required. We used this mechanism to mock tagging of our tests to run on bots (or skip tests) like this:

Screen Shot 2016-05-05 at 2.52.11 PM

Bots can run on various simulators and iOS versions in parallel, giving visibility to version/device specific issues. There is also pre and post build scripts that can be leveraged to send HipChat messages, email committers when build breaks, test status or artifact handling.

Reporting is integrated in XCode, which is highly useful in stack traversing and debugging for both QA and Developers. A bonus – Snapshots at test failure.

Conclusion

XCode UI testing has proven be more than a mere winner for iOS so far. As a viable functional testing approach, UI Testing achieved:

  • Behavior driven tests (BDD) – With the help of XCTest Gherkin
  • Faster speed of execution and writing new tests – 91% faster test execution for us ; UI Recording to aid faster test writing
  • Reliable Continuous integration of these tests on each commit – XCode Bots
  • Integrated reporting – XCode integrated with screen snapshots at test failure

Leave a Reply

Your email address will not be published. Required fields are marked *