Hotwire Tech Blog

Scribes from Hotwire Engineering

Problem

There are many test case management tools available out there but if the automation team does not select the right tool, they could face one or more of the below limitations:

  • Cannot update test case execution results automatically.
  • Cannot easily integrate with our defect tracking tool.
  • Cannot create custom reports.
  • Cannot notify test results to interested parties automatically.

Evolvement

The Android team has used various tools for managing test cases ranging from simple ones like Word, Excel to full blown ones like TestLink. With Word, Excel etc. we can just list the test cases but cannot integrate these with any defect tracking tools and it’s hard to keep up-to-date status of test results and the corresponding graphs. TestLink solves most of these issues and the team used it for a few months but its UI is outdated and there is no easy way to integrate with our automation framework and our bug tracking tool JIRA.

Hello TestRail

TestRail was easy to get started with. We were able to create sample test cases and execute them immediately. It has a modern UI, and it’s easy to copy, move & edit test cases. We can also edit test cases during test runs and the edits show automatically in all test runs. But all of our current test cases are present in TestLink and TestRail cannot recognize them unless we migrate them in the right format. 

TestRail Overview

 Test Rail Overview  

TestRail sample reports screen

 TestRail Sample Reports

TestLink to TestRail migration experience

The TestRail team has created a script for migration and it is present here. After execution however, we found the below issues:

  • All the test cases lost their type.  Here is the list of our test case types: “Manual”, “Automated” & “Ready To Automate”.  We had to add the following line in testlink2testrail.php script to preserve the test case type:

           xml_write_tag($handle, ‘type’, $type);

  • All the test cases lost their steps, to preserve the steps we added the following code to the script:

           $steps = null;
           $expectedresults = null;
           foreach ($case->steps->step as $step)
           {
                  $steps .= ($step->actions);
                  if (isset($step->expectedresults) &&
                      (string) $step->expectedresults)
                 {
                            $expectedresults = $step->expectedresults;
                 }
            }
            xml_write_html_tag($handle, ‘steps’, $steps);
            xml_write_html_tag($handle, ‘expected’, $expectedresults);

  • TestRail creates new ID for each test case. Along with the new IDs we also wanted to preserve the existing TestLink IDs to update the tags in Cucumber scenarios in our test framework, to achieve this we made the following changes to the script:

           $externalid = “NA-AND-” . $case->externalid;
           xml_write_tag($handle, ‘references’, $externalid);
           foreach ($case->keywords->keyword as $keyword)
           {
                   $type = $case->keywords->keyword[‘name’];
           }

  • All the tests with “Read To Automate” test case type lost their type, to preserve this we had to replace the type from “Ready To Automate” to “To Be Automated” as TestRail doesn’t have the former type.
  • In testlink2testrail.php all the above changes were done in the function write_case. Below is the modified function code:

/*********************************************************************/
function write_case($handle, $case)

{
   if (!isset($case[‘name’]))
   {
     return; // Cases without title/name are ignored
   }

   global $case_count;
   $case_count++;

   $title = $case[‘name’];

   xml_write_opening_tag($handle, ‘case’);
   xml_write_tag($handle, ‘title’, $title);

   $externalid = “NA-AND-” . $case->externalid;
   xml_write_tag($handle, ‘references’, $externalid);

   foreach ($case->keywords->keyword as $keyword)
   {
       $type = $case->keywords->keyword[‘name’];
   }
   xml_write_tag($handle, ‘type’, $type);
   xml_write_tag($handle, ‘template’, ‘Test Case (Steps)’);

   xml_write_opening_tag($handle, ‘custom’);
   if (isset($case->preconditions) &&
     strlen(trim($case->preconditions)) > 0)
   {
     xml_write_html_tag($handle, ‘preconds’, $case->preconditions);
   }

   if (isset($case->summary) && strlen(trim($case->summary)) > 0)
   {
     xml_write_html_tag($handle, ‘summary’, $case->summary);
   }

   if (isset($case->steps->step))
   {
     $steps = null;
     $expectedresults = null;
     foreach ($case->steps->step as $step)
     {
         $steps .= ($step->actions);
         if (isset($step->expectedresults) &&
                       (string) $step->expectedresults)
           {
               $expectedresults = $step->expectedresults;
            }
     }
     xml_write_html_tag($handle, ‘steps’, $steps);
     xml_write_html_tag($handle, ‘expected’, $expectedresults);

   }
   xml_write_closing_tag($handle, ‘custom’);

                   xml_write_closing_tag($handle, ‘case’);
                }

After migrating all the test cases to TestRail we wanted to integrate it with our automation test framework. TestRail team has provided the procedure to setup the integration and it can be found here.

Sample tear down code to update statuses of test cases:

TestRailAPIClient testRailAPIClient = new TestRailAPIClient(“https://<server>/testrail/”);
testRailAPIClient.setUser(“xxxxxxx”);
testRailAPIClient.setPassword(“xxxxx”);
JSONArray jsonArray = (JSONArray) testRailAPIClient.sendPost(“add_results_for_cases/:run_id”, updateResults);

Structure of updateResults object:

{

        “results”: [

                  {

                           “case_id”: 1,

                           “status_id”: 5,

                           “comment”: “This test failed”

                  },

                  {

                           “case_id”: 2,

                           “status_id”: 1,

                           “comment”: “This test passed”

                       },

                       ..

We also wanted to upload screenshots when test case fails but then found that their API does not support attachments/file uploads.

Where we are now

The Android team at Hotwire has integrated TestRail with their current automation framework. Using TestRail API all our test cases get automatically updated daily through Jenkins CI.

Summary

  • By using TestRail we saved lot of time in writing & managing test cases.
  • TestRail has modern UI with very nice features including real time editing of test cases and projecting test cases completion time.
  • TestRail team has provided script to migrate test cases from TestLink but it has few issues to be fixed.

Leave a Reply

Your email address will not be published. Required fields are marked *