Skip to content
GitLab
Open
Issue created by Simon Warren@swarren12

Original test class name is lost when running a JUnit test suite

When running buck test with the --xml argument, the resulting XML file contains the name of the test class that was run in a <test> element and the name of each individual method within that class in child <testresult> elements. However, when running a JUnit test suite, the name of the test class is replaced with the name of the test suite in the <test> element. This can make it difficult (/impossible) to differentiate tests that are in different classes but have the same method name.

Expected behaviour: The original test class name should be recorded when using a test suite Actual behaviour: The original test class name is replaced by the name of the test suite Steps to reproduce: Run buck test --xml on a JUnit test suite

For example, given a contrived example with the following test classes / methods:

  • TestCase1 => test1(), test2(), test3()
  • TestCase2 => test1(), test2(), test3()
  • TestCase3 => test1(), test2(), test3()
  • TestCase4 => testA()

... running these as a suite might generate an XML such as the one attached. As you can see, the report generated by buck makes it impossible to test which version of test1(), test2() and test3() failed. It isn't even clear if they all came from the same test class, although one might assume that they did given that they are together.

The original JUnit XML that was parsed in order to create the output.xml file does contain the both the suite name and the class name, so I don't see any particular reason why the class name shouldn't make its way across to the buck-generated file also?

However, I'm not sure what the ultimate desired behaviour should be when using a test suite - logically I would expect it to mimic the JUnit output of nesting the test classes under the test suite, but I don't know if this structure lends itself well to the other types of tests that buck supports. The alternatives would be to either:

  1. drop the suite information completely and just parse the JUnit XML as if it was the individual tests that had run (this would be my preference) or
  2. use an approach similar to the JUnit XML where the original class name is included as an extra attribute on the tag (less desirable to me as that's a change to the XML schema)

I have knocked together a change that implements option 1 and I'm happy to raise this as a PR if there is agreement that using a suite should work in this way. Alternatively, happy to look at making a different change if there's some consensus that it should work in a different way.