QA/test automation

2012-08-20T02:26:57Z

Liuzhe:

QA/vclauto

2012-08-17T00:42:15Z

Liuzhe:

{{Template:Documentation/Note|This page will introduce VCLAuto tech background. If you want to get started with test automation, please read the [[QA/test_automation_guide|test automation guide]].}}
==Introduction==
VCLAuto is a java library to automate GUI testing for Apache OpenOffice.

TODO: write some text to describe how the API works.

2012-08-15T05:53:04Z

Liuzhe: moved QA/junit rules to QA/test automation rules

{{DISPLAYTITLE: Best Practice Rules for Test Automation}}
[[Category: Quality Assurance]]
[[Category: TestAutomation]]

== Best practice rules for JUnit 4 ==
{{Template:Documentation/Caution|MUST READ THE SECTION BEFORE COMMITTING AUTOMATION CODE}}
'''Rule.''' Recommend to follow [http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html Java code conventions] 
'''Rule.''' Don't miss assertions. One test method should include at least one assertion, otherwise the test will always pass. 
'''Rule.''' Don't add assertions in @Before/@BeforeClass/@After/@AfterClass methods. 
'''Rule.''' Use assertions with an informative message. 
''Bad''
<source lang="java">
assertEquals(expected , actual);
assertNull(actual);
assertNotNull(actual);
assertTrue(actual);
</source>.
''Good''
<source lang="java">
assertEquals("actual should equals expected", expected , actual);
assertNull("actual should be null", actual);
assertNotNull("actual should not be null", actual);
assertTrue("actual should be true", actual);
</source>
'''Rule.''' Don't use assertTrue wrongly. 
''Bad''
<source lang="java">
assertTrue("actual should be same with expected", expected == actual);
assertTrue("actual should equals expected", expected.equals(actual));
assertTrue("actual should be null", actual == null);
assertTrue("actual should not be null", actual != null);
</source>
''Good''
<source lang="java">
assertSame("actual should be same with expected", expected, actual);
assertEquals("actual should equals expected", expected, actual);
assertNull("actual should be null", actual);
assertNotNull("actual should not be null", actual);
</source>
'''Rule.''' Use assertArrayEquals instead of assertEquals for array data. 
''Bad''
<source lang="java">
assertEquals("actual[0] should equals expected0", expected0, actual[0]);
assertEquals("actual[1] should equals expected1", expected1, actual[1]);
assertEquals("actual[2] should equals expected2", expected2, actual[2]);
</source>
''Good''
<source lang="java">
String[] expected = {"0", "1", "2"};
String[] actual = someMethod();
assertArrayEquals("actual should equals expected", expected, actual);
</source>
'''Rule.''' Avoid negation in an assertTrue or assertFalse test. 
''Bad''
<source lang="java">
assertTrue("not empty", !r.isEmpty());
</source>
''Good''
<source lang="java">
assertFalse("not empty", r.isEmpty());
</source>
'''Rule.''' Don't catch unexpected exceptions. 
''Bad''
<source lang="java">
public void testCalculation() {
try {
deepThought.calculate();
assertEquals("Calculation wrong", 42, deepThought.getResult());
}
catch(CalculationException ex) {
Log.error("Calculation caused exception", ex);
}
}
</source>
''Good''
<source lang="java">
public void testCalculation() throw CalculationException {
deepThought.calculate();
assertEquals("Calculation wrong", 42, deepThought.getResult());
}
</source>
'''Rule.''' Avoid too complex test. Simple test is easy to understand and maintain. One test should not contain too many asserts. Consider breaking the test scenario into multiple, shorter test scenarios. 
'''Rule.''' Avoid external dependencies. Others can run test by a single command or a one button click without extra configurations. e.g. Data file should be placed in the svn with test code, so that newcomers don't need to manually prepare it. 
''Bad''
<source lang="java">
public void testFile() {
String testFile = "C:\\test\\test.odt";
//Do test against on testFile
}
</source>
''Good''
<source lang="java">
public void testFile() {
// test.odt is placed with test code together. Given a relative path, prepareData will return the file's real absolute path.
String testFile = prepareData("test.odt");
}
</source>
'''Rule.''' Make test method indepdent, not affect or be affected by others. Test methods can be executed in any particular order. So do necessary setup/teardown in @Before/@BeforeClass/@After/@AfterClass method to avoid side effects. 
'''Rule.''' Log the generated data when you use random. 
'''Rule.''' Name tests properly. "testLoadFileFromServer" is better than a simple word "test". 
'''Rule.''' Consider parameterized test if you want run the same steps against a bunch of different test data.Use @RunWith(value = Parameterized.class) on class to enable parameterized test. Use @Parameters annotation on one static method to generate a collection of parameters. Each parameter will be passed into class constructor during test. 
''Example''
<source lang="java">
@RunWith(value = Parameterized.class)
public class ParameterizedTest {

private int number;
public ParameterizedTest(int number) {
this.number = number;
}

@Parameters
public static Collection<Object[]> data() {
Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
return Arrays.asList(data);
}

@Test
public void testSteps() {
// use number to do test
}
}
</source>

== Best practice rules for GUI testing ==
'''Rule.''' Don't directly use shortcut keys. Make them as an customized shortcut. Different platforms have different shortcuts for the same function. 
'''Rule.''' Don't add unnecessary sleep. 
''Bad''
<source lang="java">
//... some steps
control.waitForExistence(5, 1); // The line is not necessary.
//sleep(5); // The line is not necessary.
control.click(); // The method impliedly does several times of sleep to wait the control to exist.
</source>
''Good''
<source lang="java">
//... some steps
control.click();
</source>

'''Rule.''' Test testing codes on all platforms before committing. OpenOffice may have different behaviors on different platforms. 
'''Rule.''' Don't assert if a control exists, if the next operation imply the control must exist. 
''Bad''
<source lang="java">
//... some steps
assertTrue("Control exists", control.exists(5, 1)); // The line is not necessary.
control.click(); // click method imply the control must exist, so you have add the verification point impliedly.
</source>
''Good''
<source lang="java">
//... some steps
control.click();
</source>

QA/test automation rules

2012-08-15T05:52:19Z

Liuzhe:

QA/test automation rules

2012-08-15T05:41:59Z

Liuzhe: /* Best practice rules for GUI testing */

{{DISPLAYTITLE: Best Practice Rules for JUnit 4}}
[[Category: Quality Assurance]]
[[Category: TestAutomation]]

== Best practice rules for JUnit 4 ==
{{Template:Documentation/Caution|MUST READ THE SECTION BEFORE COMMITTING AUTOMATION CODE}}
'''Rule.''' Recommend to follow [http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html Java code conventions] 
'''Rule.''' Don't miss assertions. One test method should include at least one assertion, otherwise the test will always pass. 
'''Rule.''' Don't add assertions in @Before/@BeforeClass/@After/@AfterClass methods. 
'''Rule.''' Use assertions with an informative message. 
''Bad''
<source lang="java">
assertEquals(expected , actual);
assertNull(actual);
assertNotNull(actual);
assertTrue(actual);
</source>.
''Good''
<source lang="java">
assertEquals("actual should equals expected", expected , actual);
assertNull("actual should be null", actual);
assertNotNull("actual should not be null", actual);
assertTrue("actual should be true", actual);
</source>
'''Rule.''' Don't use assertTrue wrongly. 
''Bad''
<source lang="java">
assertTrue("actual should be same with expected", expected == actual);
assertTrue("actual should equals expected", expected.equals(actual));
assertTrue("actual should be null", actual == null);
assertTrue("actual should not be null", actual != null);
</source>
''Good''
<source lang="java">
assertSame("actual should be same with expected", expected, actual);
assertEquals("actual should equals expected", expected, actual);
assertNull("actual should be null", actual);
assertNotNull("actual should not be null", actual);
</source>
'''Rule.''' Use assertArrayEquals instead of assertEquals for array data. 
''Bad''
<source lang="java">
assertEquals("actual[0] should equals expected0", expected0, actual[0]);
assertEquals("actual[1] should equals expected1", expected1, actual[1]);
assertEquals("actual[2] should equals expected2", expected2, actual[2]);
</source>
''Good''
<source lang="java">
String[] expected = {"0", "1", "2"};
String[] actual = someMethod();
assertArrayEquals("actual should equals expected", expected, actual);
</source>
'''Rule.''' Avoid negation in an assertTrue or assertFalse test. 
''Bad''
<source lang="java">
assertTrue("not empty", !r.isEmpty());
</source>
''Good''
<source lang="java">
assertFalse("not empty", r.isEmpty());
</source>
'''Rule.''' Don't catch unexpected exceptions. 
''Bad''
<source lang="java">
public void testCalculation() {
try {
deepThought.calculate();
assertEquals("Calculation wrong", 42, deepThought.getResult());
}
catch(CalculationException ex) {
Log.error("Calculation caused exception", ex);
}
}
</source>
''Good''
<source lang="java">
public void testCalculation() throw CalculationException {
deepThought.calculate();
assertEquals("Calculation wrong", 42, deepThought.getResult());
}
</source>
'''Rule.''' Avoid too complex test. Simple test is easy to understand and maintain. One test should not contain too many asserts. Consider breaking the test scenario into multiple, shorter test scenarios. 
'''Rule.''' Avoid external dependencies. Others can run test by a single command or a one button click without extra configurations. e.g. Data file should be placed in the svn with test code, so that newcomers don't need to manually prepare it. 
''Bad''
<source lang="java">
public void testFile() {
String testFile = "C:\\test\\test.odt";
//Do test against on testFile
}
</source>
''Good''
<source lang="java">
public void testFile() {
// test.odt is placed with test code together. Given a relative path, prepareData will return the file's real absolute path.
String testFile = prepareData("test.odt");
}
</source>
'''Rule.''' Make test method indepdent, not affect or be affected by others. Test methods can be executed in any particular order. So do necessary setup/teardown in @Before/@BeforeClass/@After/@AfterClass method to avoid side effects. 
'''Rule.''' Log the generated data when you use random. 
'''Rule.''' Name tests properly. "testLoadFileFromServer" is better than a simple word "test". 
'''Rule.''' Consider parameterized test if you want run the same steps against a bunch of different test data.Use @RunWith(value = Parameterized.class) on class to enable parameterized test. Use @Parameters annotation on one static method to generate a collection of parameters. Each parameter will be passed into class constructor during test. 
''Example''
<source lang="java">
@RunWith(value = Parameterized.class)
public class ParameterizedTest {

private int number;
public ParameterizedTest(int number) {
this.number = number;
}

@Parameters
public static Collection<Object[]> data() {
Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
return Arrays.asList(data);
}

@Test
public void testSteps() {
// use number to do test
}
}
</source>

== Best practice rules for GUI testing ==
'''Rule.''' Don't directly use shortcut keys. Make them as an customized shortcut. Different platforms have different shortcuts for the same function. 
'''Rule.''' Don't add unnecessary sleep. 
''Bad''
<source lang="java">
//... some steps
control.waitForExistence(5, 1); // The line is not necessary.
//sleep(5); // The line is not necessary.
control.click(); // The method impliedly does several times of sleep to wait the control to exist.
</source>
''Good''
<source lang="java">
//... some steps
control.click();
</source>

'''Rule.''' Test testing codes on all platforms before committing. OpenOffice may have different behaviors on different platforms. 
'''Rule.''' Don't assert if a control exists, if the next operation imply the control must exist. 
''Bad''
<source lang="java">
//... some steps
assertTrue("Control exists", control.exists(5, 1)); // The line is not necessary.
control.click(); // click method imply the control must exist, so you have add the verification point impliedly.
</source>
''Good''
<source lang="java">
//... some steps
control.click();
</source>

QA/test automation rules

2012-08-15T05:37:44Z

Liuzhe: /* Best practice rules for JUnit 4 */

{{DISPLAYTITLE: Best Practice Rules for JUnit 4}}
[[Category: Quality Assurance]]
[[Category: TestAutomation]]

== Best practice rules for JUnit 4 ==
{{Template:Documentation/Caution|MUST READ THE SECTION BEFORE COMMITTING AUTOMATION CODE}}
'''Rule.''' Recommend to follow [http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html Java code conventions] 
'''Rule.''' Don't miss assertions. One test method should include at least one assertion, otherwise the test will always pass. 
'''Rule.''' Don't add assertions in @Before/@BeforeClass/@After/@AfterClass methods. 
'''Rule.''' Use assertions with an informative message. 
''Bad''
<source lang="java">
assertEquals(expected , actual);
assertNull(actual);
assertNotNull(actual);
assertTrue(actual);
</source>.
''Good''
<source lang="java">
assertEquals("actual should equals expected", expected , actual);
assertNull("actual should be null", actual);
assertNotNull("actual should not be null", actual);
assertTrue("actual should be true", actual);
</source>
'''Rule.''' Don't use assertTrue wrongly. 
''Bad''
<source lang="java">
assertTrue("actual should be same with expected", expected == actual);
assertTrue("actual should equals expected", expected.equals(actual));
assertTrue("actual should be null", actual == null);
assertTrue("actual should not be null", actual != null);
</source>
''Good''
<source lang="java">
assertSame("actual should be same with expected", expected, actual);
assertEquals("actual should equals expected", expected, actual);
assertNull("actual should be null", actual);
assertNotNull("actual should not be null", actual);
</source>
'''Rule.''' Use assertArrayEquals instead of assertEquals for array data. 
''Bad''
<source lang="java">
assertEquals("actual[0] should equals expected0", expected0, actual[0]);
assertEquals("actual[1] should equals expected1", expected1, actual[1]);
assertEquals("actual[2] should equals expected2", expected2, actual[2]);
</source>
''Good''
<source lang="java">
String[] expected = {"0", "1", "2"};
String[] actual = someMethod();
assertArrayEquals("actual should equals expected", expected, actual);
</source>
'''Rule.''' Avoid negation in an assertTrue or assertFalse test. 
''Bad''
<source lang="java">
assertTrue("not empty", !r.isEmpty());
</source>
''Good''
<source lang="java">
assertFalse("not empty", r.isEmpty());
</source>
'''Rule.''' Don't catch unexpected exceptions. 
''Bad''
<source lang="java">
public void testCalculation() {
try {
deepThought.calculate();
assertEquals("Calculation wrong", 42, deepThought.getResult());
}
catch(CalculationException ex) {
Log.error("Calculation caused exception", ex);
}
}
</source>
''Good''
<source lang="java">
public void testCalculation() throw CalculationException {
deepThought.calculate();
assertEquals("Calculation wrong", 42, deepThought.getResult());
}
</source>
'''Rule.''' Avoid too complex test. Simple test is easy to understand and maintain. One test should not contain too many asserts. Consider breaking the test scenario into multiple, shorter test scenarios. 
'''Rule.''' Avoid external dependencies. Others can run test by a single command or a one button click without extra configurations. e.g. Data file should be placed in the svn with test code, so that newcomers don't need to manually prepare it. 
''Bad''
<source lang="java">
public void testFile() {
String testFile = "C:\\test\\test.odt";
//Do test against on testFile
}
</source>
''Good''
<source lang="java">
public void testFile() {
// test.odt is placed with test code together. Given a relative path, prepareData will return the file's real absolute path.
String testFile = prepareData("test.odt");
}
</source>
'''Rule.''' Make test method indepdent, not affect or be affected by others. Test methods can be executed in any particular order. So do necessary setup/teardown in @Before/@BeforeClass/@After/@AfterClass method to avoid side effects. 
'''Rule.''' Log the generated data when you use random. 
'''Rule.''' Name tests properly. "testLoadFileFromServer" is better than a simple word "test". 
'''Rule.''' Consider parameterized test if you want run the same steps against a bunch of different test data.Use @RunWith(value = Parameterized.class) on class to enable parameterized test. Use @Parameters annotation on one static method to generate a collection of parameters. Each parameter will be passed into class constructor during test. 
''Example''
<source lang="java">
@RunWith(value = Parameterized.class)
public class ParameterizedTest {

private int number;
public ParameterizedTest(int number) {
this.number = number;
}

@Parameters
public static Collection<Object[]> data() {
Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
return Arrays.asList(data);
}

@Test
public void testSteps() {
// use number to do test
}
}
</source>

== Best practice rules for GUI testing ==
'''Rule.''' Don't directly use shortcut keys. Make them as an customized shortcut. Different platforms have different shortcuts for the same function. 
'''Rule.''' Don't add unnecessary sleep. 
'''Rule.''' Test testing codes on all platforms before committing. OpenOffice may have different behaviors on different platforms. 
'''Rule.''' Don't assert if a control exists, if the next operation imply the control must exist. 
''Bad''
<source lang="java">
//... some steps
assertTrue("Control exists", control.exists(5, 1)); // The line is not necessary.
control.click(); // click method imply the control must exist, so you have add the verification point impliedly.
</source>
''Good''
<source lang="java">
//... some steps
control.click();
</source>