User:Foral/automation/testcase

From Apache OpenOffice Wiki
Jump to: navigation, search

After automation test

After automation test.you should find out the reason why you get error or warnings. you may find out the reason from log if you only get warning(that time just do step 3) but you have to do some if you get error and sometime you have to retest it which you had error.

  • step 1:Make Summary
  • step 2:Test again one by one
  • step 3:See the Log
  • step 4-A:See the Testcase and Do it by your hand and filling out the reason.
  • step 4-B:Debag with TestTool

Step 1:Make Summary

To get summary,you can use tools/analyze/make_summary.pl and make a summary of test result. and you can explore the test result.

this sample from 3.1.1rc1 release testing. i tested on Fedora 11 with OOo3.1.1rc1 Linux without JRE version.

bash-4.0$ ./make_summary.pl ../../errorlogs/*.res

Filename                      Errors  Warnings  Warnings_during_initialization
============================  ======  ========  ==============================
c_import_general.res               0         0         0
c_updt1.res                        1         1         0
c_updt2.res                        0         0         0
c_xml_scenario.res                 0         0         0
ch2_flexible_source_range_se       2         2         0
ch2_losa_ods_to_ods.res            0         0         0
ch2_losa_ods_to_xls.res            0         0         0
ch2_losa_sxc_to_ods.res            0         0         0
ch2_losa_sxc_to_sdc.res            0         0         0
ch2_lvl1_wizard.res                0         0         0
ch2_updt_calc.res                  1         1         0
ch_xml_japanese_candlestick.       0         0         0
closeoffice.res                    1         0         0
d_export_graphic_a.res             0         0         0
d_export_graphic_b.res             0         0         0
d_opt.res                          0         0         0
d_updt.res                         0         0         0
dba_Main.res                       0         0         0
dba_ReportBuilder.res              0         0         0
dba_frm_FormFilter.res             0         0         0
dba_frm_Forms.res                  1         3         0
dba_misc_RegisterDatabase.re       0         0         0
dba_rpt_DateTime.res               0         0         0
dba_rpt_ExecuteReport.res          0         0         0
dba_rpt_Formating.res              0         0         0
dba_rpt_Grouping.res               0         0         0
dba_rpt_PageNumber.res             0         1         0
dba_rpt_PropertyBrowser.res        0         0         0
dba_xf_Submission.res              0         0         0
dba_xforms_Main.res                0         0         0
e_fileopen.res                     0         0         0
e_location.res                     0         0         0
e_options.res                      0         0         0
e_update.res                       0         0         0
f_basic_dialogs.res                0         0         0
f_basic_issues.res                 0         0         0
f_basic_protected_libraries.       0         0         0
f_basic_shared_modules.res         0         0         0
f_configuration.res                0         2         0
f_extras_labels.res                0         0         0
f_extras_tableautoformat.res       0         0         0
f_extras_templatepreview.res       0         0         0
f_filedlg_cjk_files.res            0         0         0
f_filedlg_reserved_names_uni       0         0         0
f_first.res                        0         0         0
f_help_browser.res                 0         0         0
f_help_compare_applications.       0         0         0
f_help_search.res                  0         0         0
f_printer_administration.res       0         0         0
f_script_organizers.res            0         0         0
f_security_certification_dia       0         0         0
f_security_dialogs.res             0         0         0
f_security_incorrect_passwor       0         0         0
f_security_trusted_path.res        0         0         0
f_standard_toolbar.res             0         0         0
f_tools_customize.res              0         0         0
f_tools_options.res                0         0         0
f_topten.res                       0         0         0

Filename                      Errors  Warnings  Warnings_during_initialization
============================  ======  ========  ==============================
f_window_functions.res             0         0         0
f_wizards.res                      1         1         0
g_autocorrection.res               0         0         0
g_character.res                    0         4         0
g_customshapes.res                 0         0         0
g_export_html.res                  0         0         0
g_group.res                        0         0         0
g_imagemap.res                     0         0         0
g_line.res                         0         0         0
g_mediaplayer.res                  0         0         0
g_ole.res                          0         0         0
g_paragraph.res                    0         0         0
g_print.res                        0         0         0
g_shaddow.res                      0         0         0
g_tools.res                        0         0         0
g_zoom.res                         0         0         0
i_opt.res                          0         0         0
i_slidecopy.res                    0         0         0
i_updt_1.res                       0         0         0
i_updt_2.res                       0         0         0
m_lvl1.res                         0         0         0
m_updt.res                         0         0         0
oasis_graphics_update.res          0         0         0
resetoffice.res                    0         0         0
w_numbering.res                    0         5         0
w_section.res                      2        16         0
w_shortcuts.res                    0         0         0
w_sorting.res                      0         0         0
w_spellcheck.res                   0         5         0
w_updt.res                         0         1         0
writer_xml_7_export.res            0         0         0
==============================================================================
Total 89 files                     9        42         0

step 2:Test again one by one

sometimes error/warning come from just timing or setting so just test again one by one and some of error/warning might be gone.

especially you should test again when you get a lot of error/warning.

  • c_updt1.res
    • error has gone.
    • it was timing error.(navigater Rendering is too slow)
  • ch2_flexible_source_range_se
    • error has gone.
    • it was timing error.(navigater Rendering is too slow)
  • ch2_updt_calc.res
    • error has gone.
    • it was timing error.(navigater Rendering is too slow)
  • dba_frm_Forms.res
    • error has gone.
    • it was timing error.
  • f_wizards.res
    • error has gone.
    • it was timing error.

step 3:See the Log

now you knew which test script made error and warnings. so you should find out the log from result file with tools/analyze/translater_res_file.pl or use testtool.

I recommend you to use testtool because of it make you more easy to trace the error.

bash-4.0$ ./translate_res_file.pl ../../errorlogs/ch2_flexible_source_range_selection.res |grep ^ERROR -B 5
MESSAGE: Close document
TEST_CASE: 
TEST_CASE: tDataRangeDialogWriter
MESSAGE: Open test document gTestToolPath/chart2/required/input/ch_flexrange07.odt
MESSAGE: Select chart using navigator
ERROR: Auswahlliste : Entry number 5 is too large in Select. Max. allowed is 0
--
WARNING: BORDERWINDOW "無題 1 - OpenOffice.org Writer TTD :localhost.localdomain[12479]" geschlossen, RType = BORDERWINDOW, UId = Active
TEST_CASE: 
TEST_CASE: tDataSeriesDialogWriter
MESSAGE: Open test document gTestToolPath/chart2/required/input/ch_flexrange08.odt
MESSAGE: Select chart using navigator
ERROR: Auswahlliste : Entry number 5 is too large in Select. Max. allowed is 0

this log show us

  • testcase [tDataRangeDialogWriter] had a error
  1. Open test document gTestToolPath/chart2/required/input/ch_flexrange07.odt
    1. Success
  2. Select chart using navigator
    1. Fail:this happen may be come up on the Navigator or near and error message help you to get truth.
  • testcase [tDataSeriesDialogWriter] had a error
  1. Open test document gTestToolPath/chart2/required/input/ch_flexrange08.odt
    1. Success
  2. Select chart using navigator
    1. Fail:this happen may be come up on the Navigator or near and error message help you to get truth.


source of this log

bash-4.0$ cat ch2_flexible_source_range_selection.res |grep "tDataRangeDialogWriter\|tDataSeriesDialogWriter" -A 5
1;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;438;27;61;"tDataRangeDialogWriter"
4;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;443;4;65535;"Open test document gTestToolPath/chart2/required/input/ch_flexrange07.odt"
4;~chart2\tools\ch_tools_common.inc;163;4;65535;"Select chart using navigator"
2;~global\tools\includes\required\t_doc2.inc;294;48;65535;"Auswahlliste : %ResId=22021%%Arg1=Select%%Arg2=5%%Arg3=0%"
3;~global\tools\includes\required\t_doc2.inc;294;48;65535;"--/home/goodday/OOO310_m18/testautomation/global/tools/includes/required/t_doc2.inc: fSelectFirstOLE%()"
3;~chart2\tools\ch_tools_common.inc;164;4;65535;"--/home/goodday/OOO310_m18/testautomation/chart2/tools/ch_tools_common.inc: fOpenTestDocumentAndSelectDataRange(Byref sInputFile$)"
3;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;294;48;65535;"--/home/goodday/OOO310_m18/testautomation/chart2/required/includes/ch2_flexible_source_range_selection01.inc: tDataRangeDialogWriter()"
3;~chart2\required\ch2_flexible_source_range_selection.bas;58;12;65535;"/home/goodday/OOO310_m18/testautomation/chart2/required/ch2_flexible_source_range_selection.bas: main()"
4;~global\system\includes\master.inc;561;12;65535;"   **    Error in TestExit -Routine      **"
5;~global\system\includes\master.inc;562;12;65535;"DOCKINGWINDOW "ナビゲータ" geschlossen, RType = DOCKINGWINDOW, UId = ViewNavigator"
5;~global\system\includes\master.inc;562;12;65535;"BORDERWINDOW "無題 1 - OpenOffice.org Writer TTD :localhost.localdomain[12479]" geschlossen, RType = BORDERWINDOW, UId = Active"
1;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;596;222;234;""
1;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;600;28;63;"tDataSeriesDialogWriter"
4;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;609;4;65535;"Open test document gTestToolPath/chart2/required/input/ch_flexrange08.odt"
4;~chart2\tools\ch_tools_common.inc;163;4;65535;"Select chart using navigator"
2;~global\tools\includes\required\t_doc2.inc;294;48;65535;"Auswahlliste : %ResId=22021%%Arg1=Select%%Arg2=5%%Arg3=0%"
3;~global\tools\includes\required\t_doc2.inc;294;48;65535;"--/home/goodday/OOO310_m18/testautomation/global/tools/includes/required/t_doc2.inc: fSelectFirstOLE%()"
3;~chart2\tools\ch_tools_common.inc;164;4;65535;"--/home/goodday/OOO310_m18/testautomation/chart2/tools/ch_tools_common.inc: fOpenTestDocumentAndSelectDataRange(Byref sInputFile$)"
3;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;294;48;65535;"--/home/goodday/OOO310_m18/testautomation/chart2/required/includes/ch2_flexible_source_range_selection01.inc: tDataSeriesDialogWriter()"
3;~chart2\required\ch2_flexible_source_range_selection.bas;59;12;65535;"/home/goodday/OOO310_m18/testautomation/chart2/required/ch2_flexible_source_range_selection.bas: main()"
4;~global\system\includes\master.inc;561;12;65535;"   **    Error in TestExit -Routine      **"
5;~global\system\includes\master.inc;562;12;65535;"DOCKINGWINDOW "ナビゲータ" geschlossen, RType = DOCKINGWINDOW, UId = ViewNavigator"
5;~global\system\includes\master.inc;562;12;65535;"BORDERWINDOW "無題 1 - OpenOffice.org Writer TTD :localhost.localdomain[12479]" geschlossen, RType = BORDERWINDOW, UId = Active"
1;~chart2\required\includes\ch2_flexible_source_range_selection01.inc;784;222;234;""

step 4-A:See the Testcase

search the testscript which have a testcase

you can find out from log or this kind of command.

bash-4.0$grep -f *.inc |grep tDataRangeWizardWriter
ch2_flexible_source_range_selection01.inc:' #1 tDataRangeWizardWriter                 ' Updatetest for tabpage DataRange via chart wizard
ch2_flexible_source_range_selection01.inc:testcase tDataRangeWizardWriter

get test case from script

testscripts are written in OpenOffice.org Basic. so the comment for this langage is [']. if you want to know about basic you can read OpenOffice.org Basic Guide

sometimes the style of writing is deference as this sample. it depend on person who write testscript.

bash-4.0$cat ch2_flexible_source_range_selection01.inc |grep "testcase*\|'"
testcase tDataRangeWizardWriter
    '///Open test document <i>gTestToolPath</i>/chart2/required/input/ch_flexrange05.odt
        '///Choose the Chart type <i>line</i> (should be the fifth in the top-down list)
        '/// Click Next button
        '///Check data range in edit field value (ChartFlexRange.A1:E5)
        '///Check if -Data series in rows- is enabled
        '///Check if -Data series in columns- is enabled and chosen
        '///Check if -First row as label- is enabled and checked
        '///Check if -First column as label- is enabled checked
        '///Enter invalid range ChartFlexRange.A1:E
        '///Check if -Data series in rows- is disabled
        '///Check if -Data series in columns- is disabled
        '///Check if -First row as label- is disabled
        '///Check if -First column as label- is disabled
        '///Check that Back, Next and Finish does not work
        '///Enter valid range ChartFlexRange.A1:E5
        '///Check if -Data series in rows- is enabled
        '///Check if -Data series in columns- has been enabled and chosen
        '///Check if -First row as label- is enabled and checked
        '///Check if -First column as label- is enabled checked
        '///Press Cancel button
        '///Close document
'
'-------------------------------------------------------------------------
'
testcase tDataSeriesWizardWriter
    '///Open test document gTestToolPath/chart2/required/input/ch_flexrange06.odt
        '///Choose the Chart type <i>area</i> (should be the fourth in the top-down list)
        '/// Click Next button
        '/// Click Next button
        '///Check if the data series listbox is enabled and contains four entries
        '///Check if the ADD button is enabled and click the button
        '///Check if there are now five entries in the data series listbox
        '///Check if the new data series was added below the highlighted one
        '///Check if the REMOVE button is enabled and click the button
        '///Check if there are now four entries in the data series listbox
        '/// Select third entry
            '/// The Move Up button should be enabled
            '/// Click the Move Up button twice
        '/// Select second entry
            '/// If Move Down button is enabled
            '/// Click Move Down button twice
        '///Select first entry in Data Ranges
        '///Verify that the range matches the one in edit field
        'GetSelText(line,column) is a new parameter since CWS:SRC680gh13
        'Instead of 1,2 there must be 1,4 used becqause there are two invisible
        'label-columns in that treelist-box        
        '///Select second entry in Data Ranges
        '///Check that the range in the edit field has changed, too
        'GetSelText(line,column) is a new parameter since CWS:SRC680gh13
        'Instead of 1,2 there must be 1,4 used becqause there are two invisible
        'label-columns in that treelist-box        
        '///Change the range in edit field 
        '///Check that the range in Data Ranges has changed, too
        'GetSelText(line,column) is a new parameter since CWS:SRC680gh13
        'Instead of 1,2 there must be 1,4 used becqause there are two invisible
        'label-columns in that treelist-box        
        '/// Set ChartFlexRange.A2:A4 as range for categories
        '///Enter invalid range ChartFlexRange.A2:A
        '///Check that the BACK button is disabled
        '///Enter valid range ChartFlexRange.A2:A5
        '///Press Cancel button
        '///Close document
'
'-------------------------------------------------------------------------
'
testcase tDataRangeDialogWriter
    '///Open test document <i>gTestToolPath</i>/chart2/required/input/ch_flexrange07.odt
        '///Check data range in edit field (ChartFlexRange.A2:A3;ChartFlexRange.B1:E3)
        '///Check if -Data series in columns- is enabled and chosen
        '///Check if -Data series in rows- is enabled
        '///Check if -First row as label- is enabled and checked
        '///Check if -First column as label- is enabled and checked
        '///Enter invalid range ChartFlexRange.A1:E
        '///Check that -Data series in rows- is disabled
        '///Check that -Data series in columns- is disabled
        '///Check if -First row as label- is disabled
        '///Check if -First column as label- is disabled
        '///Check that OK button is disabled
        '///Enter valid range ChartFlexRange.A1:E3
        '///Check if -Data series in columns- is enabled and chosen
        '///Check if -Data series in rows- is enabled
        '///Check if -First row as label- is enabled and checked
        '///Check if -First column as label- is enabled and checked
        '///Press Cancel button
        '///Close document
'
'-------------------------------------------------------------------------
'
testcase tDataSeriesDialogWriter
    '/// Open test document <i>gTestToolPath</i>/chart2/required/input/ch_flexrange08.odt
        '/// Switching to tab page Data Series
        '///Check if the data series listbox is enabled and contains four entries
        '///Check if the ADD button is enabled and click the button
        '///Check if there are now five entries in the data series listbox
        '///Check if the new data series was added below the highlighted one
        '///Check if the REMOVE button is enabled and click the button
        '///Check if there are now four entries in the data series listbox
        '///Use UP button and verify that it it disabled on first entry
        '///Use DOWN button thrice and verify that it is disabled on last entry
        '///Select first entry in Data Ranges
        '///Verify that the range matches the one in edit field
        'GetSelText(line,column) is a new parameter since CWS:SRC680gh13
        'Instead of 1,2 there must be 1,4 used becqause there are two invisible
        'label-columns in that treelist-box        
        '///Select second entry in Data Ranges
        '///Check that the range in the edit field has changed,too
        'GetSelText(line,column) is a new parameter since CWS:SRC680gh13
        'Instead of 1,2 there must be 1,4 used becqause there are two invisible
        'label-columns in that treelist-box        
        '///Change the range in edit field to ChartFlexRange.B2:B3
        '///Check that the range in Data Ranges has changed, too
        'GetSelText(line,column) is a new parameter since CWS:SRC680gh13
        'Instead of 1,2 there must be 1,4 used becqause there are two invisible
        'label-columns in that treelist-box        
        '///Edit range for Categories (ChartFlexRange.A2:A3)
        '///Enter invalid range ChartFlexRange.A2:A
        '///Check that the OK button is disabled
        '///Enter valid range ChartFlexRange.A2:A4
        '///Press Cancel button
        '///Close document
'
'-------------------------------------------------------------------------
'
    '///Menu Insert Object Chart to open chart wizard

Do it by your hand and find out the reason

try this testcase by my hand. sometimes you need to see the testscript. because this comment of testscript does not have detail.

testcase tDataRangeDialogWriter
    '///Open test document <i>gTestToolPath</i>/chart2/required/input/ch_flexrange07.odt
        '///Check data range in edit field (ChartFlexRange.A2:A3;ChartFlexRange.B1:E3)
        '///Check if -Data series in columns- is enabled and chosen
        '///Check if -Data series in rows- is enabled
        '///Check if -First row as label- is enabled and checked
        '///Check if -First column as label- is enabled and checked
        '///Enter invalid range ChartFlexRange.A1:E
        '///Check that -Data series in rows- is disabled
        '///Check that -Data series in columns- is disabled
        '///Check if -First row as label- is disabled
        '///Check if -First column as label- is disabled
        '///Check that OK button is disabled
        '///Enter valid range ChartFlexRange.A1:E3
        '///Check if -Data series in columns- is enabled and chosen
        '///Check if -Data series in rows- is enabled
        '///Check if -First row as label- is enabled and checked
        '///Check if -First column as label- is enabled and checked
        '///Press Cancel button
        '///Close document
'
'-------------------------------------------------------------------------

step 4-B:Debag with TestTool

set the break point and find out the error. sometime it is easy way for get the reason.

  1. open the testtool.
  2. run the testscript.
  3. after that ,you may get error.
  4. click above line from error message
  5. set the breakpoint on the testscript.
  6. run the testscript again.
  7. it will be stop on the break point.
  8. you can try the testscript step by step (short cut key is "F8")
sometimes you can not see the error on debug mode.it is fine.

knowledge

ERROR: Auswahlliste : Entry number 5 is too large in Select. Max. allowed is 0

normally this error`s reason is timing. the script can not access correctly to control if the control rendering soon.

MESSAGE: Select chart using navigator
ERROR: Auswahlliste : Entry number 5 is too large in Select. Max. allowed is 0
  • Try it 1:Just try again this testscript.
  • Try it 2:open the navigater before starting the test.
  • Try it 3:modify the testscript add wait() or sleep() function.(your system might be need some waiting time.)
  • Try it 4:Do the testcase by your hand

ERROR: DocumentConverter : Window/Control could not be accessed.

normally this error come from environment on automation test. the script might be get no Expected window or Control and the script could not access expected window/control.

  • try it 1:Delete all of the work file ,configuration file and retest.
  • try it 2:See the screen saver/power management configuration on system and retest.
  • try it 3:check security software/configuration.

Glossary

  • Auswahlliste(German)
    • shortlist(English)
    • 選択リスト(Japanese)

Any Quotation

please contact qa-dev mailing list.

Personal tools