Release QA Tracking Tool
Original proposal by Andrea Pescetti - please discuss below rather than editing this box.
I will realize a new web-based tool for tracking QA of localized OpenOffice.org builds. It will feature clear, informative and attractive screens for the general public, easy management and updating, IssueZilla integration, notification by e-mail and RSS feeds and easy extensibility.
The tool will help the OOo community during all steps of the QA process for localized builds. The focus will be on having a system, easy to use and maintain, for the QA volunteers and on guaranteeing fast and reliable diffusion of information to the general public.
The tool will automatically maintain a main page, which summarizes the status of every build, and specialized subpages. It will assist in every step of the QA process as follows.
- Localized builds are made available by known providers via the tool
- Links to the (untested) builds appear on the main page
- Interested Native-Lang QA Leads are informed by e-mail and RSS
- A Native-Lang QA Lead notifies beginning of tests to the tool
- The build appears as "In QA" on the main page
- If possible, links to the suitable TCM pages are provided (to authorized users and for supported locales) to track QA progress
- Optionally, a closing date for tests can be specified and shown
Step 3-A (if the build passes tests):
- The QA lead notifies success to the tool
- The build appears as "QA passed" on the main page
- The QA lead, assisted by the tool with a pre-filled IssueZilla template, notifies success and asks for distribution to mirrors
- A notice appears in RSS feeds to inform about approval
- When the distribution issue is closed, the QA lead notifies the tool
- The build appears as "Released" on the main page
- Download link is optionally changed
- E-mail notifications and RSS feeds inform of the release availability
Step 3-B (if the build fails tests):
- The QA lead notifies failure to the tool
- The build appears as "QA Failed" on the main page
- Links to the blocking (global or localization) issues are shown on the page
- A notice appears in RSS feeds
Main page for the general public:
- A table will summarize all information about builds for each language and platform
- Default sorting will be by language (English first, then all languages for which a localized build is available, then all other languages); other sorting options will be available
- Subpages will be available for the single builds; they will contain more verbose descriptions and more detailed information
There will be three types of users:
- ADM, Administrators: add languages/platforms, manage users
- BPR, Build providers: provide localized binaries for testing
- QAL, Native-Lang QA Leads: one or more per language, report test results
ADMs have access to the Administrative interface, providing:
- User management: ADMs can add/modify/remove any user
- Language management: ADMs can add new language name/code pairs
- Platform management: ADMs can add new platforms
- Moreover, an ADM has all BPR and QAL privileges
BPRs have access to the Build submission interface:
- BPRs can inform about builds in progress and set the "Build in progress" status for platform/language pairs
- When builds are ready, BPRs publish them through the tool
- If a new build is submitted and an older one is still in QA for the same language and platform, both builds will be displayed on the main page
- Every time a new English build is submitted, all BPRs are notified through the usual channels (e-mail, RSS feeds)
- The tool supports simultaneous submission of many builds for the same platform via wildcards or heuristics
- Optionally, mirror sites for the same build can be specified
QALs manage the rest of the process:
- QALs can add new QALs for their language, in order to delegate work
- QALs interact with the tool in all steps of the process
Technical details and requirements
- Authentication: If at all possible, users should be able to login via their OpenOffice.org website/IssueZilla account, as it happens in EIS (Environment Information System). This will also make Step 3 (opening distribution release issue) smoother. So "adding a new user to the tool" will actually mean "granting access to the tool to an existing OOo account".
- Technology: The tool will be written in PHP and will operate on a MySQL database.
- License: I would like to make the tool available under the GPL license.
Development will take about 15 weeks. The period will be divided as follows:
- In week 1: Get suggestions (ml/wiki) from people doing OOo 2.0.3 QA
- By week 3: Basic setup; see if common authentication is feasible
- By week 5: Administrative interface completed
- By week 7: Notification system (e-mail, RSS feeds) completed
- By week 10: Interface for Build Providers completed
- By week 13: Pages and interface for QALs completed; public testing
- By week 15: Layout improvements; bug fixing
I would like to know whether the project above suits the needs of the community, based on the experience of people involved in OOo QA. Adjustments to the project can be done if necessary.
If some QA teams would like to have early access to the tool, and to help with testing, they are welcome to contact the developer (see below).
- C: the tool needs to be able to handle a "global" QA state. E.g. the english (base) RC will block all other RC's. If english failes testing, all other RC's will fail as well (unfortunately this happened with 2.0.3RC5)
- Pescetti: It should not be forbidden to complete tests in localized RCs to find problems: the Italian team, for example, completed all tests on 2.0.3RC6 even if 2.0.3RC6 English had already been declared not OK. So the original idea was to just notify Native-Lang teams. Anyway, since no localized builds will be approved if the English build isn't, I will implement this suggestion: when an English RC is rejected, all the corresponding localized builds will be displayed as rejected.
- C: License should be LGPL (although the tool would not be part of the OOo Source Code, we should go in line with the project's licenses)
- Pescetti: OK, both GPL and LGPL are acceptable to me.
- C: To make the tool aware of new builds some automatizm should be introduced. E.g. a post to a specific URL would notice the tool about a new directory for builds, that would be scanned automaticaly. Filenames should be analysed, so that the tools detects platform, version, OS itself. (The best way was to have "API"/Webservice like access to the tool for that, but this might conflict with the user authentification)
- Pescetti: Yes, that was the idea. I know that there is a standard for filenames and I will look up documentation for it. Then BPRs will be able to provide a directory and give hints (i.e., "scan for Linux builds, languages: de, it, fr", or "find all files named OOo_2.0.3rc7_060622_LinuxIntel_install_LG_wJRE.tar.gz") or have the tool just perform a bunch of requests and add whatever standard-named build it finds.
- C: the tool needs to be aware, that multiple builds for the same platform / version at the same time (e.g. 2.0.3RC7 Linux/RPM, built by Sun and build by Pavel). The tool should be able to list both builds.
- Pescetti: Interesting. This is feasible. Actually, if I recall correctly, once some Pavel's builds had to be rejected due to a problem not appearing in the official RCs, so a separate QA process makes sense.
- C: language Packs should be listed by the tool as well
- Q: The stages seem good. However, where are the actual tools or checklists for step 2 ?
- Pescetti: Well, Step 2 is just beginning of tests. The testing process will be handled by the TCM. So in Step 2 any QAL whose team is ready to start tests will just login and change the build status to "In QA". It will be possible to specify other details, such as closing date for the tests and, maybe, links to the relevant TCM pages. The tool will take care of this.
- C: If multiple builds per platform / version exists, the Native-Lang QA Lead must be able to identify, what build is in test
- Pescetti: OK. If builds are different, QA on them is handled separately.
- C: TCM has the ability to generate reports, that can be accessed from Outside. QA lead should be able to enter the URL to the report.
- Pescetti: Yes, if reports are accessible with no authentication it will be possible to enter an URL to the TCM reports.
- C: QA leads should be able to change the state for a group of builds at the same time. E.g. Linux, Windows (with and w/o) Java are approved at the same time -> this should only result in one Issue for dsitribution
- Pescetti: OK. Actually, issue filing will require some interoperability with IssueZilla. This is one of the points where it would be easier to share access with IssueZilla. There are several workarounds, like creating a qatrack IssueZilla account or have the user login to IssueZilla before the issue is filed.
How to follow development
A first announcement was sent to the dev@qa mailing list on June 19.
A Subversion repository has been created.
A development blog is available.
A web page that will soon contain mock-ups of the tool in development (not a usable system, just the layout) is available.
- Find out if a common authentication is feasible.
Contacts and Acknowledgements
- Andrea Pescetti (pescetti -at- math.unipr.it)
- André Schnabel (Andre.Schnabel -at- gmx.net)
- Mailing list
- The relevant mailing list is dev@qa and all announcements will be sent on that list. You can browse the archives.
- This project is supported as an OpenOffice.org Internship.