Difference between revisions of "AOO UX Research Strategy"
(Adding new content to AOO UX page) |
(Adding new content to AOO UX page) |
||
| Line 8: | Line 8: | ||
The AOO user experience research effort seeks to characterize the current state of the product experience by applying a toolkit of qualitative and quantitative research methods which could range from system logs and instrumentation to surveys and interviews to usability testing and end user engagement. Research activity execution is proportional to community participation and support. While we strive for robust sample sizes, important and valid findings can come from studies with a wide range of participants. User research insight will help us to not only understand existing usability risks but also understand the value of the product offering. Data collected through user research can enable the community to understand the most important user tasks and goals, assess how well products support those goals and having experienced the product, what is the user's overall impression of the product and the brand. A combination of behavioral data and attitudinal data, along with leveraging standardized metrics will enable us to provide a comprehensive grade to our products which can be used as a baseline measure for future research. | The AOO user experience research effort seeks to characterize the current state of the product experience by applying a toolkit of qualitative and quantitative research methods which could range from system logs and instrumentation to surveys and interviews to usability testing and end user engagement. Research activity execution is proportional to community participation and support. While we strive for robust sample sizes, important and valid findings can come from studies with a wide range of participants. User research insight will help us to not only understand existing usability risks but also understand the value of the product offering. Data collected through user research can enable the community to understand the most important user tasks and goals, assess how well products support those goals and having experienced the product, what is the user's overall impression of the product and the brand. A combination of behavioral data and attitudinal data, along with leveraging standardized metrics will enable us to provide a comprehensive grade to our products which can be used as a baseline measure for future research. | ||
| + | |||
| + | ==User Experience Research Data Sources== | ||
| + | |||
| + | Both structured and unstructured research activities are great sources of user experience data. While structured data sources may, at times, be more statistically significant, both unstructured and structured data sources can help us better understand the needs of desires of the Apache OpenOffice user.<br /> | ||
| + | |||
| + | =====Unstructured Data Sources===== | ||
| + | |||
| + | * Ad hoc feedback | ||
| + | * Download demographics | ||
| + | * Social sentiment | ||
| + | ** Email lists | ||
| + | ** Forum comments | ||
| + | ** Social media | ||
| + | |||
| + | =====Structured Data Sources===== | ||
| + | |||
| + | * Contextual inquiry | ||
| + | ** Direct observation | ||
| + | ** Journey mapping | ||
| + | * Interviews | ||
| + | * Focus groups | ||
| + | * Surveys | ||
| + | * User session | ||
==User Experience Metrics== | ==User Experience Metrics== | ||
| Line 24: | Line 47: | ||
The combination of % of Top Tasks meeting user expectations, along with the SUS and NPS scores can provide an overall user experience "'''Health Index'''" for each product along with a targeted set of usability issues that, if fixed, should help move the individual and collective scores higher and result in a better user experience. The Health Index scores can be tracked in regular intervals to measure progress over time. | The combination of % of Top Tasks meeting user expectations, along with the SUS and NPS scores can provide an overall user experience "'''Health Index'''" for each product along with a targeted set of usability issues that, if fixed, should help move the individual and collective scores higher and result in a better user experience. The Health Index scores can be tracked in regular intervals to measure progress over time. | ||
| − | ===Basic Calculation=== | + | =====Basic Calculation===== |
*Health Index = ((TTS) * 33% weighting) + ((SUS) * 33% weighting) + ((NPS) * 33% weighting) | *Health Index = ((TTS) * 33% weighting) + ((SUS) * 33% weighting) + ((NPS) * 33% weighting) | ||
| − | ===Metric Weightings=== | + | =====Metric Weightings===== |
The exact weighting of metrics within a health index are not critical to the index value. What is critical is that longitudinal analysis, comparing one release to another, should be mindful to assess the health index using consistent weightings. | The exact weighting of metrics within a health index are not critical to the index value. What is critical is that longitudinal analysis, comparing one release to another, should be mindful to assess the health index using consistent weightings. | ||
Revision as of 08:16, 22 June 2012
Contents
User Experience Research Strategy
To help make informed design and development decisions, the OpenOffice community needs user data!
The AOO user experience research effort seeks to characterize the current state of the product experience by applying a toolkit of qualitative and quantitative research methods which could range from system logs and instrumentation to surveys and interviews to usability testing and end user engagement. Research activity execution is proportional to community participation and support. While we strive for robust sample sizes, important and valid findings can come from studies with a wide range of participants. User research insight will help us to not only understand existing usability risks but also understand the value of the product offering. Data collected through user research can enable the community to understand the most important user tasks and goals, assess how well products support those goals and having experienced the product, what is the user's overall impression of the product and the brand. A combination of behavioral data and attitudinal data, along with leveraging standardized metrics will enable us to provide a comprehensive grade to our products which can be used as a baseline measure for future research.
User Experience Research Data Sources
Both structured and unstructured research activities are great sources of user experience data. While structured data sources may, at times, be more statistically significant, both unstructured and structured data sources can help us better understand the needs of desires of the Apache OpenOffice user.
Unstructured Data Sources
- Ad hoc feedback
- Download demographics
- Social sentiment
- Email lists
- Forum comments
- Social media
Structured Data Sources
- Contextual inquiry
- Direct observation
- Journey mapping
- Interviews
- Focus groups
- Surveys
- User session
User Experience Metrics
The following metrics can be used to measure and track product health:
- OpenOffice Download Demographics
- Top Task Score (TTS)
- System Usability Scale (SUS)
- Net Promoter Score (NPS)
User Experience Health Index
User satisfaction survey data helps us understand how satisfied users are with our overall offering, as well as specific functional areas within the product. OpenOffice user research focused on identifying the core uses cases and top tasks for each product, validating the prioritization of those tasks with end users and evaluating how well those tasks could be done, will help us identify opportunities for improvement that will have the biggest positive impact. In addition to task performance as a behavioral metric, we should also consider adding two industry standard attitudinal measures. The Usability Scale (SUS) is a standardized set of survey questions that allows us to benchmark our application against other offerings. The Promoter Score which is a measure of how willing someone is to refer our product to a friend. Industry norms exist for SUS and NPS scores from which we can compare OpenOffice products.
The combination of % of Top Tasks meeting user expectations, along with the SUS and NPS scores can provide an overall user experience "Health Index" for each product along with a targeted set of usability issues that, if fixed, should help move the individual and collective scores higher and result in a better user experience. The Health Index scores can be tracked in regular intervals to measure progress over time.
Basic Calculation
- Health Index = ((TTS) * 33% weighting) + ((SUS) * 33% weighting) + ((NPS) * 33% weighting)
Metric Weightings
The exact weighting of metrics within a health index are not critical to the index value. What is critical is that longitudinal analysis, comparing one release to another, should be mindful to assess the health index using consistent weightings.
The OpenOffice weightings could be adjusted to reflect our open source orientation. For example, SUS weightings could be lower given the reduced budget and limited resources under which the OpenOffice operates. NPS weightings could be higher given the viral nature of this product. However, to keep measurement and management simple, it is best to maintain a simple 33% weighting per attribute.
Delivery
While the methodology and delivery of individual research activities is subject to the research context and unique needs of each research activity, consider the following:
- user research is never finished
- any opportunity to capture data can help enhance our understanding
- we should seek to delivery research activities that will generate an aggregated health index snapshot in support of each release
- while download logs provide robust demographic metrics, we should still include a number of standardized demographic questions as part of every user research activity to correlated data
Reporting
To facilitate more effective cross-community communication of user experience research insights, research report availability will be promoted via the community mailing lists. Each report should have a unique wiki page which includes a summary of what user research was done and the person to contact for additional details.
In addition, the aggregated health index snapshot for each release should be captured on a health index dashboard.
