Website Usability Comparison
Project Overview
​
In this study this project I have used methodologies as defined by my Professor at Bentley University, Bill Albert, in his book Measuring the User Experience and Microsoft Excel as analysis tool.
​
My task was to analyze the data that my team gathered from a web usability study of two websites about inventors and inventions:
http://www.enchantedlearning.com/inventors/
http://www.invent.org/
​
Each participant in the study was presented with 6 randomly selected and ordered tasks out of a total set of 10 tasks*:
-
Who invented the gas mask?
-
Who invented the safety pin?
-
Who invented Kevlar, the material for bullet-proof vests?
-
Who improved on Whitcomb Judson's earlier work to make a commercially successful zipper?
-
What is Josephine Cochran credited with inventing?
-
What is Richard G. Drew credited with inventing?
-
What is George de Mestral credited with inventing?
-
What is Percy Spencer credited with inventing?
-
What is Helen Blanchard credited with inventing?
-
Who invented the parking meter?
* Please note that if you want to replicate this study, it was conducted a few years ago. The content of these sites may have changed.
​
Task success and task time data were collected. In addition task ease, ease of finding information, visual appeal and SUS ratings were recorded.
Measuring the Usability
​
1. Task Success Analysis
I have calculated the average success rate percentage for each task which can be seen on the chart below. Because the data is in binary format error bars are added to the chart, they need to be calculated using the binominal distribution with the Adjusted Wald Method.
​
We can see that the average success percentages for the Task 9 on the Enchanted website and Tasks 4 & 10 on the Invent website are below 70%. To be improved, these task need to be analyzed further for specific problems.
In addition I have aggregated the task success rate percentages for the both sites. There is 1 participant in Enchanted data and 1 participant in Invent data that completed 0 successful tasks, so data cleanup would not change the results significantly.
​
For the Enchanted website 45 participants have performed 242 task successfully out of 270 total, which accounts for 90% success rate.
For the Invent website 39 participants have performed 189 task successfully out of 232 total, which accounts for an 81% success rate.
Because the confidence intervals overlap, I have used χ2 (chi square) test (see below).
​
The result: CHI-square = 0.008868 is less than 0.05 (95% confidence), that confirms that the participants achieved better success on Enchanted than Invent website.
2. Time-On-Task
Again we see that Task 9 on the Enchanted website and Task 10 on the Invent website are significantly lagging behind the other task times, which confirms the need for further analysis.
Because the confidence intervals overlap on the site wide aggregated data, I have used T-Test test assuming independent samples.
​
The result of 0.53510 signifies that the two means are not significantly different.
3. Task Ease Ratings
I have summarized the Task Ease Ratings in the charts below.
4. "Ease of Finding Information" and "Visual Appeal" ratings
I have summarized the "Ease of Finding Information" and "Visual Appeal" in the charts below.
5. SUS Ratings
6. Summary
In summary, comparing the data for the 2 websites, we can say that for the type of tasks used for the testing, the Enchanted web site performs better in all aspects, except the visual appeal. For the Invent website 4 of the tasks have less than 80% success (relatively low threshold), which is considered low performance and needs improvement. Enchanted - 9 out of 10 tasks are completed successfully; Invent - 8 out of 10 tasks are completed successfully; confirmed by χ2 (chi square) test.
​
It appears that for both sites Task Time and Task Success are in negative correlation, which is normal.
Task 9 on the Enchanted website and Task 10 on the Invent website need further analysis for problems, in order due to low scores in all tests.