The hypothesis
Each of the projects in the Activity Data programme strand were asked to establish a hypothesis that we would test throughout the project. For RISE the hypothesis we chose is:
“That recommender systems can enhance the student experience in new generation e-resource discovery services”
This hypothesis was chosen quite carefully for a number of reasons. We’ve only recently implemented our Ebsco Discovery Solution aggregated search system so we are still in an evaluation stage and are really still assessing how students at the OU will get the best out of the new system. We have a particular perspective at the Open University in that the use that students make of our search systems varies widely from course to course. So we will particularly want to look at whether there is variation between the levels of students in their reaction to recommendations.
How do we plan to evaluate the hypothesis?
We are planning to approach the evaluation in three ways.
- By establishing some website metrics to allow us to assess how user behaviour is affected by the recommendations. We expect to build two versions of the search interface, one with recommendations and one without. This will allow us to A/B test the interfaces, so we can track the impact that different options make on users behaviour. Using Google Analytics we will track where users click and where they go.
- We will track the use that users make of the rating feature to see whether there is evidence that it is actively being used.
- We will actively encourage user feedback on the tools, carry out surveys with students and run a short series of focus groups to test the hypothesis.
As part of our evaluation work we will be looking to assess whether there are variations that can be ascribed to course or course level in how useful students find the recommendations to be, and whether there are circumstances where they are not useful. We will also be testing a variety of different types of recommendations and will aim to assess which are found to be most useful.
The evaluation report will detail the activities and results of the work to test the hypothesis and we will look at using Quora to record evidence.
Interesting hypothesis, though a fairly complex one to evaluate as it is so broad. I am very interested in how the evaluation addresses the hypothesis.
Also, with the first method of evaluation how will users either decide or be allocated to the particular interface?
Hi Tom
Yes, you are right it is a fairly broad hypothesis and we probably should have made it more specific as time is short. We are only going to be using one of the new generation discovery solutions as our platform rather than testing several of them. But I think the general principle would hold true regardless of which system you were using. I’ve not seen anything specific to any of the systems that are out there (and we’ve fairly recently gone through an evaluation of four of them) that would suggest that recommendations that might work in one system wouldn’t work with others.
To talk in a bit more detail about the A/B testing. It is possible to handle this within Google’s tools by coding the elements of the page so they are displayed ‘randomly’ to users and the different options are shown the same amount of times. So typically we would show different types of recommendations. So one user might be shown recommendations such as people on your course are looking for this, another might see ‘you might like these similar articles’. This page on google explains a bit more http://www.google.com/support/websiteoptimizer/bin/static.py?page=guide.cs&guide=29619&topic=29621
What we would be looking at is to see how user behaviour changes. Which types of recommendations are looked at most often, how often do users go for the recommendations rather than the basic search results.
We intend to back this up with some more focused testing with groups of users, where we know more about the courses they are studying. What we also have is the recommendations database that we are building and that will give us some evidence of user behaviour which we will be looking at.
Pingback: Search focus groups | RISE