RISE measuring success

As we reach the end of the RISE project it’s a good time to reflect back on the success of the project.  At the start we said that we were going to measure the success in several specific ways (shown in the table below).  So how have we done?

  How measured What success looks like 
User response Survey and informal feedback from students and academics. Analytics data. Majority of users agree that recommendations are useful and enhanced their use of the search system. Analytics shows positive impact.
Take-up of tools and data Usage of tools and data, downloads of tools and data. Tools are being downloaded several times a week and there are some comments about the tools.
Community feedback Feedback. Wider discussions with community about potential of tools & ways to use the data.

User response
We think this has been amongst the strongest part of the project.  So we’ve had engagement with users through a survey and through a series of 1:1 evaluations with users.  As you can see from the graph of survey results, the majority of users are finding that recommendations are useful. RISE Are recommendations useful graph

 When we asked people about the relevance of the recommendations then we found that a high proportion were relevant (50%) with 31% not relevant. That may reflect that the system had been running for only a short period of time and may benefit from more data.

RISE Survey How relevant are the recommendations graph

We’ve setup Google Analytics to be able to track which types of recommendation and which number recommendation is being used.   We’ve done some basic work in looking at the analytics data but there is much more that could be done.  The data shows that search recommendations are more likely to be used than other types (but the caveat with RISE is that not all recommendations are being shown equally)

RISE recommendations analytics results

Although in comments users have suggested that we show more recommendations, analytics clearly shows that the first two recommendations are much more likely to be viewed than any others.Which RISE recommendations are used?

Take up of tools and data
We haven’t been able to release any data but both the RISE web interface and RISE Google Gadget have been available for a few months.  Usage of the tools shows a steady stream of users even though we haven’t done too much promotion of it given the prototype nature.  With over 11,000 page views (12% of them through the Gadget) we have reached a good number of users in a short period of time.

RISE interface usage graph

Downloads and use of the Gadget hasn’t been so easy to track even though there is a Google Gadget Dashboard.  We haven’t however had any comments or ratings by users.  We are expecting to publish the Gadget on the OU Gadget directory in the near future so this will drive the uptake of the Gadget significantly.

RISE Google Gadget dashboard

Community feedback
We’ve had a little over 20 comments on blog posts and some feedback at Activity Data events.  We’ve also had 25 people at the Innovations in Activity Data for Academic Libraries event at the Open University in July.  RISE was also asked to present at a ‘Subscribed Resources’ workshop that formed part of the SCONUL Shared Services programme.

Overall the RISE project blog has had just under a thousand visits and nearly two thousand page views from users in 32 countries.  Much of the traffic is coming via google.  The most popular posts/pages have been Innovations in Activity Data, the Technical Resources page and the February project update.

Google Analytics dashboard for RISE project blog

One of the advantages with the Activity Data projects is that we have had the Synthesis project http://www.activitydata.org/ actively working alongside us.  We’ve also had to leave until later in the project some of the dissemination activities.  But it has seemed difficult to get as much engagement with the wider community as we would have liked.

Overall
We’re happy with the engagement with users, something that is often difficult to achieve bearing in mind that we are a distance-learning instituion.  We probably would have hoped for more engagement with the community but many of the people who are working in this area are already pretty busy with other projects on activity data.  But overall, within the constraints of a six month project we are reasonably satisfied with what we’ve been able to do.

Posted in Benefits, Wins and fails (lessons along the way) | Tagged , , , , , | 1 Comment

EZProxy and activity data

EZProxy pros and cons
What RISE has demonstrated to us is that using proxy server logfiles from EZProxy RISE EZProxy log record exampleas the source of your recommendations has some major limitations (in comparison with OpenURL data at least).  In part this is due to limitations in the data that is being handled, but particularly in the way that we, at the OU, are using EZProxy.

The first limitation relates to how we use EZProxy and particularly how we use it now we have implemented the Ebsco Discovery Solution.  At the Open University most of our students use our services off-campus, so we push every electronic resource we can through EZProxy.  So when we came to define the project EZProxy seemed like a good place to draw our recommendations from as it saw the greatest coverage of our overall traffic.

Now, at the time when we defined the project we were using a federated search system and just swopping to a discovery system.  With federated search each of the search targets appeared individually within the EZProxy logfiles with their own URLs so an analysis of the logfiles would show which search target was supplying your content.   But, when we switched over to the discovery solution we decided that we would put that through EZProxy.  So most of our searches now go to EBSCO and that pulls the full text of the article from the content supplier.  Consequently as far as our EZProxy logfile is concerned all it sees is a search to EBSCO not to the final content provider.

As far as recommendations are concerned that isn’t a major issue but it does mean that analysing the logfiles to find out useful usage data may not work for us (so we need to test it to be sure).

EZProxy and article level metadata
On the plus side having the Ebsco Discovery Solution API has meant that we are at least able to do something that addresses a major limitation of the EZProxy logfile data.  Generally there is very little blibliographic metadata within the logfile (certainly in comparison with OpenURL logfiles).  To be able to display sensible recommendations you do need to be able to show some descriptive element to help users understand what is being recommended.  As a minimum you would want to show an article title and ideally you would want to show a journal title, date and maybe authors and a DOI.

RISE recommendations textYour EZProxy logfile data already has a URL you can use to link to the content but some form of bibliographic description is essential as otherwise users cannot choose which recommendations are relevant.

Now to be able to display an article title for your recommendations if you don’t have that data in your original logfile requires you to do some post-processing.  In the case of RISE, because the majority of our logfile data relates to EBSCO then we can use the Ebsco Discovery Solution API to retrieve some basic metadata about the article, such as the DOI or article title.

But this starts to raise some complications, especially if your end-game is to be able to openly release your search data (more later).   Under our license terms we aren’t permitted to store that data within the RISE database.  Now theoretically we already have an internal record ID so we could technically pull the article title in real-time using the API and display it within the RISE interface.  However with API response times typically being 3-4 seconds it isn’t practicable to send up to a dozen API calls just to populate a single page of recommendations and results.

So we’ve ended up at the moment with using the EDS metadata as a key to retrieve data from Crossref that we are licensed to store locally.  Fortunately we have found quite a high overlap between the data sources so have been able to get data for most of our recommendations.  So article level metadata, where you can get it from and what you can do with it, seems to be a major issue.

Open article level metadata
There does however seem to be some differences of opinion between providers of article level metadata (although in the case of aggregators it may be that they themselves are actually licensing it rather than creating it) and Rights and Legal experts over exactly what you can and cannot do with article level metadata.  Whether as essentially a statement of fact it is possible to restrict what can be done with this data and whether extracting selected data into another database is allowable or not.

Certainly for RISE it brings in added complications.  We’ve pretty much run out of time to do too much more.  We can think of a couple of alternative approaches using OpenURL data from EDINA or data from Mendeley that might allow us to match data to the RISE recommendations in a way that would allow the full dataset to be openly released. But realistically that may not be able to be achieved by the time the project ends this month.  At the moment we are left with potentially being able to release the EZProxy data without bibliographic data and that may be of limited value.  But we will get as far as we can.

Posted in Licensing & reuse of software and data, Open data, Technical and Standards, Wins and fails (lessons along the way) | Tagged , , , , | Leave a comment

Benefits – what we expected and what we’ve been able to achieve

For some reason we didn’t manage to finish off our blog post on the intended benefits of the RISE project.  Like the Users post we thought it might be more useful to do it towards the end of the project so we could compare what we hoped we would achieve with what we actually achieved.

Before we started the project the expected benefits were:

Benefits for OU
The project links to the Open University’s strategic priority Focus Area 2 Learning and Teaching Efficiency by improving the search experience for users and developing tools that can be used across multiple platforms.  Within the Library the project links to Strategic Priorities to ‘improve the search experience across all e-collections by implementing intuitive and integrated search’.

By exploiting data that can be recorded by the existing EZProxy system the OU can start to explore whether providing recommendations to users of e-resources will increase the use of e-resources, broaden the range of resources being used, help users to find material that they would otherwise not have located and improve the search and discovery experience.

Benefits for wider community
By using the EZProxy system, which is in widespread use across the HE sector and worldwide, as the source for recommendations data the project will ensure that there is increased value to the community as users of that system will have access to a ready-made toolkit to allow them to exploit this data.

With over 100,000 annual unique users of e-resources the OU e-resource search data will provide a large pool of openly accessible attention data about the use of e-resources, this is likely to be of high value to other institutions planning recommendation services, of use to any national/regional scale initiatives, and as the MOSAIC developer competition showed, of value to the wider community in discovering new and innovative ways to use the data.

The attention data of e-resource searches may be of interest to support the development of Shared Services around the management of e-resources, e.g. as part of the SCONUL Shared Services initiative. The reports of the processes and issues will be of value to those in the community seeking to follow this route.

The Google Gadget developed will be freely available to be adapted by other institutions to access their own search systems.

So how far have we been able to get in realising these benefits?
We’ve been able to develop both a search interface and a Google Gadget.  The search gadget will be taken up by the OU alongside the other Google Gadgets created by the DOULS project in a student dashboard of tools.  RISE has certainly allowed us to do much more work in evaluating the place of recommendations as a tool to support the use of search.

We haven’t yet been able to judge whether recommendations increases the use of e-resources, but comments in the evaluations would seem to indicate that users are getting some resources they would not otherwise have found.  In general users like the idea of having search recommendations.

We have had some interest from other EZProxy users in using the tools and will be making the code available before the end of the project.  We have built a configuration tool to allow users to specify the format of their logfiles when they setup the code.

Although RISE hasn’t yet been able to release any search data, RISE has taken part in and presented at a workshop on Subscribed Resources run with JISC and the SCONUL Shared Services initiative.

The Google Gadget has been created and released.  It is available here http://www.google.com/ig/directory?type=gadgets&url=library.open.ac.uk/rise/google_gadget/risesearch.xml

In a short period of time we think we achieved much of what we set out to do.  We certainly know a lot more about recommendations and what people want, how they can be used to improve the user experience of discovery systems and what the challenges are around EZProxy data.

Posted in Benefits | Tagged , , , | Leave a comment

RISE EZProxy parser step by step

We thought that it might be useful to set out step by step how we get from an EZProxy logfile entry to a set of bibliographic data that we can use in the display of our recommendations.  As a minimum when you make a recommendation users will want to see the article title so they can judge whether it is relevant.  The parser we use to process the daily EZProxy logfiles carries out the following steps:

  1. Extract the Ebsco accession number from the EZProxy url
    We are able to do this because we push access to our Ebsco Discovery Solution (EDS) through EZProxy.  Consequently most of the records in the EZProxy logfile will be Ebsco urls.
  2. Use the Ebsco accession number as a key to obtain bibliographic data from the EDS API.  
    We query the Ebsco API with the Ebsco accession number and look for a DOI, ISSN, Volume number, Issue number and start page.  We aren’t allowed to store this in the RISE database so we then have to obtain some article level metadata from a source that allows us to store bibliographic data within the RISE database.
  3. Use the Ebsco data, ideally the DOI but if there is no DOI then use the ISSN, Volume number, Issue number and start page to query Crossref. 
    At this stage we are trying to obtain a match for an article from the Crossref database so we can retrieve some bibliographic metadata that we can store in the RISE database.  Ideally we want to match against the DOI but if we can’t then we look for other combinations of data. 
  4. From Crossref retrieve the article title, journal title, ISSN, volume and issue details and start page.
    Once we have some relevant bibliographic data then we store them in the RISE database along with the DOI, if present.

Why are we using Crossref for the bibliographic data?
Crossref’s terms and conditions allow libraries to store the data locally,  ‘the Library may cache the DOIs and metadata and incorporate DOIs and metadata into their content and library systems’ http://www.crossref.org/03libraries/33library_agreement.html  Unfortunately our understanding of Crossref’s terms are that they would prevent the data that is derived from Crossref being openly released. 

What other approaches could be adopted?
It may well be possible to adopt other approaches.  Two spring to mind.  EDINA have recently openly released a set of OpenURL data and it would be interesting to try to match RISE content against that dataset.  Another alternative would be to use the Mendeley API to do a similar exercise.  It would be interesting to see which might give the best result.  In both cases the bibliographic data is openly available so would mean that a RISE dataset could be released that could include some bibliographic data

Posted in Technical and Standards | Tagged , , , , , | 1 Comment

June update

Activities during June and early July
A bit later than expected owing to a week spent largely talking to people about RISE, Activity Data and Shared Services.

June and early July has been spent mainly in finishing off the evaluation sessions and in planning for and delivering the RISE Innovations in Activity Data event  We are in the process of writing up the evaluations and they will form part of the Users post for the Project. 

Innovations in Activity Data for Academic Libraries was conceived as a small event that would give a chance for librarians to hear about the library-related projects in the programme strand and to think about how they might be able to use activity data themselves.  In the end we had people from six other Universities (not including those at institutions running other Activity Data projects).   It was quite a lot of effort to get everything together for even a small event but everything went well on the day, even the online presentation and we coped with the various challenges thrown at us.  Thanks to everyone who attended, presented or helped with the day.

Activity Data programme meeting
Last week the RISE project also went to the Activity Data programme meeting, again in Milton Keynes on 5 July.  The RISE presentation for this event is available here on Slideshare. 

The programme meeting included both the Activity Data and Business Intelligence projects and it was particularly interesting to see the BI projects and their different focus.  Seeing some of the work those projects have just started with dashboards, visualisations and work around looking at measuring student success (or lack of success) was really valuable for us to see. The Activity Data programme manager Andy McGregor has blogged about the programme meeting here.

Posted in Benefits, Update, Wins and fails (lessons along the way) | Leave a comment

Presentations and comments on the Innovations in Activity Data in Academic Libraries event

Innovations in Activity Data
RISE event photoMonday 4th July saw the RISE team running a small activity data event on campus at the OU. Aimed specifically at academic libraries, the event, attended by around 25 people was the opportunity for people to hear some of the latest work from the JISC Activity Data programme with presentations from three of the library-related projects and an overview of both the programme and day from the Activity Data Synthesis project.  It was also a chance to think about some of the potenial and challenges of activity data in a world cafe-type event, and to have their horizons expanded by hearing about data visualisation tools and techniques.

Presentations
A few of the presentations are available online and we will link to them from here

What are the challenges around activity data in libraries?
World Café style workshop exercise
As part of the workshop we ran a world-cafe style exercise to get delegates to think about some of the practical aspects of activity.  We covered three aspects:

  • What data?, How much?, Where is it?, How do you get at it?
  • What to do with it?
  • What are the challenges?

If you aren’t familar with this style of activity it’s an informal exercise where participants write their thoughts onto a tablecloth.  The idea is that there is a topic under discussion at eaRISE event world cafe tableclothch table and people walk around from table to table talking to people at the table and writing their thoughts about the issue onto the tablecloth.  Hopefully the comments written on the tablecloth encourage people to add their own thoughts that might confirm or dispute the comments made.  To help move things along we used facilitators at each table to encourage people to write their thoughts down. 

Over the next few days we will be writing up the comments and make them available through this blog.

Comments
If you were at the event and want to comment or blog about it we are happy to link to your thoughts from here.  First off the mark is Paul Stainthorp from Lincoln here.  Thanks from the RISE team to everyone who presented, helped out or came along on the day.  We hoped you had a great time.

Posted in Innovations in Activity Data | Tagged , , , , , , , | Leave a comment

RISE Presentation from Innovations in Activity Data for Academic Libraries event

The RISE presentation from the Innovations in Activity Data for Academic Libraries event at the Open University in Milton Keynes on Monday 4th July 2011 is now up on Slideshare here

or can be downloaded from this blog RISE presentation for workshop 2011-07-04

Posted in Innovations in Activity Data | Tagged , , | 1 Comment

Innovations in Activity Data workshop 4 July 2011 The Open University, Milton Keynes

Innovations in Activity Data workshop

Outline
A one-day workshop aimed at Higher Education library services who are interested in practical applications of activity data, what can be collected, how it can be used, visualised and presented.  The workshop will be an opportunity to hear from library projects working on the JISC Activity Data programme and from practitioners working in this area.

Location
Christodoulou meeting rooms, The Open University, Walton Hall, Milton Keynes

Date
4 July 2011

Cost
Free to attend.  Refreshments will be provided.

Programme

9.45am                 Registration

10.15am               Welcome and Introduction
Nicky Whitsed, Director of Library Services, The Open University

What activity data can you use?  Examples from JISC Activity Data programme

10.30am SALT project ‘Surfacing the Academic Long Tail ‘– MIMAS Joy Palmer, Janine Rigby

11.10am             Coffee break

11:30am RISE project ‘Recommendations Improve the Search Experience’ – Open University, Richard Nurse

12.10am               LIDP project ‘Library Impact Data Project’ – University of Huddersfield, David Pattern (via video)

12.50pm               Lunch break

How can you use the data?

1.50pm                 What are the challenges around activity data in libraries?

World Café style workshop exercise

What data?, How much?, Where is it?, How do you get at it?
– What to do with it?
– What are the challenges?

2.40pm                 How can you visualize activity data? Tony Hirst, Lecturer, Department of Communication and Systems, Open University

3.10pm                 Tea break

3.30pm                 Wrap-up session – JISC Activity Data Synthesis project, David Kay, Sero Consulting

4.00pm                 Close

Who should attend? Librarians, leaders and managers, practitioners, developers and advocates from academic libraries,  who want to understand the potential of activity data to shape, guide and improve services, to inform users and to deliver innovative new services.

Register by email to: RISE-Project@open.ac.uk

Posted in Innovations in Activity Data | Tagged , | 2 Comments

Presentation from JISC Activity Data Online event 2 June 2011

Posted in Uncategorized | Tagged , | Leave a comment

May update

After the flurry of technical activity earlier in the project May has been a quieter month that we’ve spent mainly arranging the evaluation work that starts in June, and looking at some of the early feedback from the on-going user survey.  

User evaluation work
Any research with students at the OU has to be approved by an ‘ethics’ committee, the Student Research Project Panel.  So we complete a fairly lengthly template that outlines the research we plan to do, who we will involve, how we will go about the research and what Data Protection processes we have in place.  That goes off to the panel for assessment and all being well you get approval for your evaluation activity. 

At the OU, apart from dealing with the ethical basis of the research the process also acts to regularise the contacts with students so they aren’t deluged with requests and emails.  As a distance learning institution a lot of contact with students is by email so it’s important that students can control the amount of material that is sent to them as the pace of study can be intensive.  So students can opt-in to being available for research of this type. 

Once the project is approved then we get sent a list of contact details for the students we are allowed to contact to take part in the evaluation.  For RISE we’ve had quite a good response and have been able to arrange the first few one to one interviews starting tomorrow.  We’ve also had people saying that they are interested in checking out MyRecommendations online and will complete the feedback. 

Feedback so far
When we setup the RISE interface we added a feedback link to a survey using SurveyMonkey  This has allowed us to collected some user responses more immediately. 

RISE feedback People on your course viewed

So we’ve asked questions about each of the different types of recommendations that we are providing to get people to tell us how useful they are. 

For course recommendations i.e. ‘People on your course viewed’   more than 40% saw them as Very or Quite useful.  It should be noted that if you aren’t on a course you don’t get any course recommendations so that should account for the 33% who said ‘Not applicable’.  Course recommendations are based largely on the EZProxy logfiles so have the largest amount of data to draw on. 

RISE feedback These resources may be related to others you've viewed 

The second type of recommendation, which tries to relate articles you’ve viewed with similar articles by postulating that there is a relationship between articles that a user views sequentually, shows that 50% thought them to be Very or Quite Useful, but with a larger number seeing them as not useful.  There does seem to be a ‘marmite’ effect where recommendations are either relevant or not.  That could be down to the quantity of recommendations data as RISE currently relies on data collected since the interface went live. 

RISE feedback People using similar search terms often viewedThe third type of recommendation relates to the search terms that are used and the articles viewed.  Agian 50% saw these are Very or Quite useful, but a smaller percentage saw them as Not useful.  Again these recommendations are being powered by search terms entered into the RISE interface as we don’t have the search terms used for the EZProxy data. 

RISE feedback How relevant were the recommendations 

  

The final question we asked was to try to understand a bit more about the relevance and quality of the results.   Here there was a much more definite Not Relevant response at 42% but 50% saw the results are Very or Quite relevant. So again a bit of a ‘marmite’ response that bears more detailed investigation to undestand why.

EDINA OpenURL data openly released
A few days ago came the great news that EDINA have released their OpenURL data http://openurl.ac.uk/doc/data/data.html  So we’ve been having a look at the data to see how it could help us with RISE recommendations.  The size of the dataset at nearly 300,000 rows is larger than we have with RISE and although there aren’t any search terms included we think there are ways that we can use it with RISE so have scheduled some time to setup a RISE parser to ingest the data and test it later this month.  A great example to us all though and it will be interesting to see what can be done with the data.

Posted in Update, Users | Tagged , | Leave a comment