Skip to Main Content

CSCU Assessment Toolkit

This Toolkit was developed by the CSCU Library Consortium’s Assessment Team in response to their charge of recommending best practices of implementing assessment data and analysis toward student success.

Introduction

These "Usage Data Collection Best Practices" were developed by the consortium's Acquisitions and Eresources Team. They offer a guide on effective methods for collecting and analyzing usage data. This resource is designed to support library staff in making informed decisions about resource management and assessment.

What the data can be used for:

  • acquisitions decisions
  • internal dashboards
  • external reports (IPEDS, ACRL)
  • resource promotion/outreach

Data Collection

Method: how do we get it

  • SUSHI/COUNTER: Libraries use the SUSHI (Standardized Usage Statistics Harvesting Initiative) protocol and COUNTER code of practice to ensure standardization across manual and automatic usage data collection, which allows us to make apples to apples comparisons in collection assessment.
  • How to download COUNTER reports:
    • Option 1: Download COUNTER 5 reports directly from vendor platform. See instructions on individual vendor sites or reach out to representatives for assistance.
    • Option 2: Use Alma’s SUSHI harvester to automatically harvest reports. COUNTER 4 and 5 harvesting is available. COUNTER 5 became available in 2019, so many vendors will not have R5 data prior to then.
      • Setting up SUSHI Harvesting in Alma
        • Check this list of vendors who offer SUSHI harvesting and make note of any special instructions:
        • Log in to vendor admin portal to locate SUSHI credentials or contact vendor rep.
        • In Alma, view the vendor record and select the “Usage tab.”
        • Click Add Sushi Account, and select R5
        • Fill in all information retrieved from the vendor admin platform (credentials in vendor record)
        • Click Add Report Types and select all needed reports
        • Click Test Connection and view the generated JSON file in Excel
        • The text should say: "Service_Active":true,"
        • If it returns "false" there is something wrong and you need to troubleshoot the supplied information. Try removing any non-required info, check the API key, etc. If it still doesn't work, contact the vendor for assistance.
  • Alma fulfillment/scanning for INH
    • In Alma, select the "choose location" icon to the left of the search bar
    • In the "I'm physically at" drop-down menu, select Checkout Desk
    • In the left side bar, navigate to Acquisitions > Post-Receiving Processing > Scan In Items
    • For "automatically print slip" select "No"
    • Make sure "register in house use" is checked
    • Click into the text box for "Scan Item Barcode" and make sure the cursor is blinking
    • Either scan or type the barcode into the field.  If your scanner doesn't automatically enter the date, press the enter key to submit the barcode.
    • Repeat until all the volumes are listed. When finished, click "Exit" in the upper right corner. Remember to set the location back to whatever you normally use.
  • Alma Analytics
    • E-Inventory subject area: only resources in COUNTER report AND repository
      • Can’t make report for cost/usage by calendar date
      • Uses TR_J1 Unique Title Requests for usage
        • Total_Item_Requests: Counts all article full‐content views across all formats like HTML and PDF. This is equivalent to the Release 4 usage count.
        • Unique_Item_Requests: Counts unique article full‐content views in a given session regardless of format. If a user views an article PDF and HTML in the same session this would only count as 1. This is the metric COUNTER wants librarians to evaluate.
    • Usage Data subject area: everything from every COUNTER report
  • Vendor platform reports: Use your administrator credentials to login to vendor platforms and download reports as available.  To set up an administrator account, reach out to your vendor representative.
    • Download COUNTER reports if available. Some vendors are not COUNTER compliant and offer alternative reports or don’t track usage at all.
    • Manually load reports downloaded from vendor platforms into Alma so that the data is available in Analytics. See Ex Libris instructions for Manually Uploading and Deleting COUNTER Data.
    • Access SUSHI credentials to set up automatic SUSHI harvesting in Alma
  • What formats should/can be assessed + associated reports
    • E-resources (journals, books, databases, streaming film): Use E-Inventory category in
    • Ill requests: Access the Journals Borrowed Report from WorldCat Collection Dashboard to identify highly requested journals: https://help.oclc.org/Librarian_Toolbox/OCLC_Usage_Statistics/140Assessment_Tools
    • Turnaways: Use the COUNTER 5 report Journal Access Denied (TR_J2) – Reports on Access Denied activity for content where users were denied access because their institution did not have a license for the title.
    • Print materials (circulation/in house): See scanning for INH use instructions above. Track non-barcoded items manually.

Data Management

Storage/sharing--Make it easy for others to pick up the trail!

  • Usage data is part of the library’s history of activity, and thus may have a long life. While internal uses may have a short cycle, for example, annual reviews of resources from a budgetary perspective, external uses may have a much longer cycle, for example, reports for 10-year institutional reaccreditation processes by NECHE and specialized program accreditors. The staff referring to the usage reports may not be the staff who originally generated them, and may only use such reports periodically. We should make it as easy as possible for others to pick up the trail and understand the data.
  • Usage data should be maintained in a shared location, with appropriate permissions for the staff who may need to refer to it or update it in the future. Data maintained on a personal drive may become inaccessible if staffing changes. Wherever the data is maintained, it should be in a location that is regularly backed up.
  • For ease of use across time, folder and file names should include date information. For example, the folder level might refer to the academic year, and the file name might include the academic year or date that the report was generated. Including this information in the file name avoids the problem of multiple files with the same name should the folder structure become messed up.
  • Files should be maintained in a format that supports long term use, and that is usable without specific expertise. PDF ensures long-term access, while Excel is a commonly-used tool that will allow later users to report and query without needing to rekey or convert the data, reducing the chance of errors.

Documentation - Capture your criteria!

 Later users of your data need to know the criteria used. Note the report type (COUNTER 5 or other), the date range, and any other relevant details. If possible, embed this info into the report output itself. If that is not possible, add a separate document to your folder as a read-me or with another descriptive filename like "About Report X.” If you combine or otherwise modify data in Excel, when possible use formulas that can later be viewed by others, rather than keying in calculated data. If a report requires data to be combined from multiple sources, document each of the sources so it can be compiled consistently in the future.

Data Analysis & Review

Cost-per-use (CPU) = cost divided by the number of uses. This metric assigns a financial value to each recorded use of a resource and is helpful in systematically identifying resources that may have a low return on investment. One rule of thumb is to review all resources with a CPU higher than the cost of interlibrary loan, since those represent cases where it would have been more cost effective to request the item through interlibrary loan for each recorded use than to pay for a subscription. Your institution can determine a CPU threshold that makes sense for your community’s patterns of use and resource needs.

CPU is just one aspect to consider in collection assessment.  Since our primary goal is to serve an academic need and CPU is just one way to measure the effectiveness of our service, it is not the only metric that should be considered when making collection development decisions.  If CPU is low, that shows the resource is cost effective, but if CPU is higher than the target threshold, this does not indisputably indicate that a subscription is not of value. Some resources are required for accreditation in certain departments or could be absolutely essential to the teaching and research of a group too small to contribute to a CPU equal to or greater than the desired threshold.  Therefore, these calculations should be used to flag titles for further investigation and discussion with stakeholders rather than signal a straightforward need for cancellation.

Accessibility Statement