Conference Report
1 October 2004
New York City
On the 1st of October, 2004 the NFAIS sponsored a forum in New York City entitled "Online Usage Statistics: Current Trends and Future Dirctions in Meeting User Needs." Approximately 60 individuals attended, representing vendors and publishers (38%), academic institutions (21% - both LIS programs and libraries), government agencies and libraries (14%), corporate information managers (10%), as well as professional associations, public libraries and other library related service and consulting operations. The following summarizes the content of the individual sessions.
Power point presentations for most speakers are available from the NFAIS web site (clicking on title of presentation will download and open the presentation): http://www.nfais.org/events/event_details.cfm?id=26
"Facts are stubborn but statistics are more pliable." – Mark Twain
Judy Luther - President, Informed Strategies
Tracing History through
ICOLC
ARL
NISO
COUNTER
Purposes for usage statistics
Libraries (To Know, To Show)
Publishers (Marketing, Sales, ID potential)
COUNTER Review
Factors Affecting Usage
- Marketing and access
- content
- interface "Thank god you didn’t have to learn how to cook in order to eat." - JL
Measuring Use
Journals
E-ref
e-books
next generation
Outcomes v. output (performance v. stats)
Tripping over data
Integration with academic software (WebCT and Blackboard)
"No number stands alone."-JL
Dr. David Goodman - Associate Professor, Long Island University)
NISO – library mgt and comparative stats
ICOLC- desired practices for negotiating contracts
COUNTER – usage data for pub and lib
NISO – data coll/display by type (TOC, Abs, FT, Search)
COUNTER– data coll/display by journal (AA, BB, CC)
COUNTER
What we do
- Investigate need
- Definitions
- Data coll protocols
- Define report formats
- Assist w/ compliance
- Certifiy compliance
- Guidelines for audit
- Certify auditors
What we don’t do
- Don’t establish definitions (use NISO and ICOLC, and judy’s paper)
- Don’t collect stats or store (no repository ) – encourage, but don’t do
- Do not report usage
- Don’t audit data (independent auditors – CPAs or equivalents
Why Started
- Publishers wanted to avoid stats per each requirements
- Libs/consort didn’t want to deal with different details
- Inability of self-contained to measure TOTAL use (global data)
- need to demonstrate validity of pub data (definitions)
AUDITING
- principles agreed
- guidelines/test scripts written
- rfps for COUNTER approved auditors read to go out
- took longer to anticipated – complexity revealed as scripts written
ISSUES BEING RESOLVED
- providing toolkit (set of macros in Excel for combining data) **FR**
- role of COUNTER
- role of XML DTD
- develping COP for e-books
- allocating responsibility for indirect providers
- open access journals (who is customer??)
Combining the stats is the responsibility of the library.
ISSUES YET TO BE RESOLVED
- combining stats for html, pdf and ft requests...? (systems are diverse)
- providing aggregated and indiv counts to consortia (difficult for many publishers)
- specifying what a "product" is, for compliance
- finding appropriate tolerance elvels for audits
More about disagreements about how to measure things (differences between products/services)
"Numbers are like people, torture them enough and they’ll tell you anything."
Chris Martire - PALINET
- Challenges increasing b/c products up significantly
- Not all products conform to COUNTER **FR** w/ Chris
- A lot of data manipulation being done for libraries (time+)
- Problems for consortia and large research libraries
WHY USAGE RPT?
- Support Selection Guidelines and Policies
- Usability issues and training
- New areas for collection development
- Offsite print location strategies
- Monitor availability, access and ensure availability of purchased items
- Confirm high use resources
- Print cancellation targets/migration to e
- Gateway measurement (catalogue)
- JUSTIFY EXPENDITURES
CATEGORIES OF USAGE & STATS
- content: per journal to compare titles
- content summaries: across all content, over time
- content summaries: across consortia or groups
- general summaries: web metrics, active IPs, sessions, pg views, registered user, searches
COMING SOON
- COUNTER for e-books and reference
- Stats for multi-media (music, images, video, etc.)
- Evaluation of USAGE is not VALUE
M.E. Brennan - Lucent Technologies
Spinoff from Bell - global telecommunications
Her Organization group: Integrated Information Solutions
- Market analyst rpts
- Comp intel newsbriefs
- Customer comptetior & tech resources
- Information research svcs
- Intelle Prop, content & copyright mgt
- Library Services
"We are currently in a period of transition, which may be interesting, but is rarely entirely pleasant, for those involved." A. J. Meadows, Communications Research
DECISION MAKER CONCERNS
- spiraling costs for materials
- determining value for dollars expended
- driven by cmty
- justifying the expense
- finance people – "it’s on the web, it’s free"
- others – "electronic will be cheaper"
- lack of standardization in reporting
- granularity – as fine as we can get
- emerging concern – identify users/user communities (we use dynamic IP addresses.. )
Dr. Nancy Wright - Defense Technical Information Center
Defense Technical Info Ctr – WWII to track captured German technical reports, defense dept repository
Host web sites – incl DefenseLink and STINet search service
- public Stinet
- private Stinet – defense cmty (stuff that is avaialable US Govt and/or contractors, DOD, etc.)
- introducing "classified StiNet" soon
break-even on all – no profit
WHY COLLECT
No unified requiremnents for reporting usage stats across govt. or in any manner. Most usage stats data and reports prepared as part of compliance with general performance related acts including:
- Govt performance and results act of 1993
- Klinger Cohen Act 1995
- E-Gov act of 2002
Purposes of performance and usage statistics analysis/evaluation:
- Report mission delivery
- Justify programs and costs to agency and customers (taxpayers, Congress)
- Monitor activities to respond to changes in usage
- Assure system security
HOW USED?
- Reports to Leadership
- Monthly to Pentagon
- Briefings on demand
- Budget to congress
- Site management
- Review of specific products
- DTIC – home page
- MCTL – Militarily Critical Technologies List
- Private Stinet
- Public Stinet
- Bandwith availablitiy
- Maintain surge capacity (MOAB example – popular download identified and capacity adjusted)
- Support for browsers
- Security
- Detect intrusion
- Evidence of harvesting (story of “middle of the night” IP user from another hemisphere turned out to be a library school)
- Searchable audit trail
Timothy Curran - Ovid Technologies, Inc.
Use for trend analysis, strategic planning, society reporting, product development, mktg and sales
4 levels of publishers requests:
- FT requests
- Most popular issues/articles
- Content Coverage (departments, reviews, commentary, etc.)
- Other areas
- Advertising
- What is most beneficial.
- Traffic data
- Supplementary content
DELIVERY (consolidated, consistent, integration, make it easy)
FUTURE
- more on usage patterns
- more on purchase patterns
- more on individual users (scary)
"Statistics to a librarian are like a lamppost to a drunk – they are used more for support than for illumination." – source from Seiler
Oliver Pesch - EBSCO Information Services
Activity captured – is it meaningful?
- not all accesses are positive (publisher gets credit even if doc stinks)
- usage and value are relative (journal w/ only 5 downloads not reason to ditch – may be used by professor bringing in big grants)
- we record actions not intentions
user interface can skew results
- EBSCO allows customer to set up automatic FT on cite display – so you’d get FT stats also
- If viewing HTML FT, PDF, e-mails to self, e-mails to 2 colleagues (5 ft, vs. in lib envir. only one)
- how to filter out multiple clicks of impatient users HTML (10 seconds), PDF (30 seconds)
- different sessions if other simultaneous users click in between each other
COUNTER
- Updates on Release 2
- Discussion of audit procedures
- upcoming consideration of e-reference
Beyond COUNTER (discussion of other types of metrics)
Metasearching challenges
RECOMMENDATIONS
- Use off the shelf tools
- Hybrid or custom
- Adopt standards
- Store enough/roll up enough, but not to far (allow for flexibility)
- Capture elements to meet upcoming needs
David Lange - Weil, Gotshal & Manges, LLP
Framework/ "Privacy Principles"
- Notice – tell em
- Choice/consent – give control (More important if personal info is involved, More important for children)
- Integrity/accuracy – of info being collected (reliable sources)
- Security/confidentiality – hot topic for FTC (Microsoft & drug companies that have inadvertently reveals stuff they – cc vs. bc)
- Access/correction – controversial in US, Europe requires that individuals be at the table and have ability to correct information. US companies wary of burdens this could impose if everyone had right to review and change
- Onward transfer – disclosure to third party must be consistent with the terms of original agreement
- Enforcement/redress – self-regulatory bodies on the web, ombudsman, etc.
Federal and State protections
Sector specific (children, health, finance)
International - Europe much more restrictive
RECOMMENDATIONS
Things to share with users:
- id of the entity doing collection
- nature of the data and means
- use it will be put to
- third party
- automatic or voluntary
- can you refuse
- steps for protection from inadvertent
Personal Request
- get input from as many parts of your company/institution
- technology
- marketing/strategy
- hr
- security
- legal
- know answers to all questions above (recommendations)
- draft flexibly, plans/strategies change – if they have to change, then they have advise everyone, or violate their privacy policy
General Tips
- write simple policies
- inform your own company of policies
- checks and balances
- employee training
- take advantage of technology
- review policies to make sure you are compliance
- consequences for violation of policy
"the thing I love about statistics is never having to say you are certain." – seiler for source
Jeff Shim - Florida State University
Comparison of Usage Report Statistics and Report Delivery (In 2003 only 3 compliant, in 2004 now 11 plus Ovid and Wilson yesterday.)
Basic problem with usage statistics from my point of view, is that most vendors provide one report format serving thousands of libraries who have to deal with 101 different reports.
"If you believe that COUNTER will take care of usage statistics once and for all – that’s not true."
COUNTER will not take care of everything. Reports may come in a standardized formats, but you still have to manipulate them.
No baseline data to compare against. Need major scale study to determine or guide librarians to make judgments about acceptable levels of use.
Separate login for usage reports (vs. admin/mgt of subscription) – can be useful, but when you share that info, threatens the security of your statistics.
Correlating usage with cost data is not effective in most cases.
The more information, interface got so complicated that time frame for processing requests is very long
COUNTER not interest in collecting data or building repository. Who will do it?
Ultimately librarians have to do some work. Librarians should have similar meeting like this – "Best Practices for using usage statistics"
Are we going to wait until all dust settles? No, we must start even if data is incomplete.
John Law
"Be careful what you ask for... because it comes with a cost." Complications translate into cost. Obviously not charging premium, but there is a cost and it gets distributed over products.
Complicating Issues for consortia case study:
- usage reporting structure dependent upon segmented user authentication
- authentication constraints, conflicts
- original accounts setup defines structure (usage reporting needs not often well known at that point)
- consortia changes members or constraints
- reporting of roll-up statistics complicated
- size of reports can be excessive
- varying requests for different access arrangements for viewing or privacy
portion of customers using reports: roughly 50-60% (some thought it would have been higher, some thought lower)
next step to advance usage reporting
- COUNTER XML reports for consistency in report formats
- Methods for retrieving reports vary significantly across providers
- Standards needed for consisten automated method for retrieving reports
- Facilititates collection of statistics and enables automated solutions for aggregating and reporting across multiple resources vendors
Repository for usage statistics will not come about without the library community advocating for it – e.g. there is no business model for it.
Patricia Brennan
Internal development process
Discussions
- definitions?
- How data used?
- What if we look bad?
System Decisions
- build or buy?
- How to log data?
- Manage "session" information? (seed model)
- Define and report turnaways?
- Level of data provided – layered or just top-level?
Display and Delivery Considerations
- on the fly reports?
- Formats?
- Export options?
- Frequency of update?
Outcome – Goal/Approach: made usage reporting a "product" worthy of same attention
Features - (review PPT if available), but not much unique.
Platform level –
- number of sessions
- avg session length
- max actual users
- max contractural users
- turnaways
Session data analyzed by
- time of day sessions occur
- sessions as % of total
CHALLENGES
- training and communications – Internal and External
- Data to collect – Internal v. External
- What do libraries need for decision making?
- What do we need to develop our products?
- Data to collect – Nice to know vs. must provide
- Librarians may not need as much info as we originally assumed?
- Usage information needs change as our products and the industry evolves
- Industry Initiatives – complexity of COUNTER compliance
- System differences – logging events v. weblogs
- Display formats – changes in COP requirements
Deborah Loeding
Objectives for Statistics
- Sales Retention
- High usages underscores need
- Low usages alerts sales rep re: training
- UP-sell opportunities, SUs
- Sale of related databases
Case study - turnaway issue
Judy Luther - President, Informed Strategies
Interested to see such a large group this morning and really realize how far along we are. Really reached the level of "reality statistics" – now we are actually able to get into the guts of it and see how complex it is
Interesting to see the vendor examples, and how the planning system went and the complexities with consortia
Usage used to be a librarian responsibility, but with the electronic world that has shifted to the publishers. Impressed with what vendors have done in taking that on. But nice to see that they are actually getting the value out of it for their own use.
We’ve been focusing on the quanitative, but its important to remember the qualitative.
Also appreciative of the people who spoke from the different sectors to get a sense of the variety of data needed and used.
From audience: Interesting to see how as federated searching improves and the development of ontologies, it will be interesting to see how it will affect usage statistics.
Judy – yes, good point. We are now in a position to do benchmarking. Before we make changes we should take these measures so we can track it.
Top of Page
|
This work is licensed under a Creative Commons License
by Caryn L. Anderson.
