Jump to content

GLAM/Indicators of Success

Add links
From Outreach Wiki

This page provides an overview of metrics and indicators of success that can possibly be used to evaluate GLAM-Wiki cooperations. A large part of this information was gathered during a GLAM-Wiki evaluation workshop at Wikimania 2013.

Please be aware that using quantitative methods is meaningful only in the case of relatively large numbers. For example, a comparison of retention statistics for two workshops with 25 participants each is very unlikely to yield a statistically significant result. Furthermore, measuring certain aspects may be more or less costly or more or less difficult. Indicators of success should therefore be chosen carefully on a case by case basis.

 
Indicators Measurement Methods Tools Comments
Number of pictures / media files uploaded Count of the number of files in a given Commons category (before/after) Category page; GLAMorous The number of files is given as a default information on category pages; alternatively, the GLAMorous tool also indicates the number of photos contained in child-categories. Use a dedicated Commons category to mark pictures of interest.
Count of the number of pictures uploaded by a given user / group of users within a given time frame Programs and Events Dashboard Requires user names from partners.
Number of pictures / media files used in Wikipedia articles File count GLAMorous
Number of Wikipedia articles featuring the uploaded pictures / media files Article count GLAMorous, BaGLAMa In order to include a category in BaGLAMa, you need to make a service request on the JIRA tool. As of fall 2013, it appears from discussions on Magnus' talk page that the data used for the analysis is not 100% reliable.
Number of page views of the articles containing the uploaded pictures / media files Page view statistics BaGLAMa 2 Requires a request from Magnus' talk page
Audio/video files: number of visits to the audio/video file Page view statistics ?
Number of webpages (outside the Wikipedia/Wikimedia domain) featuring the uploaded pictures / media files Website count NOT AVAILABLE

Indicators related to article creation and improvement

  
Indicator Measurement Method Tool Comment
Number of (pertinent) articles created Programs and Events Dashboard Apply it across different language versions if required.
Number of (pertinent) articles that were substantially improved Pre-post article quality comparison: quality assessment by the community according to Wikipedia article quality criteria Programs and Events Dashboard Apply it across different language versions if required.
Number of page views of the articles created and/or improved Page views of pages with a particular template Create templates to be applied to the articles of interest or their discussion pages; log/track the templates.
Coverage of a topic that is important to a particular GLAM and/or under-represented in Wikipedia (pre/post) Analysis of coverage of concrete topics/lemmas (before/after); recommend using WikiProject Tracking
Amount of new content added to Wikipedia Amount of content added by a given set of participants during an event or a certain time-frame Programs and Events Dashboard By comparing this indicator to the length of an event or its financial cost, the "efficiency" of the event itself in terms of content production can be measured. User names of participants need to be collected.
 
Indicator Measurement Method Tool Comment
Number of new contributors that remain active over a certain time period Contribution statistics Programs and Events Dashboard, Wikimetrics Make a list of event participants (usernames); generate participants' contribution statistics thanks to Wikimetrics (before/after a given date).
Edit rate of contributors participating in GLAM activities (pre/during/post) Contribution statistics Programs and Events Dashboard, Wikimetrics User names of participants need to be collected
Rate of long-term retention of contributors who participate in GLAM activities or are invited to GLAM activities (compared to other contributors) Contribution statistics Programs and Events Dashboard, Wikimetrics User names of participants need to be collected
Number of contributors who edit articles in a given topic area they would otherwise not have contributed to Survey among participating Wikipedians
 
Indicator Measurement Method Tool Comment
Number of successful interactions between GLAM staff and Wikipedians (e.g. number of feedbacks given and taken into account, number of requested media files that were provided by the institution, number of requested pieces of information that were delivered, etc.) Use a wiki page to register and track requests or feedback by Wikipedians Alternatively: keep a work log on the staff side.
Satisfaction of the GLAM regarding feedback quality Interview(s) with GLAM staff
Satisfaction of the Wikipedia/Wikimedia community regarding the GLAM's feedback quality and/or the GLAM's responsiveness
Number of edits from accounts used by GLAM staff during their working time Edit count Programs and Events Dashboard, Wikimetrics Make sure that GLAM staff avoids COI issues.
Knowledge among GLAM staff with regard to the functioning of Wikipedia and Wikipedia culture (pre/post) Survey among / interviews with participating GLAM personnel
Extent to which GLAM staff know about "free" copyright licenses and licensing requirements of Wikipedia/Wikimedia Interviews, observation
Knowledge among participating Wikipedians regarding internal processes of the GLAM (pre/post) Survey among participating Wikipedians
Extent of the GLAM's (senior) management support Project reporting / documentation
Free copyright licenses applied to content on GLAM's website Inspection of the partner's website
 
Indicator Measurement Method Tool Comment
Number of people participating in a particular event Head count Count people at a particular moment or based on a participants' list.
Ratio of active contributors among event participants Compare contribution statistics against a head count Programs and Events Dashboard, Wikimetrics This is most easily done based on a participants' list containing user names. Depending on what type of contribution is to be measured, other methods may be applied (e.g. self-declaration, inspection of contribution logs etc.)
 
Indicator Measurement Method Tool Comment
Number of media reports mentioning a particular event / the cooperation Media review
Type/scope/depth of media coverage Media review
Number media reports about a particular event / the cooperation mentioning the chapter name (or any other important information) Media review Can be applied to any other important piece of information that one intends to convey to a larger public.
Number of contributions on the GLAM web page or in GLAM social media channels referring to a particular event or to the cooperation as such Inspection of the GLAM web site and social media channels
Interview with / documentation from the GLAM's communications manager
 
Indicator Measurement Method Tool Comment
Number of visits/visitors to the partner GLAM's website (pre/post) Web analytics Web analytics tool of the partner GLAM
Number of visits/visitors to the partner GLAM's physical premises (pre/post) Visitors statistics
Number of page views via QRcode Page view statistics QRpedia Statistics (adapt the path) The QRpedia statistics doesn't seem to allow for a differentiation with regard to a particular QRcode display (if the same article is addressed via QRcodes in different locations, they will show up in the same statistics)
Number of visitors that explain Wikipedia content (e.g. accessible via QRpedia code) to children during their visit to the GLAM Interviews with visitors; observation
 
Indicator Measurement Method Tool Comment
Financial cost Accounting Financial report of the project / Analytical accounting
Amount of donated resources
Number of staff hours spent
Number of volunteer hours spent
 
Indicator Measurement Method Tool Comment
The extent to which the project was further elaborated / expanded after a given instance Project monitoring
Number of new GLAM partnerships initiated thanks to the cooperation or to a particular event (presentation/course/workshop) Monitoring of GLAM-WIKI partnerships
Number of participants becoming organizers of future events Project monitoring
Number of joint applications for external funding Number of applications / number of money gained for the movement This will likely happen many months after the original project / event.
Retrieved from "https://outreach.wikimedia.org/w/index.php?title=GLAM/Indicators_of_Success&oldid=156425"