GLAM/OpenGLAM Benchmark Survey/Concept

Note that the webpages related to the OpenGLAM Benchmark Survey are no longer maintained; an archived version is available at Zenodo.

Main page

Concept

Countries

Coordination

Data

FAQ

Implementation

Partners

Discuss

Background

In fall 2012, Bern University of Applied Sciences carried out a pilot survey among heritage institutions in Switzerland focussing on open data and crowdsourcing. The goal was to gain an initial overview of the extent to which cultural heritage institutions in German-speaking Switzerland are adapting to the new trends and also to obtain the information required to be able to develop tailor-made support programmes for heritage institutions as and where needed.

The results of the survey have been distributed among interested GLAMs in Switzerland, presented at various conferences (London, Hong-Kong, Geneva), and published in form of a report in English and German. A summary of the results has also been published in the Swiss professional journal for the librarian and documentalist communities, and an academic paper has been published in the Journal of Theoretical and Applied Electronic Commerce Research.

The survey has been very well received across the board – by GLAM representatives, by scholars, and by open data / free knowledge advocates. Last but not least, the survey has been instrumental in acquiring new cooperation partners both for GLAM-Wiki cooperation projects and for R&D projects. Also, several people from different countries have manifested their interest in adapting the survey for their country. This has lead to the idea of carrying out an international benchmark survey, whereby the same questionnaire would be administered in several countries simultaneously.

Purpose of the OpenGLAM Benchmark Survey

The main purpose of the survey is to:

• Measure the state of advancement of OpenGLAM (digitization, inter-organisational cooperation involving the exchange of metadata, open data, crowdsourcing, linked data) in a given country. Identify the main challenges.
• Inform the GLAM community about the latest developments in the area of OpenGLAM and relate them to the state of advancement of OpenGLAM in their country.
• Identify potential partners for open data and/or crowdsourcing projects (R&D projects, GLAM-Wiki cooperations, open cultural data hackathons, consulting mandates, etc.)
• Use the study report as a communication instrument to promote OpenGLAM within the local GLAM community and related sectors.
• Use the study report as an instrument for lobbying activities in favour of OpenGLAM.

In addition, the international benchmark study provides international comparisons, which:

• Allow each country to see where it stands compared to other countries.
• Provides the international OpenGLAM community with a tool that helps it to better understand the particularities of each country, to put insights gained in a country into a broader perspective, and to better adapt strategies and best practices to the specific situation of each country.

Time Frame / Duration

The project starts in January 2014 with the kick-off meeting and ends in August 2015 with the publication of the various country reports and the international report.

Follow-up activities (e.g. contacting survey participants that have indicated an interest in OpenGLAM) that are expected to be triggered by the project in the various countries may take place beyond the official end of the project.

Project Organization

The survey is carried out in a federative manner: Tasks are as much as possible delegated to national teams, while the international coordination is assured thanks to an international core team and regular coordination meetings.

The following aspects of the project must be coordinated across countries in order to allow for a meaningful comparison of the results:

• Improved and updated version of the questionnaire (must)
• Minimal number of valid responses per country to be included in the international comparison (must)
• Timeframe in which the questionnaire is administered (must)
• Data analysis and report writing for the benchmark study (must)

following aspects should be coordinated across countries to benefit from economies of scale and to increase the value of the benchmark study (quality assurance; homogeneous approach towards data cleansing; better understanding of country particularities):

• Survey tool (should)
• Translation quality (should)
• Data cleansing (should)
• Data analysis country by country (should)

The following points are at the discretion of the local team in each country:

• Amount of effort the team wants to put into the analysis of country particularities and country-specific analysis as well as the corresponding country report. Theoretically, it is possible that a country participates in the international benchmarking study without having its own country report (although this probably amounts to missing a great opportunity to communicate about OpenGLAM in that specific country)
• Collection and selection of e-mail addresses of GLAMs to be included in the survey (although it would be advantageous in view of the international comparison if all countries used a similar approach in selecting their GLAMs)
• Partnerships with umbrella organizations, research institutions, etc. in their country
• Follow-up on the GLAMs that indicated that they want to be contacted.

Minimal requirement for a country to join the project

With regard to the minimal requirement for a country to join the benchmark study two cases can be differentiated:

a) For the country in question we can use the same version of the questionnaire that is already used elsewhere (translation provided by another local team):

In this case it is sufficient if a local contact person or a local team can provide a sufficient number of valid e-mail addresses (addresses of 500-3500 GLAMs) and answer questions of survey participants during the administration of the questionnaire.

b) For the country in question another language version of the questionnaire is needed:

In this case, a local team needs in addition to provide the translation of the questionnaire and help debugging the questionnaire once it is set up in the survey tool.

There are various reasons why such a minimal solution is not ideal and should not really be aimed for. However, from a risk management perspective, this may well be considered a fall back solution if a stronger engagement turns out to be impossible (due to a lack of funding, for example).

Main Project Activities

The table below contains the main project activities. The timeline is indicative and may be adapted.

As of October 2014, we are running 2-3 months late compared to the initial time schedule shown in the table.

What?	Who?	When?	Remarks
Select the survey tool	tbd	January 2014 Should be completed ca. 3 months before the administration of the questionnaire starts in the first country.
Develop an improved and updated version of the questionnaire.	All the partners interested in participating in the organization of the study. (Lead: Beat Estermann)	By the end of February 2014.
Research particularities regarding open data and crowdsourcing among GLAMs in each of the participating countries.	Partners in the different countries	By the end of February 2014	Ideally, this small research is carried out in parallel to the development of the new version of the questionnaire, in order to be able to include important insights in the way the questions are asked. Some further-reaching research may be useful when writing the study report.
Create partnerships at the international level	tbd	As soon as possible! Needs to be completed ca. 1 week before the administration of the questionnaire starts.
Enter partnerships with national umbrella organizations, research institutions, etc. for the purpose of the survey.	Partners in the different countries	As soon as possible! (for risk managment reasons) Needs to be completed ca. 1 week before the administration of the questionnaire starts.	Having partners on board that have a high reputation among the GLAM institutions in a given country is likely to have a positive impact on the response rate. These partners will be mentioned in the invitation e-mail as well as in the report.
Collect e-mail addresses of the survey participants (GLAMs)	Partners in the different countries	As soon as possible! (for risk management reasons) Should be completed ca. 1 week before the administration of the questionnaire starts	Having a sufficient number of addresses from a coherent or representative subset of GLAMs in a given country is very critical for the success of the survey. It is therefore recommended to undertake this task as soon as possible. It might be helpful to enter partnerships with organizations that have good GLAM registries for the country in question. Assuming a response rate of 30%, at least 333 valid e-mail addresses are needed to collect 100 responses. Having e-mail addresses of 500-3500 GLAMs would be ideal (depending on the size of the country and its total number of GLAMs).
Set-up the English version of the questionnaire in the survey tool	tbd	February 2014 Should be completed ca. 2 months before the administration of the questionnaire starts in the first country.
Run a pretest on the English version of the questionnaire and adapt the questionnaire accordingly	tbd	March 2014 Should be completed ca. 1 month before the administration of the questionnaire starts in the first country ||
Translate the questionnaire	Partners in the different countries	Between April and September 2014. Should be completed ca. 1 month before the administration of the questionnaire starts.
Set-up the different language versions of the questionnaire in the survey tool	tbd	Between May and October 2014 Should be completed ca. 2 weeks before the administration of the questionnaire starts.
Administer the questionnaire	Partners in the different countries using the common survey tool	Between June and November 2014	Takes about 4 weeks, with two reminders at 10 days interval
Export and cleanse the data	tbd, with the support of the local research teams	Between July and December 2014 Can be done within the weeks following the end of survey administration in a given country	Data cleansing is probably best done by the same person for each country, with the support from people who are fluent in the language in question and are familiar with the particularities of the country.
Analyse the data on a country by country basis (country study)	tbd, with the support of the local research teams	Between August 2014 and March 2015. Can be started around 1 month following the end of the survey administration in a given country.	A standard set of data analysis is probably best done by the same person for each country, with the support from people who are fluent in the language in question and are familiar with the particularities of the country. Further country specific analyses can be done by the local research teams (facultative).
Make the country comparisons (benchmark study)	tbd	Between January and March 2015.
Write the country report (country study)	Local research teams	April - June 2015
Write the international report (benchmark study)	tbd, with the support of local research teams to account for country particularities	April - June 2015
Publish the country report (country study)	Local research teams	July/August 2015
Publish the international report (benchmark study)	tbd	July/August 2015
Follow-up on the GLAMs which indicated that they want to be contacted	Local teams	Starting from August 2015 at the latest Can be started right after data export and cleansing

SWOT Analysis

Strengths	Opportunities
The OpenGLAM Benchmark Survey is the first endeavour to measure the state of advancement of OpenGLAM (digitization, inter-organisational cooperation involving the exchange of metadata, open data / open content, crowdsourcing, linked data) at an international level. The OpenGLAM Benchmark Survey is complementary to the ENUMERATE project which has been established over the past year in Europe in order to measure the state of advancement of digitization in the heritage sector and to address questions related to digitization. The project has been initiated from within two communities (Wikimedia and Open Knowledge Foundation) which have an international culture and have proven their ability to disseminate approaches quickly across many countries. The project team is composed of people from various backgrounds (researchers, GLAM practitioners, open knowledge / open data advocates) An agile project management approach allows to move quickly at the beginning, to integrate new people as the project proceeds and to adapt the working rhythm to the potential of the team. The informal project set-up fits well with the volunteer culture of the communities that make up a large part of the project team.	The project provides a great opportunity to bring people involved in GLAM outreach in the various countries together and to establish stronger ties both at a national and at an international level. The survey provides an excellent opportunity to identify heritage institutions that are willing to open up their collections and would appreciate it if someone from the Open Knowledge / Open Data community got in contact with them. The survey provides an excellent opportunity to collect the contact addresses of heritage institutions that would like to be regularly informed about the advancement in the area of OpenGLAM. The survey encourages the national teams to get in touch with the umbrella organisations (archives, museums, libraries) and to seek partnerships with the leading heritage institutions in their country. The survey report can be used to inform heritage institutions in the participating countries about the latest developments regarding OpenGLAM. The survey report provides the national teams with an excellent instrument to support lobbying activities in favour of OpenGLAM and the advancement of free knowledge in their country. The first round of the international survey can be used to institutionalize the approach and to have a benchmark survey every 2-3 years.
Weaknesses	Risks
An agile project management approach makes it more difficult to integrate well-established (bureaucratic) organizations that are used to more long-term planning. The informal project set-up may not be appropriate when it comes to managing external funds and to institutionalizing the survey over the coming years. So far, there is no centrally managed funding that would contribute to holding the team together, and there is no funding at the international level to cover travel expenses and accommodation cost for real-life coordination meetings.	Quality issues regarding various aspects of the survey: insufficient quality of the questionnaire (certain questions don’t work as intended; excessive length leads to poor response rates) insufficient translation quality (the data contains a lot of artefacts due to translation issues) methodological weaknesses (e.g. incoherent sampling methods; bad questionnaire design) lead to less useful results samples in some countries are too small bad acceptance of the survey within the heritage sector leads to poor response rates in some countries technical problems lead to a loss of data technical problems / inappropriate data handling lead to a breach of commitments regarding the non-disclosure of responding institutions’ identity. the report doesn’t correspond to generally accepted quality standards (content, presentation, language) the target audiences are not reached in all the participating countries, because the report isn’t properly distributed / not translated into local languages Conflicts: within national teams (causing a failure to perform as required to achieve the project goals) at the international level / within the international team (causing a failure to deliver key project outputs) Insufficient resources: at the national level, for locally managed tasks at the international level, for centrally managed tasks (causing a failure to deliver key project outputs) Roles are not sufficiently clear / insufficient expectation management (leading to conflicts / dropouts / unhappy partners) The goal of carrying out a benchmark survey that allows for country comparisons cannot be reached because there are not enough countries joining The workload of centrally managed tasks is too heavy because of too many countries joining National teams don’t profit from the opportunities offered by the survey because of a lack of resources and/or commitment by their members.