Student Contributions to Wikipedia

by Amy Roth

Ongoing assessment of the Public Policy Initiative covers several areas. The students provided feedback via individual interviews, group video interviews conducted by Georgetown's Center for New Designs in Learning and Scholarship (CNDLS), focus groups, and a survey by WestED. The survey had a 25% response rate. Through the interviews and survey students expressed motivations of contributing work toward a useful public good and having a global audience for that work. The research also assessed the quality of work that students contributed to Wikipedia.

At the start of the Public Policy Initiative, foundation staff worked with Wikipedia community members to develop a quantitative article quality metric. This metric assigns weights and values to Wikipedia policies regarding article quality; the assessor applies the metric and generates a numeric quality score which is translatable to the Wikipedia 1.0 rating system (Featured Article, Good Article, A, B, C, Start, and Stub ratings). (Only 0.1% of rated articles in Wikipedia earned a Featured Article status.) Research through the initiative made a scientific comparison of public policy expert scores and Wikipedian scores. Using a linear regression model (Total score = A + Wikipedia Expert + article + error), Wikipedians on average scored article quality 4 points lower than public policy experts with a p-value much less than 0.01 and an R^2 value of .67 measuring model fit.

The assessment team is made up of both policy experts and experienced Wikipedians. The policy experts all have graduate degrees in public policy or public affairs; they are graduates of universities including: Berkeley, Boise, Brandeis, Georgetown, Pennsylvania, Pepperdine, Sacramento, and Syracuse. We have policy experts who have worked in policy for over 30 years, emeritus professors of public policy, several recent graduates, and some currently active in the public policy field. Policy experts were recruited through alumni email lists of many universities. Wikipedians were recruited through direct invitation on their user talk pages and were selected for quality assessment experience, a non-confrontational style, and interest in public policy topics. In the fall there were six policy experts and fifteen Wikipedian assessors (1:2.5, PE:WP). This spring there were twenty-five policy experts and eleven Wikipedians (2.3:1, PE:WP). Over the course of both terms there have been a total of twenty-eight policy experts and seventeen Wikipedians on the assessment team.

The assessment team performed multiple assessments on a randomly selected set of articles from each term. In most cases, three individual Wikipedians and three individual public policy experts assessed each article. For clarity I will call the group of Wikipedian and expert assessments a review. For pre-existing articles, article reviews were performed on one version of the article before the student work and then another version after the student work. For the new articles that students created, the articles only received one review which was conducted after the student contributions.

• Over the course of the project a total of 140 articles were randomly selected from the WikiProject United States Public Policy Course tab and assessed. 84 were pre-existing and 56 were new.
• There was 64% increase in all articles, the average score went up 5.8 points (9.0 before, 14.8 after)

• There was a 50% increase in pre-existing articles. The average score went up 4.8 points (9.7 before, 14.5 after)

• New articles had an average score of 15.4 which translates to an average B rating in the Wikipedia scale.

• Over the course of two university terms, approximately 800 students participating in the Initiative contributed over 8.8 million bytes to the Wikipedia namespace.

This is preliminary data, and there may be errors or omissions. There are not likely to be any substantial changes, but there will likely be some minor differences when these results are submitted to journals for publication.