Public Policy Initiative – Progress Report (November 2010)

Date: November 26, 2010
By: Wikimedia Foundation, Public Policy Initiative team

PURPOSE OF REPORT

This report serves as an informal progress report to the Stanton Foundation on the initial stages of the Wikipedia Public Policy Initiative. The intention of this report is to account for what has taken place to date, to share general lessons learned, and to broadly outline next steps and adjustments to the project plan. It also serves as a strong recommendation for the continuation of the project and grant, based on success and perceived impact thus far.

We feel it is important to highlight upfront a significant and overarching lesson learned in this significant undertaking:

The creation of an infrastructure that is strong enough to produce year-over-year results and to make Wikipedia an integral part of public policy education is more beneficial to the goal of creating public policy content (and more sustainable) than recruiting "natural" Wikipedians who are knowledgeable about public policy. Therefore, the Wikimedia Public Policy Initiative aims to implement and institutionalize Wikipedia in public policy education. We consider this to be the most sustainable approach we have seen which will lead to improvement of public policy article content on Wikipedia. If the professors involved in this project continue to use Wikipedia over a longer time span, and if we maximize the usage of Wikipedia in public policy education across many institutions we will have better outcome in this subject area than trying to turn people into Wikipedians.

ACTIVITIES

Stage A (summer 2010)

We began by hiring the project team, as outlined in the initial proposal. The Wikimedia staff who put the project in motion trained the team, and the team worked to establish collaborative norms and essential infrastructure for working together. Over the summer (Stage A), we prepared for the fall semester by undertaking the following work:

• Continued outreach to professors, with a focus on six professors who committed to participating in the fall semester;
• Recruitment of Wikipedia Online Ambassadors (initially 20) and Campus Ambassadors (14) to support students and professors in the project;
• Establishment of Advisory Board (initially called the "Steering Committee");
• First meeting of Advisory Board, in San Francisco;
• Creation of a research plan; and establishment of a rubric for article quality assessment, and assessment of educational impact;
• Development and implementation of a project communication plan, including blog posts, newsletters, and interviews;
• Design and implementation of a work space ("WikiProject") on Wikipedia to enable collaboration between students and Wikipedians, and facilitate communication of course- related information to the Public Policy Team and all participants;
• Completion of print and online training materials and educational resources for students, professors, and ambassadors;
• A three day Campus Ambassador training session at George Washington University;
• Responding to a high volume of inquiries from interested academics and policy organizations. Early media coverage over the summer included Wall Street Journal and Newsweek blogs.

Stage B1 (fall 2010)

With the beginning of fall classes (Stage B1), the ambassadors engaged with students and professors in the participating classes. As general awareness of the project grew in the academic communities, additional professors teaching in the public policy topic area became interested and motivated to incorporate Wikipedia editing into their classes immediately. Instructor and volunteer interest was much stronger than anticipated: five additional classes were added during the semester.

The ambassadors who were recruited and trained in Stage A helped the participating professors modify existing course curricula to incorporate Wikipedia assignments. They also taught key Wikipedia concepts and editing skills to each class, and distributed learning resources from the Bookshelf project. They served as mentors during the article creation and content development phases of course assignments, answering questions and providing feedback.

Students began by creating Wikipedia user accounts, and reviewing public policy-related articles on Wikipedia. Working individually or in groups, students modified existing articles or created new articles. In total, students worked on 190 Wikipedia articles, 81 of which are about U.S. public policy and the rest of which fall into the broader scope of topics related to or deriving from public policy (See "Lessons learned" section below for further detail).

In order to pave the way for self-sustaining Online and Campus Ambassador programs, an ambassador Steering Committee was created. The committee is currently composed of three Wikimedia staff and four volunteers. The Ambassador Steering Committee meets weekly to discuss the progress of the initiative and plan for work going forward. A second ambassador training will take place in early January, before the spring semester begins. Some of the current ambassadors will be brought to San Francisco for a train-the-trainer session. The training will prepare ambassadors to return to their schools and present regional trainings for other new ambassadors.

Also during Stage B1, the project staff continued outreach to additional schools, developing professor and ambassador prospects for Spring 2011.

Media attention continued to grow as the project got underway. During the fall semester, we were interviewed on National Public Radio's "All Things Considered", and covered by the two top publications for higher education: Inside Higher Ed and the Chronicle of Higher Education.

PARTICIPANTS

Professors

For a full list of professors, see Appendix A.

As outlined in the proposal, the initial plan was to work with 3 - 5 schools in the Fall semester, which would indicate significant interest and potential in the program. As of this report, we have exceeded this milestone: the program includes 13 professors at 10 schools teaching 14 courses in the fall 2010 (Stage B1). Harvard University, Georgetown University, and the University of California at Berkeley are among the participants.

Tenured professors interested in the Public Policy Initiative primarily served to connect us with Associate, Assistant, and Adjunct professors, who had more time to devote to the initiative and whose classes were more adaptable. Senior faculty members became our allies and spokespeople within their respective university departments, facilitating communication between their more junior colleagues and the Public Policy Initiative team. Participants range from tenured faculty members to teaching assistants. None of the professors had experience editing Wikipedia prior to beginning the Initiative, but all are encouraging their students to contribute to public policy content. Senior faculty have been effective advocates for the project within their respective schools.

Students

Students, as the producers of Wikipedia article content, are at the core of the Public Policy Initiative. Because the incentive for students to write to fulfill a class requirement is a well established part of academic life, the main focus of the project has been to establish the conditions that define and support the students’ participation: (1) expectations set by professors within their classes; and (2) support resources available to students.

A concurrent examination of student progress is essential to understanding the progress of the Public Policy Initiative. Based on course enrollment numbers, about 250 students created user accounts and participated during the fall semester. In the classes, each of which covers an area of public policy, students learn to:

• Write for and edit Wikipedia;
• Write in a fact-based, encyclopedic style (in contrast to a persuasive, essay style);
• Find and evaluate appropriate references and citations;
• Engage and collaborate with Wikipedia volunteers to implement new content.

By undertaking this work, students explore and engage in public policy topics while using Wikipedia assignments to research and write articles in collaboration with other Wikipedia editors.

One student’s experience clearly illustrates the potential of the program. Claire, a junior at Syracuse University, is a public policy major and a student in Carol Dwyer’s “Wikipedia and Public Policy” class. Editing Wikipedia did not come naturally to Claire and her classmates. Most students in the class had never made even a small edit to Wikipedia. Two Campus Ambassadors attended class sessions and introduced students to the technical details of editing Wikipedia, Wikipedia culture, and the key policies that guide Wikipedia (such as neutral point of view and verifiability). Claire selected an Online Ambassador who guided her through the details of how to structure her Wikipedia article, how to use wiki code to format and link it appropriately, and how to interact with other Wikipedians. When ambassadors weren't available, Claire used the "Welcome to Wikipedia" brochure from the Bookshelf materials as a reference.

Claire selected the Food Quality Protection Act article to develop. Prior to Claire’s work, the article, which had existed since 2007, had fewer than 150 words, only two citations, and a long list of the Act’s requirements. Claire expanded it, building an extensive, informative, and carefully-sourced article. The improvement was substantial enough to qualify the article for the “Did You Know?” section on the main page of Wikipedia, driving a mass audience to her article.

Volunteers

As outlined in the Public Policy Initiative proposal, we created a Wikipedia Ambassador program to facilitate work with both professors and students. The Campus and Online Ambassadors currently support and mentor 13 professors and 14 public policy-related classes. Ambassadors consult with professors on appropriate Wikipedia assignments, guiding them on how to integrate Wikipedia assignments into their course curriculum. Additionally, Campus Ambassadors teach students the basics of editing Wikipedia, how to evaluate Wikipedia articles and help with the in-class Wikipedia questions. Online Ambassadors are available virtually to help students with any questions they have about their articles.

PERCEIVED IMPACT

Improvements to public policy topic area

At this stage of the project, several formal assessments are under way to help us assess the improvements to Wikipedia coverage of public policy articles that will result from this project:

• Quality metric test: public policy experts and Wikipedia experts will evaluate the same revision of the same article. Results will be compared to explore variance in assessment between the two groups of experts. This data will reveal how accurate the assessment metric is for the other assessments.
• Quality evaluation of specific articles: A comparison of articles before and after student work will determine whether the articles targeted by the university classes improved or not.
• Ripple effect: A random sample of articles that link to and from the articles that students worked on will be explored, to see whether quality changed over time.
• Article Feedback Tool evaluation: Evaluators will give an assessment of a random sample of articles at one point in time. Their evaluations will be compared to the results of the new Article Feedback Tool, a four-category quick assessment available to readers of articles.

Preliminary results from these tests will be available in the interim project report to the Stanton Foundation, in February 2011.

Two additional questions will be explored going forward, after more evidence has been established:

1. Did the Public Policy Initiative establish a successful model of collaboration between the Wikimedia Foundation, academic experts, and the Wikipedia community for the purposes of content improvement in U.S. public policy and integrating Wikipedia as a university teaching tool? Is this model transferable to other subject areas?
2. What evidence exists that the deliverables of the Public Policy Initiative will continue in the future? Are the results sustainable?

Impact in educational practice

The Public Policy Initiative has generated discussion among a broader audience of educators and academics about Wikipedia's usage in the public policy classroom and beyond. A wide range of schools, colleges, and universities have expressed interest in participation. Expressed interest from both State and U.S. Departments of Education, and the Open Educational Resources community, infers that we are on the forefront of education, and that the initiative and public policy content on Wikipedia are gaining positive attention.

At conferences and workshops for academics and experts in education, there has been an active discourse on Wikipedia's usage as an effective teaching tool in the higher education classroom. To name a few examples:

• Princeton University hosted a workshop on October 7 titled “Edit: How Wikipedia Changes the Way We Debate, Govern and Teach,” led by two Public Policy Initiative team members.
• Professor Anne McNeil led a workshop at the University of Michigan’s Center for Research on Learning and Teaching on editing Wikipedia in higher education courses.
• Campus Ambassadors at Indiana University gave a "Teaching with Wikipedia" workshop on their campus in October.
• A workshop proposal for the American Political Science Association’s annual Teaching and Learning conference has been accepted. Wikimedia Foundation staff will teach political science and public policy professors how to integrate Wikipedia in their courses, and present the research results on student learning and article quality improvement.

While the resources of the grant and project team are dedicated to the scope of the initiative and the improvement of the public policy topic area, the model of using Wikipedia in the higher education classroom is gaining considerable momentum of its own accord as a result. Academics recognize that Wikipedia's usage in educational practice informs education reform, learning objectives and skills that are essential in today's classroom and beyond.

Media and raising awareness

Positive media coverage in both mainstream and education-oriented publications has provided considerable benefits for the reputation of Wikipedia as a high quality, free knowledge resource.

The Public Policy Initiative has been featured in the two top higher education news sources, the Chronicle of Higher Education and Inside Higher Ed. National Public Radio’s "All Things Considered" and the blogs for Newsweek and the Wall Street Journal have also run stories on the Initiative. Outreach to student newspapers and university communications departments has led to several articles and features in smaller publications, university publications, and related blogs and websites.

Media attention has positively impacted the project team's efforts to recruit new professors of public policy to the initiative. Also, media coverage has led to interest from professors outside of the discipline of public policy (who are directed to the Wikimedia volunteer community where appropriate).

ASSESSMENT OF ORIGINAL ASSUMPTIONS

Overall assessment

The grant proposal outlined a number of expectations and assumptions about what might constitute an effective program wherein subject matter experts contribute their knowledge to Wikipedia articles on public policy. While the approach was experimental, the outcomes in the fall semester classes strongly support most of the expectations, and the general efficacy of the planned approach. With a few specific assumptions, our expectations were substantially exceeded, and the model worked well overall. Adjustments based on the outcomes of the fall semester should substantially increase the output. We expect the spring semester (Stage B2) to be a great deal more productive as a result of these lessons and the resulting adjustments, than originally planned. See “Looking ahead” section for detail on adjustments to plan.

Generation of content by students

The students in the program have produced high quality content. In some cases, they have earned accolades from the Wikipedia community (most notably in the form of links from Wikipedia’s front page). Some classes have produced a great deal more content than others, reflecting the different approaches taken by the various professors. Also, fewer students than expected have focused on articles on U.S. public policy; rather, they have tended to focus on an interdisciplinary and broad scope of articles related to public policy.

Given the experimental nature of this project, we are encouraged to see a strong model for collaboration emerging in the current work, even though the differences in approach from one class to another has yielded inconsistent results.

Students have responded well to the "Welcome to Wikipedia" brochures, and to the Campus Ambassadors and Online Ambassadors. Pairing an enthusiastic, outgoing Wikipedia newcomer with an experienced Wikipedian proved to be the ideal combination for Campus Ambassadors in each classroom.

Topic area focus

Some assumptions built into the original project plan proved sub-optimal, which we knew was probable at the outset given the experimental nature of this project. Namely, even though every class was part of the project “U.S. public policy,” many professors left the choice of content up to their students, and did not restrict them to this topic area. We feel this issue, among others, can generally be addressed by establishing clearer expectations at the outset of the next semester. See next section for further detail.

LOOKING AHEAD

The project elements that have proven very successful to-date (such as the ambassador program) will be further implemented and improved in 2011. We also anticipate adjustments to plan, based on what we have learned to date, for the spring semester 2011 (Stage B2).

• Expectations will be set so that assignments primarily concern U.S. public policy, are mandatory, and span at least half the term, with specific milestones. Also, assignments will be clearly defined as individual and group assignments (the former produces more content; the latter may be better from a pedagogical standpoint). To ensure these guidelines are followed, we will clarify expectations of professors before the classes start. To this end, we have refined two different Memoranda of Understanding (MOU's), reflecting different levels of partnership between Wikimedia and each professor.
• We will implement more thorough course planning materials, including a Guidelines for Course Design document and a sample syllabus. The Guidelines document, based on the best practices we identified during Stage B1, clearly spells out expectations for the course.
• We will debut a “Wikipedia Teaching Fellowship,” awarded to professors who agree to follow the Guidelines document and who share their assignment with us prior to the beginning of the term.
• There will be strong focus on supporting Wikipedia as an effective tool for teaching within the educational community. To this end, we will work with Public Policy departments and universities to institutionalize Wikipedia use in Public Policy curricula. Additionally, we will encourage more professors to promote the model with other faculty members and universities.
• We will experiment with new incentive and rewards systems to motivate students to significantly contribute to public policy articles and also to encourage ambassadors to achieve expected outcomes for support of new contributors.
• Ambassador involvement will increase: we anticipate that the ambassador program will move towards self-sustainability. In 2011, ambassadors will take on some of the tasks currently handled by the project team (e.g. professor and new ambassador recruitment for the spring semester; training of new ambassadors, etc.) For the second Campus Ambassador training, we will use a decentralized regional training model. This approach will provide flexibility for adding new ambassadors throughout the semester. It will also enable a greater number of new ambassadors to be trained, leverage current ambassadors as trainers, allow more professors to attend the training if desired, and help establish a sustainable ambassador training model for the future. We will pair each student with an Online Ambassador early in the semester.

CONCLUSION

Stage B1 requirements for continuation of project (as defined in the grant proposal) have been met: expectations for involvement by schools have been exceeded; the articles students have worked on so far show significant improvement, and it is expected that the final analysis will indicate that the quality of public policy articles has improved. Although data analysis cannot be completed until the end of the fall term when students have finished their work on articles, there are many indicators that this model is working. This will be further elaborated on in the interim report to the Stanton Foundation, due in February 2011.

We are confident the Wikipedia Public Policy Initiative has been a success to date, and look forward to seeing it through to the next stage.

APPENDIX A: LIST OF FALL 2010 PROFESSORS

• Brian Carver, Assistant Professor, University of California at Berkeley
• Michael Cornfield, Adjunct Professor, George Washington University and George Mason University
• Rochelle Davis, Assistant Professor, Georgetown University
• Carol Dwyer, Community Benchmarks Program Director, Syracuse University
• Donna Infeld, Professor and Director of the Master's in Public Policy Program, George Washington University
• Chaim Kaufmann, Associate Professor, Lehigh University
• Max Klein, DeCal Instructor, University of California at Berkeley
• Peter Linquiti, Lecturer, George Washington University
• Nicco Mele, Adjunct Lecturer, Harvard University
• Barry Rubin, Professor, Indiana University
• Matt Senate, DeCal Instructor, University of California at Berkeley
• Jeffrey Tang, Assistant Professor, James Madison University
• Beverly Thompson, Assistant Professor, Siena College