GLAM/Newsletter/May 2021/Contents/Special story

This Month in GLAM – Volume XI, Issue V, May 2021

Special story

Wikimedia Hackathon report: Upgrading GLAM tech tools and PAWS

Major upgrades

PAWS new notebook options: OpenRefine and SPARQL

PAWS is also a general one-click computing container, so it can run a variety of different open source packages. Some of the important innovations that resulted from the Hackathon for GLAM Wiki users:

OpenRefine - OpenRefine is used for reconciliation and data cleaning, primarily to see how a data set matches (or not) Wikidata's items. Of main interest to GLAM folks: you can now run OpenRefine on the Wikimedia cloud servers, via PAWs, instead of downloading it to a local computer and installing it. This is useful for folks who cannot install software on locked-down institutional computers, or those who want to do training for OpenRefine and have been hampered by installation woes for individual users. Having folks login to PAWS and run OpenRefine in the same turnkey environment and version is a huge plus. Another benefit of having this in the cloud: after running a reconciliation session in OpenRefine on PAWS, you can share your working data with others with a public link. If you are logged in to PAWS you can choose OpenRefine from the "New" menu, or you can access it with this link:
- https://hub.paws.wmcloud.org/user/user-redirect/openrefine/

SPARQL kernel - If you've ever run a series of Wikidata SPARQL queries but wanted to save them or show a progression of several queries, PAWS can now save SPARQL queries in a Jupyter notebook format. You can make a new SPARQL notebook from the New menu. A sample can be seen here:
- https://public.paws.wmcloud.org/User:Fuzheado/SPARQL/SPARQL%20kernel%20example.ipynb

JupyterLab - PAWS has a more advanced JupyterLab mode that is more sophisticated in handling multiple files and extensions. By default, PAWS loads up the classic "notebook" view but thanks to the work at the Hackathon, PAWS can now run JupyterLab, and is useful if you're working with multiple files. PAWS starts up in classic notebook mode, but you can visit this URL to run it in JupyterLab mode:
- https://hub.paws.wmcloud.org/user/user-redirect/lab

PAWS notebooks as apps - A PAWS notebook can interactively show how code can make Wikipedia/Wikidata edits. But you can also develop a usable app with a point-and-click user interface in PAWS. Thanks to the work of User:Chicocvenancio and User:Yuvipanda, the extension Voilá is now available in PAWS, which allows you to execute a fully functioning standalone app on Wikimedia servers, run under your account.

PAWS with the Voilá option to run as a standalone app.

For an example of a project published with this system, see the USA report writeup for this month, where User:Fuzheado describes his Wikidata Graph Browser project. You can also click on this link to launch the tool using Voilá to see how it works.

R and RStudio - PAWS has previously been limited to the notebooks using the Python programming language, but during the hackathon support for R was added. R is a programming language focused on statistical computing and graphics which a be powerful tools when one needs to do data analysis and visualization. In addition to using R in classic notebooks PAWS now also comes with RStudio, an "integrated development environment" which allows you to work with multiply files, tools to debug your code, built-in tutorials, and much more. To get started with R in RStudio, use the link below:
- https://hub.paws.wmcloud.org/user/user-redirect/rstudio

Why I wanted to learn to use Jupyter

Those working with GLAM know the gap between bots and the tools that have been developed for non-coding data wranglers to upload images to Wikimedia Commons or data to Wikidata. Sometimes these tools get outdated. For the bots, on the other hand, there are not that many that can create them, and this skill requirement becomes a barrier for non-coders to accomplish reasonable results. Jupyter notebooks are coding done in the wiki way. With them, you can achieve almost anything that the bots can do, but you can build upon notebooks that others have done, copy and steal code from others and see the results in real time. They are called notebooks because you can scribble text between the chunks of code, and use the notebooks for explaining what the code does.

My idea going to the hackathon was to be able to reconcile tens of thousands of geographic items to data already in Wikidata. I wanted to find items that are located in the same place and only after that start to compare their names in different languages and other properties. There are many tools that could be tweaked to do that, but the data in Wikidata for my items was so heterogenous that I should be able switch between different approaches. So I started to create a notebook for reconciling based on geographic data. It's just a start. The notebook is not working yet, but I have a sense that it will work! I am eager to learn myself and together with others, and create a library of recipes we can exchange with one another to do the things we need. The story continues... – Susanna Ånäs

Next "Special story" →

← Previous "Special story"