hangingtogether.org

Syriac scripts added to VIAF cluster for Ephraem

We have just loaded into the Virtual International Authority File (VIAF) the second set of personal names from a scholarly resource, the Syriac Reference Portal hosted by Vanderbilt University.

Syriac is a dialect of Aramaic, developed in the kingdom of Mesopotamia in the first century A.D. It flourished in the Persian and Roman Empires, and Syriac texts comprise the third largest surviving corpus of literature from the Roman Empire, after Greek and Latin. The Syriac Reference Portal is a collaborative digital reference project funded by the National Endowment for the Humanities and the Andrew W. Mellon Foundation involving partners at Vanderbilt University, Princeton University and other affiliate institutions.

This addition represents the first time we see Syriac scripts (there are variants) as both the “preferred form” and under “alternate name forms” in a VIAF record. The Syriac Reference Portal also contributes additional Arabic and other scripts as alternate names, but selects a Syriac script form as a preferred form for people who wrote or were written about in Syriac.

The Syriac names join the Roman and Greek personal names we loaded from the Perseus Catalog last November and blogged about here as part of our Scholars’ Contributions to VIAF activity. Together they demonstrate how scholarly contributions can enrich existing VIAF clusters—generally comprising contributions from national libraries and other library agencies— by adding script forms of names that previously lacked them, as well as adding new names. Scholars benefit from using VIAF URIs as persistent identifiers for the names in their own databases, linked data applications and scholarly discourse to disambiguate names in multinational collaborations and using VIAF as a means to disseminate scholarly research on names beyond scholars’ own communities.

Adding these scholarly files demonstrates the benefits of tapping scholarly expertise to enhance and add to name authorities represented in VIAF. We look forward to more such enhancements from other scholars’ contributions.

Today we released a brand spanking new web site for library coders. It has some cool features including a new API Explorer that will make it a lot easier for software developers to understand and use our application program interfaces (APIs). But seen from a broader perspective, this is just another way station on a journey we began some years ago to enable our member libraries to have full machine access to our services.

When I joined OCLC in May 2007, I immediately began collaborating with my colleagues in charge of these efforts, as I knew many library developers and had been active in the Code4Lib community. As a part of this effort, we flew in some well-known library coders to our headquarters in Dublin, OH, to pick their brains about the kinds of things they would like to see us do, which helped us to form a strategy for ongoing engagement.

From there we hired Karen Coombs, a well-known library coder from the University of Houston, to lead our engagement efforts. Under Karen’s leadership we engaged with the community in a series of events we began calling hackathons, although we soon changed to calling them “mashathons” in response to the pejorative nature the term “hack” had in Europe. In those events we brought together library developers to spend a day or two of intense learning and open development. The output of those events began populating our Gallery of applications and code libraries.

Karen also dug into the difficult, but very necessary, work to more thoroughly and consistently document our APIs. Her yeoman work in this regard helped to provide a more consistent and easier to understand and use set of documentation from which we continue to build upon and improve.

When Karen was moved into another area of work within OCLC to better use her awesome coding ability, Shelley Hostetler was hired to carry on this important work.

In this latest web site release I think you will find it even easier to understand and navigate. One essential difference is it is much easier to get started since we have better integrated information about, and access to, key requesting and management when those are required (some services do not require a key).

Although this new site offers a great deal to developers who want to know how to use our growing array of web services, we recognize it is but another step along the road to developer nirvana. So check it out and let us know how we can continue to improve. As always, we’re listening!

Today in Cape Town, South Africa, at the OCLC Europe, Middle East and Africa Regional Council (EMEARC) Meeting, my colleagues Richard Wallis and Ted Fons made an announcement that should make all library coders and data geeks leap to their feet. I certainly did, and I work here. However, viewed from our perspective this is simply another step along a road that we set out on some time ago. More on that later, but first to the big news:

1. We have established “work records” for bibliographic records in WorldCat, which bring together the sometimes numerous manifestations of a work into one logical entity.
2. We are exposing these records as linked open data on the web, with permanent identifiers that can be used by other linked data aggregations.
3. We have provided a human readable interface to these records, to enable and encourage understanding and use of this data.

Let me dive into these one by one, although the link above to Richard’s post also has some great explanations.

One of the issues we have as librarians is to somehow relate all the various printings of a work. Think of Treasure Island, for example. Can you imagine how many times that has been published? It hardly seems helpful, from an end-user perspective, to display screen upon screen of different versions of the same work. Therefore, identifying which works are related can have a tremendous beneficial impact on the end user experience. We have now done that important work.

But we also want to enable others to use these associations in powerful new ways by exposing the data as linked (and linkable) open data on the web. To do this, we are exposing a variety of serializations of this data: Turtle, N-Triple, JSON-LD, RDF/XML, and HTML. When looking at the data, please keep in mind that this is an evolutionary process. There are possible linkages not yet enabled in the data that will be later. See Richard’s blog post for more information on this. The license that applies to this is the Open Data Commons Attribution license, or ODC-BY.

Although it is expected that the true use of this data will be by software applications and other linked data aggregations, we also believe it is important for humans to be able to see the data in an easy-to-understand way. Thus we are providing the data through a Linked Data Explorer interface. You will likely be wondering how you can obtain a work ID for a specific item, which Richard explains:

How do I get a work id for my resources? – Today, there is one way. If you use the OCLC xISBN, xOCLCNum web services you will find as part of the data returned a work id (eg. owi=”owi12477503”). By striping off the ‘owi’ you can easily create the relevant work URI: http://worldcat.org/entity/work/id/12477503

In a very few weeks, once the next update to the WorldCat linked data has been processed, you will find that links to works will be embedded in the already published linked data. For example you will find the following in the data for OCLC number 53474380:

schema:exampleOfWork http://worldcat.org/entity/work/id/12477503

As you can see, although today is a major milestone in our work to make the WorldCat data aggregation more useful and usable to libraries and others around the world, there is more to come. We have more work to do to make it as usable as we want it to be and we fully expect there will be things we will need to fix or change along the way. And we want you to tell us what those things are. But today is a big day in our ongoing journey to a future of actionable data on the  web for all to use.

Introduction

Since entering the library world, I’ve been fascinated by the substream of uber-geeks which are the Dewey Decimal Classification (DDC) nerds. It was the enthusiastic and technical-leaning banter of Editor-in-Chief Michael Panzer, that enticed me. When I learned that the DDC is designed to be able to describe all possible concepts, and then also that is not a strict hierarchy, but really a web of complex relations, I was intrigued by the foresight of the approach. I see the DDC as a best effort 100 years before anyone would utter “crowdsourcing,” – which is how a modern analog the Wikipedia Category systems works. Using Graph Theory we develop a new way to look at, and compute with DDC and then connect it to Wikipedia Categories.

(In this post we assume some knowledge of Graph Theory, so if you need a quick primer, it’s at the end: Graph Theory tl;dr.)

Representing the Dewey Decimal Classification in Graph Theory

In order to represent DDC using Graph Theory we have to translate its components into graph theory. That means,

• Classifications become the “nodes”.
• Relationships between Classifications become the “edges.”
• Relation colourings:
  • Notational hierarchy – Red
  • See References – Green
  • Class elsewhere – Blue

We are actually defining a directed graph because the relations we have are not in general reciprocal. That is, if a classification A is below B in hierarchy,  or  A has a “see also” reference to B, B does not necessarily have that reference to A. (Although there are some reciprocal “class elsewhere” relations for example 391 and 646.3  in the 4 hop graph below). Note also that colourings – the mathematical parlance – indicates that relationships are different without assigning the relationship a numerical value.

Visualizations

This has all been quite abstract so far so let’s look at something concrete. We’ll pick classification 394.1, whose caption is the enjoyable “Eating, drinking; using drugs” .  The distance to other nodes is how many edges we travel from our source. We will look at all the connections that are n-hops from 394.1. Below is the evolution of visualizations from distances 2, 3, 4, and 5. By 5 hops you can see that the graph begins to be too complicated to visually represent in a single image.

Weighting the Graph

One of the powerful aspects of rendering the classification system as a graph is that we can determine the shortest distance between two nodes. However, as mentioned, our colourings have no intrinsic value, so in order to to find the shortest paths, we have to assign weights to colours. This is variable, but for the rest of the examples I am using the settings: Notational Hierarchy = 0.5, See Reference = 1, Class_elsewhere = 2.  That means that our shortest path between 394.1 and 341.33 (“Diplomatic Law”) is 394.1 -> 394-> 395  -> 327.2 -> 341.33, which has a total value of 5.5.

Transforming Wikipedia Categories

Consider a Wikipedia Category. It can contain many Wikipedia  articles, each of which has zero or more citations of books. Some of those book citations include the ISBN, which is precisely what the OCLC  Classify API is hungry for. Classify will transform an ISBN or OCLC number into a DDC, based on how  it is most popularly cataloged in WorldCat. That means we can arrive at a set of classifications associated with the category. From this set it would be instructive if we could somehow average them into one representative classification. In Graph Theory terms this is known as finding the center, and there are several ways to approach this problem. The method we employ here is the betweenness centrality. That is, we calculate the weighted shortest paths between all the classifications associated with the category, and find the nodes that were most touched in all the shortest paths. This is rather abstruse without some examples. We run the algorithm on some categories and below are the top 5 between nodes of:

• Category:Cooking techniques:
  1. 641 – Food and Drink
  2. 641.5 – Cooking
  3. 641.3 – Food
  4. 613.2 – Dietetics
  5. 641.815 – Bread and bread-like foods

• Category:Feminism
  1. 305.42 – Social role and status of Women
  2. 305.43 – Women by occupation
  3. 305.9 – People by occupation and miscellaneous social statuses
  4. 305 – Social Groups
  5. 305.4 – Women

This shows there is some basis for proving that this technique works. The centre of the classifications we find are around the same topic of that organizes the classification. We usually finds a strand or run in the system that hover around the main ideas. One caveat though is that for both of these Categories only about 25% of Wikipedia articles actually contain an ISBN citation. Yet of articles that do include at least one ISBN citation, they average 2.7 citations per article. With that in mind we continue to look at some set of articles that are not organized by topic, and are also more ISBN-rich. What better category than Featured Articles?

• Category:Featured articles
  1. 930 – History of the ancient world c. 499
  2. 940 – History of Europe
  3. 361 – Social problems and welfare in general
  4. 800 – Literature and Rhetoric
  5. 362 – Social welfare problems & services
  6. 580 – Plants (Botany)

Featured articles, of which there were ~4,000 gave lead to 25,000 ISBNs, which related to 17,000 Classifications. There are only approximately 50,000 standard Classifications. This at first would seem impressive coverage, but distribution of the coverage in the graph was not calculated, so its still difficult to say how diverse the featured article set is. Computing on this set was very intensive, in fact just computing the centre of these 17,000 Classifications took 9 hours parallelized over 16 cores. These topics seem to be more abstract, and less interrelated. Without being able to infer some sort of connect, next we run the algorithm not on a category, but on the list of articles that your blogger has ever edited (307 articles).

• Contributions of User:Maximilianklein
  1. 930 – History of the ancient world c. 499
  2. 800 – Literature and Rhetoric
  3. 302.23 -  Media (Means of communication)
  4. 910.45 – Ocean travel and seafaring adventures
  5. 361 – Social problems and welfare in general

Uncovered here is a latent passion for Seafaring adventures! This is probably owing to the fact that travel articles may have cited ISBN-having travel books which were disproportionately focused around sea-travel. More importantly what arises here is that several of the central Classifications that describe the edit history are similar to the featured articles results. This would seem to suggest that there are some very central nodes that most connections are channelled through if they lie at disparate regions of the graph. The implication is that the problem of classifying an arbitrary group of object remains difficult! Of course these results are only for one possible set of values for our weights dictionary, where these top level channels would be more “expensive” to hop-through. So future research should try and calibrate these values to produce the most varied set of central nodes.

This work is available as an Ipython notebook and on github, and comments welcome
–notconfusing

Graph Theory? tl;dr

We are not talking about the colloquial meaning of “graph”, but rather the mathematical concept.

A Graph is a collection of:

• “Nodes” or “Vertices” – the objects
  • Here letters A through F
• “Edges” – relations (directed or undirected)
  • Here the arrows between letters

Not talking about this kind of graph. (Image By Inductiveload [Public domain], via Wikimedia Commons)

An example of a “Graph Theory” Graph (By David W. at de.wikipedia (Original text : David W.) [Public domain], from Wikimedia Commons)

from The Visual Thesaurus

Folks may already have seen the list of top love stories we put out yesterday for St. Valentine’s day. It was pulled together very quickly by JD Shipengrover and Diane Vizine-Goetz.

This one is quite nice and we are pleased with the response.

It is programmatically generated and based on a fiction subset of Worldcat(which includes movies); it is FRBRized so it is at the work level and consolidates editions etc; the items are identified by genre data in the MARC records; and it is ranked by holdings.

We are going to do more lists in future.

When our colleagues at OCLC Research Library Partner institutions were asked for topics that they’d like to see addressed, they kept saying, “Do something about Digital Humanities,” but they didn’t say what needed doing. When we said at staff meetings, “We should do something about Digital Humanities,” we were asked what exactly and demurred. What did we have to contribute? Then Jen and I started looking around and realized that a lot has been written by librarians about DH and libraries, and librarians often go to meetings to talk to other librarians about DH. We decided to try to steep ourselves in DH from the researchers’ point of view and see what might come of it. After lots of reading, several conferences, and a couple of focus group sessions, we’d learned a lot. We synthesized what we heard and shared it with several colleagues who thought it would be useful to library directors who are not already in the thick of DH.

Humanités Numériques [crop] Calvinius CC BY-SA 3.0

We extrapolated from what we heard from the DH scholars to what we thought it could mean for library directors. The result was a spectrum of possible ways for libraries to engage with DH. It ranged from making sure DH scholars knew about services you were already offering that could be helpful to them, to full-bore immersion in a DH center, and with lots of other possibilities in between. There are ideas about offering training, involving DH scholars in digitization initiatives, helping to make their work discoverable, enabling them to enhance metadata, preserving the outcomes of their work, and making sure that it’s all sustainable. Whether or not a library has one staff member, several, or a center devoted to DH is largely a local issue. That may be based on how much is wanted by the researchers and how much is doable or affordable — but these things will change over time. Where a library begins on the spectrum is not at all where they might end up.

From our conclusion:

Humanities scholars have always been a central constituency for research libraries. The digital humanities constitute an evolving approach to research, and directors must support this work as a component of the university’s research mission. Libraries offer many useful services to digital humanists. Where the need is clear and DH scholars are receptive, libraries can offer various dedicated services to further DH efforts. In some cases, a full-blown DH center may be warranted…

…No matter which approaches to supporting the digital humanities you opt to take, keep in mind that what we call “The Digital Humanities” today will soon be considered “The Humanities.” Supporting DH scholarship is not much different than supporting digital scholarship in any discipline. Increasingly, digital scholarship is simply scholarship.

Since our essay, “Does Every Research Library Need a Digital Humanities Center?” was released last week there has been a lot of important discussion on Twitter and on various blogs. The conversation has gone in many useful directions. The ACRL blog dh+lib has offered to continue the conversation.

In my last post I identified the most edited records in WorldCat, which, no surprise, were all serials. Someone who read the post asked about this information by format (e.g., books, maps, scores, etc.). I doubt that I will get to all of the various formats, but I decided to take a look at books.

Unlike serials, for which I noted those that had 60 or more edits, for books I had to lower the threshold to 40 to get any at all (the most edited item had 58 edits). So here are the book records which have been edited more than 39 times in WorldCat (in no particular order):

• Harry Potter and the Deathly Hallows
• Naruto [a graphic novel series]
• Mockingjay
• Working Americans, 1880-2005 [a multi-volume series]
• Catching Fire
• Fullmetal Alchemist [a graphic novel series]
• The Hunger Games
• Religion Past & Present: Encyclopedia of Theology and Religion [a multi-volume work]
• Bleach [a graphic novel series]

An inevitable conclusion from the above seems to be that the more libraries that hold a book the more likely a cataloger will be to touch the record for it, which would explain how Harry Potter and the Hunger Games books made it on the list.

Recently I’ve been doing a large pile of data processing jobs that has me working in cycles of 20 minutes or so. In other words, I do some edits, kick off a job on our compute cluster (fondly named “Gravel” — don’t ask) and about 20 minutes later I do roughly the same thing. Yeah, I know, you’re thinking “why doesn’t he automate it?”. And I would, except that this is a shared resource and rather than kicking off my monster list of jobs that could keep the cluster running from now until…well…a long while from now I think it’s better to introduce some variability in load.

All of that is a long introduction to how I came to discover the most edited records in WorldCat. To fill in those 20 minute blocks I took up some “mini investigations” that do not take as long to perform.

For one such investigation I looked into how often WorldCat Records have been edited and by whom. I will be blogging about this in an upcoming post, but a small slice of this investigation was a closer look at the records that have been edited a lot. Since we keep track of the cataloging symbol of every institution that has edited a record, these can stack up for records that require updates on a regular basis — in other words, serials.

All of the records for these serials were edited more than 60 times over their life in WorldCat, and in no particular order:

• THE Journal: Technological Horizons in Education
• Journal of Geophysical Research
• Books in Print
• Encyclopedia of Associations
• US News & World Report
• Journal of the National Cancer Institute
• Occupational Outlook Handbook
• Health, United States
• Federal Register
• Budget of the United States Government
• Statistical Abstract of the United States
• Morbidity and Mortality Weekly Report
• Social Security Bulletin

Take a bow, serials catalogers, you’ve clearly earned your pay.

This is another installment in my continuing series of eclectic, peripatetic, and yes, let’s just say it: “pathetic” data investigations. The most recent identified the top countries of publication for WorldCat records. For whatever reason, I got it into my head to determine which English words appear the most in the main title of WorldCat items.

Clearly there are at least two ways to go about this: a) a formal, well-designed, highly replicable and ultimately near perfect investigation, or b) a slapdash, fast, seat-of-your-pants investigation of questionable merit. When given such a choice, I find the latter completely irresistable.  So I took part of my day today and did exactly that.

Since I already had code on our research cluster affectionately named “Gravel” that could extract a specific subfield, I powered it up and sucked out all of the 245 $a fields from WorldCat. As part of that process, I extracted only unique strings. The sharp ones among you have likely noticed a couple flaws already: 1) I was too lazy to filter based on language, and 2) I was too careless to normalize the title strings.

Flaws have never stopped me before, so I blazed on as if nothing was amiss. Then I threw that monster file onto another computer where I didn’t have to worry about interfering with any of the actually useful work that my colleagues where doing on Gravel (you’re welcome). There I wrote a special-purpose Perl script to take each title string, split it into individual words, lowercase them, and count up the occurrences. I dabbled in creating a “stop-words” list of useless words like “a” and “an” and “and” and “the” (ad infinitum) but that quickly began looking like a rabbit hole. As I was only really interested in identifying the top 30 or so words I figured my human eyeball would be sufficient to trap those in the end. Likewise with the foreign words.

That was really about it. Well, except for all the time I spent on Facebook waiting for the operations to complete. Did I say that out loud?

Anyway, without further ado (thank god) here are the top occurring meaningful English title words in WorldCat:

2020380 new
1853252 report
1431184 study
1159042 development
1069940 analysis
1004554 history
978681 county
968097 international
929294 state
890928 guide
856935 system
789983 education
778732 school
756569 united
748894 national
736474 management
706559 social
700137 book
688993 states
688328 studies
687695 general
687665 american
679083 systems
678582 public
677286 water
671552 research
666407 life
661707 health
645966 plan
644212 world
642100 effects


OK, now move along, nothing to see here.

I’m a data geek. I just love processing data in various ways to see what I can find out. So recently I decided to look into the countries of publication as recorded in the 300+ million MARC records in WorldCat. Just for kicks I did some processing of the 260 $a subfield, which is  the “Place of publication, distribution, etc.” as it appears on the piece, or noted in various other ways if it doesn’t.

As you might imagine, what results from such an investigation is a complete dog’s breakfast, with a large variety of punctuation marks, typographical errors, imaginative spellings, and just plain junk. No, it is much better to parse bytes 15-17 of the 008 field, which at least are supposed to only contain values from this list maintained by the Library of Congress. Progress.

That is, until one discovers that this “Code List for Countries” is not exactly that. If you happen to be in a certain select part of the world (mostly the United States, Canada, and Australia), you can also select state or province-specific codes. So before I used this table to translate the codes for actual countries I first had to translate the table, so that the code for “California” translated instead to “United States”. Progress.

Oh, and then countries have this tiresome tendency to change over time. The Soviet Union broke up. Czechoslovakia split into two. And don’t even get me started about the hot mess that used to fall under the general term of “Micronesia”. So I had to make some executive (and no doubt indefensible) decisions about how to deal with those. By and large, if I could identify some geography (e.g., Uzbekistan) that had a former life that could also be identified (e.g., Uzbek S.S.R.), I translated them both into the current entity. But lord only knows how many items that don’t have this distinction end up being miscounted. But progress of some sort nonetheless.

Oh, and places like “West Berlin” got their own code. How quaint. But now I’m just whining.

In the end I had the table translated into my twisted view of reality and could run my program against the entirety of WorldCat, parsing out the precious three bytes from the 008 and running my undoubtedly flawed translation on the result. I just love that “Unknown” came out on top. Somehow, after this journey, it seemed fitting.

With no further ado, here are the top 25 “countries” of publication from the records in WorldCat:

74,330,023  Unknown
52,460,566  United States
34,014,675  Germany
24,374,828  United Kingdom
21,009,805  France
 9,142,988  Japan
 8,706,853  China
 7,950,373  Spain
 6,649,599  Italy
 6,312,625  Netherlands
 6,142,256  Canada
 5,641,525  Switzerland
 3,725,639  Russia
 3,516,374  Australia
 3,310,194  Poland
 2,923,655  Denmark
 2,739,910  Sweden
 2,219,850  India
 1,996,800  Slovenia
 1,936,800  Austria
 1,612,948  Belgium
 1,518,478  Israel
 1,514,824  Brazil
 1,412,034  Mexico
 1,197,454  Finland

The full list is here. Knock yourself out. I sure did.