How reliable are our initial results?

Having spent time this spring refining our methods for pre-processing input data (book records from Stationers' Company and from the copyright libraries) and the algorithm for matching records of the same book across datasets, we've recently started a new phase we're calling "match categorisation". In short, the goal is to understand how reliable or unreliable the outputs of the matching algorithm are. The algorithm can't actually determine whether two records refer to the same book or not. Instead, it gives scores for the similarity of book title, author name and publisher name. By examining the results from our sample year of 1863, we've started grouping these scores into three categories:

- TRUE MATCH: the similarity is so close that we can safely assume the books are the same

- FALSE MATCH: the similarity is so distant that we can safely assume the books aren't the same

- CHECK MATCH: the similarity is close enough that it's worth a human checking and deciding whether the books are the same

If the title, author and publisher scores are each 100 out of 100, we're obviously in TRUE. If the scores are each close to 0, we're in FALSE. But how far below 100 do scores need to be before it's useful for a human to check? How far above 0? What if we have a high score in one or two of the fields but low in the other(s)? How do we evaluate a "null" score, which is different from 0 because it means that information was missing in one of the inputs (Stationers' Register doesn't always specify author names; at least a quarter of the National Library of Scotland's entries don't specify a publisher)?

The motivation here is that we're dealing with thousands or tens of thousands of books per year, and eventually five distinct libraries plus Stationers' Register, so manually checking every match isn't feasible. We want to direct our human labour towards the items in our results where it's most necessary. Looking at sample results helps us get a sense of how reliable a given set of scores are, and also - crucially - how many items fall into which categories. Ideally, we want the CHECK category to be a modest proportion of the overall results. Happily, having so far examined scores between 80 and 100, only 29 out of 4338 potential matches would qualify as CHECK. Of course, this proportion is likely to get larger as our scores get lower. Still, it's good to know that in this upper range, we can have high confidence in the algorithm's matches.

Browsing results also has the added benefit of highlighting intriguing titles. Some that stood out to me were:

- 450 questions on the French language (why so many, and yet such a round number?)

- The power of the tongue; or, Chapters for talkers (something paradoxical about a silent book dedicated to "the power of the tongue" - what kind of "talkers" is this book for anyway?)

- Ought working-men to be fined for claiming the franchise? (worrying that that question even needed to be asked - a reminder that in Britain, as in many countries, the right to vote was hard won)