PERL Data File Search Script

In the Powershell - Get Inventory Script process I posted last month I briefly mentioned a web based server script that we use to search the data output files we copy to our intranet server. Because I have noticed a significant boost in traffic to the website hosting the Get Inventory Script, I have cleaned up and rewritten the PERL script I developed for searching these files and I am making it available here. I used PERL for this task as the process runs for us, on a Linux intranet server, which of course has PERL natively installed.

This PERL script is designed to locate all the HTML files in a single directory. These files are in fact named for the hostname of the workstations that originally generated them when they were created by the Powershell - Get Inventory Script, so the script either searches for computer names and displays the complete data sets for each computer on one page, or it searches each file to locate the data string requested and displays the results. Which type of search is in fact done, is dependant on the users choice in the original search (Hostname or Keyword). If a Keyword search is selected any HTML tags in the source code are removed, before comparing the remaining text data with the string being sort. This reduces the number of false hits!

By searching all the data files harvested (copied) from the workstations there is no need to set up a database to store the data and the interaction overhead this causes. Because we are seaching ALL the files content we are able to effectively isolate discrete data quickly. For example: If I am asked for the number of installations of EndNote we have. I can set up a Keyword search for EndNote the returning data will advise me of the number of workstations with Endnote (124 as of writing) and then list for me the names of the workstations followed by the version details of EndNote on each workstation.

Links:

Download and implement the PERL Data File Search Script.

View the Powershell - Get Inventory Script post.

Compare your Library with LibraryThing

Tim Splading of LibraryThing just release and new set of data, An export of all the ISBN's that LibraryThing knows about.

Tim writes:

Over on Next Generation Catalogs for Libraries, NCSU's Emily Lynema, asked me:

"Do you have any idea of the coverage of non-fiction, research materials in LT? Have you done any projects to look at overlap with a research institution (or with WorldCat)?"

No, we haven't. And I'm dying to find out, both for academic and non-academic libraries.

So I decided to see how hard it would be to write a script to compare the LibraryThing dataset against a simple export from our library system. It turns out it didn't take to long. And I have posted the perl source code on my personal website so you no longer have that as an excuse for not helping Tim out.

Here are the stats for The University of Waikato Library:

Out of approximately 500,000 Bib records in our database I found only about 178,460 unique ISBNs. LibraryThing has 1,774,322 ISBNs so they have ten times as many as us! Note: This was found to be an error during normalisation. The number is now 292,073


UoW Library and LibraryThing have 45,259 73,377 ISBN's in common, which means that LibraryThing only has about 15% of the ISBN's we have or in other words 75% of our ISBN's are ones that LibraryThing doesn't have. This seems like a surprisingly large number given how much larger LibraryThing's database is. Tim may have the right idea though, as he said he suspects LibraryThing users tend to have the paperback (cheaper) copies of books rather than the more expensive hardcover versions that libraries tend to buy. It would be interesting to see if that is infact the reason, or if we just have very different set of resources from what is cataloged in LibraryThing.

Database	Total ISBNs	Unique ISBNs	Percentage Unique
University of Waikato	292,073	218,696	74.88%
LibraryThing	1,774,322	1,700,943	95.86%

Total ISBNs in common: 73,377


I figure since they asked the question, NCSU Libraries should be next...

External Links:

• Perl Script for comparing two lists of data
• LibraryThing's Thingology Blog

Update: Updated Figures after discovering I had dropped a whole bunch of ISBN's when normalising them.