Advanced Searching in ShipIndex.org

Written by:
Published:
Categories: Research Help

Our “Introduction to Searching ShipIndex.org” Research Help Guide provides an overview of the basic parts of searching ShipIndex.org. There are some advanced features, as well, and they’re worth knowing about.

It seems straightforward – type in a ship name, get results. Actually, there’s a whole lot more that you can do in the database, and it is, admittedly, kinda wonky and a bit tricky. Please watch the following video to better understand how to use the search features more effectively.

Here’s an overview of the most important points:

  • Remember that your search only brings back that ship name. There’s lots more after you click on the “See other matching ships” button.
  • To limit a search to just the ship name field, use the term “@ship_name” and then the name of the ship. For example, “@ship_name union“.
  • To search for a ship name that starts or ends with a specific word, use these characters: use “^” (the carat character) in front of a word, to indicate that the ship name must start with that word; or “$” (dollar sign) at the end of the word, to indicate the ship name must end with that word. For example:
    • “@ship_name ^union” to display ships whose name starts with “union” – Sample search
    • “@ship_name castle$” to display ships whose name ends with “castle” – Sample search
  • Use “*” (asterisk) to expand a search, but you must start with at least two characters.
    • A search for “acht*” returns ships where a word that starts with “acht” appears in the ship name or the citation. In this case, it includes “Acht Gebroeders”, “Achtienhoven”, “Martini Achter”, and “Y’acht Tu Put Family First”. Sample search. It does not look for “acht” within a word; it only looks for words that start with “acht”. (The apostrophe in “Y’acht” appears to create a new word; note that no entries with the full word “Yacht” appear in this search.)
    • If you search for “@ship_name *acht” you’ll get lots of citations for ships with names like “Deltagracht”, “Dotte Yacht”, and more. – Sample search.
    • And, if you put an asterisk before and after “acht”, you’ll get a combination of both searches from above, plus ship names where “acht” just appears within a word, as in “A Lil Naughty Yachty”. – Sample search.
  • You can even exclude specific words from ship names by putting a “!” (exclamation point) in front of the terms you want ignored. A search with the phrase “@ship_name ^mary !annie” will return all entries where the ship name starts with “mary” except for those that also have “annie” in the name. – Sample search

The full help page will give you more guidance as well. But it’s clear that there’s a lot you can do when searching the database.

An important concept here is that the ShipIndex.org database serves as a de facto authority file for vessels. There’s no larger collection of vessel names out there, and if you’re having a hard time finding information about a ship with the name you have, you might do some searching to see if there’s a different version of it. At a genealogy conference in England, a person came to me seeking information about a ship named “maid of sussex”. We found no ship with that name. Now, it might have still existed, but the ShipIndex.org database is pretty comprehensive at this point. By playing with the database, and searching for just “sussex”, for instance, then clicking on “See other matching ships”, we found “Sussex Maid,” which was, most likely, the ship she was seeking.

There are many ways of using and searching the ShipIndex.org database, even if you’re not subscribing. Give it a try, and let me know if you have questions or comments.

Introduction to Searching ShipIndex.org

Written by:
Published:
Categories: Research Help

The video below shows effective ways of using the ShipIndex.org database. It’s good to know what the database can, and cannot, do.

Here’s an overview: ShipIndex.org tells you what books, journals, websites, databases, resources, and more, mention the ships you want to learn more about.

There are, basically, two versions of the database: the free database and the subscription database. You’ll find over 150,00 citations, and lots of great content, from many significant books in maritime history, in the free database. You’ll also find links to subject headings in WorldCat, though have significantly declined in value, due to a change by OCLC, WorldCat’s owner, this past month. Read more about that here.

The subscription database has over 3.2 million citations in it, and it’s always growing. The subscription database contains lots more online content than the free database, plus many, many new books in maritime history. All content that’s been added since 2009 has gone into the subscription database, except for updates to the WorldCat file.

Watch this video to learn more about the specifics of searching the database, and a bit more about what you can expect:

Subscribing to the database is inexpensive and easy — you can subscribe for a set period of time, from two weeks to a year, or you can subscribe on a monthly basis, in which the subscription will continue until you cancel.

You might also find that an academic or public library near you subscribes to the database. If you’d like to suggest that your local public or academic library consider a subscription to ShipIndex, please take a moment to tell them about the database! We can set up a trial for them, and if they let patrons access resources during a trial, you could use it then. Ask your librarian to contact us about options for subscribing!

We believe strongly in the importance of maritime museums, so we offer free access to the full database for all full members of the Council of American Maritime Museums (CAMM). A museum staffperson can contact us to get access set up on their campus and in their buildings. If they so choose, they can offer this access to museum visitors, as well, such as in their library or through their wifi network.

Basic advanced searching

When you search for a ship name, you’ll often see the following phrase at the top of a results page: “See other matching ships”.

This means that, in this case, you’re looking at ships whose full name is “Eagle”, excluding any prefixes like “HMS”, “USS”, or of course, “USCGC”.

When you click on the “See other matching ships” link, which is in the red box shown above, you get this result:

Each ship name here either contains the word “Eagle” in the name, or somewhere in the citation. With a common one-word name like “Eagle”, this can be overwhelming. The next step is to narrow down the search, by using this search term: “@ship_name eagle“, to get this result:

Now every citation has the word “Eagle” in it, which definitely narrows down results. But there’s more that can be done to get to better results. See our video on advanced searching, to learn more.

WorldCat (April) Fools

Written by:
Published:
Categories: New Content, Uncategorized
Tags: ,

This is the first of a few new blog posts. It’s April 1, April Fools Day, but there is, alas, no foolin’ around here. It’s just bad news, start to finish, with the WorldCat subject entity links that have been in the free ShipIndex database since 2009. Read on, to learn more.

When ShipIndex switched from a personal project to a real company, back in 2009, I put all of the citations that had been in the “project” database, into the free database. Anything new was going to go in to the subscription database. I had been in contact with researchers at OCLC, the very large library cooperative that ostensibly helps libraries manage their resources, and shares those holdings, via their publicly available database called WorldCat. I worked with several remarkable people there, who through the years generated a list of all of the “identities” for ships in WorldCat.

This meant we could find books or manuscripts that were by or about ships. So, a book about a ship is easy enough to imagine – the book The Royal Yacht Britannia: The Official History is clearly about that vessel. Having a specific subject heading about that specific yacht makes it easier to differentiate between vessels with the same name. It also created links to books by ships, which often meant logbooks our individually-kept personal journals by people who were on board a vessel. It was a great way of uncovering a lot of useful content about ships that wouldn’t be found otherwise.

But the folks at OCLC said this content needed to be in the free database, not in the then-nascent subscription database. That was fine with me; it was worth including that content and keeping it freely available. The file has been updated occasionally over the past few years, and has always been in the completely free database.

Two or three weeks ago, I was doing some searching, and looked at WorldCat records. I saw notices indicating that the OCLC Identities project, on which these links were based, was going away. This past week, all the links to WorldCat failed. OCLC has ended this project, and with it, links to lots of content that used to be in the database. They’ve also removed linking by Library of Congress Control Number. You’re just searching by phrase now – this seems like the total antithesis of the ideals behind Linked Data.

I have figured out a way to make these links mostly work. The links are now searching by subject headings, rather than by control numbers or identities. As a result, in many cases, they won’t work effectively. In the old file, there was a search to an identity for a ship named “104”, and it specifically went to the entry for a specific ship with that name. Now, the search is for any entry that has both terms “104” and “ship” in a subject heading, so instead of one or two specific results, you get 38 results. Some refer to ‘cruise 104’ of a different vessel. It’s really too bad. Searches for ships like “Mary” are going to terrible, because they’ll include ships named “Mary Rose”, “Mary Ellen”, “Mary & Frank”, “Mary Smith”, and any other ship that has ‘mary’ as just part of its name – instead of going directly to the ship you’re researching. A search for a single, common word ship name, like “Eagle” or “Union” or “James” or “Monitor” or “Wasp” is going to return any record that has that word anywhere in the list of subject headings, even if the term doesn’t have anything to do with a ship name. Connections we’ve made, between specific vessels represented in WorldCat and other citations for those specific vessels, are probably no longer relevant.

OCLC did some work in creating Virtual International Authority File (VIAF) records for some ships, as well. Again, this was great in differentiating between ships with different names. But as far as I can tell, that is also all wiped out.

I’m disappointed and frustrated by this change, as I am with most of what OCLC has done to WorldCat over the past few years.

I’ll leave with this image I collected from WorldCat a few weeks ago, telling me that a copy of a book I wanted was at the State Library of South Australia, but that library is further than the distance to the moon:

My frustration with WorldCat – and OCLC – is ancient news, but it does just keep getting worse. This is really unfortunate. This is NOT a good April Fools joke.

ONE THOUSAND RESOURCES!

Written by:
Published:
Categories: Books, New Content, Website Improvements

Wow – it’s been quite a while since I last posted anything to the blog. One would be forgiven for thinking we’d disappeared. But we haven’t. In fact, we’ve been working away, adding new content, adding new (mostly backend) functionality, trying new marketing work, and more. But first, we’ve hit a big milestone — yesterday, we loaded our ONE THOUSANDTH resource! I’ll admit, I’m pretty astounded at that. Here’s a list of what we’ve added since the last blog post:

So, we’ve been working hard at adding new content. Getting to ONE THOUSAND resources is HUGE, in my book! We’ve got more to go, I assure you.

On the technology side, we have converted most of our subscription processing from PayPal to Stripe. We think that’s better for us, and better for our customers. If you have an opinion otherwise, I’d be glad to hear it; for now, we think it will make things easier for users. We have some more work coming soon, this time on the login process.

As always, if you have suggestions for content to add to the database, or questions about how it works, please contact me at comments (at) shipindex (dot) org, and share your thoughts, suggestions, and ideas. Until then, fair winds!

Specific resources that have been removed from the ShipIndex database

Written by:
Published:
Categories: Data Correction

In the past I haven’t kept a running tally of content that has been removed from the database, but I have mentioned it. The database takes a huge hit when I have to remove 380,000 citations in one go! As mentioned in the prior post, we’ve looked over all online resources, just to make sure they’re working. Many are not. I am going to make a note of the ones that I’m deleting from the database on the table below; I’ll be updating this over the next month or two as a work through all of the problematic online resources.

RESOURCE NAME NOTES ABOUT THE RESOURCE
National Small Boat RegisterAs noted here, the database has been taken offline for an unknown period. Let’s just hope it does, eventually, return.
Blue World Web MuseumGoogle still shows the underlying data, but it’s not available. This collection had great links to images of ships from artwork.
Containership-info.comJust not there anymore…
NOAA History: NOAA Coast and Geodetic ShipsI could not find any of the NOAA history pages anymore. A few pages may exist on the NOAA pages, but they’re not organized in the same way as in the past.
Union List of Historic Vessels in North AmericaThis one threw me for a loop,

I’ll keep adding to this list as I work through our data.

More resource additions, and a few deletions, too

Written by:
Published:
Categories: Data Correction
Tags:

I provided a list of new resources added to the ShipIndex.org database, back in November. We’re always adding new content, so I’ll include a list of the new stuff at the bottom of this post in an upcoming post. But I also need to address the fact that we have to remove some stuff, as well.

Monographs, or books, are great as resources, because once they’re added, we know they’re not going anywhere. Those books are in libraries and collections around the world. You may not be able to access them right away, but eventually, you’ll be able to do so. Online resources are great because you can link to them RIGHT NOW. Boom, click, done. Except, when that doesn’t work.

Online resources are great for convenience, but not for reliability. They change and disappear all the time. For some reason, website publishers still don’t realize that if they’re going to change their URLs, they’re going to break access for repeat users. They can include redirects, but rarely do. Too often, website publishers switch from a straightforward linking and searching structure to some fancy search tool that removes prior direct links, and makes new direct links impossible. Tim Berners-Lee, the creator of the World Wide Web, defined five stars for Open Data. One of those is making sure that people can point to your stuff. That is, make sure they can link to it easily. If you are required to do a search to get data, rather than also having a direct link that would get a person to your content, then you’re doing it wrong.

As one example, there’s a brand new “Royal Navy Loss List searchable database” at https://thisismast.org/research/royal-navy-loss-list-search.html. It’s nice that this data is here, and you can do a search for, say, “Indefatigable”, and find a record. But you cannot provide a direct link to the “Indefatigable” results, without going through that search page, which is really annoying, at least for those who care about open data.

Unfortunately, in this case, the MAST Loss List database only meets one of Sir Tim’s five stars toward Open Data. They could — and should — do much better.

But even worse is the total disappearance of online resources. Our data team recently reviewed all online resources in the database, and found quite a few which have disappeared, or are currently offline. We discovered a lot of problems that we’ll need to address. In some cases, the fix is pretty easy because there’s an obvious change to the URLs in the database. This was the case for the Bremen Passenger Lists; we fixed them, and they’re accessible again.

For others, though, we see bigger problems. Take the UK’s “National Small Boat Register”, for instance, which was hosted by the National Maritime Museum in Cornwall. At https://nmmc.co.uk/explore/databases/, you can see that the museum reports in an undated note, “The NSBR is currently offline whilst we create a new and improved website. We will have it up and running again as soon as possible. Please check back for further updates.” WHAT???

I’m all for thoughtful and improved websites, but why take down the old one when you’re building the new one??? Why not just keep it up until the new one is live and working?? The old one worked, didn’t it?? (It obviously did, at one point, when we added it to the database.)

There’s nothing to do but delete the National Small Boat Register contents from the ShipIndex database, and hope we’ll discover the replacement database when — if — it is ever put back online.

The Blue World Web Museum recently disappeared, as did other smaller resources. If you manage a vessel database that you can no longer keep online, please, please, please, contact me at comments (at) shipindex (dot) org, and give me a chance to see if we can save that resource for you.

I think I’ll start a separate blog post that lists the online databases that have disappeared; if you know of new sites for any of these, or contacts for folks who might be willing to offload that work to ShipIndex.org, please do let me know.

This got quite long, so I’ll create a separate post that lists the recently-added new content, in a day or three. (After making the post about lost databases, I suppose.)

Last few months of new content!

Written by:
Published:
Categories: Books, New Content

Goodness, it’s been a while since I added a blog post. We do have a lot of new content that’s been added, and some new great functionality, as well — I need to write something about that, since it is its own big step forward.

But for now, let’s list the content that has been added to the ShipIndex database since May:

There’s more to add soon, and maybe enough to get us over indexes to 1000 resources in the database, before the end of the year. We’ll see!

We have a lot of files still left to process, and we’re going to be adding a lot of new files, too, soon. So, as always, there’s more content coming soon. If you know of a title whose index should be added to the database, please do let me know, at comments (at) shipindex (dot) org — now’s a great time to get some more titles on the list, so they’ll be processed soon!

Recently added content, May 2021

Written by:
Published:
Categories: Uncategorized

OK, so “recently” isn’t necessarily accurate here; I think that this list covers content added in the past year, actually. We are always adding content to ShipIndex.org, but sometimes it’s slow going. So here’s a list of the content that has been added since my last content post,

New content:

As you can see, it’s a lot of content, even if we haven’t been bragging about what we’ve added through the year. As always, please send a note to comments (at) shipindex.org if you know of a title that you think should be added to the database.

Most Popular Vessel Names in the US

Written by:
Published:
Categories: New Content, Uncategorized
Tags:

I updated the Merchant Vessels of the United States database today. That’s a big file (~375k entries) and it serves as an interesting collection of personal and merchant vessels.

(There’s a minor error in the import, in that about 10% of the entries – in the Os through Rs – are duplicated. I’m working on correcting that problem. Also, apologies about the layout in this blog post, particularly with the tables. Not sure what the problem is, but I’ll try to correct it.)

Unfortunately, the US Coast Guard has changed their system, and NOAA has dropped their version of the database altogether, so you can no longer link directly to a specific ship. This is very frustrating, but I can’t control other sites’ setups. The URL will take you to the search page, and you can search again for the ship name that you’d found in ShipIndex.

The Coast Guard has also removed tons of personal information about owners of recreational vessels. The remaining information will still be useful to some.

MVUS also creates an interesting opportunity to look at a really large data set, and get a good sense of what vessel names are most appealing to the most people in the US.

Continue reading →

Unique Vessel Identifiers Added to ShipIndex.org – the short version

Written by:
Published:
Categories: Data Correction, Website Improvements

 

I wrote a very long blog post about our new use of Wikidata identifiers, and how it changes the ShipIndex database. It might be too long for some. Here, I hope to limit my overview of these changes to just four paragraphs (not including this one).

The ShipIndex database has grown a lot over the past decade, and now has over 3.5 million citations in it. For very common ship names, this makes it too hard to find information about a specific ship. How do you find information about the right America, when there are 2,378 different citations to work through?

The solution is unique identifiers – basically a specific identifier for each hull. This also allows us to bring together citations for the same ship as it changes names. To make this work, we’re using identifiers from Wikidata, plus local identifiers when Wikidata doesn’t yet have one. Wikidata makes it easy to use Linked Data, so that we can uniquely identify and share items and concepts: the identifier Q82925, represented at https://www.wikidata.org/wiki/Q82925, specifically refers to the author Joseph Conrad, while Q1278752, at https://www.wikidata.org/wiki/Q1278752, specifically refers to the ship named after Conrad, now at Mystic Seaport Museum. It also refers to the original name of that ship, Georg Stage, but differentiates it from the current ship with that name. Similarly, the identifier Q838125 refers to USS Hornet (CV-8), and Q1141355 refers to USS Hornet (CV-12).

When you look at the ShipIndex page for Hornet now, you’ll see eleven different ‘cards’, each one referring to a specific hull, or vessel. Click on any of those ship names, and you’ll see only the citations that have been associated with that unique identifier. And note the URL for each card – it includes the Q-identifier, so others can easily link to it as well, without needing to create some new URL. In addition, many citations have not been associated with any card, because we cannot determine to which vessel the citation refers – at least, not without going back to the original source.

In the future, I hope to offer ways for individuals to help grow this resource, and maybe create a way that people can share their own information – images, reminiscences, comments, online links – about specific vessels. Until then, I’ll be working away at associating as many citations as possible to specific vessels. I hope that this improves your experience with ShipIndex, and helps everyone do more and better maritime history research.

For more information about what we’ve done here, check out the much-longer blog post.