A few days ago my earlier post about metadata, with the conclusion that “we don’t need metadata,” caught the attention of Daniel Tunkelang. It just so happens that Daniel is the chief scientist at Endeca, a prominent enterprise information company. You know, like, he knows what he’s talking about.

Long story short, Daniel commented on my post here and then, after I responded, mirrored our discussion in Does Metadata Matter, a new thread at The Noisy Channel, his blog that hosts vigorous discussions of search design and other enterprise information issues way above my pay grade. Daniel and others there critically but constructively took me to task because of the empirically unsupported general conclusion that metadata is not needed, and the specific failure to make clear the anecdotal basis for our conclusion about metadata on this project. As you can see from the discussion at Daniel’s blog, they killed me on the empirical point. In response, I better explained there than I did in my original post why our conclusion about not needing metadata on this project makes sense.

The discussion there ended on Christopher’s grace note, something that the TIG program and others within the legal services community will appreciate:

I also want to applaud your project which I should have done before, the mission your team has set is very admirable. It’s gratifying to see technology being put to use to help people who may not otherwise be able to have access to the same advances we do.

If you’re interested in enterprise search, you should consider paying The Noisy Channel a visit.

• Metadata matters.
• Metadata adds worth to data.
• Documents are data.
• Keywords are the essential data in documents.
• Keywords in context create knowledge.
• Documents have worth because they contain knowledge.
• Enterprise search finds keywords.
• Findable keywords yield documents.
• Findable keywords in context yield documents with knowledge.
• Knowledge in documents has worth.
• Metadata is not essential for enterprise search.
• We don’t need metadata.

What’s our point? Before answering that question, we invite understanding of the context: This project is about implementation of enterprise search within a large but not humongous non-profit organization. We’re talking about 170 paid employees, with easily an equal number of volunteers of one kind or another. So let’s say for purposes of context that we have 350+ real people using our networked infrastructure. We have two — count ‘em, two — IT guys. We’re not talking Fortune 500 here. We’re not even talking Fortune 500,000. That’s our world.

Working on this project, we have evaluated what we need from metadata as part of enterprise search implementation. Our conclusion? We don’t need metadata.

Or better said, we don’t need to add metadata for a Google Search Appliance (GSA) to accomplish what we want to accomplish with enterprise search. We could use metadata more — and there are several very impressive features in a GSA that can exploit external metadata and metadata biasing of search results — assuming the organization has the resources to organize and manage metadata. But as a practical matter, do we have the resources to go down that path and, ultimately, do we need it? No.

In fact, as part of this project, we have put a metadata model in place, a simple “labeling” or tagging system. It exploits our Sharepoint server installation with a practical (if kludgy) way to add metadata to files saved to a shared document repository. For example, when saving a file in a directory in our structural taxonomy, as the user navigates — say, to the Income Maintenance folder…

…a dialog box pops up with a prompt to add one or more optional “LSNC labels” to the file, associating the file with additional folders or categories in our taxonomy:

In the above example, an Excel spreadsheet with unemployment data is being saved to the “Unemployment Insurance” folder, a subfolder under the “Income Maintenance” top-level directory, but is also marked or tagged as “Data-Statistics-GIS” and “Employment.” Even then, this kludge only works with Microsoft applications, which is to say Sharepoint doesn’t work as cooperatively with other applications we rely on, like WordPerfect, Adobe Acrobat and others.

Regardless, is the addition of metadata to documents a good thing? Obviously, yes. Metadata matters. (Taxonomy matters, too… yet to what purpose?) Do you need to add metadata to documents for effective enterprise search, and specifically with a Google Search Appliance? Not really, not for what we are doing. Why not? Because improvements in search algorithms are such that metadata is not needed to help the search.

The poster child for these gains in enterprise search algorithms is, not surprisingly, Google whose GSA has matured considerably. Google is a verb. Microsoft (or Sharepoint) are not. A principal reason for that is Google years ago broke out early from the search-engine pack and raised the bar in terms of quality of search results. Google became what the average person now expects from search. That is why it is a verb. It is what most people do. They Google. Another reason is that Google simplifies search.

In the context of our project, at the scale and with the resources available to even a fairly large non-profit, what is practical or impractical in using metadata? And even if used, does it affect the quality of enterprise search results enough to warrant those additional costs in time and money?

So far, we don’t see it.

Much more is coming. These first several steps simply set the stage for getting into deployment of the Google Search Appliance (GSA), what we think are the more interesting and challenging parts. Among other things, upcoming are documentation of how to install and configure the connector to the GSA, and walkthroughs explaining how we have configured our GSA to accomplish the goals of the project.

(Kudos to Ed Lachgar, IT extraordinaire at Legal Services of Northern California for writing up these first 16 sections.)