We have now posted a further revised Jing video with audio providing a brief, 4-minute overview of the LSNC Shared Portal. This is the actual intro overview video we circulated internally to provide all staff with a basic visual and feature orientation, before our more extended, in-house live demos to be conducted next week.

It’s not so easy to do a public video demo of our new Pika 4.0 case management system design changes, because of confidentiality issues, but we will post select screenshots reasonably soon so you can get a visual idea of changes we have made to that application.

For those interested, here is the recently approved TIG final evaluation report for The Findability Project.

This TIG project was funded for an 18-month period from January 2008 through June 2009. Much of the report will ring familiar to those who have followed the project here, since much of what has already been posted mirrors what would be required in a TIG evaluation report. Essentially, this public project site enabled us to give others in the legal services community an ongoing, if lagging, report of progress on the project, while at the same time considerably easing the process of writing up the evaluation report at the end of the project since we had already written most it as we went along.

We’re winding things down here, but we will continue to post here at least through the next TIG conference in early 2010. Among other things, we will be detailing how in finalized form we are integrating our project’s GSA test frontend functionality into a more expansive shared organization portal, part of our current deployment of a heavily customized version of Pika 4.0. We have finished the LSNC redesign of Pika 4.0 as well as a new LSNC shared portal “front door” (built on WordPress), both of which are scheduled to be in place and in use by LSNC staff the day after the Labor Day break.

Stay tuned, people!

Many months of fish to fry ahead, including bearing down on LSNC’s customized rebuild of Pika 4.0 and working out a solution for integrating our Google Search Appliance. So, laying low until October 1. See you back here this Fall and in early 2010 at the next Austin TIG, when we serve up the whole enchilada.

Think of it as the Revenge of the Empiricists.

A few days ago my earlier post about metadata, with the conclusion that “we don’t need metadata,” caught the attention of Daniel Tunkelang. It just so happens that Daniel is the chief scientist at Endeca, a prominent enterprise information company. You know, like, he knows what he’s talking about.

Long story short, Daniel commented on my post here and then, after I responded, mirrored our discussion in Does Metadata Matter, a new thread at The Noisy Channel, his blog that hosts vigorous discussions of search design and other enterprise information issues way above my pay grade. Daniel and others there critically but constructively took me to task because of the empirically unsupported general conclusion that metadata is not needed, and the specific failure to make clear the anecdotal basis for our conclusion about metadata on this project. As you can see from the discussion at Daniel’s blog, they killed me on the empirical point. In response, I better explained there than I did in my original post why our conclusion about not needing metadata on this project makes sense.

The discussion there ended on Christopher’s grace note, something that the TIG program and others within the legal services community will appreciate:

I also want to applaud your project which I should have done before, the mission your team has set is very admirable. It’s gratifying to see technology being put to use to help people who may not otherwise be able to have access to the same advances we do.

If you’re interested in enterprise search, you should consider paying The Noisy Channel a visit.

These are our contentions:

• Metadata matters.
• Metadata adds worth to data.
• Documents are data.
• Keywords are the essential data in documents.
• Keywords in context create knowledge.
• Documents have worth because they contain knowledge.
• Enterprise search finds keywords.
• Findable keywords yield documents.
• Findable keywords in context yield documents with knowledge.
• Knowledge in documents has worth.
• Metadata is not essential for enterprise search.
• We don’t need metadata.

What’s our point? Before answering that question, we invite understanding of the context: This project is about implementation of enterprise search within a large but not humongous non-profit organization. We’re talking about 170 paid employees, with easily an equal number of volunteers of one kind or another. So let’s say for purposes of context that we have 350+ real people using our networked infrastructure. We have two — count ‘em, two — IT guys. We’re not talking Fortune 500 here. We’re not even talking Fortune 500,000. That’s our world.

Working on this project, we have evaluated what we need from metadata as part of enterprise search implementation. Our conclusion? We don’t need metadata.

Or better said, we don’t need to add metadata for a Google Search Appliance (GSA) to accomplish what we want to accomplish with enterprise search. We could use metadata more — and there are several very impressive features in a GSA that can exploit external metadata and metadata biasing of search results — assuming the organization has the resources to organize and manage metadata. But as a practical matter, do we have the resources to go down that path and, ultimately, do we need it? No.

In fact, as part of this project, we have put a metadata model in place, a simple “labeling” or tagging system. It exploits our Sharepoint server installation with a practical (if kludgy) way to add metadata to files saved to a shared document repository. For example, when saving a file in a directory in our structural taxonomy, as the user navigates — say, to the Income Maintenance folder…

…a dialog box pops up with a prompt to add one or more optional “LSNC labels” to the file, associating the file with additional folders or categories in our taxonomy:

In the above example, an Excel spreadsheet with unemployment data is being saved to the “Unemployment Insurance” folder, a subfolder under the “Income Maintenance” top-level directory, but is also marked or tagged as “Data-Statistics-GIS” and “Employment.” Even then, this kludge only works with Microsoft applications, which is to say Sharepoint doesn’t work as cooperatively with other applications we rely on, like WordPerfect, Adobe Acrobat and others.

Regardless, is the addition of metadata to documents a good thing? Obviously, yes. Metadata matters. (Taxonomy matters, too… yet to what purpose?) Do you need to add metadata to documents for effective enterprise search, and specifically with a Google Search Appliance? Not really, not for what we are doing. Why not? Because improvements in search algorithms are such that metadata is not needed to help the search.

The poster child for these gains in enterprise search algorithms is, not surprisingly, Google whose GSA has matured considerably. Google is a verb. Microsoft (or Sharepoint) are not. A principal reason for that is Google years ago broke out early from the search-engine pack and raised the bar in terms of quality of search results. Google became what the average person now expects from search. That is why it is a verb. It is what most people do. They Google. Another reason is that Google simplifies search.

In the context of our project, at the scale and with the resources available to even a fairly large non-profit, what is practical or impractical in using metadata? And even if used, does it affect the quality of enterprise search results enough to warrant those additional costs in time and money?

So far, we don’t see it.

There is a great deal of work proceeding behind the scenes and several key project elements are converging as we move toward finalizing this public project. Among other things, we have been working through modest but practical solutions for better placement and targeting of our existing 300,000+ repository documents, while solidifying all the additional Google Search Appliance (GSA) targets in our enterprise search sights, described in Part Two. At the same time, we are in our own March (and April and May) Madness as we mount a rapid-fire round of trainings for each of our eight remote offices (spread out over 50,000 square miles of Northern California) on their new role in making Google Enterprise search work for them, which is to say for all of us. And as mentioned in earlier posts, we are working every bit as earnestly on our latest in-house build of the Pika CMS.

The infusion of new, future content into the simplified structural taxonomy we created is a separate challenge we will be posting about later. Dealing with our existing files is more immediate, more concrete. Groking those files has been one of the more interesting, at times hilarious parts of this project.

For those legal services programs interested in how the existing files from our eight local offices break out, here are the percentages for the seven most common “document” types:

• 67% – WordPerfect (WPD)
• 18% – Word (DOC)
• 9% – Portable Document Format (PDF)
• 3% – Excel (XLS)
• 1% – Text (TXT)
• 0.9% – Rich Text Format (RTF)
• 0.6% – PowerPoint (PPT)

(Discretion prohibits us from detailing the other file flotsam discovered on local office servers. That said, allow us to observe that some within our organization have extraordinarily good taste in photos taken by National Geographic, and not such good taste in music.)

We are totally on track for targeting most of our planned GSA targets: The existing office archive files listed above have long been targeted (although we still have a lot of work left to fit them into our structural taxonomy); over the last several months we have worked very hard to refresh and update (and remove, as warranted) the targeted content at LSNC’s various public websites; and we are very pleased with the quality of the GSA results we are getting out of Google Sites.

This is all good news. In addition, we are putting in place a few more content channels: Targeting the content in our organization’s seven private Google discussion groups, and a program-wide canvass for select hard-copy training resource materials for digital conversion and addition to the shared repository. It’s all good.

We have had one major disappointment: We discovered that there are significant, unanticipated technical challenges unique to the Pika CMS that thus far have prevented effective use of the GSA to target Pika content. The problem is not the GSA itself or configuring the GSA to target Pika. The GSA by design performs wholly benign, non-destructive crawls as it indexes targeted records. We did a huge amount of target testing and SERP evaluation, and we were very pleased — actually, thrilled is a better word — with the results we were getting from Pika. The unanticipated problem is that the current version of Pika is not well optimized for use as an enterprise search target. There are code anomalies in Pika that, among other things, cause it to auto-generate new case intakes and case records when it is crawled by the GSA.

After an assessment by Pika Software, it is now apparent it will take something in the neighborhood of 200 hours of work to make Pika more receptive, shall we say, to an external crawl by the GSA. (Ouch!) So for now, we have to put that part of the project to the side. File under: Lessons Learned.

As part of the current reporting cycle for our LSC TIG milestones, the primary funding source for this project, we did three things to evaluate our TFP test-bed installations in Sacramento and Chico. First, we talked to staff in both offices to hear them out about how the basic orientation to enterprise search and the temporary search portal went for them. We’ve already written up some about that here.

Second, in Sacramento we also took the time to observe some staff actually using the temporary search portal to see what their experience was actually like. Why? Well, not only is there is simply no substitute for actually talking to people about an application you are striving to evaluate, more importantly, there is no substitute for observing them while using it. Jakob Neilsen aptly sums up the differences in this regard as the contrast between “what people say” and “what people do.”

Third, we know that Bristow Hardin at LSC is going to want data because he has a Ph.D. in social science (from my alma mater; go Slugs!) and data is his life. So we also used SurveyMonkey.com to conduct a survey with an intentionally light touch to get some basic data, to see how users in our two test-bed locations were dealing with this whole enterprise (pun intended).

We put the survey up for one week and got a modest 17 respondents from the two offices. The survey had 18 questions and participation dropped off from 17 to 11 respondents from the beginning to the end of the survey. Among the results, in the order asked:

• 86% knew how to locate the TFP search portal; 14% did not.
• Asked their reaction to the elements on the temporary search portal page, they said they were “organized” (33%), “well organized” (42%) or “superbly organized” (17%); no one reported the portal page as anything less than organized.
• 92% were confident they understood what the elements on the search portal are and why they are there.
• When asked what they would want added to a search portal, only one person made a suggestion: “Can you add something so I can sort by file type”. (We have already added this filter feature to our next iteration of the GSA search results page.)
• When asked about locating the file servers where shared documents reside, 100% of 12 respondents said they knew how to locate the local file server directories; 11 of 12 said they also knew how to locate the program-wide shared file server.
• 100% knew how to locate their personal user directory and the user directories of others on the shared file server in the local office. (Okay, not a surprising result but we thought we should ask. You never know.)
• Asked how well they understood how the directory structures are organized on the program-wide shared document repository, one person out of 12 respondents said they understood some but not all of the directory names. All other respondent’s said they “get it” (50%); the organization “makes sense but I would do some of them differently” (17%); or the directory structure “makes perfect sense” (25%).
• Asked whether they are familiar with how to search using Google, 100% selected either “yes” (83%) or “are you serious?” (17%). (We have yet to encounter anyone within our organization who does not know what Google is or how to use if for search. Hey, but again, we thought we should ask.)
• We also asked whether the use of the TFP search portal was similar or dissimilar to using Google on the Web. Of 11 respondents, two thought it was “nothing like Google” and two others thought it was “something like Google.” The remainder thought it was “way Googley” (56%) or “exactly like Google” (9%).
• A key question, of course, is whether there is proof in the search pudding. Again, of 11 people who responded, one person felt searching was “not successful at all” and two were only “partially successful.” Of the remainder, four reported their searches were “successful” (37%); two reported their searches were “very successful” (18%); and two felt their searches were “dead-on successful” (18%), with an overall success rate of 73%.
• Finally, we asked staff whether they understood the search results. One person said “not at all.” Two said they “only partially understand” the search results. The rest said they either “do understand the search results” (9%); have a “very good understanding” of the results (55%); or “absolutely understand” the search results (9%), again an overall rate of 73% for those “understanding” the search results.

That’s the rough-cut survey, without any actual polish of our installation. We will be doing other surveys once we get everything in place, including the newly tweaked shared portal, and complete our roadshow to assure everyone knows how to exploit the bells and whistles we will be adding in over the next few months. Stay tuned.

[Update: In response to a question I got after posting this, there are 31 total staff in our local Sacramento and Chico offices. So, about 55% (17/31) of the staff participated in the survey and 35% (11/31) finished it, for a completion rate of 65% (11/17).]

We are at the last stages of our GSA “blunt instrument” mode. Within a few weeks, at the outside, we should have a spiffier, notably more user-friendly shared search portal in place for all LSNC staffers. By January, we should (hopefully) have a few select Google Sites properly targeted and the basic beginnings of a nice set of user-friendly file-type and collections filters in place. With those in place, we then plan on rolling out a formal roadshow to all our local branch offices to train everyone on the new shared search portal, the new search enhancements, and assist directly with the offices to get targeted files in place and properly organized on our shared document repositories.

Before getting to that point, to be sure, we have had some modest hesitation about unleashing the GSA on our staffers in its current, incomplete state of implementation. But the staff experiences with our temporary shared portal have been very positive, as detailed in the earlier post, Talk about zero learning curve. (And see the next post for the data results of our “test-bed” evaluations in our Sacramento and Chico offices.)

My instinctive temperament is to remain skeptical, even when faced with some measure of success. The day after we did the “blunt instrument” training, one of the younger, hot-shot lawyers stopped by my office and raved about how effective the GSA was at finding stuff. (It is sooooo hackneyed for any of us to emote this way, but at least a couple of times in the conversation he exclaimed, “This is sooooo awesome.”) Okay… but this is also an advocate who is only two doors down from me and the two IT staff I supervise, so he’s always picking up the positive, can-do, tech vibe that radiates out from the paid IT staff. (Remember, we’re a 140+ employee but still seriously tech-understaffed legal aid program.) He saw some of the stuff we were doing with the GSA even before we showed it to anyone else, so he was already on board with the project, big time. It works for him. But what about everyone else?

This real-world example from today tells me it is:

On the other side of the building, one of our stalwart paralegals was working on a client’s case involving an alleged overpayment in the CalWORKs cash-aid program. [CalWORKs is California's name for the TANF program.] He needed to quickly find a good example of a request for administrative rehearing to review the alleged overpayment. The paralegal went to the temporary search portal and, quite sensibly, typed in the search box the following, without quotes: “request for rehearing overpayment”.

He got about 500 search results and found a good, usable example on the first page of results. He also made this observation to me in a subsequent email: “I used several of the documents in the search. I did not have to download them. I just used the text version. Which is very useful and quick.”

Tell me about it.

In an earlier post about selecting Google Search Appliance (GSA) targets for this project, the narrative definitely edged toward the more abstract. We highlighted four principal sets of GSA targets: files on our newly created “shared document repositories”; repurposed intranet content being moved over from an MediaWiki installation on an old server; cherry-picked content available on LSNC’s varied public websites (LSNC maintains 13 distinct public websites and subsites); and select records in Pika CMS, the secured, web-based case management system used by all our advocates.

As far as it goes, this abstract list of GSA target sets fairly summarizes what we, as an organization, want to make transparent via enterprise search, which is to say make “findable” in ways not practicable without the GSA. This abstract list of GSA targets, however, fails to convey what we have done at non-abstract, practical level to make those targets useful to our larger search goals.

So, let me hit a few notes about several practical decisions we’ve made at launch as we target the GSA at real files offering real search results.

As described in Part One, when we first unpacked our GSA and aimed it, uh, somewhat aimlessly at any and every file on one of our local servers, the GSA did its job in killer fashion… and blew out our file limit. While one can proceed that way, we were always mindful that we had to sort out how to organize and structure the shared content that we seriously wanted to make searchable and findable. So, one of the first tasks we confronted on this project was to work out our thinking about “taxonomy,” resulting in the basic directory structures we have adopted.

That taxonomic “organization” step was essential to this project, but completing that particular project objective doesn’t translate directly into searchable content organized in a particular way. You see, there is this pesky little detail: Real people need to actually identify the existing and/or newly created files to be included and then somehow get the files in the directories on the shared document repository that are the target of the GSA.

Easier said than done.

In our case — particularly given the limited IT and support staff resources available to us as a typical legal services field program — we had to come up with some practical approaches to move existing files from any number of different locations to the designated shared locations or document repositories. (I will discuss how we handle adding newly created files, in a later post.) Here’s what we did with our existing files to fold them into the content targeted by our GSA:

1. Initially, include all existing “staff-specific” content, with an opt-out

We did find ourselves on the receiving end of a lot of staff enthusiasm about this project. Truly. But it is impractical and unrealistic to expect your individual legal services advocates and other staff to comb through all their thousands of files and then move them over to a different file server location. (Maybe it should be realistic to expect them to do this, but in our experience it just ain’t gonna happen. No way.) But there are tons of content gold in them thar files, so we had to figure out a way to initially get all that good stuff in place, even if not parsed out in a taxonomic sense, so we could target it.

To accomplish this, we first vetted with, and got buy-in from, all our local offices to do the following:

On the local project file server for each local office, we created a special project “archive” directory. Then each local office manager copied each individual staff member’s files wholesale over to a user-specific directory in this so-called archive directory. Having an unequivocal “opt-out” option was important to the success of this approach. Again and again, in formal meetings and informal discussions, we reminded office staff that they could ask that any or all files to be removed from these initial archive-file targets. No questions asked. There were a few such requests, but not a lot: One advocate asked that her files not be targeted at all, so we removed all her files; two others had less than a dozen files they wanted removed as targets, so we did so. No biggie.

The net effect is that this makes the targeted advocate files initially non-taxonomic, but in short order you have a huge repository that has a (allow me to exaggerate here, for literary effect) 99% chance of including pretty much everything the individual staff members would add if they “woulda, coulda, shoulda,” so to speak. In our case, this initially amounts to about 300,000 document files, the vast majority of which are advocate-generated files.

At launch, this does mean that these office-specific, bulk compilations of existing files added as targets include a significant number of drafts and duplicates that one would normally not include as a shared file if it were being added as a newly created file. For example, within our office culture it is not only common but actually expected that advocates not work in isolation on major cases. (We discourage the “lone eagle” model.) So, our early search-results testing shows that often the same file shows up in more than one target location because more than one advocate has a copy of the file in their archive.

It bears mentioning two other factors we kept in mind as part of this initial targeting of shared files: We double- and triple-checked with all management staff to assure nothing management-sensitive or -confidential was moved to a location where it could be targeted. Also, before we moved anything over wholesale, as described, we asked all staff to remove certain types of files that no one would reasonably expect to be part of the searchable content. Examples: Family photos, MP3 music downloads, YouTube videos, yada yada yada. Enough said.

We do have an approach in mind for “peeling off” these office-specific archives over time, to separate out the drafts and duplicates and place them within our taxonomic directory structures. More about that later.

2. Using Google Sites as the platform for our existing intranet content

I recall having a passing conversation with Gabrielle Hammond at last year’s TIG conference about how we were holding off on further intranet development while waiting to see how Google implements its JotSpot-based wiki application, now known as Google Sites.

Well, people, we now know what Google Sites is all about and we love it! For the last several years we had been using MediaWiki as the publication platform for our intranet, but we are in the process of replacing it for our internal wiki needs. We are about half way through that process, which should be completed shortly after the first of the year.

One big bonus of moving our intranet content to Google Sites is that it is quasi-tailor made to work with both the GSA and Google Analytics. I say “quasi” because the interactions between them are good but hardly optimal at the moment. For example, only days ago Google Analytics for Google Apps was rolled out, but the quality of the data we are getting so far is not so easy to get a handle on. More importantly, Google promises GSA integration with Google Sites, but it is still a buggy implementation. We have easily targeted test site pages within our domain’s Google Sites, but have hit a wall with getting the GSA to properly return search results on the indexed content within files uploaded to Google Sites. Turns out we are one of several organizations that have identified this problem and Google Enterprise support assures it will have a fix with its next software upgrade, in about a month or two. We (and our GSA consultant) are confident this will work in due time, but it’s one of those details we have to wait on for now.

3. Updating our public web content

Over the last 10 years, LSNC has placed an enormous amount of its advocate content out on the public Web. But one recent example is the California Food Stamp Guide, a prime example of public content that our advocates can search at that site, but would want to be able to search directly via our GSA shared portal. It is also one example of a content cluster that can be part of or its own GSA collection. (“Collections are logical views of information in the index, as defined by URL patterns. This allows you, for example, to index the entire contents of your intranet, but then divide it up into logical groups of content.”)

Implementation of The Findability Project has prompted some public housekeeping. Our target testing of our public content, predictably, reveals that we have stuff out there that is, well… past its shelf-life, shall we say. So we are working on a systematic way to thoroughly review and clean up that public content. It is obvious but important: Current and correct public content means better search results via the GSA. (Apologies to the larger legal services community for not doing it sooner.)

4. Targeting our case management system

We consider our Pika case management system a key, long-term GSA target. But we are not there yet. We have prioritized getting all the other targeted content organized and in position, with clear protocols in place. We also are busy reworking on our shared portal, which will integrate the GSA search functions and provide users with (hopefully) intuitive ways to filter their search results, search select content collections, and provide the users with some nice Google GSA touches like OneBox searches, among other features.

That all said, being able to target our case management system is a total no-brainer and perhaps the most practical of necessities. In a given day, there is likely nothing more common or more vital to our work for clients than the search for information within our case management system. The native search functions built into the current version 3.07 of Pika are good. But we are optimistic that we can exploit the GSA to make those searches even better. And certainly more integrated with everything else in our new enterprise search universe.

This is a non-extra credit read, somewhat tangentially related to “taxonomy.” But, hey, this project is hard work and I’m entitled, as are you, to have some fun, no?

I previously alluded to how I sat down with each of the advocates in our flagship Sacramento office to view and discuss how each organized their files. I’m not suggesting you need to do this with everyone in your organization. But doing so with at least a fair cross-section of your people will teach lessons not likely learned otherwise. Let’s call it “reality.”

Three particular experiences in doing this are favorites of mine.

The first relates to the same advocate who, the good sport that he is, agreed to let me post a photo of his hard-copy file organizational scheme. When I sat down with him to take a look at how he had organized his files on a local server, it was a gloriously indulgent vision of horizontal organization. The guy (who is one of our best welfare lawyers) had 595 MB in 2,623 document files … wait for it … in one folder. Whew, talk about going “broad-and-shallow”! Because his file-naming conventions include the relevant client name, I really can’t give you a screenshot of this Ripley’s moment. There was something really extraordinary about this encounter, almost anthropological about it, akin to witnessing an indigenous tribe in the primeval, untouched by the outside world.

The second involves the polar opposite, another highly regarded lawyer who is hyper-organized. And well he should be, with 3,616 work files totalling 3 GB tightly organized in 671 folders and subfolders. Peter Morville would not likely approve of his organization scheme, I don’t think, since this advocate went for a “narrow-and-deep” hierarchy, with nine top levels and 662 subfolders, as many as five levels down.

And then there is the third anecdote, my favorite of all. As I went through this attorney’s files, I was authentically impressed by how sensible and well organized her directory structure was. While I organize my personal directories differently, hers were organized much the way many advocates in the program do (by cases or projects or substantive area), easily understood and well suited to how she works. Broad enough to hit all her bases, yet with enough subfolder depth for her to “navigate” to find particular files. A good, functional result for her.

As I showed her the project taxonomy, she was fine with the top-level selections. She understood immediately and instinctively why those choices had been made and had no quarrel with them. But when I showed her that the subfolder organization only went one-level deep, her facial expression changed noticeably. She said nothing but I could see her anxiety. So I asked her, “You look worried, a bit. What are you thinking?”

She paused and then she asked, “If you organize the shared directories this way, with only one level below the topics, how can anyone ever find anything? I don’t work that way.”

This was my response:

“Have you ever used Google?” (She good-naturedly looks back at me with her best “give me a break” smirk on her face.) “Well,” I continued, “when you search with Google you are usually able to find what you are looking for, right?”

“Of course,” she answered.

Then I said, rhetorically, “Do you think Google ‘organizes’ the Web in subfolders like you do?”

“Oh, I get it,” she said. “Everything doesn’t have to be organized that way if I have a way to Google it, right?”

(with permission, but anonymously to protect the guilty)