ECM systems and Search - Either-Or?

So you want to manage your content and then you want to find it too. Sounds reasonable. Must you pick just an enterprise content management (ECM) or a Search system, one or the other but not both? That seems like a crazy proposition. Although most people (myself included) consider Search and Enterprise Content Management solutions complementary, you’d be surprised what some others think, especially when there is a shortage of funds. In this era of tight budgets, it isn’t surprising to find some who reason “well, if I can’t manage my stuff, at least I can find it when I need to manage it.” A shortsighted view at best, you may rightly think. If you don’t manage your files, and you do search and fine them, then you may find a great many versions and not be sure which one is the authentic version. Oh yes, you can always check the file creation date. And if that file is a Word document, and someone clicked “save” after printing a Word document, then that date itself isn’t meaningful. So you can find things but you can't be sure about the value of what you've found.

And what about the reverse: Managing your files but having no search solution? Luckily that is less of an issue. Every ECM system, from SharePoint to Documentum to Alfresco to name-your-favorite has a built-in search system. Why? Because it doesn’t do any good to store and manage you’re your files if you can’t locate them. Findability is easy when there are only a few hundred files and a few people storing them. That would be the first hour of the first day when an ECM system is first set up. Findability gets hard really fast after that.

Instead of thinking of the "ECM or Search" choice as an either-or, think about the critical benefits each provides the other. Search and ECM together are better than the the combination of each separately. To get the most out of an ECM system (even with its built-in search system) it is worth thinking about how search improves ECM and vice-versa. Ultimately, neither is effective without some good understanding of this symbiotic interplay, and each requires a commitment to improving its use, oversight and continuous improvement. When you appreciate this interplay, and the responsibilities that follow after you get an ECM system running, then you might even consider stepping up to a search system that goes beyond what you normally get for “free” with ECM. Interestingly, two of my favorite ECM systems, Documentum from EMC and Alfresco, both now use Lucene/Solr, and this open source search system doesn’t skimp on power. For more about the benefits of search to ECM, and ECM to search, go to my Guident blog post. You’ll be surprised how these two systems work together, the whole being bigger than the sum of their parts. I’d welcome your comments and thoughts there (and here too).

Thoughts on Enterprise Search Summit 2011

It has been a couple of weeks since ESS 2011 in NYC, and I've had a chance to collect my thoughts about the conference.
The conference, as usual, was a don't-miss opportunity for anyone interested in search systems, search projects, or practical ways to improve search satisfaction. There were more attendees than last year. I found a surprising dichotomy among the attendees and vendors. As to the attendees, they were either new to search or long-time search professionals. Vendors included only one big name (Google, who else?) and many smaller vendors from both the US (such as Basis Technology, H5, and Vivissimo) to vendors from Europe and Australia (Raytion –Germany—and SpringSense –Australia).
As to the themes, they were many. Some could have been from a conference 5 years ago. The enduring themes dealt with such topics as Search projects, and bringing failing search projects back on track (my own presentation) to newer themes of integrating search with social media and search on mobile devices. I was very surprised and pleased to see eDiscovery as a topic and to see at least two vendors offering eDiscovery products and services (H5 and Clearwell).
About 1/3 of the attendees at my presentation requested a copy of Guident's free "Findability Checklist," now expanded with attributed quotes anyone can use in their own search presentation. I've expanded that to include general ECM quotes too. If you want one too, send me an email.
Other observations:

• Google was somewhat cocksure about its position in the commercial search market. This is the second year in a row when I found their presentation hard to understand, hard to hear (speakers need professional presentation training), and as much marketing as new material. Nice water bottles at their booth though ;-). I still believe their search appliance inside the firewall is up against competition, from vendors small and large.
• Improving Search user satisfaction. These systems must be intuitive, and in this respect, Google sets the standard.
• No Bing. No surprise.
• Delivering search on mobile devices, although that is still a nascent theme inside the firewall.
• Personalizing search also remains a holy grail.
• Search systems still have plenty of differentiation, and there is plenty of room for vendors (such as SpringSearch) to add value to others' systems.

Best of show, IMHO, was RealStory's "Search Vendors in 30 minutes." The only disappointment (and a big one) is that they did not make their presentation available after the show.
All in all, a show worth attending.

Justifying eDiscovery Systems

As I said in my Information Insider October 2009 column, "The landmark 2006 Federal Rules of Civil Procedures Rule 26 and its updates make all electronic stored information (ESI) subject to legal discovery, and ESI continues its unbridled growth." Given the nation's increasing litigiousness, and the exploding amount of electronic information everywhere that could be subject to subject to 2006 FRCP rule 26, I am surprised how little we've heard about such litigation. Is it simply that our attention is elsewhere (whether the US Health Care debate, 2 wars, Global Warming –or is it Global Cooling?, the earthquake in Haiti…)? Or is eDiscovery yet another ticking time bomb that will burst onto the news when we least expect it? Well the vendors supplying eDiscovery solutions have plenty to say about that.

And what is special about eDiscovery? Why not just buy the very best search system available, and use it to do all the "e-lectronic" discovery that you want? After all, isn't it all about "search"? I spoke with Ursula Talley, VP Marketing of Stored IQ, to gather expert opinions on this subject. Here are excerpts from her comments about this, which I find pretty illuminating.

First, "Enterprise Search and eDiscovery Search technology do share a set of core capabilities, specifically crawling, indexing and searching data across a multitude of various applications and storage systems. Enterprise Search is designed to assist knowledge workers with information access and retrieval. The end result is that a user can find some files with information that can help that user complete a task." So what's the difference? Ursula went on to say "eDiscovery Search is designed to support a workflow that can be legally defended in court. The end result is a set of data files that is preserved (saved to a new, target location without any changes to the metadata and recording every system and location for each data file was originally located)." This kind of quarantining of content goes over and above what you can do with any enterprise search system. Moreover, she says that search performed by eDiscovery systems must also be very robust. Such eDiscovery searching can require queries with between 25 – 300 search terms. Moreover (for those of you who have ever posed a complex query on an enterprise search system, then went to have a cup of coffee while you waited for the result to return) eDiscovery search must be able to copy large volumes of content that has been found, "if necessary hundreds to thousands of gigabytes, without disrupting user productivity."

While it's at it, robust eDiscovery systems such as those from StoredIQ can provide de-duplication of email and user files (saving space and attorney time pouring over the same redundant files), while keeping a record of every location where those items originally resided – in case the judge asks. Lastly, searching just email systems can be a real pain, since they are so big and are threaded. Even the best search often is like sorting through low-grade ore, tons of it. eDiscovery systems also can extract both metadata and content from email and export this into a database format that can be queried and re-used into legal document review applications.

So how do you go about justifying the purchase of an eDiscovery system? Not by claiming you can add features to an existing or new Enterprise Search system. Instead, focus on the other features that you'll need if a lawsuit comes a calling. Unfortunately, getting your eDiscovery house in order may be like getting your electronic records management house in order – really hard to justify until after the lawsuit. Still, at least you can avoid the trap of thinking that Enterprise Search can do all you need to find and quarantine your information for a credible eDiscovery defense.