Christopher Even on SharePoint Search

True faceted searching and SharePoint. Is it possible?

Everything is possible. It is just a matter of cost and time. I have been having an ongoing discussion with Lar's of Mondosoft about faceted searching and here is summary of what we came up with:

First off you could just license an external search engine that already has faceted searching and plug it into SharePoint replacing the standard SharePoint search. But what we are discussing is the possibility of doing true faceted searching with the SharePoint Search Engine.

See http://sharepointsearch.com/pages/bigresourcelist.aspx or ask.com for for an example of what I mean.

Here are the requirements:

• a facet can be both static ( metadata property) or dynamic (conceptually determined) 
• all facets are displayed and searchable (meaning clickable to further narrow the search).
• counts are displayed for each facet entry which represents the total search result entries that contain the entry.
• The facet entries and their counts are accurate which means they honor the current search, current security of that search and are not just a subset of the returned results. If a search returns a million results, all results are used to determine the facet entries not just the first 20k.
• The interface for facet display must be clean and configurable by users so as to allow them to hide unnecessary facets.

Here are the barriers:

• Currently only possible to do static facets as SharePoint doesn't have the ability to do conceptual searching ( see Engenium for possible future availability) 
• Impossible to reliably get large( 100k+)  result sets back from SharePoint as the new security trimmer model will block that. Although you could target specific content sources that don't have a real time security trimmer and just do faceted searching within that set.
• Since behind the scenes SP Search is combining a SQL search and a file based full text search there is alot of work going on and it is optimized to return the first few thousand results back fast. Beyond that it will start to slow down and getting all million results out will certainly exceed user time requirements.
• Current SP Web parts and search UI is locked down hard and not extendable enough to work with for this purpose, so you have to write your own search interface from scratch or work with another vendor who has one like Mondosoft or BA-Insight.

Ideas for approach ( they can be used in any combination):

• read ALL the results back from a Search without limits and read the whole dataset into memory, then programatically ( or through xslt) generate accurate facets. This is possible if you own the search interface like Ontolica, but will still face issues with performance and be at the mercy of the security trimmers.
• Create a duplicate SSP that crawls the same data but has Security Trimming turned off ( can be done through object model easily enough)  and use that to generate facets. Numbers may be off but think it is acceptable.
• Use your result set to return only the workIDs (document ids) back and use them to run a query directly against the SP Search SQL database to get the metadata property counts.
• Create an OLAP cube from the SP Search SQL database with all the facets predefined and generated. Include the workid as primary key and run OLAP queries with the working universe from the SP Search to get totals. - This sounds like a fun approach but not sure of its merit without some testing.

Will continue to think about this.