I have been a huge fan of the Grails Searchable plugin, and its one of the main selling points when I talk to people about Grails. But something has made me go sour on it recently. A great blog post from Boris Goykhman about the plugin has prompted me to write my own.
For those of you that don’t know; Searchable plugin is a plugin wrapper in the Grails Framework for the Compass Search engine that abstracts much of the domain class to search index mappings. And Compass is an extended implementation of the robust Open Source Lucene search engine. But its the “Domain Model to Search Index” abstraction that has cause for concern.
So lets start – I think Searchable is absolutely brilliant!!! It simplifies the process of adding a search engine to a Grails application. Although apart from some straight forward indexing and searching, I don’t believe its designed to handle complex search scenarios. Sure, you can spend a lot of time bastardising your application to support many of the elements (such as searchable component and searchable reference) to try and get deep nested relationships correctly mapped, but more often than not, I find myself also bastardising my domain model to handle what I want search to do!!
I will continue to use searchable for basic type search mappings, but I have recently decided that it is more efficient to separate the concerns between domain data and search indexes. Domain models are inherently a tree structure, there are relationships, abstractions and extensions. Whereas your search index is a flattened version of this. And trying to do this flattening through decorators and mapping files is not effective.
In a recent project, I took the approach to handling the domain to search indexing data structure conversion myself. And its already started to demonstrate pay offs! I have moved to the Solr Search engine, which is another emerging implementation of the Lucene Engine. I favour it these days for its added features particularly in Geo Spatial Search capabilities.
Quite simply, my application has a SolrService that takes domain objects that I pass it and convert them to a flattened Solr Object (that I also implemented), that is then passed to the Solr Search Engine through SolrJ. All this is quite simply managed through the afterInsert, afterUpdate and afterDelete triggers in the Domain objects. But it means that I don’t have to write complex mapping files to try and map my domain structures to the search index I want to create.
What are the benefits? Most of you would state that it would take more time doing this, but I found it to be the contrary. I have spent so much time (sometimes weeks), tweaking searchable configurations and mappings to see how it would affect my index. Looking at indexes with the Luke tool to try and determine what is actually going on and then trying to perform searches against that data to see if I get back what I am after.
I have found that the search indexes are much smaller and much more performant because of it. Because you have much more control over the indexes, I create my indexes so that I don’t have to do any GORM queries at all when rendering search results. Which you can also do in searchable, but its much more complex to understand the translation.
Soon I will try and post some code on how I actually did this.
I’ve started this a long time ago: https://github.com/lucastex/gsolr and looked for some people int he grails-user list to help/get involved. But things cooled after that.
Anyway, I’m using it as a nice way to declare solr servers (embedded or http) and configure its access. The plugin injects automatically spring beans for each solr server defined in the config.
Queries are not handled, but we can evolve this together if you want!
Thanks Rob,
That’s really interesting Lucas – I’ll check it out.
My only concern would be that I don’t think that indexing domain classes is the correct approach. Fundamentally, that is the issue with Searchable. So I think the approach with the plugin may be to create some kind of configuration mapping document that describes what an index should look like, and THEN how it should be populated with domain object data fields. I am not sure if this is possible, but we should investigate!
This is the reverse of all the approaches I have seen so far, but I believe it would make the indexes much more useable, faster and smaller.
When you sai “indexing a domain class it not the correct approach”, this is exactly right! (and sometimes even difficult to convince people of that.
The best way is to define a un-normalized (is this the correct word in en?) model, containing info of all data you’ll search, more like a regular document-database approach, as mongo and others.
In the e-commerce I’ve worked we did this, and in the index, we had data from 5 or 6 “different domains”, but all data was “linked” in the main concept of our app, the product being sold.
Thanks Rob,
That sounds like a sensible approach. Like I said, the approach seems to always have been to skew domain models to indexes, but they are different concepts IMO. We have been doing some work towards doing what I am saying at Posse. I’ll check out the work you have done, and maybe I can contribute some of the stuff we have ben doing!!
This is a good explanation of the limitations of Searchable. I would also point out that the mirroring causes several problems with complex models. On top of that, it only works with Hibernate, so you can forget about it with non-SQL data stores.
That said, it is a great way to get simple full text search into an application and shouldn’t be dismissed solely on the grounds of the difficulties associated with moderately complex domain models and awkward indexing requirements. I would say that as soon as you find your indexing requirements diverging from the domain model, it’s best to manually manage the indexing as you describe. But that’s no reason not to start with Searchable.
An alternative to Solr is Elastic Search, which has been developed by the guy that wrote Compass. It’s schemaless and operates as a separate search service with a JSON API. A bit like MongoDB in that sense.
BTW I hope your work goes into a Solr plugin!
Hi Rob, how much luck are you having rehydrating GORM objects as search results, using your method?
Thanks for your comments Peter. As I mention in my post I certainly think that Searchable is brilliant and should be your first attempt to implement search if your requirements are simple. Its architecture for enabling in an app is pure art – just simple and understandable, and that is one of its most attractive points.
I do like Elastic Search as well and did a spike with it. In fact, Elastic Search is easy enough to use that sometimes the plugin can be somewhat limiting as it takes the same path as Searchable does (mapping domain objects to search indexes).
In the end, Solr was my choice as it includes some additional functionality that I need or plan to use, such as geoSearch capabilities and faceting.
I have looked at the great work Mike Breevoort has done on the Solr grails plugin, and he has made some great inroads with the scripts, but again, it takes the approach of mapping domain models to search indexes, which is what I want to move away from.
My thinking at the moment (although not quite the implementation), would be to create SearchIndex classes much like we create Domain classes in grails, and map fields to Domain Objects and their properties. Then let the plugin handle the magic of unravelling the mapping the other way around. But I need to think this through a little more before painting myself into a corner
As you have probably seen in previous comments with Lucas Teixeira, he has been working on a plugin that seems to be taking a similar approach I am thinking, so maybe that is where the work can go to unify a plugin effort.
As much as possible, I am currently trying to capture as much of the data I require in the search index and avoid having to go back to the domain objects. Although, thre are some scenarios where this is not possible.
But in general, I am not finding much friction using this approach, in those instances you simply do a GORM lookup as you would with any other object. As long as ou index your PrimaryKeys with the indexed document, you should be fine.
Hi Rob. I’ve only used solr as a separate webapp servicing search calls for another Spring/Hibernate webapp, so I can’t judge the efficiency and simplicity of the existing Grails Solr plugin compared to Searchable plugin. So I’m looking forward to your code examples!
Totallt agree with you… A lot of people using Grails plugins think that they are like black boxes that provide unlimited functionality on top of an already existing API, and this is not true for many of them; otherwise, they would end up being just another layer of complex API on top of another complex API. And this applies perfectly to the Searchable plugin, IMHO.
Thank you for putting to words for what i’ve been thinking for a long time. I’m using searchable, but I’ve made so many tweaks, that I’m no longer benefiting from it as a plug-in. It would be nice to share experiences as we move to solr/elastic search.