Experience In Using IndexTank & Its various Facets

IndexTank got acquired by LinkedIn just a while back and I received a newsletter saying that they may discontinue services in 6 months. My tweet about this got a few responses from Pat Allen (@pat – the creator of Flying Sphinx) and others, which ended up with me promising a brief comparison between Thinking Sphinx (TS), WebSolr (Solr) and IndexTank and why I love IndexTank!. There are plenty of documents and posts on how to setup, configure and use TS and Solr. This post does give information about using IndexTank and the various facets available with it that I find awesome and/or non-existent in TS or Solr.

My experiments with Full-text searching

When I first set out needing full text searching, I used Solr. It was pretty good though re-indexing took ages and to ensure consistency, I had to re-index every day via cron. Then I found Thinking Sphinx – and loved it because it managed delta indexes! Wow – no more daily re-index cron jobs. Even the re-indexing was way quicker.

TS was easy to configure and it generated a production.sphinx.yml, which I could tweak (at my own risk) to get it to index what I want – a bad practice but works well if you know what you are doing.

The big issue with both Solr and TS was that it required tight integration with models and my database. For example – in TS, if a relationship was changed, I had to ensure to trigger the parent / child delta index in order to ensure it gets indexed too.  Both TS and Solr add methods to ActiveRecord, which I find a little annoying.

These nuances gets my code too dependent on TS or Solr and switching from them to something else becomes a big pain!

IndexTank makes an entrance.

IndexTank is a NoSQL hosted service for full text search. What I liked best about IndexTank is that I had my application integrated in about 15 minutes – no frills, 3rd party storage. Wow!

The first thought is ‘performance hit’ … when I checked from Rails console — result was totally acceptable! Of course performance is relative but when you consider that the server load is reduced and a third party is doing a full text index on my data — the cost is covered.

Accessibility is awesome. I can create and update documents on IndexTank at will! That is I dont need to index / re-index or add delta’s to my database, because IndexTank is not dependent on my database or even my schema!

I can now index what I want and how I want. Here is a small example:

# config/initializers/indextank.rb
CLIENT = IndexTank::Client.new('http://:67xN9mHBV7BV8w@iej.api.indextank.com')
INDEX = CLIENT.indexes('idx')

I like the fact that there are no rake tasks for configuration, starting and stopping. Since its a 3rd party service and not a separate process on my server, my server resources are not utilized. The icing on the cake is that we can manage multiple indexes on IndexTank and hence segregate the entire data, not just subset it!

Using it is pretty simple:

# app/models/user.rb
INDEX.document("User:id:#{self.id}").add(:text => "#{self.name} #{self.address}")
# basically what ever I want

It’s interesting to note that self.name could be a attribute accessor but self.address could be a method which returns a GeoCoder formatted address – the point being that I can send processed output to IndexTank! Now, when I want search results, I simply do this

results = INDEX.search("something")
=> {"matches"=>2, "query"=>"something",
"facets"=>{}, "search_time"=>"0.009",
"results"=>[{"docid"=>"User:id:33", "query_relevance_score"=>-2217269.0},
{"docid"=>"User:id:38", "query_relevance_score"=>-2739353.0}]}

So, I now get the number of matches, the objects which matched sorted according to their relevance! ‘Relevance’ – what is that you ask? Its an amazing feature of IndexTank called scoring functions. The default scoring function is by time of creation but we can easily register new scoring functions so that we get the objects in the order we want. Wowie!

Note that ‘text’ is the default keyword for IndexTank for storing data. But there is a LOT more to this. I can not only provide my own keys instead of text, I can also use some facets like ‘categories’ to limit my search. For example, I can modify my code earlier for insertion like this:

INDEX.document("User:id:#{self.id}").add(:text => "#{self.name} #{self.address}",
:categories => {:type => 'admin'})

I added the ‘admin’ category for users. And when I search using the same query as above, I get this result

results = INDEX.search("something")
=> {"matches"=>2, "query"=>"something",
"facets"=>{'type' => {'admin' => 1, 'user' => 1}}, "search_time"=>"0.009",
"results"=>[{"docid"=>"User:id:33", "query_relevance_score"=>-2217269.0},
{"docid"=>"User:id:38", "query_relevance_score"=>-2739353.0}]}

Notice this line in particular in the results, that filters my search result.

"facets"=>{'type' => {'admin' => 1, 'user' => 1}}

A slight modifications in my search call renders sub-set results:

results = INDEX.search("something", :category_filters => { :type => 'admin'}))
=> {"matches"=>1, "query"=>"something",
"facets"=>{'type' => {'admin' => 1}, "search_time"=>"0.007",
"results"=>[{"docid"=>"User:id:33", "query_relevance_score"=>-2217269.0}
]}

Not that this is not doable in TS or Solr – but its not as usable as IndexTank. Please educate me otherwise. This post is getting long already, so I plan to detail out IndexTank faceting and scoring functions in my next post and keep this one as a comparison between IndexTank and TS or Solr.

Heroku Setup

First and foremost IndexTank has the basic version free! At the time of writing this post, the lowest version of FlyingSphinx is ‘wooden’ version (12$ per month) and WebSolr has the ‘Silver’ version. (20$ per month). This is pretty steep when it comes to prototyping, basic building of small apps. IndexTank free version has upto 100,000 documents storage which is free – which is more than enough to support a small application.

Further-more, for both TS and WebSolr, we need workers which are run by in the daily cron. This does increase the cost further.

The FlyingSphinx and Websolr configuration on Heroku is no simple task either. There are quite a few caveats to cater too and it can still cause some hiccups. Its indeed worth mentioning the the support you get from FlyingSphinx and WebSolr is indeed awesome. But I would expect it if I were paying a fee for the setup 😉

Hope this stirs up a healthy debate – (and secretly hoping that IndexTank services continue even after 6 months).