Field Collapsing in SOLR

Field Collapsing
Field collapsing allows something akin to a “group by” in SOLR, so that the number of results returned reflect a logical grouping rather than another total.??
Faceting can be used in conjunction. Facet counts reflect subsets within results, where-as collapse counts are group by counts.
This means that Field Collapsing could be used for certain analytics, as well as the common use-case of nesting and grouping results. To use effectively, I found it helpful to “pre-collapse” certain fields, so that a new, unique string was created that could be used to easily group, since I believe you can only field collapse on a single field. (If I’m wrong, please let me know!)
Special Setup
This assumes you are or will be running a development version of SOLR (trunk via SVN).
Field collapsing is not yet available in SOLR trunk and you must apply a patch file to SOLR and build again.
If pulling in from trunk, download the patch found at https://issues.apache.org/jira/browse/SOLR-236 in to your solr source code directory.

wget https://issues.apache.org/jira/secure/attachment/12440108/SOLR-236-trunk.patch
patch -p 1 -i SOLR-236-trunk.patch

And rebuild using supplied Apache Ant scripts.
Read more on these sites or in the comprehensive “Solr 1.4, Enterprise Search Server” page 191.
Why SOLR
SOLR has a been a great tool for BetterLesson.org. Because our primary database is MySQL, we looked at around 8 full-text indexers – but the two finalists were Sphinx [1] and SOLR. Sphinx had very tight integration with MySQL, so the learning curve seemed less. ??SOLR required a JVM, an app server, and quite a lot of configuration.
When we were deciding, an excellent SOLR book came out just when we were choosing. Further the SOLR IRC channel and mailing list for SOLR are friendly and quite active. We even had the option for commercial support through Massachusetts’ own Lucid Imagination. So I dove in.??While the configuration is non-trivial, but the configuration parameters have proven very powerful.??
More background:
I had written a half-dozen or so custom faceted search interfaces – almost entirely using MySQL, and even one used Sesame (an RDF store – and it eventually worked pretty well). Skipping the stories of pain, confusion and suffering on the road to enlightenment – SOLR has been great.??Used extensively at Netflix.com, Zappos.com, CitySearch.com, Reddit.com, Wego.com, Whitehouse.gov, Drupalgardens.com and others [2], ??supported by Apache, based on Lucene, SOLR provides a scalable, distributed search and has good data import from MySQL, including delta queries.
[1] – Sphinx is used by Craigslist and http://www.sphinxsearch.com/powered.html

6 thoughts on “Field Collapsing in SOLR

  1. If you’ve downloaded the patch into the solr directory, I believe the patch command that people want is: "patch -p 1 -i SOLR-236-trunk.patch"

  2. Hi, I followed the instructions posted here and when i apply the patch, i get the following errorthe source code i applied it on is the latest source code from the trunk. # patch -p 1 -i SOLR-236-trunk.patchpatching file src/java/org/apache/solr/handler/component/CollapseComponent.javapatching file src/test/test-files/solr/conf/solrconfig-fieldcollapse.xmlpatching file src/java/org/apache/solr/search/fieldcollapse/collector/DocumentGroupCountCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/FieldValueCountCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/DocSetHitCollector.javaHunk #1 FAILED at 17.Hunk #2 succeeded at 29 (offset 1 line).1 out of 2 hunks FAILED — saving rejects to file src/java/org/apache/solr/search/DocSetHitCollector.java.rejpatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/AggregateFunction.javapatching file src/test/test-files/solr/conf/solrconfig.xmlHunk #1 FAILED at 406.1 out of 1 hunk FAILED — saving rejects to file src/test/test-files/solr/conf/solrconfig.xml.rejpatching file src/java/org/apache/solr/handler/component/QueryComponent.javaHunk #1 FAILED at 523.1 out of 1 hunk FAILED — saving rejects to file src/java/org/apache/solr/handler/component/QueryComponent.java.rejpatching file src/java/org/apache/solr/search/fieldcollapse/collector/CollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/AverageFunction.javapatching file src/test/org/apache/solr/search/fieldcollapse/FieldCollapsingIntegrationTest.javapatching file src/test/org/apache/solr/search/fieldcollapse/AdjacentCollapserTest.javapatching file src/solrj/org/apache/solr/client/solrj/response/FieldCollapseResponse.javapatching file src/test/test-files/fieldcollapse/testResponse.xmlpatching file src/test/org/apache/solr/search/fieldcollapse/DistributedFieldCollapsingIntegrationTest.javapatching file src/java/org/apache/solr/search/DocSetAwareCollector.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/CollapseCollector.javapatching file src/test/org/apache/solr/client/solrj/response/FieldCollapseResponseTest.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/AggregateCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/SumFunction.javapatching file src/java/org/apache/solr/search/fieldcollapse/CollapseGroup.javapatching file src/test/org/apache/solr/handler/component/CollapseComponentTest.javapatching file src/java/org/apache/solr/search/fieldcollapse/AbstractDocumentCollapser.javapatching file src/test/org/apache/solr/search/fieldcollapse/NonAdjacentDocumentCollapserTest.javapatching file src/common/org/apache/solr/common/params/CollapseParams.javapatching file src/java/org/apache/solr/search/fieldcollapse/AdjacentDocumentCollapser.javapatching file src/java/org/apache/solr/search/fieldcollapse/util/Counter.javapatching file src/solrj/org/apache/solr/client/solrj/response/QueryResponse.javaHunk #1 FAILED at 47.Hunk #2 FAILED at 63.Hunk #3 succeeded at 122 with fuzz 2 (offset -8 lines).Hunk #4 succeeded at 328 with fuzz 2 (offset 25 lines).2 out of 4 hunks FAILED — saving rejects to file src/solrj/org/apache/solr/client/solrj/response/QueryResponse.java.rejpatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/MinFunction.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/DocumentFieldsCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/AbstractCollapseCollector.javapatching file src/java/org/apache/solr/util/DocSetScoreCollector.javapatching file src/java/org/apache/solr/search/SolrIndexSearcher.javaHunk #8 succeeded at 873 (offset -2 lines).patching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/MaxFunction.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/CollapseContext.javapatching file src/java/org/apache/solr/search/fieldcollapse/DocumentCollapser.javapatching file src/solrj/org/apache/solr/client/solrj/SolrQuery.javaHunk #1 FAILED at 17.Hunk #2 FAILED at 50.Hunk #3 FAILED at 76.Hunk #4 FAILED at 148.Hunk #5 FAILED at 197.Hunk #6 succeeded at 510 (offset -155 lines).5 out of 7 hunks FAILED — saving rejects to file src/solrj/org/apache/solr/client/solrj/SolrQuery.java.rejpatching file src/test/test-files/solr/conf/schema-fieldcollapse.xmlpatching file src/java/org/apache/solr/search/fieldcollapse/DocumentCollapseResult.javapatching file src/java/org/apache/solr/search/fieldcollapse/NonAdjacentDocumentCollapser.java[root@germinait06 Solr1.4]# patch -p 1 -i SOLR-236-trunk.patchpatching file src/java/org/apache/solr/handler/component/CollapseComponent.javapatching file src/test/test-files/solr/conf/solrconfig-fieldcollapse.xmlpatching file src/java/org/apache/solr/search/fieldcollapse/collector/DocumentGroupCountCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/FieldValueCountCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/DocSetHitCollector.javaHunk #1 FAILED at 17.Hunk #2 FAILED at 28.2 out of 2 hunks FAILED — saving rejects to file src/java/org/apache/solr/search/DocSetHitCollector.java.rejpatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/AggregateFunction.javapatching file src/test/test-files/solr/conf/solrconfig.xmlHunk #1 FAILED at 406.1 out of 1 hunk FAILED — saving rejects to file src/test/test-files/solr/conf/solrconfig.xml.rejpatching file src/java/org/apache/solr/handler/component/QueryComponent.javaHunk #1 FAILED at 523.1 out of 1 hunk FAILED — saving rejects to file src/java/org/apache/solr/handler/component/QueryComponent.java.rejpatching file src/java/org/apache/solr/search/fieldcollapse/collector/CollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/AverageFunction.javapatching file src/test/org/apache/solr/search/fieldcollapse/FieldCollapsingIntegrationTest.javapatching file src/test/org/apache/solr/search/fieldcollapse/AdjacentCollapserTest.javapatching file src/solrj/org/apache/solr/client/solrj/response/FieldCollapseResponse.javapatching file src/test/test-files/fieldcollapse/testResponse.xmlpatching file src/test/org/apache/solr/search/fieldcollapse/DistributedFieldCollapsingIntegrationTest.javapatching file src/java/org/apache/solr/search/DocSetAwareCollector.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/CollapseCollector.javapatching file src/test/org/apache/solr/client/solrj/response/FieldCollapseResponseTest.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/AggregateCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/SumFunction.javapatching file src/java/org/apache/solr/search/fieldcollapse/CollapseGroup.javapatching file src/test/org/apache/solr/handler/component/CollapseComponentTest.javapatching file src/java/org/apache/solr/search/fieldcollapse/AbstractDocumentCollapser.javapatching file src/test/org/apache/solr/search/fieldcollapse/NonAdjacentDocumentCollapserTest.javapatching file src/common/org/apache/solr/common/params/CollapseParams.javapatching file src/java/org/apache/solr/search/fieldcollapse/AdjacentDocumentCollapser.javapatching file src/java/org/apache/solr/search/fieldcollapse/util/Counter.javapatching file src/solrj/org/apache/solr/client/solrj/response/QueryResponse.javaHunk #1 FAILED at 47.Hunk #2 FAILED at 63.Hunk #3 FAILED at 130.Hunk #4 succeeded at 84 with fuzz 2 (offset -219 lines).3 out of 4 hunks FAILED — saving rejects to file src/solrj/org/apache/solr/client/solrj/response/QueryResponse.java.rejpatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/MinFunction.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/DocumentFieldsCollapseCollectorFactory.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/AbstractCollapseCollector.java
    patching file src/java/org/apache/solr/util/DocSetScoreCollector.javapatching file src/java/org/apache/solr/search/SolrIndexSearcher.javaReversed (or previously applied) patch detected! Assume -R? [n] nApply anyway? [n] yHunk #1 FAILED at 17.Hunk #2 FAILED at 530.Hunk #3 FAILED at 586.Hunk #4 FAILED at 610.Hunk #5 FAILED at 663.Hunk #6 succeeded at 798 with fuzz 2 (offset 93 lines).Hunk #7 FAILED at 809.Hunk #8 FAILED at 968.Hunk #9 succeeded at 1260 (offset 2 lines).7 out of 9 hunks FAILED — saving rejects to file src/java/org/apache/solr/search/SolrIndexSearcher.java.rejpatching file src/java/org/apache/solr/search/fieldcollapse/collector/aggregate/MaxFunction.javapatching file src/java/org/apache/solr/search/fieldcollapse/collector/CollapseContext.javapatching file src/java/org/apache/solr/search/fieldcollapse/DocumentCollapser.javapatching file src/solrj/org/apache/solr/client/solrj/SolrQuery.javaHunk #1 FAILED at 17.Hunk #2 FAILED at 50.Hunk #3 FAILED at 76.Hunk #4 FAILED at 148.Hunk #5 FAILED at 197.Hunk #6 FAILED at 665.Hunk #7 succeeded at 620 with fuzz 1 (offset -101 lines).6 out of 7 hunks FAILED — saving rejects to file src/solrj/org/apache/solr/client/solrj/SolrQuery.java.rejpatching file src/test/test-files/solr/conf/schema-fieldcollapse.xmlpatching file src/java/org/apache/solr/search/fieldcollapse/DocumentCollapseResult.javapatching file src/java/org/apache/solr/search/fieldcollapse/NonAdjacentDocumentCollapser.java

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s