Solr provides ability to customize handling of Solr requests by extending built in or by writing completely new Request handlers and Search Components. Search components define the logic for running Solr queries and make the results available in the response which will be returned to users. There are several default search components that work with all SearchHandlers without any additional configuration and these are executed by default, in the following order
If you register a new search component with one of these default names, the newly defined component will be used instead of the default. The search component is defined in solrconfig.xml separate from the request handlers, and then registered with a request handler as needed. By default Solr uses a search component named “query” for performing search operations. Below sample configuration is for overwriting default search component with a new search component ‘MyQueryComponent’.
/products/solrconfig.xml
|
1 2 3 4 5 6 7 8 9 10 11 |
<searchComponent name="query" class="solr.MyQueryComponent"> <lst name="invariants"> <str name="rows">10</str> <str name="fl">*,score</str> </lst> <lst name="defaults"> <str name="q">*:*</str> <str name="indent">true</str> <str name="echoParams">explicit</str> </lst> </searchComponent> |
Executing multiple search queries
By default QueryComponent can execute one Solr search per request. For supporting multiple Solr queries we need to write a search component either by extending QueryComponent or SearchComponent.
Write a class by extending QueryComponent
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
public class MultipleQueryComponent extends QueryComponent { protected static Logger log = LoggerFactory.getLogger(MultipleQueryComponent.class); public static final String NUMBER_OF_QUERIES_PARAM = "qn"; public static final String MULTI_QUERY_PREFIX = "mQry"; public static final String COMPONENT_NAME = "multiQuery"; @Override public void prepare(ResponseBuilder rb) throws IOException { ….. } @Override public void process(ResponseBuilder rb) throws IOException { log.debug("Process started: "); ….. } } |
Overwrite prepare() and process() methods to customize our needs.
prepare() method:
We can write logic for validating request and prepare component for processing queries. The custom query component we are writing do not support grouping and validating mandatory query parameters.
process () method:
Process method first extracts raw solr queries (buildRawQueries()) and build SolrParams (parseRawQueries) out of raw Solr queries. Each SolrParams instance resembles one query. Execute one query at a time and the query result is added to ResponseBuilder. The logic for query execution is delegated to QueryComponent as there is nothing much we need to override for our requirement (processQuery). Queries are executed in order specified and until we retrieve max rows required. If rows parameter is passed -1 all queries are executed.
Prepare the final result by merging all the results of all queries executed and return to user.
Sample Solr query:
http://localhost:8983/solr/products/select?qn=3&mQry1=…&mQry2=….&mQry3=…&rows=25&bebug=true&sort=…&qt=multiQuery
Parameters:
qt : Name of query handler that process multi query Solr request .
qn : Number of Solr queries in the request
mQryX: Solr query which can be executed independently. Where X in parameter name should be an integer which denotes the order of execution of query. If ‘qn’ parameter is 4, we have to pass mQry1, mQry2, mQry3 and mQry4 parameters.
debug: Defaults to false. If passed true Solr query debug information is returned in response.
sort: Default Sort parameter if sort parameter is not available in individual queries (mQryX).
Register custom search component in solr by adding the following in solrconfig,xml
|
1 2 3 4 5 6 7 8 9 10 11 |
<searchComponent name="multiQuery" class="com.nbos.solr.handler.component.MultipleQueryComponent"> <lst name="invariants"> <str name="rows">10</str> <str name="fl">*,score</str> </lst> <lst name="defaults"> <str name="q">*:*</str> <str name="indent">true</str> <str name="echoParams">explicit</str> </lst> </searchComponent> |
MultipleQueryComponent has to be register with RequestHandler to be used. We can configure a new request handler using existing StandardRequestHandler or write a custom request handler.
Use built-in request handler and configure MultipleQueryComponent:
|
1 2 3 4 5 6 7 |
<requestHandler name="multiQuery" class="org.apache.solr.handler.StandardRequestHandler"> <arr name="components"> <str>multiQuery</str> <str>stats</str> <str>debug</str> </arr> </requestHandler> |
As a custom request handler:
|
1 2 3 4 5 6 7 |
<requestHandler name="multiQuery" class="org.nbos.solr.handler.component.MultipleQuerySearchHandler"> <arr name="components"> <str>multiQuery</str> <str>stats</str> <str>debug</str> </arr> </requestHandler> |
Code of MultipleQuerySearchHandler:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
package com.nbos.solr.handler; import java.net.MalformedURLException; import java.net.URL; import java.util.ArrayList; import java.util.List; import org.apache.solr.handler.StandardRequestHandler; import org.apache.solr.handler.component.DebugComponent; import org.apache.solr.handler.component.StatsComponent; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class MultipleQuerySearchHandler extends StandardRequestHandler { protected static Logger log = LoggerFactory.getLogger(MultiQuerySearchHandler.class); @Override protected List<String> getDefaultComponents() { ArrayList<String> names = new ArrayList<String>(6); names.add("multiQuery"); names.add(StatsComponent.COMPONENT_NAME); names.add(DebugComponent.COMPONENT_NAME); return names; } @Override public String getDescription() { return "Multiple query request handler"; } @Override public String getSource() { return "$URL: https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_1/solr/core/src/java/org/apache/solr/handler/StandardRequestHandler.java $"; } @Override public URL[] getDocs() { try { return new URL[] { new URL("http://blogs.nbostech.com/2015/09/<span id="editable-post-name" title="Temporary permalink. Click to edit this part.">multi-search-s…uery-component</span>/") }; } catch (MalformedURLException ex) { return null; } } } |
Complete code of MultipleQueryComponent:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 |
package com.nbos.solr.handler.component; import java.io.IOException; import java.util.ArrayList; import java.util.HashMap; import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.Set; import org.apache.commons.lang.ArrayUtils; import org.apache.commons.lang.StringUtils; import org.apache.solr.common.SolrException; import org.apache.solr.common.params.CommonParams; import org.apache.solr.common.params.GroupParams; import org.apache.solr.common.params.MultiMapSolrParams; import org.apache.solr.common.params.SolrParams; import org.apache.solr.common.util.NamedList; import org.apache.solr.handler.component.QueryComponent; import org.apache.solr.handler.component.ResponseBuilder; import org.apache.solr.request.SolrQueryRequest; import org.apache.solr.response.ResultContext; import org.apache.solr.search.DocIterator; import org.apache.solr.search.DocList; import org.apache.solr.search.DocSlice; import org.apache.solr.servlet.SolrRequestParsers; import org.slf4j.Logger; import org.slf4j.LoggerFactory; public class MultipleQueryComponent extends QueryComponent { protected static Logger log = LoggerFactory.getLogger(MultipleQueryComponent.class); public static final String NUMBER_OF_QUERIES_PARAM = "qn"; public static final String MULTI_QUERY_PREFIX = "mQry"; public static final String COMPONENT_NAME = "multiQuery"; @Override public void prepare(ResponseBuilder rb) throws IOException { SolrQueryRequest req = rb.req; SolrParams params = req.getParams(); // does not support grouping boolean grouping = params.getBool(GroupParams.GROUP, false); if (grouping) { throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Grouping is not supported by MultipleQueryComponent "); } Integer numberOfQueries = params.getInt(NUMBER_OF_QUERIES_PARAM); if (numberOfQueries == null || numberOfQueries <= 0) { throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Number of queries param is not set or invalid. Params value: " + numberOfQueries); } Integer rows = params.getInt(CommonParams.ROWS); if (rows == null || rows <= 0) { throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Number of rows param passed is invalid. rows value: " + rows); } List<String> queries = buildRawQueries(params); if (queries == null || queries.size() == 0) { throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Require at least one query." + rows); } log.debug("Number of Queries: " + queries.size()); } @Override public void process(ResponseBuilder rb) throws IOException { log.debug("Process started: "); SolrQueryRequest req = rb.req; // original params SolrParams originalParams = req.getParams(); // find queries List<String> rawQueries = buildRawQueries(originalParams); // create SolrParams for solr queries List<SolrParams> allSolrParams = parseRawQueries(rawQueries); // find other query params for processing request Map<String, String[]> otherQueryParams = new HashMap<String, String[]>(); buildOtherParams(originalParams, otherQueryParams); // max number of rows final Integer maxRows = originalParams.getInt(CommonParams.ROWS); Set<Integer> docIds = new HashSet<Integer>(); // process all queries for (int queryCount = 0; queryCount < allSolrParams.size() && docIds.size() < maxRows; queryCount++) { SolrParams solrParams = allSolrParams.get(queryCount); // convert SolrParams to Map of params for modifying or adding new values Map<String, String[]> rawParams = SolrParams.toMultiMap(solrParams.toNamedList()); // add other params rawParams.putAll(otherQueryParams); // reset rows parameter rawParams.put(CommonParams.ROWS, new String[] { maxRows - docIds.size() + "" }); // add id filter to exclude already fetched docIds if (docIds.size() > 0) { rawParams.put(CommonParams.FQ, addIdFilter(docIds, rawParams.get(CommonParams.FQ))); } // rawParams.put(CommonParams.FL, new String[]{"*,score"}); solrParams = new MultiMapSolrParams(rawParams); req.setParams(solrParams); log.debug("Processing query index: q" + queryCount + " Query:" + solrParams.toString()); processQuery(rb); if (rb.getResults() != null) { // if last query returned any results docIds.addAll(getDocIdsSet(rb.getResults().docList)); } } buildResponse(rb, maxRows); } private void buildOtherParams(SolrParams originalParams, Map<String, String[]> otherParams) { Iterator<String> itr = originalParams.getParameterNamesIterator(); while (itr.hasNext()) { String paramName = itr.next(); if (!paramName.startsWith(MULTI_QUERY_PREFIX)) { otherParams.put(paramName, originalParams.getParams(paramName)); } } } public void buildResponse(ResponseBuilder rb, Integer maxRows) { log.debug("Processing results of all queries to generate final response"); SolrParams params = rb.req.getParams(); NamedList response = rb.rsp.getValues(); Iterator itr = response.iterator(); ResultContext aggregateResults = new ResultContext(); List<Integer> allDocIds = new ArrayList<Integer>(); List<Float> allScores = new ArrayList<Float>(); // we will have multiple responses if we execute multiple QueryComponents // aggregate all results from all responses and prepare one final result to // return to client for (int i = 0; i < response.size(); i++) { if (response.getName(i) == "response") { extractDocDetails(allDocIds, allScores, ((ResultContext) response.getVal(i))); } } // remove all old responses while (response.get("response") != null) { response.remove("response"); } int retainDocs = allDocIds.size() >= maxRows ? maxRows : allDocIds.size(); allDocIds = allDocIds.subList(0, retainDocs); allScores = allScores.subList(0, retainDocs); int[] docIds = ArrayUtils.toPrimitive(allDocIds.toArray(new Integer[0])); float[] scores = ArrayUtils.toPrimitive(allScores.toArray(new Float[0])); // all the final result to response which will be sent to client aggregateResults.docs = new DocSlice(0, allDocIds.size(), docIds, scores, 0, 0.0f); response.add("response", aggregateResults); } private void extractDocDetails(List<Integer> allDocIds, List<Float> allScores, ResultContext resultContext) { DocIterator itr = resultContext.docs.iterator(); while (itr.hasNext()) { int nextDocId = itr.nextDoc(); if (!allDocIds.contains(nextDocId)) { // we want to retain unique results allDocIds.add(nextDocId); if (resultContext.docs.hasScores()) { allScores.add(itr.score()); } else { allScores.add(0.0f); } } } } private String[] addIdFilter(Set<Integer> docIds, String[] existingFilterQueries) { String idFilter = "-id:(" + StringUtils.join(docIds, " ") + " ) "; if (existingFilterQueries == null) { existingFilterQueries = new String[0]; } String[] newFilterQueries = new String[existingFilterQueries.length + 1]; newFilterQueries[newFilterQueries.length - 1] = idFilter; System.arraycopy(existingFilterQueries, 0, newFilterQueries, 0, existingFilterQueries.length); return newFilterQueries; } private Set<Integer> getDocIdsSet(DocList docList) { Set<Integer> docIds = new HashSet<Integer>(); DocIterator itr = docList.iterator(); while (itr.hasNext()) { docIds.add(itr.next()); } return docIds; } private List<SolrParams> parseRawQueries(List<String> rawQueries) { List<SolrParams> allSolrParams = new ArrayList<SolrParams>(); for (String query : rawQueries) { allSolrParams.add(SolrRequestParsers.parseQueryString(query)); } return allSolrParams; } protected void processQuery(ResponseBuilder rb) throws IOException { rb.setQuery(null); rb.setQueryString(null); super.prepare(rb); // prepare query parsers super.process(rb); // delegate query processing to QueryComponent } protected List<String> buildRawQueries(SolrParams params) throws SolrException { // find queries List<String> queries = new ArrayList<String>(); int numberOfQueries = params.getInt(NUMBER_OF_QUERIES_PARAM); for (int n = 1; n <= numberOfQueries; n++) { String nthQuery = params.get(MULTI_QUERY_PREFIX + n); if (nthQuery == null || nthQuery.trim().length() == 0) { throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Query param: " + MULTI_QUERY_PREFIX + n + " is missing in Multi-Search Query"); } nthQuery = java.net.URLDecoder.decode(nthQuery); queries.add(nthQuery); } return queries; } } |
Handling HTTP 413 error status from Solr while executing multiple Solr queries.
When sending multiple queries in one request, Solr may return HTTP 413 status code.
Solr returns HTTP 413 status code when the client sends Solr queries in GET request and total number of characters are more than allowed. The maximum number of characters allowed depends on server configuration Solr is running. Mostly all servers allow 1024 characters in GET request. To avoid HTTP 413 error, clients should send Solr queries using POST method when request payload contains more than 1024 characters.