SOLR 4.1 Out Of Memory error After commit of a few thousand Solr Docs
we are testing solr 4.1 running inside tomcat 7 and java 7 with following options
JAVA_OPTS="-Xms256m -Xmx2048m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+ParallelRefProcEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/ubuntu/OOM_HeapDump"
our source code looks like following:
/**** START *****/
int noOfSolrDocumentsInBatch = 0;
for(int i=0 ; i<5000 ; i++) {
SolrInputDocument solrInputDocument = getNextSolrInputDocument();
server.add(solrInputDocument);
noOfSolrDocumentsInBatch += 1;
if(noOfSolrDocumentsInBatch == 10) {
server.commit();
noOfSolrDocumentsInBatch = 0;
}
}
/**** END *****/
the method "getNextSolrInputDocument()" generates a solr document with 100 fields (average). Around 50 of the fields are of "text_general" type. Some of the "test_general" fields consist of approx 1000 words rest consists of few words. Ouf of total fields there are around 35-40 multivalued fields (not of type "text_general").
We are indexing all the fields but storing only 8 fields. Out of these 8 fields two are string type, five are long and one is boolean. So our index size is only 394 MB. But the RAM occupied at time of OOM is around 2.5 GB. Why the memory is so high even though the index size is small? What is being stored in the memory? Our understanding is that after every commit documents are flushed to the disk.So nothing should remain in RAM after commit.
We are using the following settings:
server.commit() set waitForSearcher=true and waitForFlush=true
solrConfig.xml has following properties set:
directoryFactory = solr.MMapDirectoryFactory
maxWarmingSearchers = 1
text_general data type is being used as supplied in the schema.xml with the solr setup.
maxIndexingThreads = 8(default)
<autoCommit>
<maxTime>15000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
We get Java heap Out Of Memory Error after commiting around 3990 solr documents.Some of the snapshots of memory dump from profiler are uploaded at following links.
http://s9.postimage.org/w7589t9e7/memorydump1.png
http://s7.postimage.org/p3abs6nuj/memorydump2.png
can somebody please suggest what should we do to minimize/optimize the memory consumption in our case with the reasons? also suggest what should be optimal values and reason for following parameters of solrConfig.xml
- maxwarmingsearchers- number - spellcheck-on/off?
- omitNorms=true/false?
- omitTermFreqAndPositions?
- mergefactor? we are using default value 10
- java garbage collection tuning parameters ?
链接地址: http://www.djcxy.com/p/92504.html
上一篇: Solr文本字段和字符串字段