* Bug 15451 fixed: reduce Lucene API usage 92/20392/3
Clement David [Wed, 15 Jan 2020 10:57:58 +0000 (11:57 +0100)]
This commit has been checked with lucene 8.4.0 and 5.2.1.

 http://bugzilla.scilab.org/15451

Change-Id: I9f49d3a2bf9f882fe58f571d7acbcd74cc0a5773

scilab/CHANGES.md
scilab/etc/classpath.xml.vc
scilab/modules/helptools/helptools.iss
scilab/modules/xcos/src/java/org/scilab/modules/xcos/palette/PaletteIndexer.java
scilab/modules/xcos/src/java/org/scilab/modules/xcos/palette/PaletteSearchManager.java
scilab/modules/xcos/src/java/org/scilab/modules/xcos/palette/PaletteSearcher.java
scilab/scilab-lib.properties.vc

index a9fe15e..cf983c7 100644 (file)
@@ -54,6 +54,7 @@ Compilation
 Dependencies
 ------------
 
+* Lucene has been updated to 8.4.0; API compatibility is preserved down to 5.2.1.
 
 Packaging & Supported Operating Systems
 ---------------------------------------
@@ -185,14 +186,16 @@ Bug Fixes
 * [#14863](http://bugzilla.scilab.org/show_bug.cgi?id=14863): In Xcos, the default ending time was unhandily high (100000), reduced it to 30.
 * [#14982](http://bugzilla.scilab.org/show_bug.cgi?id=14982): `msprintf` segmentation fault was caught due to wrong size
 * [#14985](http://bugzilla.scilab.org/show_bug.cgi?id=14985): scilab crashed if a .bin file was not found.
-* [#15087](http://bugzilla.scilab.org/show_bug.cgi?id=15087): Deleting rows or columns from a matrix is slow (regression)
+* [#15087](http://bugzilla.scilab.org/show_bug.cgi?id=15087): Deleting rows or columns from a matrix is slow (regression
+)
+* [#15248](http://bugzilla.scilab.org/show_bug.cgi?id=15248): `lsq()`was leaking memory.
 * [#15269](http://bugzilla.scilab.org/show_bug.cgi?id=15269): `xgetech` was poor and stiff compared to any combination of `gca()` properties `.axes_bounds`, `.data_bounds`, `.log_flags`, and `.margins`. It is removed.
 * [#15271](http://bugzilla.scilab.org/show_bug.cgi?id=15271): `bitget` needed to be upgraded.
 * [#15321](http://bugzilla.scilab.org/show_bug.cgi?id=15321): `lu()` was leaking memory.
 * [#15368](http://bugzilla.scilab.org/show_bug.cgi?id=15368): `freson()` silently returned frequencies not corresponding to a maximum, or returned [] instead of some still computable maxima frequencies.
 * [#15425](http://bugzilla.scilab.org/show_bug.cgi?id=15425): The Kronecker product `a.*.b` failed when `a` or `b` or both are hypermatrices, with one or both being polynomials or rationals.
+* [#15451](http://bugzilla.scilab.org/show_bug.cgi?id=15451): The code was not adapted to use `lucene 4.10` in Debian.
 * [#15523](http://bugzilla.scilab.org/show_bug.cgi?id=15523): `%ODEOPTIONS(1)=2` didn't work with solvers 'rk' and 'rkf'
-* [#15248](http://bugzilla.scilab.org/show_bug.cgi?id=15248): `lsq()`was leaking memory.
 * [#15577](http://bugzilla.scilab.org/show_bug.cgi?id=15577): `edit` did not accept a line number as text, as with `edit linspace 21`.
 * [#15668](http://bugzilla.scilab.org/show_bug.cgi?id=15668): `save(filename)` saved all predefined Scilab constants %e %pi etc.. (regression)
 * [#15715](http://bugzilla.scilab.org/show_bug.cgi?id=15715): `%nan` indices crashed Scilab.
index 7ca970b..ec367d9 100644 (file)
@@ -33,9 +33,9 @@ The option is "disableUnderMode" and can be:
     <path value="$SCILAB/thirdparty/looks-2.1.1.jar"/>
     <path value="$SCILAB/thirdparty/commons-logging.jar"/>
     <path value="$SCILAB/thirdparty/jhall.jar"/>
-    <path value="$SCILAB/thirdparty/lucene-core-5.2.1.jar"/>
-    <path value="$SCILAB/thirdparty/lucene-analyzers-common-5.2.1.jar"/>
-    <path value="$SCILAB/thirdparty/lucene-queryparser-5.2.1.jar"/>
+    <path value="$SCILAB/thirdparty/lucene-core-8.4.0.jar"/>
+    <path value="$SCILAB/thirdparty/lucene-analyzers-common-8.4.0.jar"/>
+    <path value="$SCILAB/thirdparty/lucene-queryparser-8.4.0.jar"/>
 
     <!-- Load when needed -->
 
index 1375738..0c8a608 100644 (file)
@@ -47,9 +47,9 @@ Source: thirdparty\xml-apis.jar;DestDir: {app}\thirdparty; Components: {#COMPN_S
 Source: thirdparty\xml-apis-ext.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
 Source: thirdparty\xmlgraphics-commons.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
 Source: thirdparty\batik-all.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
-Source: thirdparty\lucene-analyzers-common-5.2.1.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
-Source: thirdparty\lucene-core-5.2.1.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
-Source: thirdparty\lucene-queryparser-5.2.1.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
+Source: thirdparty\lucene-analyzers-common-8.4.0.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
+Source: thirdparty\lucene-core-8.4.0.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
+Source: thirdparty\lucene-queryparser-8.4.0.jar;DestDir: {app}\thirdparty; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
 ;
 Source: modules\{#HELPTOOLS}\license.txt; DestDir: {app}\modules\{#HELPTOOLS}; Components: {#COMPN_SCILAB} and {#COMPN_JVM_MODULE}
 ;
index c0cc71b..c4c51dc 100644 (file)
@@ -24,6 +24,7 @@ import java.nio.file.Path;
 import java.nio.file.SimpleFileVisitor;
 import java.nio.file.attribute.BasicFileAttributes;
 import java.util.ArrayList;
+import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
@@ -70,11 +71,8 @@ public final class PaletteIndexer {
     public void createIndex(Map<String, PaletteBlock> blockNameToPalette) {
         try {
             mgr.getIndexWriter().deleteAll();
-
-            // insert all block names
-            for (String blk : blockNameToPalette.keySet()) {
-                index(blk);
-            }
+            
+            HashSet<String> blocks = new HashSet<>(blockNameToPalette.keySet());
 
             // insert all help pages
             for (File r : roots) {
@@ -98,7 +96,8 @@ public final class PaletteIndexer {
                             }
                         } else if (fname.endsWith(".html")) {
                             // this is a regular file
-                            if (blockNameToPalette.containsKey(basename)) {
+                            if (blocks.contains(basename)) {
+                                blocks.remove(basename);
                                 index(basename, file.toUri().toURL());
                             }
                         }
@@ -107,6 +106,11 @@ public final class PaletteIndexer {
                     }
                 });
             }
+            
+            // insert all missing block names
+            for (String blk : blocks) {
+                index(blk);
+            }
 
             mgr.getIndexWriter().commit();
         } catch (IOException ex) {
@@ -161,8 +165,7 @@ public final class PaletteIndexer {
             Document doc = new Document();
 
             // add the block name
-            Field refname = new TextField("refname", basename, Field.Store.YES);
-            refname.setBoost(100f);
+            Field refname = new StringField("refname", basename, Field.Store.YES);
             doc.add(refname);
 
             // add the refpurpose
@@ -171,21 +174,20 @@ public final class PaletteIndexer {
 
                 Field refpurpose;
                 if (found.isPresent()) {
-                    refpurpose = new TextField("refpurpose", found.get(), Field.Store.YES);
+                    refpurpose = new TextField("refpurpose", found.get(), Field.Store.NO);
                 } else {
-                    refpurpose = new TextField("refpurpose", "", Field.Store.YES);
+                    refpurpose = new TextField("refpurpose", "", Field.Store.NO);
                 }
 
-                refpurpose.setBoost(10f);
                 doc.add(refpurpose);
             }
 
             // add the html content
             try (BufferedReader r = new BufferedReader(new InputStreamReader(url.openStream(), "UTF-8"))) {
                 doc.add(new TextField("content", r));
+                
+                mgr.getIndexWriter().addDocument(doc);
             }
-
-            mgr.getIndexWriter().addDocument(doc);
         } catch (IOException e) {
             Logger.getLogger(PaletteIndexer.class.getName()).log(Level.SEVERE, null, e);
         }
@@ -195,8 +197,8 @@ public final class PaletteIndexer {
         try {
             Document doc = new Document();
             doc.add(new StringField("refname", block, Field.Store.YES));
-            doc.add(new StringField("refpurpose", block, Field.Store.YES));
-            doc.add(new TextField("content", block, Field.Store.YES));
+            doc.add(new TextField("refpurpose", block, Field.Store.NO));
+            doc.add(new TextField("content", block, Field.Store.NO));
 
             mgr.getIndexWriter().addDocument(doc);
         } catch (IOException e) {
index 16516d7..1325ad2 100644 (file)
@@ -15,11 +15,16 @@ package org.scilab.modules.xcos.palette;
 
 import java.io.File;
 import java.io.IOException;
+import static java.util.Comparator.reverseOrder;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import static java.util.Map.Entry.comparingByValue;
 import java.util.logging.Level;
 import java.util.logging.Logger;
+import static java.util.stream.Collectors.counting;
+import static java.util.stream.Collectors.groupingBy;
+import static java.util.stream.Collectors.toList;
 
 import javax.swing.JScrollPane;
 import javax.swing.tree.TreeModel;
@@ -94,14 +99,24 @@ public final class PaletteSearchManager {
             indexIsOutdated = false;
         }
 
-        List<Document> found = paletteSearcher.search(query);
-        for (Document doc : found) {
-            PaletteBlock block = nameToPalette.get(doc.get("refname"));
+        List<Document> found = paletteSearcher.search(query.toLowerCase());
+        // sort results per maximum frequency of the block name
+        Map<String, Long> freq = found.stream()
+                .map(doc -> doc.get("refname"))
+                .collect(groupingBy(x->x, counting()));
+        List<String> foundNames = freq.entrySet()
+                .stream()
+                .sorted(comparingByValue(reverseOrder()))
+                .map(Map.Entry::getKey)
+                .collect(toList());
+        
+        for (String b : foundNames) {
+            PaletteBlock block = nameToPalette.get(b);
             if (block != null) {
                 view.addBlock(block);
             }
         }
-        view.setText(queryLabel + found.size() + " " + XcosMessages.MATCHES);
+        view.setText(queryLabel + foundNames.size() + " " + XcosMessages.MATCHES);
         view.revalidate();
         scrollPane.revalidate();
     }
index 2649058..334c3e5 100644 (file)
@@ -51,13 +51,13 @@ public final class PaletteSearcher {
      * @return paths to the found blocks
      */
     public List<Document> search(String str) {
-        List<Document> found = new ArrayList<>();
+        
+        ArrayList<Document> found = new ArrayList<>();
         try (IndexReader reader = DirectoryReader.open(mgr.getDirectory())) {
             IndexSearcher searcher = new IndexSearcher(reader);
 
             StandardQueryParser queryParserHelper = new StandardQueryParser();
             queryParserHelper.setAllowLeadingWildcard(true);
-            queryParserHelper.setLowercaseExpandedTerms(true);
             queryParserHelper.setAnalyzer(mgr.getAnalyzer());
             queryParserHelper.setMultiFields(new String[] {"refname", "refpurpose", "content"});
 
@@ -78,6 +78,7 @@ public final class PaletteSearcher {
         } catch (IOException | QueryNodeException e) {
             Logger.getLogger(PaletteSearcher.class.getName()).log(Level.SEVERE, null, e);
         }
+        
         return found;
     }
 }
index c3390dd..708f4d2 100644 (file)
@@ -18,11 +18,11 @@ jogl2.jar=${thirdparty.dir}/jogl2.jar
 
 jhall.jar=${thirdparty.dir}/jhall.jar
 
-lucene-core.jar=${thirdparty.dir}/lucene-core-5.2.1.jar
+lucene-core.jar=${thirdparty.dir}/lucene-core-8.4.0jar
 
-lucene-analyzers-common.jar=${thirdparty.dir}/lucene-analyzers-common-5.2.1.jar
+lucene-analyzers-common.jar=${thirdparty.dir}/lucene-analyzers-common-8.4.0.jar
 
-lucene-queryparser.jar=${thirdparty.dir}/lucene-queryparser-5.2.1.jar
+lucene-queryparser.jar=${thirdparty.dir}/lucene-queryparser-8.4.0.jar
 
 gluegen2.jar=${thirdparty.dir}/gluegen2-rt.jar