Configuration

Config files

The following configuration files are processed by smart.finder:

  • WEB-INF/classes/default-application.properties (DO NOT EDIT)
    This file contains all configuration options of smart.finder with their default values.

  • WEB-INF/classes/custom-application.properties
    If you change the working directory data.directory.location, you must adjust this file. All other adjustments are made in the following file.

  • [data.directory.location]/application.properties
    Editing this file is the recommended way to make configuration changes.
    By default, ${user.home}/.smartfinder is the working directory data.directory.location of smart.finder. The application.properties is not created automatically. You can use the WEB-INF/classes/application.properties file as a template. It is recommended to leave only the settings in the file that have been changed.

The format of the configuration files must correspond to the Java Properties file format .

Sample Properties file
# The files must be UTF-8 encoded, otherwise umlauts can cause errors!
# The safest way is to encode umlauts in Unicode syntax e.g: รค = \u00E4 (vgl. http://0xcc.net/jsescape/)

# Comments are preceded by a hash

# Common syntax:
key = value

# A value can reference another key
key1 = http://${key.with.server}/test
After changing one of the configuration files a restart of the web application or alternatively the entire Tomcat server is required.

Configuration parameters

This section describes the parameters that most often need to be changed in a default installation. Other parameters are described in comments in the WEB-INF/classes/default-application.properties file.

Both the client and the server have these configuration files. The following list does not distinguish between them. Stick to the respective default-application.properties file to change the parameters for the respective web application.
data.directory.location

Working directory where map.apps data (e.g. an HSQL database) is stored locally.

In case the Tomcat server is run as a Windows service, the .smartfinder directory is located in the user directory of the Tomcat user who started the service. Path separators must be specified as either / or \\.

Default: data.directory.location=$\{user.home\}/.smartfinder

proxy.allowedServerUrls

List of URLs that the Proxy Servlet may access.

smart.finder provides an integrated Proxy Servlet. Direct access to certain resources that are only accessible under another domain is prohibited due to security restrictions within JavaScript. This parameter defines corresponding exceptions. A ; must be placed after each URL. Each line, except the last, ends with a \.

finder.service.url

Context name of the smart.finder server.

If the name was changed during the installation of the smart.finder Server, the current name must be entered here.

Default: /smartfinder-search

solr.solr.home

Directory path to solr.home.

The path can be specified absolute or relative. You can also use placeholders that are resolved by the Web Container.

Example:

solr.solr.home = ${catalina.base}/webapps/smartfinder-search/WEB-INF/solr.home
solr.default.core.name

The standard index of the server.

This index is used if no other index is specified during job creation in the job manager.

Default: core0

Security

security.mode

Protection mode of the smart.finder administrative area.

Access to the administrative areas of the smart.finder can be protected if required. To do this, the value of this parameter must be set to INTEGRATED (otherwise: NONE).
The mode ONLY_AUTHN can only be used in conjunction with security.manager Enterprise Edition.

Allowed values: INTEGRATED, NONE, ONLY_AUTHN
Default: INTEGRATED

If INTEGRATED was selected, the mode can be specified in more detail using the following parameter set:

security.user.admin.name

Username in plain text.

security.user.admin.pw

Password of the user.

This must be described according to the selected encoding.

security.user.pwenc

Encoding of the password.

This can be plain (no coding) or by means of MD5 (Message-Digest Algorithm 5) or SHA-1 (secure hash algorithm). Some web services support the generation of MD5 or SHA-1 encrypted passwords.

Allowed Values: plain, MD5, SHA-1

security.user.use_mapped_pass

Specifies whether a password encoding other than plain has been selected.

Allowed Values: true, false

security.ssl.trustAny

Specifies whether self-signed certificates are supported for HTTPS connections.

Allowed Values: true, false
Default: false

Mailing

mailing.host

Hostname on which an SMTP server is available for sending e-mails.

mailing.port

SMTP port on the SMTP host computer.

mailing.username

Username for the SMTP server.

mailing.password

Password for the SMTP server.

mailing.senderaddress

Reply address for e-mail dispatch.

mailing.locale

Language setting for sending e-mails.

Allowed Values: de, en

Logging

The following configuration parameters are available to adjust logging settings:

logging.logger.level

Level of detail of the log.

Allowed Values: TRACE, DEBUG, INFO, WARN, ERROR
Default: INFO

logging.output.location
Deprecated since 2.0.8
Use logging.file.location instead.
logging.file.location

Location where the log file is stored.

The default value corresponds to the logs directory of Tomcat. With the value ${data.directory.location}/logs the log file is created in the working directory of smart.finder.

Default: ${catalina.base}/logs

logging.file.prefix

This parameter defines the name of the log files.

To prefix log files with the URL context path of the smart.finder installation (for example smartfinder), use the value ${webcontext.name}.

Default: ct-smartfinder

Additional logging options are described in the default-application.properties file, for example to enable or disable logging into the console, into files, and GELF logging.

Changing the location of the index

In the standard delivery, the indexes of smart.finder are stored under the following path:

%TOMCAT%\webapps\smartfinder-search\WEB-INF\solr.home

The indexes of the individual cores are stored in the subdirectories located here. To change the location of the indexes you have to change the value of the variable solr.solr.home in the file application.properties.

solr.solr.home=[absolute path]

After a restart, the indexes are stored in this directory.

Sort search results

In the standard configuration, the result of a search query to the index is sorted according to the hit probability score. In contrast, if results are to be sorted alphabetically according to a field (e.g. Title), the schema configuration of this field must be unique. This means that fields used by analyzers (including all fields of type text) can only be used for sorting search results if the analyzer generates exactly one single expression. Further restrictions for sorting can be found in the Solr documentation . To be able to sort on such fields nevertheless, these fields must be copied into a new field of a different type in the schema using copyField.

The following is an example for the field name:

  1. Change the schema.xml file.

    Adapted schema.xml
    <fields>
    [...]
    <field name="name" type="text" indexed="true" stored="true" required="false" multiValued="false"/>
    <copyField source="name" dest="name_string"/>
    <field name="name_string" type="string" indexed="true" stored="true" required="false" multiValued="false"/>
    [...]
</fields>

  2. Save the file.

  3. Restart the smart.finder server or Apache Tomcat for the changes to take effect.

Configuration of synonyms of the facet values

Sometimes the terms used in the database for one thing are not consistent. You should combine synonyms according to the instructions below.

For example, the values shape and shapefile both refer to the same format. You can always display the values of the format facet as SHP in the user interface.

To configure this, 3 steps are necessary.

Step 1: Mapping values to synonyms

Enter the synonyms in the configuration file /smartfinder-search/WEB-INF/solr.home/core0/conf/lang/synonyms.txt.

For the above example, the following line would need to be added: shape, Shapefile => SHP.

The values before the arrow are also stored in the index. You can still search for shape and Shapefile, but also for SHP. In the attribute filter SHP is always displayed. A filter on the value SHP then includes all its synonyms.

Step 2: Synonym index linking

Link the synonyms to the field in the index.
To do this, open the file /smartfinder-search/WEB-INF/solr.home/core0/conf/schema.xml and find the field for whose values you define the synonyms.

In our example this is the field format:

<field name="format" type="string" indexed="true" stored="true" required="false" multiValued="true"/>

Directly below it, create a new field with type text_synonym, for example with the name format_facet and the following attributes:

<field name="format_facet" type="text_synonym" indexed="true" stored="true" required="false" multiValued="true"/>

If documents have a value for the field format when indexed, this value is also copied to the field format_facet. This field can be used as a facet and must then also be configured in the client configuration facetFields, see siehe Selection and sequence of filter attributes

Search engine optimization (SEO)

Documents that are indexed by smart.finder, can be made accessible to internet search engines like Google or Bing. In this section we explain the necessary steps to configure smart.finder for search engines.

Creating a sitemap

To enable web crawler to find the detail pages for metadata in smart.finder, the search engines must have a record of the URLs that lead to the web pages containing the information. Since these pages are generated dynamically in the smart.finder, sitemaps are used to list them.

Sitemaps are XML files, in which all the URLs to the smart.finder detail pages are listed. These files can be read by search engines. If the Sitemap function is activated, smart.finder will create these XML files automatically and save them to the server that smart.finder runs on.

An overview of the sitemap protocol used can be found here .

Configuration parameters

Sitemap generation can be configured with the following parameters:

sitemap.enabled

Enables/disables the creation of the sitemap.

Possible values: true, false
Default value: false

sitemap.executionTime

A cron string that specifies the times at which the sitemap should be recreated with the most current documents.

Default value: 0 0 6 * * ? (corresponds to every day at 6 am)

sitemap.coreName

The name of the index for which the sitemap is to be created.

Default value: ${solr.default.core.name}

sitemap.queryString

An optional filter expression to limit the documents to be included in the sitemap. This filter is used to query an index when creating the sitemap. All documents matching the filter will be included in the sitemap. The syntax of the filter expression corresponds to the Standard Query Parser from Apache Solr.

Default value: *:*

sitemap.lastModifiedFields

A sorted list of names of those attributes that contain the date of the last modification for a referenced document.

With each URL in the sitemap, the optional element lastmod can be specified, which contains a date value. This date indicates the timeliness of the resource. There are a number of fields in the index that can be used for this purpose, e.g. 'created' or 'modified'. Which field should be taken for the timeliness date depends on the content of the index.

Example: The value of the sitemap.lastModifiedFields is 'created, modified'. When creating the sitemap, the system first looks for the field created in the current document. If this is present, its value is used for the element lastmod in the sitemap. If the field is not present, the second field in the list (here: modified) is looked up, and so on. If none of the fields is present, the element lastmod is not created in the sitemap for the current document.

sitemap.maxEntries

The maximum number of URLs per sitemap file.

Each sitemap file may contain a maximum of 50,000 entries, or have 50 MB in uncompressed size. If the maximum number of a sitemap file specified here is exceeded, a new sitemap file is automatically created for the remaining entries. If the number of sitemap files created is greater than 1, a file sitemap_index.xml is also created in the same directory, which refers to the individual sitemap files.

Default value: 45000

sitemap.baseUrl

Specifies the base URL of the server hosting the sitemap files. Usually this is the URL where the smart.finder client can be reached, e.g. https://[HOST]:[PORT]/smartfinder.

sitemap.storageLocation

The location of the sitemap file in the file system. It is advisable to specify the root directory of the smart.finder application here if possible, so that the files can be accessed directly via ${sitemap.baseUrl}.

Ensure fixed identifiers (IDs)

A fixed identifier is essential for accessing documents in the index. If resources are indexed that do not have a unique ID, the ID is automatically generated by smart.finder during indexing.

To ensure that the same identifier is generated for a document on each indexing pass, the ConditionalIdProcessorFactory must be set to the algorithm fieldhash in the solrconfig.xml of the relevant core. This algorithm calculates a unique hash value for a given number of fields. This only changes if the values of the fields change. fieldhash is the default setting in the index "core0".

If no field is found for a document, a random ID is generated as a fallback solution.

Provision of the robots.txt file

To provide a web crawler with some basic information on what contents on the website are allowed to be crawled and which are not, as well as the location of the sitemap, a so-called robots.txt file can be used. You can find an introduction to the syntax here .

In the root directory of the web application smartfinder you will find a prepared robots.txt.sample file which contains a minimal setup. Modify this file according to the web crawling requirements of your site. In any case, replace [SITEMAP_URL] with the absolute URL to the generated sitemap.

Example:

Sitemap: https://demos.conterra.de/smartfinder/sitemap-1.xml.gz

If more than one sitemap file and thus also a sitemap index file were created, insert the URL to the sitemap_index.xml file here.

Example:

Sitemap: https://demos.conterra.de/smartfinder/sitemap_index.xml

Then rename the file to robots.txt. When a web crawler visits your site, it will look for the robots.txt file first and then index your site according the settings in this file.

Dynamic rendering for smart.finder

The smart.finder web frontend depends to a large degree on JavaScript. This poses a major problem for many search engine crawlers, as these have limited support for JavaScript.

To overcome this problem, smart.finder can render a web page in advance, if the page is requested by a search engine crawler. That means the web page’s JavaScript is executed on the server and the crawler is provided with the final HTML code instead of JavaScript. Users with ordinary web browsers accessing your site will still be delivered the default page including JavaScript.

For prerendering web pages, Rendertron must be installed on your server. Rendertron is based on Node.js and Puppeteer and requires an instance of Google Chrome to be installed on your system.

Installing Rendertron

Follow the instructions on the Rendertron website to install Rendertron on your machine. When Rendertron is installed and running it will listen on port 3000 by default. The Rendertron service must be accessible by the smart.finder backend and should not be publicly available.

Configuring smart.finder

The smart.finder configuration differs depending on the way you installed it on your system (standalone or in an existing map.apps instance).

Configuring smart.finder for the standalone installation variant

If you installed smart.finder using the smartfinder.war file that shipped with smartfinder, proceed as follows:

  1. In the application.properties file of the "smartfinder" webapp, set the following property to the URL that the smart.finder backend uses to access Rendertron.

    seo.dynamicRendering.url=http://localhost:3000/render

  2. Restart Tomcat, so that smart.finder is loaded with the new settings.

Configuring smart.finder with an existing map.apps instance

If you installed the smart.finder bundles in an already existing map.apps instance, there are some more steps to do:

  1. Copy the file resources/seo/ct-finder-seo-filter-%version%.jar that shipped with smart.finder to the folder %TOMCAT%/webapps/mapapps/WEB-INF/lib (the name of the mapapps folder might differ in your installation).

  2. Open the file %TOMCAT%/webapps/mapapps/WEB-INF/classes/spring-filter-config.xml in a text editor.

  3. In this file look for the element <ref bean="jsonTojspDispatcherFilter"/> and insert a new element <ref bean="prerenderFilter"/> as follows:

    <list>
    ...
    <ref bean="jsonTojspDispatcherFilter"/> <!-- Already existing element -->
    <ref bean="dynamicPrerenderFilter"/> <!-- <- Insert this new element -->
</list>

  4. In the lower section of the same file, within the outer beans element, insert the following bean element:

    <beans>
    <!-- ... -->
    <!-- Already existing element -->
    <bean id="propertiesReplacementFilter" class="de.conterra.config.support.rest.PropertiesReplacementFilter">
        <property name="propertiesResolver" ref="configPropertiesResolver"/>
        <property name="acceptPath" value="/,/account/login/,/manager/,/manager/3.x/*,/*.html,**/index.html,**/boot.js,**/*init.js,**/*.json,/js/sample/,/js/sample/**/*.html,/js/tests/test-apps/,/js/**/tests/**/*.html"/>
        <property name="excludePath" value="/resources/**,/event/**,/proxy/**,**/$raw/**"/>
    </bean>

    <!-- +++++++++++++++++++ -->
    <!-- Add the following bean element to the file -->
    <bean id="dynamicPrerenderFilter" class="de.conterra.finder.seo.DynamicRenderingFilter">
        <constructor-arg name="renderer">
            <bean class="de.conterra.finder.seo.DefaultRenderer">
                <constructor-arg name="baseURL" value="${seo.dynamicRendering.url}"/>
                <constructor-arg name="httpClientFactory" ref="defaultHttpClientFactory"/>
            </bean>
        </constructor-arg>
        <constructor-arg name="searchBotSniffer">
            <bean class="de.conterra.finder.seo.SearchBotSniffer">
                <constructor-arg name="userAgentRegEx">
                    <array value-type="java.lang.String">
                        <value>.*bingbot.*</value>
                        <value>.*Googlebot.*</value>
                    </array>
                </constructor-arg>
            </bean>
        </constructor-arg>
    </bean>
</beans>

  5. Open the file application.properties file of your map.apps instance and add the following property at the end of the file:

    seo.dynamicRendering.url=http://localhost:3000/render

    Here, set the URL to the renderer service as it is accessible from the map.apps backend. The Rendertron service listens on port 3000 by default.

  6. Restart Tomcat to reload map.apps with the new settings.

Register smart.finder at search engines

As final step, you should register the smart.finder website at the desired search engines. For Google, you can do this with the Google Search Console. In Bing, registering a website is done through the Bing Webmaster Tools.