Indexing ArcGIS Server layers

smart.finder allows you to index layers of an ArcGIS Feature Server and ArcGIS Map Server. To do this, you must adapt the indexing schema and create a mapping of the attributes of a feature type in the Solr schema. The following example shows you the necessary steps.

smart.finder Server is delivered with a preconfigured ArcGIS Feature Server Setup. This setup consists of:

A Solr Core with preconfigured schema (smartsearch)
A sample app called full-screen-map-multicore

Both the indexing scheme and the layer configuration are already prepared for indexing a layer "Points of Interest in the city of Münster" with a total of 18739 features. You can use this example as a blueprint for setting up additional layers. The following section explains in detail what has to be done for this.

Data basis

Let’s assume that you want to index a layer for "Points of Interest in the city of Münster". A feature of this layer has the following fields:

FID (type: esriFieldTypeOID, alias: FID, SQL Type: sqlTypeOther, length: 0, nullable: false, editable: false)
osm_id (type: esriFieldTypeString, alias: osm_id, SQL Type: sqlTypeOther, length: 10, nullable: true, editable: true)
code (type: esriFieldTypeSmallInteger, alias: code, SQL Type: sqlTypeOther, nullable: true, editable: true)
fclass (type: esriFieldTypeString, alias: fclass, SQL Type: sqlTypeOther, length: 28, nullable: true, editable: true)
name (type: esriFieldTypeString, alias: name, SQL Type: sqlTypeOther, length: 100, nullable: true, editable: true)
Shape__Area (type: esriFieldTypeDouble, alias: Shape__Area, SQL Type: sqlTypeDouble, nullable: true, editable: false)
Shape__Length (type: esriFieldTypeDouble, alias: Shape__Length, SQL Type: sqlTypeDouble, nullable: true, editable: false)

The fields osm_id, code and name should be taken into account when indexing the features.

Adaptation of the Solr schema

smart.finder is delivered with a preconfigured core for this use case. The smartsearch core is located in the directory

/{smartfinder_server_name}/WEB-INF/solr.home/smartsearch

schema.xml for the smartsearch core:

<fields>
    <field name="identifier" type="string" indexed="true" stored="true" required="false" multiValued="true"/>
    <field name="title" type="text" indexed="true" stored="true" required="false" multiValued="false"/>
    <field name="layer" type="text" indexed="true" stored="true" required="false" multiValued="false"/>
    [...]
    <field name="poi_name" type="text" indexed="true" stored="true" required="false" multiValued="false"/>
    <field name="poi_code" type="text" indexed="true" stored="true" required="false" multiValued="false"/>

    <!-- copy to search field -->
    <copyField source="identifier" dest="full_text"/>
    <copyField source="title" dest="full_text"/>
    <copyField source="layer" dest="full_text"/>
    <copyField source="poi_name" dest="full_text"/>
    <copyField source="poi_code" dest="full_text"/>
    [...]
</fields>

The following fields are mandatory in the Smart Search schema:

layer: Saves the name of the indexed layer
title: Saves the title used for displaying a feature in search results

Further, you can add fields analogously to the above-mentioned procedure depending on the indexed ArcGIS Service Layer.

Linking ArcGIS Server layer and Solr schema

The link between the fields of the layer and the Solr schema is set up in the file /smartfinder-search/WEB-INF/classes/spring-feature-layer-config.xml.

The following listing shows an excerpt from that file:

spring-feature-layer-config.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                            http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">
    <!-- Default layer configuration that can reused for all layer configurations -->
    <bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="defaultIndexingConfiguration">
        <property name="arcGISServerUrl" value="..."/>
        [...]
    </bean>
    <util:list id="layerConfigurations">
        <!-- Configuration for indexing feature layer with id 0 -->
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="layerConfigurationFederalStates">
            <!-- The parent configuration that should be used -->
            <constructor-arg name="parentConfiguration" ref="defaultIndexingConfiguration"/>
            <!-- Layer id of the layer that should be indexed -->
            <property name="id" value="0"/>
            <!-- Name of that will appear in the UI (faceted search)-->
            <property name="descriptiveName" value="POIs Münster"/>
            [...]
        </bean>
        <!-- Configuration of new custom layer -->
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="layerCustomFeatureServer">
            [...]
        </bean>
    </util:list>
</beans>

The listing above and the fully functional sample included in the distribution demonstrate, how you can map the field names of an ArcGIS Server layer to the fields of the Solr schema.

The central part of the configuration is defining LayerConfiguration objects. A LayerConfiguration describes how a specific layer should be indexed. In a LayerConfiguration you define, for example, the fields of an ArcGIS Server layer that are added to the Solr schema. A LayerConfiguration is specified as a bean element inside the <util:list id="layerConfigurations">..</util:list> element. Only the `LayerConfiguration`s included in this list are taken into account when indexing.

Inheritance of configuration values

It often happens that, for example, several layers in an ArcGIS service contain the same fields. In such a case, the same configuration settings would have to be repeated again and again for every individual LayerConfiguration object. To simplify this, a LayerConfiguration can inherit the configuration of a superordinate LayerConfiguration. LayerConfiguration objects with repeating settings can therefore be defined in a much more compact way. A concrete LayerConfiguration can still overwrite values that it has inherited from the superordinate LayerConfiguration.

As can be seen in the example above, the bean element for the parent LayerConfiguration is defined outside the <util:list id="layerConfigurations">..</util:list> element. The following element is inserted into the bean elements within the <util:list id="layerConfigurations">..</util:list> element that are to inherit the configuration:

<constructor-arg name="parentConfiguration" ref="defaultIndexingConfiguration"/>

Here, the value of the ref attribute refers to the name of the superordinate LayerConfiguration.

Parameters of a `LayerConfiguration`

In most cases, the parameters for a LayerConfiguration are specified in single property elements within the bean element of the LayerConfiguration. Every property element has a name and a value attribute to set the concrete configuration value.

You can set the following parameters for the LayerConfiguration. Parameters annotated with required must be set either in the LayerConfiguration object or a referenced superordinate LayerConfiguration.

name (required)

Unique name of the indexing configuration. The name is specified with the name attribute on the corresponding bean element.

Example:

<bean class="de.[...].LayerConfiguration" name="<CONFIG_NAME>"> [...] </bean>

arcGISServerUrl (required)

URL to the layer of the ArcGIS Feature or Map Server.

Example:

<property name="arcGISServerUrl" value="http://example.de/FeatureServer"/>

indexedFieldsToSchemaMapping (required)

Mapping of the fields of the schema to the fields of the layer.
Within a map element specify an entry element for each layer field with the following attributes:

key The name of the field in the layer

value The name of the field in the Solr-Schema

Example:

<property name="indexedFieldsToSchemaMapping">
    <map>
        <entry key="osm_id" value="identifier"/>
        <entry key="code" value="poi_code"/>
        <entry key="name" value="poi_name"/>
    </map>
</property>

The fields osm_id, code and name are attributes of the feature layer, the corresponding fields identifier, poi_code and poi_name are the fields of the schema into which the values are indexed.

titleFields (required)

Defines how the title field is assembled. The title field is used to display the hits in the search box.

To create the title "- Roads (0428)" from a feature with the values name="Roads" and code="0428" use the following configuration:

 <property name="titleFields">
    <list>
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
            <property name="name" value="name"/>
            <property name="postDelimiter" value=" ("/>
            <property name="preDelimiter" value="- "/>
        </bean>
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
            <property name="name" value="code"/>
            <property name="postDelimiter" value=")"/>
        </bean>
    </list>
</property>

The properties preDelimiter and postDelimiter properties denote the characters that are prepended or appended to the field specified with the name property.

pageSize

Maximum number of requested features per request against the layer.

If this value is not set, the value globally configured for the indexing service arcgisServerContentIndexingService from the file /smartfinder-search/WEB-INF/classes/spring-indexing-config.xml is used.

Default: 50

Example:

<property name="pageSize" value="50"/>

indexGeometry

Defines whether the geometry of a feature should be indexed.
Default: true

Example:

<property name="indexGeometry" value="true"/>

translateDomainValues

Defines whether existing domain values of features should be considered as names in the index.
If false, only the code of one field is taken.
Default: false

Example:

<property name="translateDomainValues" value="false"/>

Indexing of the layer

To index the layer, create an indexing job for a resource of type URL in the Job Manager .

The URL of these jobs points to the Feature Server layer that you previously defined in the layer configuration. Related to this is the URL: <arcGISServerUrl>/<id>.

For the prebuilt example, the URL is: https://services.arcgis.com/ObdAEOfl1Z5LP2D0/arcgis/rest/services/poi_muenster_WFL1/FeatureServer/0

After starting the job, the fields of the layer are indexed according to the configuration.

If the feature layer contains the error code Error: Error performing query operation during indexing it is possible that the heap size of the SOCs in the ArcGIS server is too small to handle the feature response to be provided completely.

You can solve this in two different ways:

Reduction of pageSize in the layer configuration of smart.finder
Adjusting the heap size in ArcGIS Server, see ESRI Support Error: Error performing query operation

Result of the indexing

To see the results open the example app: https://<yourserver>/smartfinder/?lang=en&app=full-screen-map-multicore.
Select the smart search core:
After entering the search term Schule, the search result is as follows:
A click on a search result in the list shows the feature on the map.

Indexing ArcGIS Server layers

Data basis

Adaptation of the Solr schema

Linking ArcGIS Server layer and Solr schema

Inheritance of configuration values

Parameters of a LayerConfiguration

Indexing of the layer

Result of the indexing

Parameters of a `LayerConfiguration`