Indexing ArcGIS Server layers

smart.finder allows you to index layers of an ArcGIS Feature Service and ArcGIS Map Service. To do this, you must create a mapping of the attributes of a feature type to fields in the Solr schema. The following example shows you the necessary steps.

smart.finder is delivered with a preconfigured ArcGIS Feature Server Setup. This setup consists of:

A Solr Core with preconfigured schema (smartsearch)
A sample app called full-screen-map-multicore

The layer configuration is already prepared for indexing a layer "Points of Interest in the city of Münster" with a total of 18739 features. You can use this example as a blueprint for setting up additional layers. The following section explains in detail what has to be done for this.

Data basis

Let’s assume that you want to index a layer for "Points of Interest in the city of Münster". As can be seen from the metadata of the service, a feature of this layer has the following fields:

FID (type: esriFieldTypeOID, alias: FID, SQL Type: sqlTypeOther, length: 0, nullable: false, editable: false)
osm_id (type: esriFieldTypeString, alias: osm_id, SQL Type: sqlTypeOther, length: 10, nullable: true, editable: true)
code (type: esriFieldTypeSmallInteger, alias: code, SQL Type: sqlTypeOther, nullable: true, editable: true)
fclass (type: esriFieldTypeString, alias: fclass, SQL Type: sqlTypeOther, length: 28, nullable: true, editable: true)
name (type: esriFieldTypeString, alias: name, SQL Type: sqlTypeOther, length: 100, nullable: true, editable: true)
Shape__Area (type: esriFieldTypeDouble, alias: Shape__Area, SQL Type: sqlTypeDouble, nullable: true, editable: false)
Shape__Length (type: esriFieldTypeDouble, alias: Shape__Length, SQL Type: sqlTypeDouble, nullable: true, editable: false)

The fields osm_id, code and name should be taken into account when indexing the features.

Maintenance and configuration of the search index

smart.finder is delivered with a preconfigured core smartsearch.

The fields required for indexing are automatically added at runtime in accordance with the configuration described below. Manual adjustments to the indexing scheme are not necessary.

If required, manual adjustments can still be carried out as described below.

Adjusting the indexing schema manually

The smartsearch core is located in the directory

/smartfinder-search/WEB-INF/solr.home/smartsearch

When managing the index, the managed-schema file is created from an existing schema.xml file (if available) when the application is started. The original file schema.xml is renamed schema.xml.bak and serves as a backup. Only the managed-schema file is actively used, which may not be changed manually. Changes to the file can only be made via the configuration described on this page or via the Solr Admin.

To enable manual adjustments, the smart.finder must first be stopped and the following steps carried out:

Rename file schema.xml.bak to schema.xml
Optionally: Copy the content from the file managed-schema to the file schema.xml if you have made modifications via the Solr Admin.
Delete the managed-schema file
Make adjustments to the schema.xml file
Start smart.finder

After starting, the files are created or renamed again.

Linking the ArcGIS Server layer and the Solr schema

The link between the fields of the layer and the Solr schema is set up in the file /smartfinder-search/WEB-INF/classes/spring-feature-layer-config.xml.

The following listing shows an excerpt from that file:

spring-feature-layer-config.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                            http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">
    <!-- Default layer configuration that can reused for all layer configurations -->
    <bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="defaultIndexingConfiguration">
        <property name="arcGISServerUrl" value="..."/>
        <property name="id" value="0"/>
        <property name="fieldsConfig">
            <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldsIndexingConfig">
                <property name="fetchAllFieldsFromServerExcept">
                    [...]
                </property>
                <property name="fieldToSchemaMappings">
                    [...]
                </property>
            </bean>
        </property>
        <property name="titleFields">
            [...]
        </property>
    </bean>
    <util:list id="layerConfigurations">
        <!-- Configuration for indexing feature layer with id 0 -->
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="layerConfigurationFederalStates">
            <!-- The parent configuration that should be used -->
            <constructor-arg name="parentConfiguration" ref="defaultIndexingConfiguration"/>
            <!-- Layer id of the layer that should be indexed -->
            <property name="id" value="0"/>
            <!-- Name of that will appear in the UI (faceted search)-->
            <property name="descriptiveName" value="POIs Münster"/>
            [...]
        </bean>
        <!-- Configuration of new custom layer -->
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="layerCustomFeatureServer">
            [...]
        </bean>
    </util:list>
</beans>

The listing above and the fully functional sample included in the distribution demonstrate, how you can map the field names of an ArcGIS Server layer to the fields of the Solr schema.

The central part of the configuration is defining LayerConfiguration objects. A LayerConfiguration describes how a specific layer should be indexed. In a LayerConfiguration you define, for example, the fields of an ArcGIS Server layer that are added to the Solr schema. A LayerConfiguration is specified as a bean element inside the <util:list id="layerConfigurations">..</util:list> element. Only the `LayerConfiguration`s included in this list are taken into account when indexing.

Inheritance of configuration values

In ArcGIS Services, several layers often contain the same fields, which leads to repeated configuration settings in the LayerConfiguration objects. To simplify this, a LayerConfiguration can inherit the settings of a parent LayerConfiguration. This allows repetitive settings to be defined more compactly. A specific LayerConfiguration can still overwrite inherited values.

As can be seen in the example above, the bean element for the parent LayerConfiguration is defined outside the <util:list id="layerConfigurations">..</util:list> element. The following element is inserted into each bean element within the <util:list id="layerConfigurations">..</util:list> element that is to inherit the configuration:

<constructor-arg name="parentConfiguration" ref="defaultIndexingConfiguration"/>

Here, the value of the ref attribute refers to the name of the superordinate LayerConfiguration.

Configuration of a `LayerConfiguration` object

This section describes the possible configurations that can be defined on a LayerConfiguration object.

Name of a `LayerConfiguration` bean

Every bean element for a LayerConfiguration must have a unique name. This name is required to reference the bean at other locations within the Spring configuration.

The name is specified with the name attribute on the corresponding bean element.

Example:

<bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="myLayerConfiguration">
    [...]
</bean>

Defining a superordinate `LayerConfiguration`

A LayerConfiguration can inherit the settings of a parent LayerConfiguration. The superordinate LayerConfiguration is defined in the constructor-arg element of the LayerConfiguration bean. The value of the ref attribute refers to the name of the superordinate LayerConfiguration.

Example:

<constructor-arg name="parentConfiguration" ref="defaultIndexingConfiguration"/>

Parameters of a `LayerConfiguration`

The parameters for a LayerConfiguration are specified in single property elements within the bean element of the LayerConfiguration. Every property element has a name attribute that contains the name of the configuration parameter. The value of the parameter is defined either in a value attribute or as content of the property element.

You can set the parameters in the list below for LayerConfiguration objects.

Parameters annotated with required must be set either in the LayerConfiguration object or in a referenced superordinate LayerConfiguration.

arcGISServerUrl (required)

URL to the layer of the ArcGIS Feature or Map Server.

Example:

<property name="arcGISServerUrl" value="http://example.de/FeatureServer"/>

id (required)

ID of the ArcGIS Feature layer to be indexed

Example:

<property name="id" value="0"/>

fieldsConfig (required)

Defines the fields of an ArcGIS feature that are to be indexed as well as their names in the Solr index.

The value of this parameter is a bean element of type FieldsIndexingConfig. This bean element can contain the properties fetchAllFieldsFromServerExcept and fieldToSchemaMappings.

Example:

<property name="fieldsConfig">
    <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldsIndexingConfig">
        <property name="fetchAllFieldsFromServerExcept">
            [...]
        </property>
        <property name="fieldToSchemaMappings">
            [...]
        </property>
    </bean>
</property>

The properties fetchAllFieldsFromServerExcept and fieldToSchemaMappings can be used individually or in combination, but at least one of the properties must be defined.

fetchAllFieldsFromServerExcept

A list of fields that are to be excluded from indexing.

Example: Index all fields except osm_id
```
<property name="fetchAllFieldsFromServerExcept">
    <list>
        <value>osm_id</value>
    </list>
</property>
```
An empty list means that all fields are indexed.

Example: Index all fields
```
<property name="fetchAllFieldsFromServerExcept">
    <list/>
</property>
```

fieldToSchemaMappings

Explicit specification of the field names to be indexed.

The value of the property is a list element that contains a bean element of the class FieldToSchemaMapping for each field in the ArcGIS Server layer, where each bean has the following property elements:

featureFieldName (mandatory) The name of the field or attribute in the layer

indexFieldName (optional) The name of the field in the Solr schema. If specified, this name is used in the index. If no indexFieldName is specified, the field in the index gets its name from featureFieldName.

smart.finder generates the fields in the index automatically.

Example:

<property name="fieldToSchemaMappings">
    <list>
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldToSchemaMapping">
            <property name="featureFieldName" value="osm_id"/>
            <property name="indexFieldName" value="identifier"/>
        </bean>
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldToSchemaMapping">
            <property name="featureFieldName" value="code"/>
            <property name="indexFieldName" value="poi_code"/>
        </bean>
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldToSchemaMapping">
            <property name="featureFieldName" value="name"/>
            <property name="indexFieldName" value="poi_name"/>
        </bean>
    </list>
</property>

titleFields (required)

Defines how the title field is assembled. The title field is used to display the hits in the search box.

To create the title "- Roads (0428)" from a feature with the values name="Roads" and code="0428" use the following configuration:

 <property name="titleFields">
    <list>
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
            <property name="name" value="name"/>
            <property name="postDelimiter" value=" ("/>
            <property name="preDelimiter" value="- "/>
        </bean>
        <bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
            <property name="name" value="code"/>
            <property name="postDelimiter" value=")"/>
        </bean>
    </list>
</property>

The properties preDelimiter and postDelimiter properties denote the characters that are prepended or appended to the field specified with the name property.

pageSize

Maximum number of requested features per request against the layer.

If this value is not set, the value globally configured for the indexing service arcgisServerContentIndexingService from the file /smartfinder-search/WEB-INF/classes/spring-indexing-config.xml is used.

Default: 50

Example:

<property name="pageSize" value="50"/>

indexGeometry

Defines whether the geometry of a feature should be indexed.
Default: true

Example:

<property name="indexGeometry" value="true"/>

translateDomainValues

Defines whether existing domain values of features should be considered as names in the index.
If false, only the code of one field is taken.
Default: false

Example:

<property name="translateDomainValues" value="false"/>

Tips for an extensive configuration

If you want to prepare a large number of ArcGIS layers for indexing, the configuration in the spring-feature-layer-config.xml file can become very extensive and confusing. To simplify the configuration, it can be split into several files. This allows you to configure services thematically separately and the individual configuration files become smaller and easier to maintain.

The following example shows the configuration of two ArcGIS service layers (‘Tree cadastre’ and ‘Protected areas’) in two separate files. These files are then included in the central configuration file spring-feature-layer-config.xml. The procedure can be applied to any number of layers.

Please note the following points:

The name of the file spring-feature-layer-config.xml must remain unchanged.
The names of the imported files are freely selectable (here: spring-feature-layer-protected-areas.xml and spring-feature-layer-tree-catatster.xml).
The configurations described in this chapter can be applied within the files.

Contents of the file spring-feature-layer-schutzgebiete.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

	<bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="layerSchutzgebiete">
		<property name="arcGISServerUrl" value="https://services2.arcgis.com/jUpNdisbWqRpMo35/arcgis/rest/services/ATKIS_Schutzgebiete_NRW/FeatureServer"/>
		<property name="pageSize" value="50"/>
		<property name="indexGeometry" value="true"/>
		<property name="translateDomainValues" value="false"/>
		<property name="id" value="3"/>
		<property name="descriptiveName" value="ATKIS Schutzgebiete"/>
        <property name="fieldsConfig">
            <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldsIndexingConfig">
                <property name="fieldToSchemaMappings">
                    <list>
                        <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldToSchemaMapping">
                            <property name="featureFieldName" value="NAM"/>
                            <property name="indexFieldName" value="poi_name"/>
                        </bean>
                        <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldToSchemaMapping">
                            <property name="featureFieldName" value="OBJID"/>
                            <property name="indexFieldName" value="poi_code"/>
                        </bean>
                    </list>
                </property>
            </bean>
        </property>
        <property name="titleFields">
			<list>
				<bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
					<property name="name" value="NAM"/>
					<property name="postDelimiter" value=" ("/>
				</bean>
				<bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
					<property name="name" value="OBJID"/>
					<property name="postDelimiter" value=")"/>
				</bean>
			</list>
		</property>
	</bean>
</beans>

Contents of the file spring-feature-layer-baumkataster.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

	<bean class="de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration" name="layerBaumkataster">
		<property name="arcGISServerUrl" value="https://services2.arcgis.com/jUpNdisbWqRpMo35/arcgis/rest/services/K%C3%B6ln_Baumkataster/FeatureServer"/>
		<property name="pageSize" value="50"/>
		<property name="indexGeometry" value="true"/>
		<property name="translateDomainValues" value="false"/>
		<property name="id" value="0"/>
		<property name="descriptiveName" value="Baumkataster"/>
        <property name="fieldsConfig">
            <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldsIndexingConfig">
                <property name="fieldToSchemaMappings">
                    <list>
                        <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldToSchemaMapping">
                            <property name="featureFieldName" value="DeutscherN"/>
                            <property name="indexFieldName" value="poi_name"/>
                        </bean>
                        <bean class="de.conterra.finder.api.arcgis.arcgisserver.FieldToSchemaMapping">
                            <property name="featureFieldName" value="StandortNr"/>
                            <property name="indexFieldName" value="poi_code"/>
                        </bean>
                    </list>
                </property>
            </bean>
        </property>
		<property name="titleFields">
			<list>
				<bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
					<property name="name" value="DeutscherN"/>
					<property name="postDelimiter" value=" ("/>
				</bean>
				<bean class="de.conterra.finder.api.arcgis.arcgisserver.TitleField">
					<property name="name" value="StandortNr"/>
					<property name="postDelimiter" value=")"/>
				</bean>
			</list>
		</property>
	</bean>
</beans>

Integration of the files and layer configurations in spring-feature-layer-config.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">

    <import resource="classpath:spring-feature-layer-schutzgebiete.xml"/>
    <import resource="classpath:spring-feature-layer-baumkataster.xml"/>

    <util:list id="layerConfigurations">
		<ref bean="layerBaumkataster"/>
        <ref bean="layerSchutzgebiete"/>
	</util:list>
</beans>

The individual configuration files are included in the spring-feature-layer-config.xml file using the import element.

In addition, the beans from the imported files are added to the list of all layer configurations in the <util:list id=‘layerConfigurations’> element. The value for the bean attribute corresponds to the name attribute of the respective layer configuration (e.g. <bean class=‘de.conterra.finder.api.arcgis.arcgisserver.LayerConfiguration’ name=‘layerBaumkataster’>.

The layer configurations are then available for indexing.

Indexing of the layer

To index the layer, create an indexing job for a resource of type URL in the Job Manager .

The URL of these jobs points to the Feature Server layer that you previously defined in the layer configuration. Related to this is the URL: <arcGISServerUrl>/<id>.

For the prebuilt example, the URL is: https://services.arcgis.com/ObdAEOfl1Z5LP2D0/arcgis/rest/services/poi_muenster_WFL1/FeatureServer/0

After starting the job, the fields of the layer are indexed according to the configuration.

If the feature layer contains the error code Error: Error performing query operation during indexing it is possible that the heap size of the SOCs in the ArcGIS server is too small to handle the feature response to be provided completely.

You can solve this in two different ways:

Reduction of pageSize in the layer configuration of smart.finder
Adjusting the heap size in ArcGIS Server, see ESRI Support Error: Error performing query operation

Result of the indexing

To see the results open the example app: https://<yourserver>/smartfinder/?lang=en&app=full-screen-map-multicore.
After entering the search term Schule, the search result is as follows:
A click on a search result in the list shows the feature on the map.

Indexing ArcGIS Server layers

Data basis

Maintenance and configuration of the search index

Adjusting the indexing schema manually

Linking the ArcGIS Server layer and the Solr schema

Inheritance of configuration values

Configuration of a LayerConfiguration object

Name of a LayerConfiguration bean

Defining a superordinate LayerConfiguration

Parameters of a LayerConfiguration

Tips for an extensive configuration

Indexing of the layer

Result of the indexing

Configuration of a `LayerConfiguration` object

Name of a `LayerConfiguration` bean

Defining a superordinate `LayerConfiguration`

Parameters of a `LayerConfiguration`