Templates, Pipelines, Indices (manually)

Essential parts of the Elasticsearch configuration can be done via Kibana. The distribution contains files that ensure the integration of Elasticsearch and Kibana with service.monitor. This includes the configuration of the indices and index policies as well as the import of the Kibana dashboards.

Dev Console: Index Templates, Lifecycle Templates, Bootstrap Index

The delivery directory contains the folder dev-console with files prepared for execution with the Kibana Dev Tools. The Dev Tools are located in the Management menu item of Kibana. For each of service.monitor’s supported data sources there is a file in the dev-console folder:

  • ct-analytics

  • ct-arcgis-logfile

  • ct-fme-*

  • ct-log

  • ct-monitoring

Depending on which data sources are to be activated, copy the contents of the file into the Dev Tools. As a rule, there are three commands in each file:

  • Create the ILM Policy

  • Create the index template

  • Creating the Bootstrap Index

You can execute the sections by clicking on the arrow symbol from top to bottom. Usually the success of an action is answered with acknowledged in the Dev Tools in the right window.

The command to create the bootraps index does not need to be executed when updating service.monitor for the respective data source if data is already collected on the basis of rolling indices.

You have to repeat this procedure for all data sources to be used. Please refer to the configuration notes under Integration of data sources.

View of the (three) requests in the Dev Tools for the data source ct-arcgis-logfile
A better understanding of the interrelationships of the actions described here is provided by this overview.

Dev Console: Ingest pipelines

Ingest pipelines can change events just before they are saved or when they are updated according to defined rules. The delivery directory contains the folder analytics/elasticsearch/ingest with files prepared for execution with the Kibana Dev Tools and containing the ingest pipelines. Publishing the ingest pipelines is mandatory for running service.monitor.

Publish Ingest Pipeline

The configuration pages for the individual data sources contain notes on which ingest pipelines are relevant in each case.

Configuration for the current index (only for update).

This step is only necessary if it is a reconfiguration of existing indices.

Whether an index has already configured an ingest pipeline for service.monitor can be tracked in the Index Settings. ( Stack Management > Index Management > <Index> > Edit Settings)

image::installation/analytics/setup/index-verify-pipeline.png[Example view of an index with configured pipeline].

To configure the pipeline in Elasticsearch for all indexes, the pipeline must be set to the affected indexes.

Example of post-configuration for ArcGIS log data
PUT /ct-arcgis-logfile-*/_settings
{
"index.final_pipeline" : "ct-monitor-arcgis"
}
Example of post-configuration for monitoring data
PUT /ct-monitoring-*/_settings
{
"index.final_pipeline" : "ct-monitor-monitoring"
}
The pipeline can also be set directly via the edit dialogue (see above).