Elastic Lifecycle Management

Lifecyle management supports the permanent, stable operation of the Elastic Cluster by defining clear rules for limiting index sizes and discarding events once they have been collected. For each different data category of service.monitor, specifications are made for the following properties:

  • Index Template: The index template defines - if necessary - specifications for the "data model" to be used (mappings) and specifications for internal data storage and replication (shards and replicas). The ingest pipeline to be executed can also be named in a template.

  • Index Pattern: The pattern can be used to decide which index templates are applied to which new indices to be created. ILM Policy Name: The policy defines the rules of the life cycle of a set of indices.

  • Alias: With the help of an alias, many physical indices can be addressed without the need for concrete knowledge of the exact index name for a client.

  • Bootrap Index Name: The name of the first index created using an index lifecycle policy and an index template.

Topic Index template name ILM policy name index pattern Alias Boostrap index name

ct-log

ct-log

ct-log-policy

ct-log-*

ct-log

ct-log-000001

ct-analytics-app

ct-analytics-app

ct-analytics-app-policy

ct-analytics-app*

ct-analytics-app

ct-analytics-app-000001

ct-analytics-map

ct-analytics-map

ct-analytics-map-policy

ct-analytics-map*

ct-analytics-map

ct-analytics-map-000001

ct-analytics-log

ct-analytics-log

ct-analytics-log-policy

ct-analytics-log*

ct-analytics-log

ct-analytics-log-000001

ct-analytics-tool

ct-analytics-tool-server-map-other

ct-analytics-tool-server-map-other-policy

ct-analytics-tool*

ct-analytics-tool

ct-analytics-tool-000001

ct-analytics-server

ct-analytics-tool-server-map-other

ct-analytics-tool-server-map-other-policy

ct-analytics-server*

ct-analytics-server

ct-analytics-server-000001

ct-arcgis-logfile

ct-arcgis-logfile

ct-arcgis-logfile-policy

ct-arcgis-logfile-*

ct-arcgis-logfile

ct-arcgis-logfile-000001

ct-monitoring

ct-monitoring

ct-monitoring-policy

ct-monitoring-*

ct-monitoring

ct-monitoring-000001

ct-fme-jobs

ct-fme-jobs

ct-fme-jobs-policy

ct-fme-jobs-*

ct-fme-jobs

ct-fme-jobs-000001

ct-fme-log

ct-fme-log

ct-fme-log-policy

ct-fme-log-*

ct-fme-log

ct-fme-log-000001

ct-fme-jobroutes

ct-fme-jobroutes

ct-fme-jobroutes-policy

ct-fme-jobroutes-*

ct-fme-jobroutes

ct-fme-jobroutes-000001

The service.monitor data sources are operated with different specifications that can be adapted to local needs. The following ideas, among others, were the guiding principles for the default data:

  • A physical size of 10GB per index is appropriate for Elastic.

  • application log data should be deleted after two years at the latest, as it does not contain any information of long-term value

  • map.apps analytics data is kept forever (apart from Javascript console data)

ILM policy name Rollover size Rollover age Delete after rollover

ct-log-policy

10gb

365d

365d

ct-analytics-app-policy

10gb

ct-analytics-map-policy

10gb

ct-analytics-log-policy

10gb

60d

30d

ct-analytics-tool-server-map-other-policy

10gb

ct-arcgis-logfile-policy

10gb

365

365d

ct-monitoring-policy

10gb

365d

ct-fme-jobs-policy

10gb

365d

365d

ct-fme-log-policy

10gb

365d

365d

ct-fme-jobroutes-policy

24h

1m