Maintain Elasticsearch indices (map.apps Analytics data)

General considerations

The events collected by Analytics increase the file index managed by Elasticsearch. Depending on the usage intensity of the systems that are analyzed via service.monitor, this file index can grow significantly. It is not uncommon for file system requirements to grow by 1 GByte per day, posing particular challenges to the service.monitor operating infrastructure.

Sooner or later, measures will have to be taken which weigh the long-term implications of the data collected against the operational capability of the system. For this decision, it is worth taking a look at the types of events that are currently collected by service.monitor. The following types of events are available within map.apps:

  • Start of the map.apps application

  • map interactions (zoom, pan)

  • tool interactions

  • Javascript console events

Within the scope of recording events on the server side, there are event types:

  • Requests to protected services (security.manager)

  • general server requests (security.manager, map.apps, service.monitor)

The types of events serve different analysis purposes, occur at different frequencies, and take up different amounts of space in the file index. The following table contains suggestions on how to store them for a certain period of time.

Event type

Avg. event size in kb

Recommended storage time in index

Use in dashboards or widgets (selection)

Console log events

0,81

1 day - 1 month

Table of errors and warnings, errors and warnings per app and per browser

map.apps apps start events

0,7

forever

Many analyses are based on this type of event: Geo-IP localization, app usage frequency, user operating environment, …​

map.apps map interactions

0,69

6 months - forever

Used Services, Requests against ArcGIS Online, Heatmap and Display of Level of Details

map.apps tool usage

0,52

6 months - forever

used actions in the apps, users are searching, users are selecting

Server event of security.manager & map.apps

0,35

6 months - forever

Use of protected services, all server requests

Since the log events contribute the lowest value to the long-term usage analysis, but occur frequently and require the most storage space, we recommend that you regularly delete such events from an operational point of view or enable ILM on your indices (see here). This also applies to the events of map interactions and tool usage in a much weaker form. This must always be compared with your individual usage preference.

Sample Requests

Query index size

print index name and disk usage
curl 'http://localhost:9200/_cat/indices/log*?h=i,ss'

Close Index

Close index 'my_index'
curl -XPOST 'http://localhost:9200/my_index/_close'

Delete Index

Delete index 'my_index'
curl -XDELETE 'http://localhost:9200/my_index'

Subsequent anonymisation of user information according to DSGVO

The Logstash Pipeline described in the article Checking and Ensuring DSGVO Conformity is stored under the path examples/anonymize-user-dsgvo.