Maintain Elasticsearch indices (map.apps Analytics data)

The events collected by Analytics increase the file index managed by Elasticsearch. Depending on the usage intensity of the systems that are analyzed via service.monitor, this file index can grow significantly. It is not uncommon for file system requirements to grow by 1 GByte per day, posing particular challenges to the service.monitor operating infrastructure.

Sooner or later, measures will have to be taken which weigh the long-term implications of the data collected against the operational capability of the system. For this decision, it is worth taking a look at the types of events that are currently collected by service.monitor. The following types of events are available within map.apps:

Within the scope of recording events on the server side, there are event types:

The types of events serve different analysis purposes, occur at different frequencies, and take up different amounts of space in the file index. The following table contains suggestions on how to store them for a certain period of time.

Since the log events contribute the lowest value to the long-term usage analysis, but occur frequently and require the most storage space, we recommend that you regularly delete such events from an operational point of view or enable ILM on your indices (see here). This also applies to the events of map interactions and tool usage in a much weaker form. This must always be compared with your individual usage preference.

The Logstash Pipeline described in the article Checking and Ensuring DSGVO Conformity is stored under the path examples/anonymize-user-dsgvo.