Maintain Elasticsearch indices
General considerations
The events collected by Analytics increase the file index managed by Elasticsearch. Depending on the usage intensity of the systems that are analyzed via service.monitor, this file index can grow significantly. It is not uncommon for file system requirements to grow by 1 GByte per day, posing particular challenges to service.monitor’s operating infrastructure.
Sooner or later, measures will have to be taken which weigh the long-term implications of the data collected against the operational capability of the system. For this decision, it is worth taking a look at the types of events that are currently collected by service.monitor.
The following types of events are available within map.apps:
-
Start of the map.apps application
-
map interactions (zoom, pan)
-
tool interactions
-
Javascript console events
Within the scope of recording events on the server side, there are event types:
-
Requests to protected services (security.manager)
-
general server requests (security.manager, map.apps, service.monitor)
The types of events serve different analysis purposes, occur at different frequencies, and take up different amounts of space in the file index. The following table contains suggestions on how to store them for a certain period of time.
| Event type | Avg. event size in kb | Recommended storage time in index | Use in dashboards or widgets (selection) |
|---|---|---|---|
Console log events |
0,81 |
1 day - 1 month |
Table of errors and warnings, errors and warnings per app and per browser |
map.apps apps start events |
0,7 |
forever |
Many analyses are based on this type of event: Geo-IP localization, app usage frequency, user operating environment, … |
map.apps map interactions |
0,69 |
6 months - forever |
Used Services, Requests against ArcGIS Online, Heatmap and Display of Level of Details |
map.apps tool usage |
0,52 |
6 months - forever |
used actions in the apps, users are searching, users are selecting |
Server event of security.manager & map.apps |
0,35 |
6 months - forever |
Use of protected services, all server requests |
Log events make the least contribution to long-term usage analysis, but occur frequently and require the most storage space. We recommend deleting such events regularly from an operational perspective or enabling ILM on your indexes. This also applies to the events of map interactions and tool usage in a much weaker form. This must always be compared with your individual usage preference.
Subsequent anonymization of user information
The Logstash Pipeline described in the article Checking and Ensuring DSGVO Conformity is stored under the path examples/anonymize-user-dsgvo.