Analytics/Elastic

You need to apply the installation steps described in this documentation only to those service.monitor data sources in the specific usage scenario. The following data sources or analysis topics are currently supported by service.monitor.

User interaction data of map.apps or security.manager

If you activate service.monitor in map.apps and/or security.manager, concrete user interactions of the users are recorded and sent to Elasticsearch via the monitor-analytics webapp and a Logstash pipeline. You can view and analyze the collected data using the dashboards available in Kibana.

ArcGIS Enterprise logfiles

service.monitor also supports collecting ArcGIS Server and Portal for ArcGIS logfiles. Additional, it provides simple methods for visualization and analyzation of data.

ArcGIS Enterprise ArcGIS SOC Process ingesting

Using Elasticsearch Metricbeat a more detailed understanding of the load on your ArcGIS Enterprise ArcSOC processes can be achieved. service.monitor provides diagrams that show the number of active SOC processes and their load (CPU, RAM) on your ArGIS host system(s).

FME Server job information and log output

service.monitor also allows the display of FME job information and logs output that has arisen during the execution of jobs. By regularly querying the FME job repository, detailed data on the use and utilisation of the FME infrastructure is available, which can be displayed statistically and visually on the dashboards.

con terra Application log files

In recent years, central log data aggregators have become very important for the successful operation of IT systems. This is one of the core competencies of the Elasticsearch Stack. The products and software artifacts developed at con terra support the storage of application log files in such central systems instead of writing this information to the local file system. service.monitor Analytics contains a logstash pipeline that can record and modify log events and send them to Elasticsearch for storage.

Periodic monitoring tasks

service.monitor Monitoring and Analytics can be linked together via a simple data transfer. If the transfer is active, all information about individual monitoring events (service, job, URL, success, error, etc.) is sent from the monitoring to a Logstash pipeline, where it is modified and sent to Elasticsearch for storage. You can then meaningfully analyze this data using the convenient interaction options of the dashboards.

Overview of installation and installation steps

This overview should help to better understand the interrelationships of data sources and necessary installation steps and give hints on how to check the successful data flow.

Ideal typical order

  1. Installation of basic Elasticsearch components

  2. Configuring Elastic Index(-template), Lifecycle Policy and Alias

  3. Importing Kibana dashboards, queries and index patterns

  4. Configuring Logstash Pipelines

  5. Configuring Elastic Ingest pipelines

  6. further activities outside Elasticsearch

Assignment of files

Refer to the following table for the assignment of files from the delivery to the data sources.

Data source Index alias dev-console kibana logstash ingest more

ct-analytics

  • analytics-usagelog-app

  • analytics-usagelog-map

  • analytics-usagelog-tool

  • analytics-usagelog-server

  • analytics-usagelog-log

ct-analytics.txt

ct-analytics/export.ndjson

ct-analytics

ct-monitor-analytics

map.apps Bundle upload, Installation of monitor-analytics-webapp

ct-arcgis

  • ct-arcgis-logfile

ct-arcgis-logfile.txt

ct-arcgis/export.ndjson

ct-arcgis-logfile

ct-monitor-arcgis

Configuration of Filebeat on ArcGIS hosts

ct-arcgis-soc

  • <like Metricbeat>

ct-arcgis-soc.txt

ct-arcgis-soc/export.ndjson

ct-monitor-metricbeat-arcsoc

Configuration of Metricbeat on ArcGIS hosts

ct-fme

  • ct-fme-jobs

  • ct-fme-log

  • ct-fme-jobroutes

ct-fme-*.txt

ct-fme/export.ndjson

ct-fme-*

ct-monitor-fme

set FME environment parameters being used by Logstash

ct-log

  • ct-log

ct-log.txt

ct-log/export.ndjson

ct-log

ct-monitor-logging

Activate Log4J logging via GELF in con terra products

ct-monitoring

  • ct-monitoring

ct-monitoring.txt

ct-monitoring/export.ndjson

ct-monitoring

ct-monitor-monitoring

Activate transfer of monitoring events from monitor-webapp