Release Notes 4.9

What’s New

ArcGIS Enterprise Log Data Transfer & Dashboards

ArcGIS Enterprise log data is now sent directly from Filebeat to Elastic. The log events are processed in an Elastic Ingest Pipeline. This reduces the number of events that need to be sent via Logstash and standardizes the Filebeat configuration of hosts that use both ArcGIS Enterprise and FME Flow. Furthermore, improvements have been made to the ArcGIS related dashboards and new diagrams have been created for analysis.

Portal for ArcGIS Monitoring

The monitoring application now also natively supports the "Portal for ArcGIS" service type. In addition to a health check, the functionality of the federated ArcGIS Server can also be checked regularly.

Delay in notification of service failures

In certain use cases, it makes sense to only trigger and send a message to the affected persons after a certain number of failures. From now on, you can define for each job after how many errors the alerting rule should become active.

Elastic Aggregation Expectations

In the monitoring application, another expectation type can be used in connection with Elasticsearch. The administrator can compare the values of Elastic aggregation calculations with a threshold value. An alarm is triggered in the event of a failure. For example: "The average response time of the "World Map" service in the last 15 minutes is less than 0.5 seconds".

Update Notes

If you skip several versions during the update, please also follow all update notes of the intervening versions.
General notes on updating service.monitor.

Customization of the database tables

Due to the extension of the monitoring by a possible delay of the notification in case of errors, the table SM_JOB must be updated. Please note the SQL update scripts, which are located in the resources\sql\upgrade\4.9 folder of the delivery.

Deprecated Features

  • Elasticsearch 7.x (will be replaced by Elastic 8.x)

  • ONLY_AUTHN mode in /monitor web application (integration with security.manager EE)

  • FME Logstash pipelines

  • ArcGIS Logstash pipeline

  • map.apps Logstash pipeline

  • user_agent_parsed field data model will change significantly when upgrading to Elastic 8 compatibility (map.apps data flow)

Known Limitations

SRVMON-182

[Monitoring] Some UI elements are not updated after job/service deletion

SRVMON-202

[Monitoring] monitoring templates on week day basis cannot be deleted once they have been created

SRVMON-260

[Monitoring] Creation date of job is updated when job is edited

SRVMON-845

[Monitoring] Evaluation of noerror-flag fails for FME Service Type

SRVMON-868

Issue with Logging Queries for AutoRegistered Stores in search-ui

Changelog

4.9.1

Fixed Issues

SRVMON-880

[Monitoring] Static Token Credentials are not respected in monitoring request

SRVMON-881

[SQL] Upgrade Script for MS SQL Server is syntactically incorrect

SRVMON-882

[ct integration] netty-libraries are missing when trying to configure map.apps server side integration

SRVMON-888

Publishing a dashboard via Python API fails if dashboard refers to index-pattern which is not available at Elastic yet

4.9.0

New Features

SRVMON-840

[Monitoring] Allow for specifying a failure threshold count before sending a notification message

SRVMON-841

[Monitoring] Introduction of Elastic Aggregation Query Expectations

SRVMON-842

[Monitoring] Provide basic support for Monitoring Portal for ArcGIS

SRVMON-810

Replace ArcGIS Logstash Pipeline w/ Elastic Ingest Pipeline approach

SRVMON-813

service.monitor Monitoring webapplication supports con terra Identity Service

SRVMON-809

Separate Elasticsearch/Kibana maven module from Python API Code

SRVMON-795

[Setup Python API] Space-Name is hard-coded in Python script

SRVMON-801

[FME Python API] Also store workspace, repo name and job success information to FME Job log events

SRVMON-820

[FME Python API] User name pseudonomization shall create "stable" output

SRVMON-806

[FME Python-API] Allow flag for not obfuscating FME Flow user names in Job items

SRVMON-818

[Python FME API] Pipeline shall not fail if no logs are available for a certain FME job ID

SRVMON-814

Rename Kibana Rules Entry in Monitoring webapp from "active" / "Aktiv" to "enabled"…​.

SRVMON-837

Write to different indices from one filebeat config file, for arcgis and fme logfiles

SRVMON-847

Update 'ct-fme-jobroutes-policy' lifecyle policy to reflect Python driven data flow

SRVMON-849

[ArcGIS Log data] Provide grok patterns to parse german ArcGIS log messages for ags.code 7615 and 7549 in ingest pipeline "ct-monitor-arcgis-parse-servicename"

SRVMON-851

Disable ILM for FME jobroutes index

SRVMON-874

Provide Integration files for security.manager 4.19+ and map.apps 4.15+

Fixed Issues

SRVMON-700

Refreshing of ArcGIS Server Tokens does not always happen at the right time

SRVMON-819

[FME Python API] Do not fail pipeline execution when configuration is not providing verify_ssl_certs from new common subsection

SRVMON-821

ArcGIS Log entry fails publishing when a timeout value was parsed from message

SRVMON-822

[Analytics map.apps API] User authentication status is not tracked in "ct/framework/FrameworkEvent/STARTED"-event

SRVMON-823

[FME Python API] Do not fail pipeline execution if there are fails on submitting documents to elasticsearch

SRVMON-824

[FME Python API] There are indexing errors when FME jobs include empty date fields

SRVMON-826

Creation of expectation containing a '<' fails due to validation problems.

SRVMON-836

[usagelog] Property 'servicemonitor.admin.role' not defined

SRVMON-843

Missing Log Indexing for FME Filebeat and Ingest-Pipeline

SRVMON-848

[FME Flow] Log level matching fails for certain Tomcat related Log Files while ingesting

SRVMON-854

[map.apps js Bundles] sessionId-Attribute is no longer created/collected

SRVMON-863

Console warnings when using analytics_restservice in map