Release Notes 5.0

What’s New

Support for Elasticsearch 9.1 and Kibana 9.1

Starting with this version, support for Elasticsearch 9.1 and Kibana 9.1 is introduced. Please note that older versions of Elasticsearch and Kibana are no longer supported. Make sure to update your environment to the latest version to benefit from the newest features and security updates.

Support for running in Docker containers

From this version of service.monitor onwards, con terra provides Docker containers and Docker Compose instructions for the virtualized operation of service.monitor. This enables easier deployment and scaling of the application in various environments.

Please note that the use of Docker containers is subject to additional licensing costs. For more information, please contact our sales department or support@conterra.de.

Support for Safe FME Flow REST API v4

Safe FME Flow data can now be retrieved via the REST API v4. Support for monitoring via the REST API v4 has also been added.

Python Setup API

Die Python Setup API has been extended to provide more fine-grained control options for the update behavior of Elasticsearch and Kibana objects. For more information, see Prepare Elasticsearch Stack via Python.

Update Notes

If you skip several versions during the update, please also follow all update notes of the intervening versions.
General notes on updating service.monitor.

Deprecated Features

Elasticsearch Logstash

Support for using Logstash to transfer data to Elasticsearch has been removed. Please use direct communication with Elasticsearch via the HTTP endpoint instead.

Alert Notifications via X/Twitter

The channel for sending alert notifications via Twitter/X has been removed. Please use the functionality for sending alert notifications via Microsoft Teams Workflows or the Json Webhook instead.

Alert Notifications via Microsoft Teams Incoming Webhook

The channel for sending alert notifications via Microsoft Teams Incoming Webhook is no longer supported by Microsoft and has been removed. Please use the functionality for sending alert notifications via Microsoft Teams Workflows in combination with Adaptive Cards instead.

Monitoring of Safe FME Flow Rest API v2

Monitoring of Safe FME Flow instances via Rest API v2 is no longer supported. Please use Rest API v4.

Configuration Changes

Updating Elasticsearch and Kibana objects via Python script

To provide better control over the update behavior, the configuration parameter overwrite_policy has been introduced in the configuration file. The policy can be set separately for Elasticsearch and Kibana, allowing central objects (e.g., ILM and templates) to be updated independently of each other. Details are described in Configuration of parameters.

The parameter common.overwrite_objects has been removed and replaced by the new parameters elasticsearch.overwrite_policy and kibana.overwrite_policy.

Querying Safe FME Flow data via Rest API v4

Querying Safe FME Flow data is now done via Rest API v4. The previous API v3 is no longer supported in the Python API of service.monitor. Please change your configuration to refer to the HTTP endpoint of API v4.

Updating Elasticsearch and Kibana objects via Python script

To provide better control over the update behavior, the configuration parameter overwrite_policy has been introduced in the configuration file. The policy can be set separately for Elasticsearch and Kibana, allowing central objects (e.g., ILM and templates) to be updated independently of each other. Details are described in Configuration of parameters.

The parameter common.overwrite_objects has been removed and replaced by the new parameters elasticsearch.overwrite_policy and kibana.overwrite_policy.

Changes to the indexing of FME jobs

With the switch to FME Flow REST API v4, some fields that were previously included in the Elasticsearch index will no longer be included. This affects:

  • timeDelivered

  • timeSubmitted

  • request.workspacePath

  • request.TMDirectives

  • request.NMDirectives

  • result.timeRequested

  • result.numFeaturesOutput

These fields are no longer available in the new API version and can therefore no longer be indexed.

Sending notifications via Microsoft Teams Workflows

The configuration for sending notifications via Microsoft Teams Workflows has been updated. Instead of the direct URL to the webhook, the URL to a Microsoft Teams Workflow is now used. The templates for sending messages within the product have also been adapted to make use of the new possibilities offered by Adaptive Cards.

Support for Java 25 and Apache Tomcat 11

With this version of service.monitor, Java 25 and Apache Tomcat 11 are additionally supported. Support for Java 17 is no longer available.

Support for sensitive data in a separate file (secrets.properties)

This release introduces the possibility to move sensitive data from the application.properties file to a separate file called secrets.properties. If the file is viewed by several people (for example in a screen sharing) or you send the configuration file application.properties to our support or others, the sensitive data is no longer visible.

We recommend that you check your configuration and move sensitive data, such as passwords, to a secrets.properties file to increase security.

Example configuration

The file application.properties refers to the file secrets.properties, which contains sensitive data:

# Load secrets from separate file
include=./secrets.properties

# Admin user definition
security.user.admin.pw=${secret.admin.pw}

The actual sensitive data, such as passwords, are then defined in the secrets.properties file:

secret.admin.pw=fill-with-password

Important changes to ingest pipelines

The processing of ArcGIS log data has been changed so that, during the next indexing of ArcGIS data, it is mandatory to write to a new index. This is necessary because the structure of the data has changed and indexing into the old index would lead to errors. === New default value for security.user.pwenc

The property security.user.pwenc specifies which encoding should be used for the user password in INTEGRATED mode. The new default value is now plain.

security.user.pwenc=plain

Changes to ingest pipelines

The processing of ArcGIS log data has been changed so that the next time ArcGIS data is indexed, it must be written to a new index. This is necessary because the structure of the data has changed and indexing to the old index would lead to errors.

Discontinued Features

Logstash

Logstash is no longer officially supported as of this version. All data pipelines are available as Elastic Ingest Pipelines.

Notifications via X/Twitter

The channel for sending notifications via Twitter/X has been removed.

Support for Safe FME Flow Rest API v2/v3

Support for monitoring Safe FME Flow instances via the Rest API v2 has been removed.

Collecting data from Safe FME Flow instances via the Rest API v3 has been removed.

Microsoft Teams Incoming Webhook

The possibility for sending notifications via Microsoft Teams Incoming Webhook is no longer supported by Microsoft and has been removed.

Deprecated Features

  • ONLY_AUTHN mode in /monitor web application (integration with security.manager EE)

Known Limitations

SRVMON-182

[Monitoring] Some UI elements are not updated after job/service deletion

SRVMON-202

[Monitoring] monitoring templates on week day basis cannot be deleted once they have been created

SRVMON-260

[Monitoring] Creation date of job is updated when job is edited

SRVMON-965

[analytics/user interactions] Support IPv6 address obfuscation

SRVMON-1027

[FME Python API] API reads only 1000 most recent jobs from FME Flow

Changelog

5.0.0

New Features

SRVMON-972

[Elasticsearch] Support Elastic 9.1 as default service.monitor runtime platform

SRVMON-970

[FME Python API] Change FME Job ID detection mode to a more reliable mechanism

SRVMON-1028

[Monitoring] Re-Order Service Type appearance in services drop down

SRVMON-973

Elastic Ingest Pipeline Support (Elasticsearch 9.1)

SRVMON-967

[Elastic] Support for FME Flow REST API v4 - "Jobs Harvesting"

SRVMON-1009

[Elastic] Support for FME Flow REST API v4 - "Queue Harvesting"

SRVMON-977

Support Filebeat 9.1 as log shipper

SRVMON-984

[Monitoring] Support secrets.properties

SRVMON-985

[Analytics Webapp] Support secrets.properties

SRVMON-1013

[Monitoring] Add basic support for FME Flow Rest API v4

SRVMON-1015

[docker] Provisioning of an official /monitor docker image

SRVMON-1016

[docker] Provisioning of an official /monitor-analytics docker image

SRVMON-1017

[docker] provide official docker compose files for full service.monitor setup

SRVMON-1007

[Notification] Update Velocity Templates for Microsoft Teams to support Teams Workflow / Adaptive Cards rather than deprecated "incoming Teams Webhook"

Fixed Issues

SRVMON-946

[ArcGIS Dashboard] Query filter does not include all desired ArcGIS message levels for "errors & warnings"

SRVMON-958

[Analytics] [Installation] Dashboards not initialized correctly after first run of Python script (on Linux)

SRVMON-1021

[FME Python API] Log level is not correctly derived from string value

SRVMON-1024

[Ingest] ArcGIS pipeline does not follow ecs scheme for 'log.syslog.severity.code'

SRVMON-1037

[Monitoring] SampleRequest values are not being transfered to HTML Form correctly