Overview of all update notes

In the course of an upgrade, all Elasticsearch-related objects such as pipelines, templates, Kibana diagrams and ingest pipelines must be updated.

5.0

Deprecated Features

Elasticsearch Logstash

Support for using Logstash to transfer data to Elasticsearch has been removed. Please use direct communication with Elasticsearch via the HTTP endpoint instead.

Alert Notifications via X/Twitter

The channel for sending alert notifications via Twitter/X has been removed. Please use the functionality for sending alert notifications via Microsoft Teams Workflows or the Json Webhook instead.

Alert Notifications via Microsoft Teams Incoming Webhook

The channel for sending alert notifications via Microsoft Teams Incoming Webhook is no longer supported by Microsoft and has been removed. Please use the functionality for sending alert notifications via Microsoft Teams Workflows in combination with Adaptive Cards instead.

Monitoring of Safe FME Flow Rest API v2

Monitoring of Safe FME Flow instances via Rest API v2 is no longer supported. Please use Rest API v4.

Configuration Changes

Updating Elasticsearch and Kibana objects via Python script

To provide better control over the update behavior, the configuration parameter overwrite_policy has been introduced in the configuration file. The policy can be set separately for Elasticsearch and Kibana, allowing central objects (e.g., ILM and templates) to be updated independently of each other. Details are described in Configuration of parameters.

The parameter common.overwrite_objects has been removed and replaced by the new parameters elasticsearch.overwrite_policy and kibana.overwrite_policy.

Querying Safe FME Flow data via Rest API v4

Querying Safe FME Flow data is now done via Rest API v4. The previous API v3 is no longer supported in the Python API of service.monitor. Please change your configuration to refer to the HTTP endpoint of API v4.

Updating Elasticsearch and Kibana objects via Python script

To provide better control over the update behavior, the configuration parameter overwrite_policy has been introduced in the configuration file. The policy can be set separately for Elasticsearch and Kibana, allowing central objects (e.g., ILM and templates) to be updated independently of each other. Details are described in Configuration of parameters.

The parameter common.overwrite_objects has been removed and replaced by the new parameters elasticsearch.overwrite_policy and kibana.overwrite_policy.

Changes to the indexing of FME jobs

With the switch to FME Flow REST API v4, some fields that were previously included in the Elasticsearch index will no longer be included. This affects:

  • timeDelivered

  • timeSubmitted

  • request.workspacePath

  • request.TMDirectives

  • request.NMDirectives

  • result.timeRequested

  • result.numFeaturesOutput

These fields are no longer available in the new API version and can therefore no longer be indexed.

Sending notifications via Microsoft Teams Workflows

The configuration for sending notifications via Microsoft Teams Workflows has been updated. Instead of the direct URL to the webhook, the URL to a Microsoft Teams Workflow is now used. The templates for sending messages within the product have also been adapted to make use of the new possibilities offered by Adaptive Cards.

Support for Java 25 and Apache Tomcat 11

With this version of service.monitor, Java 25 and Apache Tomcat 11 are additionally supported. Support for Java 17 is no longer available.

Support for sensitive data in a separate file (secrets.properties)

This release introduces the possibility to move sensitive data from the application.properties file to a separate file called secrets.properties. If the file is viewed by several people (for example in a screen sharing) or you send the configuration file application.properties to our support or others, the sensitive data is no longer visible.

We recommend that you check your configuration and move sensitive data, such as passwords, to a secrets.properties file to increase security.

Example configuration

The file application.properties refers to the file secrets.properties, which contains sensitive data:

# Load secrets from separate file
include=./secrets.properties

# Admin user definition
security.user.admin.pw=${secret.admin.pw}

The actual sensitive data, such as passwords, are then defined in the secrets.properties file:

secret.admin.pw=fill-with-password

Important changes to ingest pipelines

The processing of ArcGIS log data has been changed so that, during the next indexing of ArcGIS data, it is mandatory to write to a new index. This is necessary because the structure of the data has changed and indexing into the old index would lead to errors. === New default value for security.user.pwenc

The property security.user.pwenc specifies which encoding should be used for the user password in INTEGRATED mode. The new default value is now plain.

security.user.pwenc=plain

Changes to ingest pipelines

The processing of ArcGIS log data has been changed so that the next time ArcGIS data is indexed, it must be written to a new index. This is necessary because the structure of the data has changed and indexing to the old index would lead to errors.

4.10

Adjustments to the Analytics indices

Since changes have been made to the use of the ingest pipelines, existing indices of the analytics-usagelog-* pattern must also be adapted via the Elastic Dev Console.

Input Elastic Dev Console
PUT /analytics-usagelog-*/_settings
{
"index.final_pipeline":"_none",
"index.default_pipeline":"ct-monitor-analytics"
}

Configuration changes

New properties concerning the connection between /monitor-analytics and Elasticsearch

User interaction data is no longer sent to Elasticsearch via the Logstash component, but directly to Elastic via HTTP.

analytics.elastic.url

URL to Elasticsearch http endpoint

analytics.elastic.username

Username for authenticating w/ Elasticsearch

Default: elastic

analytics.elastic.password

Password for authenticating w/ Elasticsearch

4.9

Customization of the database tables

Due to the extension of the monitoring by a possible delay of the notification in case of errors, the table SM_JOB must be updated. Please note the SQL update scripts, which are located in the resources\sql\upgrade\4.9 folder of the delivery.

4.8

Update Tomcat and Java

Please update you Tomcat instance to version 10. In case you have been using Java 11 so far, please update to version 17 or 21.

4.7

There are no actions to take.

4.6

Properties

The properties of the two web applications monitor and monitor-analytics have been adjusted and harmonized with the other con terra products. This concerns the properties of the mailing:

  • mail.host becomes mailing.host.
    This change must be made in external application.properties files.

Updating Elastic Index Lifecycle Management Policies & Templates

If service.monitor 4.5 is already in use, an update of the Index Lifecycle Policies and Index Templates for the used data sources is necessary (resources/analytics/elasticsearch/dev-console). In the templates, a bug with the alias configuration is fixed and the use of Elastic Ingest Pipelines is anchored.

Since version 4.5, if the data source to map.apps user interactions is already active, then the following steps must be performed (SRVMON-718):

  • Update ILM and Index Template as described above.

  • Execute each statement below, replacing -000001 with the value of the most recent index (this may be different for each index)

Update analytics usagelog indices config
POST ct-analytics-app*/_alias/analytics-usagelog-app
POST ct-analytics-app-000001/_alias/analytics-usagelog-app
{"is_write_index":true}


POST ct-analytics-map*/_alias/analytics-usagelog-map
POST ct-analytics-map-000001/_alias/analytics-usagelog-map
{"is_write_index":true}

POST ct-analytics-log*/_alias/analytics-usagelog-log
POST ct-analytics-log-000001/_alias/analytics-usagelog-log
{"is_write_index":true}

POST ct-analytics-server*/_alias/analytics-usagelog-server
POST ct-analytics-server-000001/_alias/analytics-usagelog-server
{"is_write_index":true}

POST ct-analytics-tool*/_alias/analytics-usagelog-tool
POST ct-analytics-tool-000001/_alias/analytics-usagelog-tool
{"is_write_index":true}

DELETE ct-analytics-app*/_alias/ct-analytics-app
DELETE ct-analytics-tool*/_alias/ct-analytics-tool
DELETE ct-analytics-map*/_alias/ct-analytics-map
DELETE ct-analytics-log*/_alias/ct-analytics-log
DELETE ct-analytics-server*/_alias/ct-analytics-server

In the case of using service.monitor with FME Server Jobs, manual adjustments are also necessary if the functionality is already in use (SRVMON-733). Due to an error, an incorrect assignment of indexes, aliases and patterns occurred here (ct-fme-jobs vs. ct-fme-job).

The stock data must be copied to the index pattern ct-fme-jobs-* under ILM administration. This is done by re-indexing into the alias ct-fme-jobs. Afterwards, the old index ct-fme-job can be removed.

Correct FME Jobs Index data location
# re-index pre-existing entries to index alias
POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "ct-fme-job"
  },
  "dest": {
    "index": "ct-fme-jobs"
  }
}
# once re-indexing has finished and after verifying all fme jobs have been transfered, the old index is safe to delete
DELETE /ct-fme-job

# same procedure for fme jobroutes
POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "ct-fme-jobroute"
  },
  "dest": {
    "index": "ct-fme-jobroutes"
  }
}
DELETE /ct-fme-jobroute

Notifications: Updating the monitoring database schema

Please run the SQL script for your database under resources/sql/upgrade/4.6.

Monitoring: Oracle DB data type check

If you use Oracle DB: Please check if resources/sql/upgrade/4.5/oracle-schema-changes.sql was already executed with the last upgrade.

Analytics Bundles for map.apps

The bundles for collecting user interactions now follow their own release cycle and have been renamed from e.g. usagelog_restservice to analytics_restservice. This must be taken into account when referencing the bundles in the app.json or the application.properties of map.apps.

4.5

Monitoring

Migration of Oracle DB data type from LONG RAW to BLOB

In order to replace the data type LONG RAW, which is no longer recommended for Oracle databases, with the data type BLOB, the SQL script located under resources\sql\upgrade\4.5 can be executed. The migration forces the application.property value db.type to be set to the value oracle.

4.4

Analytics

Elasticsearch 7

With service.monitor version 4.4, a major version jump was made to the base technology Elasticsearch used in Analytics. Please follow following update notes.

Monitoring

Adjusting the database tables

Due to the extension of the monitoring by the password management, a new table must be created for existing installations. Please note the SQL update scripts that are located in the update folder of the SQL directory of the installation package for monitoring.

Configurational adjustments

Java version

service.monitor supports Java Runtime Environments in version 8 and 11.

application.properties

The configuration of the database manufacturer has been simplified in the monitoring configuration. Instead of having to enter all the details of the driver, dialect, etc., it is now sufficient to set a single property (here PostgreSQL as an example):

db.type=postgresql

4.3

This section lists all changes to be considered when performing updates.

Adjusting the database tables

By extending the monitoring by service expectations, two new tables have to be created for existing installations. Please note the SQL update scripts that are located in the folder update of the SQL directory of the installation package for monitoring.

More strict SSL connection validation (since Monitoring 4.3.3)

With version 4.3.3 there is by default a more strict validation of HTTP SSL connections in place (sue to security reasons). service.monitor will deny connections to those servers which provide an untrusted, invalid or self-signed certificate. service.monitor will also send a notification once a certificate is not valid. This is also true for those (valid) certificates which are not included in the Java Virtual Machine’s keystore of your Servlet Container. Please see the documentation for importing these.

This behavior can be overridden by configuration options. See chapters Common configuration options and SSL configuration options of the installation guide.

Configuration changes

Manual adjustment of the mail subject

Using the configuration values mail.subject.de and mail.subject.en, prefixes can now be defined in the application.properties file, which precede the usual mail subject.