Configuration of Logstash Pipelines

Logstash comes with a customized configuration that meets the requirements of service.monitor Analytics. This includes multiple data pipelines, schema templates for Elasticsearch, and the central configuration file pipelines.yml. By default, this file is completely disabled, but by using the included pipelines.yml, Logstash can be used with multiple pipelines.

The pipelines.yml file and the pipelines folder are copied to the config folder of Logstash. The file contents must be adjusted (see the following explanations)

pipelines.yml (The path specifications must be adapted to the local conditions.)

- pipeline.id: ct-analytics
path.config: "/etc/logstash/pipelines/ct-analytics/*.conf"
#- pipeline.id: ct-log
#  path.config: "/etc/logstash/pipelines/ct-log/*.conf"
#- pipeline.id: ct-monitoring
#  path.config: "/etc/logstash/pipelines/ct-monitoring/*.conf"
#- pipeline.id: ct-arcgis-logfile
#  path.config: "/etc/logstash/pipelines/ct-arcgis-logfile/*.conf"

The pipelines folder contains the concrete logstash pipeline definitions. Here, adjustments must be made in the files: If the pipelines ct-log, ct-arcgis-logfile or ct-monitoring should be operated, it is recommended to comment in both lines by removing #.

\pipelines\ct-analytics\logstash-analytics-900-output.conf
\pipelines\ct-log\ct-log-900-output.conf
\pipelines\ct-monitoring\ct-monitoring-900-output.conf
\pipelines\ct-arcgis-logfile\ct-arcgis-logfile-900-output.conf

Mandatory adjustments:

path to template files
Host name of the Elasticsearch cluster, if not accessible via localhost and port 9200
Configuring username and passwords for accessing Elasticsearch

Optional adjustments:

Date pattern for index rotation
alternative port configuration for inputs (file ct-*-100-input.conf)

Logstash uses one TCP/UDP port (e.g. 12201, 12202, 12203) per data pipeline when using the service.monitor. Please check if access to this port from outside (depending on the purpose of use) is allowed by your firewall!