Troubleshooting

Data from the various data sources enters service.monitor or the Elasticsearch data storage via different paths. Therefore, there are sometimes different points at which the smooth operation of service.monitor can be tested.

List of test points for troubleshooting

If there are problems during commissioning, the functionality of the individual components must be checked and their operation ensured. The following list, which is currently not exhaustive, lists possible test points.

  • Logstash starts up successfully

  • Elasticsearch is in operation

  • Kibana is reachable via http or https

  • Filebeat is configured correctly and can send data

  • web application /monitor-analytics is in operation

  • web application /monitor can be started in the browser

  • map.apps does send user interaction data

Perform tests

Logstash starts up successfully

  • Checking the logfile of the logstash process (path e.g. /var/log/logstash/logstash-plain.log).

    • Are all defined logstash pipelines started?
      e.g. Pipeline started successfully {:pipeline_id⇒"ct-monitoring can be found in the logfile

    • Are there any error messages related to the connection to Elasticsearch?

Elasticsearch is in operation

  • Checking the logfile of the Elasticsearch process (path e.g. /var/log/elasticsearch/elasticsearch.log).

    • Are there any obvious error messages?

  • Checking the access to Elasticsearch via http.

    • Run the check e.g. with curl curl http://localhost:9200 -u <USER>:<PWD>

    • Run the check from a host running e.g. Filebeat or the Pythons scripts; e.g. with curl curl http://<FQDN>:9200 -u <USER>:<PWD>

Kibana is reachable via http or https

  • Checking if Kibana can be called in the browser and the login with username and password works.

  • Checking the logfile of the Kibana process (path e.g. /var/log/kibana/kibana.log)

    • Are there any obvious error messages?

Filebeat` is configured correctly and can send data

For example, Filebeat is configured on ArcGIS Server hosts to send log data from there to Logstash. * Examine the log file of the Filebeat process (The location of the log data may vary. On Windows it is useful to examine the call in the services administration). ** Are there any error messages regarding the sending of data to 'Logstash'?

This check applies to the integration with ArcGIS Enterprise.

web application /monitor-analytics is in operation

  • Check whether the application is accessible in the browser. This must also be possible from one of the users' browsers.

  • Check the log data of the application for any error messages. The log file is located at <tomcat>/logs/ct-monitor-analytics-webapp.log (different path possible depending on installation).

This check applies to integration with map.apps.

web application /monitor can be started in the browser

  • Check that the application is accessible to the monitoring administrator in the browser.

  • Check the log data of the application for any error messages. The log file is located at <tomcat>/logs/ct-monitor-webapp.log (different path possible depending on installation).

map.apps does send user interaction data

  • Start a map.apps application and open the developer tools of the browser. In the network view, requests to the /monitor-analytics webapp should be visible after the map.apps application has been started.

    • Are requests sent to /monitor-analytics/resources/usage/log?

    • Does the webapp respond with a 202 status code?

This check applies to integration with map.apps.

Other useful checks

Is there any data stored in the Elasticsearch indexes?

  • Open Kibana and navigate to Management > Stack Management > Index Management.

  • Sort the table by Docs count.

  • Is there already data in the index for whose data source you want to troubleshoot?