FME Flow (API v3)

You have the following options to monitor FME Flow.

FME Flow Health Check
FME Flow Engines available on server
FME Flow Engines ready for job execution
Check for delayed FME flow job execution
Job Execution Success Check from FME Flow Schedules

Almost all FME Flow checks are a combination of a specific request and the formulation of specific expectations that are made of the FME Flow. For example, if the availability of FME Flow engines is checked, the user must define how many engines are expected on the server via an expectation.

First, create a credential template for authenticating the requests against FME Flow.

Health Check

To monitor a FME Flow Health Check, follow these steps.

Register a new service with following parameters:
- Name the service.
- Choose type FME Flow (API v3).
- Define the HTTP address of the FME Flow https://fmeserver.<myhost>.org.
Create a job for the new registered service.
- Name the job.
- The Categorie is user specific.
- Choose Standard request. The corresponding request is added to the Content field.
- Define Monitoring & Notification and Quality of service according to your requirements.

Engines available on server

In order to be notified in the event of a failure of engines registered on the server, define an Expectation.

The following expectation expects two engine instances on the FME Flow:

{"type" : "FmeEngineCountExpectation", "content":[2]}

You can now define a job and choose the correct request with the standard request Engine Check. Select the created expectation in the tab Quality of service.

Engines ready for job execution

You can use the same job and an additional expectation to monitor the available capacities of server engines that are available for the job.

In the example below, one engine is expected not to be running a job at the time of the check.

{"type" : "FmeEngineAvailableExpectation", "content":[1]}

Delayed FME Flow job execution

To check delayed jobs — in execution or in the queue — use the FmeJobTimeShiftExpectation. This allows the checking of any time information within an FME job item in relation to the current time and a user defined duration.

Click to see an example FME job item in the queue.

{
	"request": {
		"publishedParameters": [
			{
				"name": "FME_SECURITY_ROLES",
				"raw": "fmeadmin fmesuperuser user:admin"
			},
			{
				"name": "FME_SECURITY_USER",
				"raw": "admin"
			}
		],
		"workspacePath": "\"admin_test/test_workspace_1/test_workspace_1.fmw\"",
		"TMDirectives": {
			"rtc": false,
			"ttc": -1,
			"description": "",
			"tag": "admin_test",
			"priority": -1,
			"ttl": -1
		},
		"NMDirectives": {
			"directives": [],
			"successTopics": [],
			"failureTopics": []
		}
	},
	"workspace": "test_workspace_1.fmw",
	"engineHost": "",
	"timeQueued": "2021-02-18T14:00:00+01:00",
	"description": "",
	"repository": "admin_test",
	"userName": "admin",
	"sourceType": "SCHEDULES",
	"id": 80959,
	"sourceName": "admin_test/create_test_log_2",
	"engineName": "",
	"timeSubmitted": "2021-02-18T14:00:00+01:00",
	"status": "QUEUED"
}

If you now define an expectation as shown below, timeSubmitted is compared with the difference of the current time and the duration (here: one hour). If current Time - duration > timeSubmitted — i.e. the job has been in the waiting hour for longer than 1 hour — you will be informed about this job.

Expectation: Job in the queue for an hour

{"type" : "FmeJobTimeShiftExpectation", "content":[{"duration":"PT1H","attribute":"timeSubmitted"}]}

Use timeSubmitted, timeStarted or timeQueued in conjunction with the standard Jobs queued and Jobs running requests to perform the desired functional checks.

Job Execution Success Check

To monitor FME Flow Schedules continuously use the FmeScheduleJobExecutionExpectation. You can check success of mission critical FME Schedules.

Identify the category and name of the schedule.
Create a new job with the example request Schedule check and add the category and name for the request.
Execute Test request and identify the periodicity of the schedule (e.g. cron string or day-based execution at a defined time).
Based on the periodicity of the schedule, design a monitoring template at whose rhythm service.monitor is to be executed and for whose time it is expected that a successful FME job execution has taken place.
Define a FmeScheduleJobExecutionExpectation that additionally defines a time duration in whose time frame a successful execution must have taken place.

The figure below shows the successful (expected) test of an FME job. The calculated reference time — the difference between job execution and the duration stored in the expectation — is before the start time of the last FME job execution.

{"type" : "FmeScheduleJobExecutionExpectation", "content":[{"duration":"PT10M"}]}

At the specific execution time of the job, the schedule is queried from FME Flow. service.monitor will then generate a request to FME Flow based on the schedule information, which will query the last successful job execution of this job (and its start time). Based on this information, it can be decided whether there is an expectation error or not.

So: If now - duration > timeStarted then a message is triggered.

Duration notation

The notation of duration follows the ISO 8601 duration format. For example, PT10M means a duration of 10 minutes, PT12H30M5S means 12 hours, 30 minutes and 5 seconds.