Workflow Transactions - Foreground and Background Jobs

The workflow engine executes process instances in a transactional way. A transaction is a logical unit of work that must succeed or fail as a whole.

The default behaviour is that a transaction completes when the engine reaches a "wait" state on every active path of execution in an instance. This is also the point at which the call that started the process will return.

This means that if a process never reaches such a state after starting, the initial transaction will not be committed and there will be no record of the process being started. It also means that if task completion fails, the process will be rolled back to the last valid state.

When you've got a basic process design, think about when it's going to commit and whether or not the default behaviour is going to make sense.

Foreground and Background Jobs

Processing within the workflow engine can be divided into foreground and background jobs.

Two Foreground Jobs

A foreground job results directly from a call into the engine, and returns when processing is complete. For example, consider the following simple workflow:

When the workflow is started through a call to startProcess (which will usually have been made by a form submission), a script block runs and an email is sent, and then the call to startProcess returns. This means that if the form reports that the workflow was started successfully, we know that the email has been sent.

Similarly, when the "User Confirmation" task is completed via a call to completeTask (again made by a form action), another script block runs, an API call is made, and then the call returns. If the form reports that the task was completed successfully, we know that the API call has been made with no exception being thrown (for the sake of this example, we'll assume that the API call has been configured to throw a BPMN error on failure and that no handler has been defined for this).

A Background Job

Now consider the following job, where the user task in the previous example has been replaced with a timer set to a five minute delay.

When the workflow is started through a call to startProcess, the script block runs, the email is sent, and then the call to startProcess returns. The instance then waits at the timer until it fires. This means that, as before, if the form reports that the workflow was started successfully, we know that the email has been sent.

When the five minutes set in the timer has elapsed, the following script block and API call are executed within a background job - a job started from within the workflow engine in response to an internal event without any direct interaction from the user. Notifications of errors in background jobs are emailed to the site administrator.

A background job will be automatically retried if it takes more than five minutes to run.

Asynchronous Continuations

You can achieve custom control over jobs (and therefore transactions) by marking tasks as asynchronous.

When an execution reaches a task that has been marked as asynchronous, the engine will commit the current transaction, and schedule a background job to continue the process from the asynchronous task.

This behaviour is useful for processes that are fully automatic but use slow APIs. The execution can return straight away, and process time consuming jobs in the background. This means that a user can receive immediate confirmation of their form submission, rather than waiting a long time for the background processing to complete.

Consider the following workflow. It will send an email, then perform an API call, then return to the user.

If the API call takes a long time to perform, we might not want the user to have to wait for it to complete before the form submission finishes. We can achieve this by marking the "Perform API call" task as asynchronous:

This splits the original foreground job into a foreground and a background job (each of which will be carried out in a separate database transaction). The form submission will complete as soon as the "Prepare API call" script block completes, without waiting for the call itself.

Retrying Jobs

Since iCM version 10.0.5.0 the automatic retrying of jobs that error has been disabled in the workflow engine. This prevents activities that may involve calls to external applications being repeated multiple times.

If you are concerned about activities failing and want to have a retry mechanism, see the External Calls - Callbacks and Polling article for an example of how you can build this into your process design.