user task forms that is provided by user task management addon
user task notifications that is provided by user task email addon
These two combined provide an excellent support for human actors participating in workflow automation. This article is about to provide some hints behind these two features of Automatiko, so let's dive into them directly.
User task forms
Forms that represent user task (a task assigned to human actors) are a common requirement in the process/workflow automation scenarios. In many cases it is expected that forms will be auto generated by the workflow engine (and some offerings on the market actually do that). But the main problem with that approach is that is is very limited. In many cases it can only support basic forms and what is even more important they are not providing proper business context behind the task. They are usually very generic and expose internal parts of the workflow engine that runs them.
To give an example - let's assume we have a simple vacation request approval task assigned to a manager. The auto generated forms are usually going to present it with a checkbox for approval decision and a button to complete the task. The reason for that is the generation of the task form is based on data types of its outputs which will be approved of type boolean and boolean is usually represented as check box. While this will work it does not show the form in expected format - meaning it would show the details of the request and then have two buttons one to approve and another to reject. Taking it even further if the reject button is used the form could ask for additional comment why it was rejected. This can't be done with auto generated forms.
The approach in Automatiko is bit different, it can still show very simple generic form but that is considered as fallback option as sometimes there is just a matter of providing kind of "For your information" type of tasks where the only thing to be done is to acknowledge it. Though for anything that requires input from end user the form should be designed and provided as part of the service.
This is realised with templates for user tasks. Each template is a fully featured HTML page that can use any kind of framework or styles. You can build a really dynamic forms with the use of JavaScript frameworks e.g. JQuery, you can style it with Bootstrap and so on. You own the entire space on how to build your forms, how to layout the forms, if you need to load data from other service to populate fields you can do that without a hassle.
Templates in Automatiko relies on Qute, a server side templating engine from Quarkus. It gives you all the power of the templating and is very well tuned for performance and fast delivery to your clients. Automatiko will give you all the details you need for given task so you can render it and make it very contextual to your users so they will directly know what is expected from them. Plus you can make it to look and feel as any other application in your organisation.
Another important aspect of the user tasks in workflow automation is to notify when a task is assigned. This is very common and almost any user of workflow automation expects this to be out of the box. Automatiko comes with this by a means of addon that will equip your service with this feature.
Emails are expected to serve as a way of notifying about task being assigned and not necessarily about the complete context behind the task. That's why a default template for emails in many cases will be good enough. But it is configurable as well so you can define an email notification template for every task separately. It uses the same approach as the user task forms - templates.
In addition, tasks can be assigned to individuals and groups so somehow there must be a way to know the email addresses for them. By default Automatiko assumes users are represented as email addresses but that is not always the case. Don't worry, there is a simple way to solve it, by implementing a single interface you can provide your way of resolving user and group identifiers to email addresses. To learn more head to the Automatiko documentation.
At the end I'd like to give you an opportunity to see it in action, have a look at this short video showing both features in action.
If you have any questions or comments feel free to reach out on twitter or mailing list (@automatiko_io) automatiko-dev@googlegroups.com).
A common scenario when working with workflows is to handle data objects and their changes. In most of the situations workflow instance will only keep the last value of it and to realise use cases like comparing what was just sent to the instance with what was already in there requires having duplicated data object definitions. This is not the best approach as it makes the workflow definition "corrupted" with details less important from the business goal perspective.
With Automatiko (since version 0.3.0) there is an alternative way to this problem. This is to version data objects by annotating it with data object tag called versioned.
So what happens when you make data object versioned?
Automatiko engine will record every change to the variable as new version. These versions are then available to be accessed as any other variable but will require additional suffix to the variable name
suffix $ will give access to complete version list of the variable e.g. person$
suffix $X where X is a number of the version to retrieve it can be a negative (-1) to fetch latest version e.g. person$5 or person$-1
Sometimes referring to versions directly might result in errors like attempting to get the version that does not exist. To make it simpler, Automatiko provides ready to use functions that can be used from
script tasks
gateway conditions
The functions you can use are as follows:
previousVersion(versions) allows to get latest version of the variable list - previousVersion(person$)
variableVersion(versions, number) allows to get variable version stored under version number - variableVersion(person$, 4) - note that this one is safe and will return null when given version number does not exist
isEqual(var1, var2) allows to easily compare two versions of the variable - isEqual(person, previousVersion(person$))
Another aspect is that you can easily create your custom functions by simply implementing io.automatiko.engine.api.Functions interface and implementing public static methods that will become functions and will be available in the workflow definition. You can read up more in Automatiko documentation.
In addition to that Process Management UI also provides quick access to variable versions and allows to also revert to given version of the data object. Once you have process management addon in your service it provides both UI and REST api to interact with versions of the data objects.
Following video shows this in action and the value it brings.
Stay tuned for more updates around Automatiko project. If you have any questions or comments join our community either on mailing list or GitHub discussions.
Here is more of a developer view on the recently published article on Automatiko blog about building Kubernetes Operators with workflows. It shows that is brings significant value to the overall visibility of the operator logic and makes it really approachable for non kubernetes gurus.
I'd like to take it a bit further and show how efficient it can be thanks to the internals of Automatiko. Automatiko takes advantage of Quarkus that provide the runtime mechanics so to say. Quarkus comes with an outstanding feature called dev mode. Everyone who heard about Quarkus most likely heard about dev mode and live reload. But there is more to it!!!
Remote dev mode - is a sibling to the dev mode but it allows you to live reload application remotely. A perfect fit for in-container development or even better in kubernetes cluster development. This brings us to the unbelievable efficient developer experience when building Kubernetes operators - you can build them directly inside the Kubernetes cluster. No need for rebuilding the image, no need to redeploying the container and so on... it just works like a charm.
Have a look at this video illustrating how you can easily work on your operator logic that runs inside the kubernetes cluster. Make modifications to the logic and try it out almost instantly (assuming your kubernetes cluster and deployed container have enough resources to make it efficient ;)).
The video showed number of features
how to configure the application to run in remote dev mode
how to make changes to the workflow definition
how to make modification to application code
automatic reload of the application that includes both workflow and java classes
All working smoothly and efficiently without much of a hassle
A huge kudos to Quarkus team for making it a fantastic piece of software that makes developer life easier... a truly cloud native as it enables direct in cluster development that speeds the work up significantly!!!
As a follow up of part 1 of the Automatiko IoT and MQTT I'd like to take you further in exploration around workflows and IoT with MQTT.
This time we look at the details of how to take advantage of some MQTT features (e.g. wildcard topics), collect sensor data into a bucket (or to make this simple - a list) and then assign user tasks based on amount of data collected instead of for every event.
In addition to that, we look into Automatiko features that makes using workflows for IoT way easier...
Wildcard topics
Let's start with MQTT feature for subscribers - this is actually what workflow in our sample is - a MQTT topic subscriber. So let's first look like how does it work under the hood in Automatiko.
Automatiko uses message events (start or intermediate) of BPMN to integrate with message broker - in this case MQTT. Message events are referencing message which describes the way how it can connect to the broker
by default uses name as the topic name
data type of the message defines what type will be used to unmarshall incoming event into
supports custom attributes to alter defaults
In this article we are going to use custom attributes to define both topic that will use wildcard and the correlation expression to extract information out of the topic instead of the message (as we did in part 1).
In the screenshot above you can see two custom attributes
topic
correlationExpression
Topic is used to define the actual topic in MQTT that the workflow definition will be connected to and listening to incoming events.
Correlation expression on the other side defines an expression that will be used for each incoming event to extract a key to be used for both identification and correlation.
Correlation expression uses special function "topic" that accepts following parameters
message - references the incoming message
index - an index that will reference different parts of the topic - it starts with 0
So for this example topic(message, 1) and the topic home/kitchen/temp the extracted correlation key will be kitchen. This in turn will be used as identifier of the workflow instance and thus you can use it in the service api calls.
You can read up more on the messaging support in automatiko in documentation.
Data bucket to collect sensor data
Next topic for today is the collection of sensor data. In part one we simply assign it to a data object of the same type. This time we expand and make sure we can accumulate the data.
So with that said there are few things that must be done
data object must be of type list
data object of type list needs to be initialized so it can easily get new items
message events received from MQTT need to be added to the list instead of assigned to data object (which would mean overridden)
Changing type is rather simple but ensuring it is initialized might be a bit more complex... but not with automatiko :) It is as simple as adding a tag on data object - auto-initialized
Configuring the message event to append to a list instead of assigning to the data object is also rather simple with automatiko, it is to use expression instead of direct mapping on the message event.
And that's it - we have now ready to use data bucket for our sensor data.
Decide when to include human actors
In this simple example we are going to receive events from MQTT and collect them into a bucket. But we don't want to involve human actors on each event. So here we can use gateways - a construct that allows us to have different paths in the workflow.
We only create a user task when the bucket has more than 5 events collected. Otherwise it simply end the path. But to prevent the workflow instance from finishing we make the workflow to be ad hoc - that allows it to stay active even though there are no active nodes in it. Marking workflow as ad hoc is done in the Properties -> Process panel.
See it in action
Have a look at this 10 min video showing all this in action. Note that this is live coding so you will see the errors as they might show up and you can imagine that this is not a scripted video where everything works from the start :)
Code can be found in github where each part of the series is a separate tag and main branch is pointing to latest version of the code base.
Conclusions
This part focused on bringing more advanced features of MQTT into the workflows to show the integration and how powerful these two can be. Using topic information as correlation, locating right workflow instances for incoming events and accumulating data is one of the most common use cases for IoT so workflows should make it simple to realize that.
Around a month ago there was the first release of Automatiko project. It aims at providing an easy to use yet powerful toolkit to build services and functions based on workflows and decision. You can read up more information about the Automatiko project at the website or blog.
This blog post is very basic introduction that attempts to address the first level of entry when starting with Automatiko.
Before you start...
So first things first... to get you up and running you need few things on your local computer
Java 11 +
Maven 3.6 +
Eclipse IDE and Automatiko plugin
Optionally docker or podman to run services as containers
The use case of this introductory sample is very simple - to connect to MQTT to receive data from the sensors that are being published there. It mainly focuses on the steps to get this working and further articles will provide more advanced features in action.
Let's get started
To get started you need to create a project, a maven project to be precise. Luckily Automatiko comes with bunch of maven archetypes that makes this task way faster. One of the is `automatiko-iot-archetype` and this is the one we are going to use today.
Archetype can be used from IDE or command line so whatever you prefer can be used and result will be exactly the same.
Once the project is there, you need to define your data model that will be used by the workflow - as data objects. In this example you can create simple POJO like class names `Temperature` that will have two fields
value - of type double that will contain the temperature value from the sensor
room - the location where the temperature was measured
Next step is to create a workflow definition that will be responsible for connection to MQTT broker. This is realised by message start event that allows not only to be the entry point for new instances but also allows to define some characteristics of the connectivity and processing.
All that is needed to talk to the MQTT broker is defined directly in the workflow definition. In our sample case, the topic in MQTT broker is simple taken from the message defined on the start event.
At the same time, data that is received from the MQTT topic is automatically converted to the data model - represented as 'Temperature` class and mapped to a `temp` data object inside the workflow instance.
Lastly, user task is also added to the workflow definition that introduces a human actor into the flow. That is mainly for demonstration purpose to show we receive data from MQTT that is properly converted into a Temperature class instance and set within the workflow instance data objects (aka variables).
A bit of advance...
As you will see when running this sample, ID of the workflow instance is auto generated (in UUID format). But that can be changed and take advantage of either data or the MQTT topic itself. By setting correlation or correlation expression on the message (via its custom attributes) you can set the id to be more domain specific.
In this case, we take the room from the message and use it as correlation. Correlation upon start of the workflow instance becomes its id - aka business key and by that it can be used in exchange with the generated ID. With that said you can use this when interacting with service API.
You can look at the introduction video that covers the content of this article.
Let's get this running...
To get this running we need a MQTT broker. Personally I find Mosquitto to be excellent choice but any MQTT compliant broker would work.
You can get Mosqiutto running with just a simple command (docker required)
docker run -it -p 1883:1883 -p 9001:9001 eclipse-mosquitto
There are other ways to run mosquitto so visit website if docker is not an option.
Run the Automatiko service
Since Automatiko generates fully functional service there is no much to do to see it in action. It is based on maven and leverages Quarkus as runtime so most Quarkus features are also available out of the box e.g. dev mode, Dev UI etc.
mvn clean quarkus:dev
and then wait a bit for maven to download all the bits....
Once it's started you should see similar log entries
You can head directly to http://localhost:8080/swagger-ui and you will be presented with nicely document API of your service
So it is now fully functional service connected to your local MQTT broker. With this you can start publishing sensor data to it and see how quickly they are consumed by Automatiko service.
You can use mosquitto client command to publish messages from command line
mosquitto_pub -t temperature -m '{"room":"livingroom", "value" : 25.0}'
use Swagger UI to see created instances based on incoming messages from MQTT.
Running as container
In case you would like to get this running as container then it is again dead simple, just run maven command with `container` profile enabled
mvn clean package -Pcontainer
And that's it. As soon as build is over you will have the container image in your local registry.
Conclusion
That's it for the first introduction - I hope it will get you interested and you will look forward for the next articles. They will come ever other week... at least that's the plan. Please share your feedback either here or via mailing list and twitter.
If you have any ideas for the use cases to cover with Automatiko don't hesitate to let us know about them as we are here to help and explore.
In this post, we’ll introduce a new jBPM API which allows for predictive models to be trained with Human Tasks (HT) data and for HT to incorporate model predictions as outputs and complete HT without user interaction.
This API will allow you to add Machine Learning capabilities to your jBPM project by being able to use, for instance, models trained with historical task data to recommend the most likely output. The API also gives developers the flexibility to implement a “prediction-only” service (which only suggests outputs) as well as automatically completing the task if the prediction’s confidence meets a user-defined prediction confidence threshold.
This API exposes the HT handling to a prediction service.
A prediction service is simply any third-party class which implements the org.kie.internal.task.api.prediction.PredictionService interface.
This interface consists of three methods:
getIdentifier() - a method which returns a unique (String) identifier for your prediction service
predict(Task task, Map<String, Object> inputData) - a method that takes task information and the task's inputs from which we will derive the model's inputs, as a map. The method returns a PredictionOutcome instance, which we will look in closer detail later on
train(Task task, Map<String, Object> inputData, Map<String, Object> outputData) - this method, similarly to predict, takes task info and the task's inputs, but now we also need to provide the task's outputs, as a map, for training
This class will consist of:
A Map<String, Object> outcome containing the prediction outputs, each entry represents an output attribute’s name and value. This map can be empty, which corresponds to the model not providing any prediction.
A confidence value. The meaning of this field is left to the developer (e.g. it could represent a probability between 0.0 and 1.0). It's relevance is related to the confidenceThreshold below.
A confidenceThreshold - this value represents the confidence cutoff after which an action can be taken by the HT item handler.
As an example, let's assume our confidence represents a prediction probability between 0.0 and 1.0. If the confidenceThreshold is 0.7, that would mean that for confidence > 0.7 the HT outputs would be set to the outcome and the task automatically closed. If the confidence <= 0.7, then the HT would set the prediction outcome as suggested values, but the task would not be closed and still need human interaction. If the outcome is empty, then the HT life cycle would proceed as if no prediction was made.
By defining a confidence threshold which is always higher than the confidence, developers can create a “prediction-only” service, which will assign predicted outputs to the task, but never complete it.
The initial step is then, as defined above, the predict step.
In the scenario where the prediction's confidence is above the threshold, the task is automatically completed. If the confidence is not above the threshold, however, when the task is eventually completed both the inputs and the outputs will then be used to further train the model by calling the prediction service's train method.
Example project
An example project is available here. This project consists of a single Human Task, which can be inspected using Business Central. The task is generic and simple enough in order to demonstrate the working of the jBPM's prediction API.
For the purposes of the demonstration, this task will be used to model a simple purchasing system where the purchase of a laptop of a certain brand is requested and must be, eventually, manually approved. The tasks inputs are:
item - a String with the brand's name
price - a Float representing the laptop's price
ActorId - a String representing the user requesting the purchase
The task provides as outputs:
approved - a Boolean specifying whether the purchase was approved or not
This repository contains two example prediction service implementations as Maven modules and a REST client to populate the project with tasks to allow the predictive model training.
Start by downloading, or alternatively cloning, the repository:
For this demo, two random forest-based services, one using the SMILE library and another as a Predictive Model Markup Language (PMML) model, will be used. The services, located respectively in services/jbpm-recommendation-smile-random-forest and services/jbpm-recommendation-pmml-random-forest, can be built with (using SMILE as an example):
$ cd services/jbpm-recommendation-smile-random-forest
$ mvn clean install
The resulting JARs files can then be included in the Business Central’s kie-server.war located in standalone/deployments directory of your jBPM server installation. To do this, simply create a WEB-INF/lib, copy the compiled JARs into it and run
$ zip -r kie-server.war WEB-INF
The PMML-based service expects to find the PMML model in META-INF, so after copying the PMML file in jbpm-recommendation-pmml-random-forest/src/main/resources/models/random_forest.pmml into META-INF, it should also be included in the WAR by using
$ zip -r kie-server.war META-INF
jBPM will search for a prediction service with an identifier specified by a Java property named org.jbpm.task.prediction.service. Since in our demo, the random forest service has the identifier SMILERandomForest, we can set this value when starting Business Central, for instance as:
For the purpose of this documentation we will illustrate the steps using the SMILE-based service. The PMML-based service can be used by starting Business Central and setting the property as
Once Business Central has completed the startup, you can go to http://localhost:8080/business-central/ and login using the default admin credential wbadmin/wbadmin. After choosing the default workspace (or creating your own), then select "Import project" and use the project git URL:
The repository also contains a REST client (under client) which allows to add Human Tasks in batch in order to have sufficient data points to train the model, so that we can have meaningful predictions. NOTE: Before running the REST client, make sure that Business Central is running and the demo project is deployed and also running.
The class org.jbpm.recommendation.demo.RESTClient performs this task and can be executed from the client directory with:
The prices for Lenovo and Apple laptops are drawn from Normal distributions with respective means of 1500 and 2500 (pictured below). Although the prediction service is not aware of the deterministic rules we've used to set the task outcome, it will train the model based on the data it receives. The tasks' completion will adhere to the following logic:
The purchase of a laptop of brand Lenovo requested by user John or Mary will be approved if the price is around $1500
The purchase of a laptop of brand Apple requested by user John or Mary will be approved if the price is around $2500
The purchase of a laptop of brand Lenovo requested by user John or Mary will be rejected if the price is around $2500
The client will then simulate the creation and completion of human tasks, during which the model will be trained.
SMILE-based service
As we've seen, when creating and completing a batch of tasks (as previously) we are simultaneously training the predictive model. The service implementation is based on a random forest model a popular ensemble learning method.
When running the RESTClient, 1200 tasks will be created and completed to allow for a reasonably sized training dataset. The prediction service initially has a confidence threshold of 1.0 and after a sufficiently large number (arbitrarily chosen as 1200) of observations are used for training, the confidence threshold drops to 0.75. This is simply to demonstrate the two possible actions, i.e. prediction without completing and completing the task. This also allows us to avoid any cold start problems.
After the model is trained with the task from RESTClient, we will now create a new Human Task.
If we create a HT requesting the purchase of an "Apple" laptop from "John" with the price $2500, we should expect it to be approved.
If fact, when claiming the task, we can see that the prediction service recommends the purchase to be approved with a "confidence" of 91%.
If he now create a task for the request of a "Lenovo" laptop from "Mary" with the price $1437, he would expect it to be approved. We can see that this is the case, where the form is filled in by the prediction service with an approved status with a "confidence" of 86.5%.
We can also see, as expected, what happens when "John" tries to order a "Lenovo" for $2700. The prediction service fills the form as "not approved" with a "confidence" of 71%.
In this service, the confidence threshold is set as 1.0 and as such the task was not closed automatically.
The minimum number of data points was purposely chosen so that after running the REST client and completing a single task, the service will drop the confidence threshold to 0.75.
If we complete one of the above tasks manually, the next task you create will be automatically completed if the confidence is above 0.75. For instance, when creating a task we are pretty sure will be approved (e.g. John purchasing a Lenovo $1500) you can verify that the task is automatically completed.
PMML-based service
The second example implementation is the PMML-based prediction service. PMML is a predictive model interchange standard, which allows for a wide variety of models to be reused in different platforms and programming languages.
The service included in this demo consists of pre-trained model (with a dataset similar to the one generated by RESTClient) which is executed by a PMML engine. For this demo, the engine used was jpmml-evaluator, the de facto reference implementation of the PMML specification.
There are two main differences when comparing this service to the SMILE-based one:
The model doesn't need the training phase. The model has been already trained and serialised into the PMML format. This means that we can start using predictions straight away from jBPM.
The train API method is a no-op in this case. This means that whenever the service's train method is called, it will not be used for training in this example (only the predict method is needed for a "read-only" model), as we can see from the figure below.
You can verify that the Business Central workflow is the same as with the SMILE service, although in this case no training is necessary.
The above instructions on how to setup the demo project are also available in the following video (details are in the subtitles):
In conclusion, in this post we’ve shown how to use a new API which allows for predictive models to suggest outputs and complete Human Tasks.
We’ve also shown a project which can use different prediction service backends simply by registering them with jBPM without any changes to the project.
Why not create your own jBPM prediction
service using your favourite Machine Learning framework, today?
In this post, we will introduce the new Prometheus Kie Server Extension, which has been released as part of jBPM version 7.21.0.Final. This extension aims to make extremely easy for you to publish metrics about your Kie Server runtime execution. Using Prometheus and Grafana has become a standard for monitoring cloud services these days, and allowing the Kie Server to expose metrics related to processes, tasks, jobs and more, becomes a powerful integration not only for you to get a snapshot of the current status inside the server but also for combining it with information from different sources such as JVM, Linux and more. Not only in terms of infrastructure monitoring, but it is also a great tool to get insights into the execution of your business process.
To get started with Prometheus, take a look in this overview and the full list of exporters and integrations. Grafana is also another powerful tool that allows you to create nice looking dashboards, combining data from multiple sources, to get started take a look here.
Here is an example based on the metrics exposed from the Kie Server:
To enable this new extension, set the Prometheus system property to org.kie.prometheus.server.ext.disabled=false. When you enable this extension, a series of metrics will be collected, including information about deployments, start time, data sets, execution errors, jobs, tasks, processes, cases, and more. For the complete list of metrics, see the Prometheus services repository on GitHub.
After the extension is started, you can access the available metrics at ${context}/services/rest/metrics.
To quickly demonstrate all the capabilities of this integration, we created a short video, with more details about how to get started, two example dashboards for Grafana, as well as a Docker compose configuration that you can use as a playground to explore all these tools working together.
Docker compose example configuration is available here and to get started, simply run:
docker-compose -f jbpm-kie-server-prometheus-grafana.yml up
After all images start, you have the following tools available:
To access the example dashboards, please login to Grafana using the default credentials: username admin and password admin. Then navigate to Dashboards ->Manage, in there you should have two examples: jBPM Dashboard and jBPM Kie Server Jobs.
As you interact with your Kie Server and Business Central instances, like deploying and starting new process instances, you should notice the metrics values changing in the dashboard. Prometheus is configured to scrape data every 10 seconds.
