Unified KIE Execution Server - Part 3

Part 3 of Unified KIE Execution Server deals with so called managed vs. unmanaged setup of the environment. In version 6.2 users went through Rules deployments perspective to create and manage KIE Execution Server instances.
That approach required to have execution server configured and up and running. Some sort of online only registration that did not work if the kie server instance was down.

In version 6.3, this has been enhanced to allow complete configuration of KIE Execution Servers inside workbench even if there is no actual instances configured. So let's first talk about managed and unmanaged instances....

Managed KIE Execution Server 

Managed instance is one that requires controller to be available to properly startup. Controller is a component responsible for keeping a configuration in centralized way. Though that does not mean there must be only single controller in the environment. Managed KIE Execution Servers are capable of dealing with multiple controllers.

NOTE: It's important to mention that even though there can be multiple controllers they should be kept in sync to make sure that regardless which one of them is contacted by KIE Server instance it will provide same set of configuration.

Controller is only needed when KIE Execution Server starts as this is the time when it needs to download the configuration before it can be properly started. In case KIE Execution Server is started it will keep trying to connect to controller until the connection is successfully established. That means that no containers will be deployed to it even when there is local storage available with configuration. The reason why it is like that is to ensure consistency. If KIE Execution Server was down and the configuration has changed, to make sure it will run with up to date configuration it must connect to controller to fetch that configuration.

Configuration has been mentioned several times but what is that? Configuration is set of information:

• containers to be deployed and started
• configuration items - currently this is a place holder for further enhancements that will allow remotely configure KIE Execution Server components - timers, persistence, etc

Controller is a component that is responsible for overall management of KIE Execution Servers. It provides a REST api that is divided into two parts:

• controller itself that is exposed to interact with KIE Execution Server instances
• administration that allows to remotely manage KIE Execution Server
• add/remove servers
• add/remove containers to/from the servers
• start/stop containers on servers

Controller deals only with KIE Execution Server configuration or definition to put it differently. It does not handle any runtime components of KIE Execution Server instances. They are always considered remote to controller. Controller is responsible for persisting the configuration to preserve restarts of the controller itself. It should manage the synchronization as well in case multiple controllers are configured to keep all definitions up to date on all instances of the controller.

By default controller is shipped with KIE workbench (jbpm console) and allows fully featured management interface (both REST api and UI). It uses underlying git repository as persistent store and thus when GIT repositories are clustered (using Apache Zookeeper and Apache Helix) it will cover the controllers synchronization as well.

Above diagram illustrates single controller (workbench) setup with multiple KIE Execution Server instances managed by it. Following diagram illustrates the clustered setup where there are multiple instances of controller sync over Zookeeper.

In the above diagram we can see that KIE Execution Server instances are capable to connect to all controllers, but they will connect to only one. Each instance will attempt to connect to controller as long as it can reach one. Once connection is established with one of the controller it will skip other controllers.

Working with managed servers

There are two approaches that users can take when working with managed KIE Server instances:

Configuration first

With this approach, user will start working with controller (either UI or REST api) and create and configure KIE Execution Server definitions. That is composed of:

• identification of the server (id and name + optionally version for improved readability)
• containers 

Register first

Let the KIE Execution Server instance to auto register on controller and then configure it in terms of what containers to run on it. This is simply skipping the registration step done in first approach and populates it with server id, name and version directly upon auto registration (or to put it simple on connect)

In general there is no big difference and which approach is taken is pretty much a personal preference. The outcome of both will be the same.

Unmanaged KIE Execution Server

Unmanaged KIE Execution server is in turn just a standalone instance and thus must be configured individually using REST/JMS api of the KIE Execution server itself. The configuration is persisted into a file that is considered internal server state. It's updated upon following operations:

• deploy container
• undeploy container
• start container
• stop container

Note that KIE Execution server will start only the containers that are marked as started. Even if the KIE Execution Server will be restarted, upon boot it will only make containers available that were in started state before server was shutdown.

In most of the case KIE Execution Server should be ran in managed mode as that provides lots of benefits in terms of control and configuration. More benefits will be noticed when discussing clustering and scalability of KIE Execution Servers where managed mode will show its true power :)

Let's run in managed mode

So that's about it in theory, let's try to run the KIE Execution Server in managed mode to see how this can be operated.

For that we need to have one Wildfly instance that will host the controller - KIE Workbench and another one that will hold KIE Execution Server. Second we already have based on part 1 of the blog series.

NOTE: You can run both KIE workbench and KIE Execution Server on the same application server instance but it won't show the improved manageability as they will always be up or down together. 

So let's start with installing workbench on Wildfly. Similar to what we had to do for KIE Execution server we start with creating user(s):

• kieserver (with password kieserver1!) that will be used to communicate between KIE Server and controller, that user must be member of following roles:
• kie-server
• rest-all
• either add following roles to kieserver user or create another user that will be used to logon to KIE workbench to manage KIE Execution Servers
• admin
• rest-all

To do so use the Wildfly utility script - add-user located in WILDFLY_HOME/bin and add application users. (for details how to do that part 1 of this blog series)

Once we have the users created, let's deploy the application to it. Download KIE workbench for wildfly 8 and copy the way file into WILDFLY_HOME/standalone/deployments.

NOTE: similar to KIE Server, personally I remove the version number and classifier from the war file name and make it as simple as 'kie-wb.war' that makes the context path short and thus easier to type.

And now we are ready to launch KIE workbench, to do so go to WILDFLY_HOME/bin and start it with following command:

./standlone.sh --server-config=standalone-full.xml

wait for the server to finish booting and then go to: 

http://localhost:8080/kie-wb

logon with user you created (e.g. kieserver) and go to Deployments --> Rules Deployments perspective. See following screencast (no audio) that showcase the capabilities described in this article. It starts with configure first approach and does show following:

• create KIE Execution Server definition in the controller
• specified identifier (first-kie-server) and name
• create new container in the KIE Execution Server definition (org.jbpm:HR:1.0)
• configure KIE Execution Server to be managed by specifying URL to the controller via system property:
• -Dorg.kie.server.controller=http://localhost:8080/kie-wb/rest/controller
• -Dorg.kie.server.id=first-kie-server (this is extremely important that this id matches one created in first step in the KIE Workbench)
• start kie server and observe controller's log to see notification that kie server has connected to it
• start container in controller and observe it being automatically started on KIE Execution Server instance
• shutdown KIE Execution Server and observe logs and UI with updated status of kie server being disconnected
• illustrates various manage options in controller and it effect on KIE Execution Server instance.

So this screen cast concludes third part of the Unified KIE Execution Server series. With this in mind we move on into more advanced cases where we show integration with non java clients and clustering. More will come soon...

Unified KIE Execution Server - Part 2

This blog post is continuation of the first of the series about KIE Execution Server. In this article KIE Server Client will be introduced and used for basic operations on KIE Execution Server.

In the first part, we have went through the details of installation on Wildfly and verification with simple REST client to show it's actually working. This time we do pretty much the same verification although we expand it with further operations and make it via KIE Server Client instead.

So let's get started. We are going to use same container project (hr - org.jbpm:HR:1.0) that includes hiring process, that process has set of user tasks that we will be creating and working with. To be able to work on tasks our user (kieserver) needs to be member of the following roles used by the hiring process:

• HR
• IT
• Accounting

So to add these roles to our user we again use add-user script that comes with wildfly to simply update already existing user

NOTE: don't forget that kieserver user must have kie-server role assigned as well.

With that we are ready to start the server again

KIE Server Client

KIE Server Client is a lightweight library that custom application can use to interact with KIE Execution Server when is written in Java. That library extremely simplifies usage of the KIE Execution Server and make it easier to migrate between versions because it hides all internals that might change between versions. 

To illustrate that it is actually lightweight here is the list of dependencies needed on runtime to execute KIE Server Client

[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ kie-server-client ---
[INFO] org.kie.server:kie-server-client:bundle:6.3.0-SNAPSHOT
[INFO] +- org.kie:kie-api:jar:6.3.0-SNAPSHOT:compile
[INFO] +- org.kie:kie-internal:jar:6.3.0-SNAPSHOT:compile
[INFO] +- org.kie.server:kie-server-api:jar:6.3.0-SNAPSHOT:compile
[INFO] |  +- org.drools:drools-core:jar:6.3.0-SNAPSHOT:compile
[INFO] |  |  +- org.mvel:mvel2:jar:2.2.6.Final:compile
[INFO] |  |  \- commons-codec:commons-codec:jar:1.4:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.9:compile
[INFO] |  +- com.thoughtworks.xstream:xstream:jar:1.4.7:compile
[INFO] |  |  +- xmlpull:xmlpull:jar:1.1.3.1:compile
[INFO] |  |  \- xpp3:xpp3_min:jar:1.1.4c:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.1:compile
[INFO] +- org.jboss.resteasy:jaxrs-api:jar:2.3.10.Final:compile
[INFO] |  \- org.jboss.logging:jboss-logging:jar:3.1.4.GA:compile
[INFO] +- org.kie.remote:kie-remote-common:jar:6.3.0-SNAPSHOT:compile
[INFO] +- org.codehaus.jackson:jackson-xc:jar:1.9.9:compile
[INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.9:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.2:compile
[INFO] +- org.jboss.spec.javax.jms:jboss-jms-api_1.1_spec:jar:1.0.1.Final:compile
[INFO] +- com.sun.xml.bind:jaxb-core:jar:2.2.11:compile
[INFO] \- com.sun.xml.bind:jaxb-impl:jar:2.2.11:compile

So let's setup a simple maven project that will use KIE Server Client to interact with the execution server

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelversion>4.0.0</modelversion>
  <groupid>org.jbpm.test</groupid>
  <artifactid>kie-server-test</artifactid>
  <version>0.0.1-SNAPSHOT</version>
  
  <dependencies>
    <dependency>
        <groupid>org.kie</groupid>
        <artifactid>kie-internal</artifactid>
        <version>6.3.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupid>org.kie.server</groupid>
        <artifactid>kie-server-client</artifactid>
        <version>6.3.0-SNAPSHOT</version>
    </dependency>
    <dependency>
      <groupid>ch.qos.logback</groupid>
      <artifactid>logback-classic</artifactid>
      <version>1.1.2</version>
    </dependency>
  </dependencies>

That's all dependencies that are needed to have KIE Server Client embedded in custom application. Equipped with this we can start running KIE Server Client towards given server instance

Following is code snippet required to construct KIE Server Client instance using REST as transport

String serverUrl = "http://localhost:8230/kie-server/services/rest/server";
String user = "kieserver";
String password = "kieserver1!";

String containerId = "hr";
String processId = "hiring";

KieServicesConfiguration configuration = KieServicesFactory.newRestConfiguration(serverUrl, user, password);
// other formats supported MarshallingFormat.JSON or MarshallingFormat.XSTREAM
configuration.setMarshallingFormat(MarshallingFormat.JAXB);
// in case of custom classes shall be used they need to be added and client needs to be created with class loader that has these classes available 
//configuration.addJaxbClasses(extraClasses);
//KieServicesClient kieServicesClient =  KieServicesFactory.newKieServicesClient(configuration, kieContainer.getClassLoader());
KieServicesClient kieServicesClient =  KieServicesFactory.newKieServicesClient(configuration);

Once we have the the client instance we can start executing operations. We start with checking if the container we want to work with is already deployed and if not deploy it

boolean deployContainer = true;
KieContainerResourceList containers = kieServicesClient.listContainers().getResult();
// check if the container is not yet deployed, if not deploy it
if (containers != null) {
    for (KieContainerResource kieContainerResource : containers.getContainers()) {
        if (kieContainerResource.getContainerId().equals(containerId)) {
            System.out.println("\t######### Found container " + containerId + " skipping deployment...");
            deployContainer = false;
            break;
        }
    }
}
// deploy container if not there yet        
if (deployContainer) {
    System.out.println("\t######### Deploying container " + containerId);
    KieContainerResource resource = new KieContainerResource(containerId, new ReleaseId("org.jbpm", "HR", "1.0"));
    kieServicesClient.createContainer(containerId, resource);
}

Next let's check what is there available, in terms of processes and get some details about process id we are going to start

// query for all available process definitions
QueryServicesClient queryClient = kieServicesClient.getServicesClient(QueryServicesClient.class);
List<ProcessDefinition> processes = queryClient.findProcesses(0, 10);
System.out.println("\t######### Available processes" + processes);

ProcessServicesClient processClient = kieServicesClient.getServicesClient(ProcessServicesClient.class);
// get details of process definition
ProcessDefinition definition = processClient.getProcessDefinition(containerId, processId);
System.out.println("\t######### Definition details: " + definition);

We have all the details so we are ready to start the process instance for hiring process. We set two process variables:

• name - of type string 
• age - of type integer

// start process instance
Map<String, Object> params = new HashMap<String, Object>();
params.put("name", "john");
params.put("age", 25);
Long processInstanceId = processClient.startProcess(containerId, processId, params);
System.out.println("\t######### Process instance id: " + processInstanceId);

Once we started we can fetch tasks waiting to be completed for kieserver user

UserTaskServicesClient taskClient = kieServicesClient.getServicesClient(UserTaskServicesClient.class);
// find available tasks
List<TaskSummary> tasks = taskClient.findTasksAssignedAsPotentialOwner(user, 0, 10);
System.out.println("\t######### Tasks: " +tasks);

// complete task
Long taskId = tasks.get(0).getId();

taskClient.startTask(containerId, taskId, user);
taskClient.completeTask(containerId, taskId, user, null);

since the task has been completed and it has moved to another one we can continue until there are tasks available or we can simply abort the process instance to quit the work on this instance. Before we abort process instance let's examine what nodes has been completed so far

List<NodeInstance> completedNodes = queryClient.findCompletedNodeInstances(processInstanceId, 0, 10);
System.out.println("\t######### Completed nodes: " + completedNodes);

This will give us information if the task has already been completed and process moved on. Now let's abort the process instance

// at the end abort process instance
processClient.abortProcessInstance(containerId, processInstanceId);

ProcessInstance processInstance = queryClient.findProcessInstanceById(processInstanceId);
System.out.println("\t######### ProcessInstance: " + processInstance);

In the last step we get the process instance out to check if it was properly aborted - process instance state should be set to 3.

Last but not least, KIE Server Client can be used to insert facts and fire rules in very similar way

// work with rules
List<GenericCommand> commands = new ArrayList<GenericCommand>();
BatchExecutionCommandImpl executionCommand = new BatchExecutionCommandImpl(commands);
executionCommand.setLookup("defaultKieSession");

InsertObjectCommand insertObjectCommand = new InsertObjectCommand();
insertObjectCommand.setOutIdentifier("person");
insertObjectCommand.setObject("john");

FireAllRulesCommand fireAllRulesCommand = new FireAllRulesCommand();

commands.add(insertObjectCommand);
commands.add(fireAllRulesCommand);

RuleServicesClient ruleClient = kieServicesClient.getServicesClient(RuleServicesClient.class);
ruleClient.executeCommands(containerId, executionCommand);
System.out.println("\t######### Rules executed");

So that concludes simple usage scenario of KIE Server Client that covers

• containers
• processes
• tasks
• rules

A complete maven project with this sample execution can be found here.

Enjoy and stay tuned for more to come about awesome KIE Execution Server :)

Unified KIE Execution Server - Part 1

This blog post initiates the series of articles about KIE Execution Server and its capabilities provided in version 6.3. Here is a short description of what you can expect:

1. Introduction to KIE Execution Server and installation notes
2. Use of KIE Server Client to interact with KIE Execution Server
3. KIE Execution Server managed vs unmanaged
4. KIE Execution Server with non java clients
5. KIE Execution Server clustering/scalability

These are just starting points as more articles most likely will follow depending on interest ... so let's start with first and foremost - the introduction and installation

KIE Execution Server introduction

In version 6.2 KIE Execution Server has been released that was targeting Drools users to provide out of the box execution environment that is accessible via REST and JMS interface. It was designed to be standalone and lightweight component that can be deployed to either application servers or web containers (with obvious limitation - no JMS on web containers).

As it proved to be a valid option as a standalone component that can be easily deployed and scaled, in version 6.3 there will be so called unified KIE Execution Server that will bring in more capabilities to the end users:

• BRM capability that is what was in 6.2 providing rules execution
• BPM capability that brings jBPM into the picture
• process execution
• task execution
• asynchronous jobs execution

All of these are provided in unified way and exposed via REST and JMS interfaces. On top of it a KIE Server Client is delivered that makes use of this server very easy in java environment.

The unification means that from end user point of view you will not have to switch between different servers to take advantage of rule or process execution, same client can be used to perform both and so on. Unified terminology was used as well to not confuse users and thus here comes the most important parts:

• server - is the actual instance of the execution server
• container - is execution representation of the kjar/KieContainer that can be composed of various assets (rules, processes, data model, etc) - there can be multiple containers on single server
• process - business process definition available in given container - can be many per container
• task - user task definition available in given container - can be many per container
• job - asynchronous job that is/was scheduled in the execution server
• query - predefined set of queries to retrieve data out from the execution server

NOTE: Very important note to take into account is that all operations that modify data like:

• insert fact
• fire rules
• start process
• complete task

must always be referenced via container to guarantee all configuration to be properly set - class loader for custom data, handlers, listeners being registered in time etc.

While access to read only data like queries is simplified and expects the minimum set of data to be given to find details. E.g. get process instance - requires only process instance id as by that it will be able to find it and will return all the details required to perform operations on it - including container id (same goes for tasks etc).

Installation

Let's start with standalone mode running on Wildfly 8.1.0.Final (8.1.0 is used as it was tested with both kie server and kie workbench so better stick to just one version of the application server at the beginning :))

So we have to start with downloading Wildfly distribution and unzipping it to desired location - referred as WILDFLY_HOME. Here we start with configuration:

• create user in application realm 
• name: kieserver 
• password: kieserver1!
• roles: kie-server

NOTE: these are the defaults that can be changed but if you decide to change them you'll need to provide changed values via system properties upon server startup. So for the sake of simplicity let's start with defaults.

To add user you can use add-user.sh (or add-user.bat on windows) script that comes with Wildfly distribution. Just go to WILDFLY_HOME/bin and invoke add-user script:

• next download EE7 version of kie execution server 6.3.0 version from here
• downloaded version shall be copied to WILDFLY_HOME/standalone/deployments
• personally I usually change the name of the war file to not include version and classifier as it will be used as context path of the deployed application making all urls much longer
• so optionally you can rename the war file to short version like kie-server.war

We are almost ready to start, last thing is to prepare set of system properties that we will use to start our server with fully featured environment:

• first of all we must start wildfly server with full profile that activates JMS support
• --server-config=standalone-full.xml
• optionally, though useful when we have many wildfly instances running on same machine, let's specify port offset for wildfly server
• -Djboss.socket.binding.port-offset=150
• next we give the kie server instance and identifier - it's optional as if not given it will generate one, though it will be less human readable so let's give it a name
• -Dorg.kie.server.id=first-kie-server
• let specify the url location that our kie server will be accessible - this is important when running in managed mode (see part 3 of this series) but it's a good practice to give it always
• -Dorg.kie.server.location=http://localhost:8230/kie-server/services/rest/server

with that we are ready to launch our kie server in standalone mode, use this command from WILDFLY_HOME/bin:

./standalone.sh  

--server-config=standalone-full.xml 

-Djboss.socket.binding.port-offset=150 

-Dorg.kie.server.id=first-kie-server 

-Dorg.kie.server.location=http://localhost:8230/kie-server/services/rest/server

Once application server (and application) starts you should be able to issue simple GET request to the server using the org.kie.server.location url to get information about running server:

When opening this page you will be prompted for user name and password, use the one you created in the beginning of installation process - kieserver with password kieserver1!

So we have kie server up and running with following capabilities:

• KieServer - this is always present as it provides deployment operations to be able to deploy/undeploy containers on kie server instance
• BRM - rules execution
• BPM - process, tasks and jobs execution

Version of the kie server is also available (in this case is 6.4.0-SNAPSHOT as already running on latest master version - though at the time of writing this 6.3.0 is exactly the same)

Unified kie server is built on top of extensions aka capabilities and they can be turned on or off via system properties if one does not need some:

• -Dorg.drools.server.ext.disabled=true - to disable BRM extension
• -Dorg.jbpm.server.ext.disabled=true - to disable BPM extension

When disabling BPM extension you will see lot less things being bootstrapped upon  server start - no persistence is involved. So let's disable BPM capability, simply shutdown the server and start it with following command:

./standalone.sh  

--server-config=standalone-full.xml 

-Djboss.socket.binding.port-offset=150 

-Dorg.kie.server.id=first-kie-server 

-Dorg.kie.server.location=http://localhost:8230/kie-server/services/rest/server

-Dorg.jbpm.server.ext.disabled=true

watch the server startup logs and then issue the same url request as previously to see the server info response:

As you can see there is no BPM capabilities any more that means any attempt to contact any of the REST/JMS api that belong to BPM will fail.

Let's get back to fully featured KIE Execution Server and deploy container to it and run some simple process to verify it does work.

To do so, I'll use REST client in Firefox that allows to execute any HTTP method towards given endpoint. So we start with creating/deploying container to running KIE Execution Server

Endpoint:

• http://localhost:8230/kie-server/services/rest/server/containers/hr
where hr is the name of the container

Method:

• PUT

Request body:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<kie-container container-id="hr">

    <release-id>

        <group-id>org.jbpm</group-id>

        <artifact-id>HR</artifact-id>

        <version>1.0</version>

    </release-id>

</kie-container>

this is one of the standard example project that comes with every version of jBPM and it's part of jbpm-playground repository. Make sure it was built at least once and is available in maven repository that your server has access to or is in your local maven repo (usually at ~/.m2/reporitory)

When request is finished successfully you should see following response being returned:

That tells us we have single container deployed and it is in status STARTED - meaning ready to accept and process requests. So let's see if it actually is ready...

First let's see what processes do we have available there

Endpoint:

• http://localhost:8230/kie-server/services/rest/server/queries/processes/definitions

Method:

• GET

When successfully executed you should find single process being available with process if hiring inside container id hr

That tells us we have some processes to be executed, so let's create one instance of hiring process with some process variables

Endpoint:

• http://localhost:8230/kie-server/services/rest/server/containers/hr/processes/hiring/instances
where hr is the name of the container and hiring is the process id

Method:

• POST

Request body:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<map-type>

    <entries>

        <entry>

            <key>age</key>

            <value xsi:type="xs:int" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">25</value>

        </entry>

        <entry>

            <key>name</key>

            <value xsi:type="xs:string" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">john</value>

        </entry>

    </entries>

</map-type>

So let's issue the start process request...

And examine response...

As we can see we have successfully created process instance of hiring process and the returned process instance id is 1.

As last verification step let's list active process instances available on our kie server instance

Endpoint:

• http://localhost:8230/kie-server/services/rest/server/queries/processes/instances

Method:

• GET

So that's all for the first article, introducing unified KIE Execution Server and it's first steps - installation and verification if it actually works. Stay tuned for more coming ... a lot more :)