środa, 2 sierpnia 2017

Managed KIE Server gets ready for the cloud

As described in this article, KIE Server can run in two modes:

  • managed, wit controller that is responsible for providing kie containers to be deployed
  • unmanaged, self contained server that allows to deploy kie containers manually 
In this article, I'd like to focus on managed mode and show some improvements in that area that will make managed KIE Server ready for the cloud.


With default configuration of managed KIE Server, both controller and kie server need to know how to communicate with each other. By default it is REST based communication and thus require to provide credentials while sending requests
  • user and password - for BASIC authentication
  • token for BEARER authentication 
These should be gives as system properties on each side:

  • org.kie.server.user and org.kie.server.password is to be set on controller jvm to instruct what credentails to use when connecting to kie server(s)
  • org.kie.server.controller.user and org.kie.server.controller.password is to be set on kie server jvm to instruct what credentials to use when connecting to controller

This configuration fits nicely in non restricted environment where both controller and KIE Server(s) don't have any limitations to talk to each other. Though it does require that user name and password used by controller to connect to kie servers is the same as it is set globally via system properties and thus will be used whenever talking to any KIE Server instance.

Though this setup can become problematic if there are any restrictions between these two. In some cases controller might be hidden behind firewall. That will then make an issue for it to communicate with KIE Server(s) when needed. Similar this becomes an issue in OpenShift environment where controller and KIE Server(s) are in different namespaces - they won't see each other internal IP.

Here we touch upon another aspect of managed KIE Servers - its location. KIE Server when running in managed mode requires following configuration parameters (given as system properties on jvm that runs the KIE Server):
  • org.kie.server.id - an id that points to server template id defined in the controller
  • org.kie.server.controller - an URL of the controller to connect to upon start
  • org.kie.server.location - an URL of this instance of the KIE Server where it will be accessible over HTTP/REST
The location of the KIE Server is expected to be unique - since this is an URL where the actual instance is accessible. Though this becomes an issue when running kie servers behind load balancer or when running in a cloud based environments. 

It puts us in situation that we either give load balancer URL and by that loose the capabilities of receiving updates from controller (as only one of them will get updates based on load balancer selection) or we bypass load balancer and then loose the capabilities of it for runtime operations. Keep in mind that the location that kie server does provide on connection to controller is then used by (so called) runtime views in workbench - process instances, tasks, etc.

In OpenShift environment that is pretty much the same issue - either public IP is provided which completely hides the individual PODs or internal IP of the POD. It has the same consequences as load balancer with one addition - internal IP won't work at all across namespaces.

Websockets to the rescue...

To resolve all the issues mentioned above, an alternative (and soon to be the default) way of communicating between KIE Server and Controller was introduced. It is based on Websockets that is now available in pretty much any JEE container (including servlet container) and solves pretty much all the issues that were identified for both on premise and in the cloud.

As illustrated on the diagram above, KIE Server is the one who initiate the communication and keeps it active as long as it's alive. That in turn removes any need from KIE Controller to know how to communicate (and by that connect to) with KIE Server instances. So there is no more need to configure any user name or password on controller jvm to talk to KIE Servers, it will simply reuse open channel to connected KIE Servers.

KIE Server is solely responsible for the connection. That means it needs to know where the controller is, how to authenticate when opening connection and how to handle lost connection (e.g. when controller goes down).

So the first two are exactly the same, given as system properties on jvm that KIE Server is going to run:
  • using either BASIC or BEARER authentication 
  • org.kie.server.controller - an URL of the controller to connect to upon start
Lost connections are handled by retry mechanism - as soon as KIE Server gets notification that the connection is closed it will start a background thread that will attempt to connect to controller every 10 seconds. Once it is reconnected that thread is terminated. It will reconnect only if the KIE Server itself is not the one who closed the connection.

Since we keep connection open between kie servers and kie controller then the location given when kie server connects does not have to be unique any more. That solves the issue with running behind load balancer or in OpenShift with different namespaces. System property that provides location (org.kie.server.location) should now be given as the load balancer or public IP in OpenShift. 

NOTE: If you don't run behind load balancer on on-premise setup (not OpenShift) then keep the location of the kie server unique regardless of the websocket being used. Similar rule applies - same public IP/load balancer should be kept for single server template only.

There is no need for any extra configuration to enable websocket based communication, it is based only on the actual URL given as controller url - org.kie.server.controller system property.


Depending where is your controller you might need to change:
  • localhost - to actual host/IP of your server where controller is deployed to
  • 8080 - to actual port number of your server where controller is deployed to
  • kie-wb - to actual context path of the controller web app 

Both protocols - HTTP/REST and Websocket are active by default and either of them can be used. Though one rule must be kept - use single protocol for all kie servers of given server template. 
Recommended is to keep single protocol across all kie servers connected to single controller.

Workbench that provides UI for process related operations (Process Instance, Process Definitions, Tasks perspectives) will utilise websocket channel only for administration operations, that is:
  • controller based operations to manage kie servers
  • data set queries registration required by runtime views
All other operations, like getting user tasks, getting process definitions or instances, will use regular REST based communication as it will call endpoints on behalf of logged in user to enforce security.

With this enhancement managed KIE Server is way nicer option to run in cloud and behind load balancer than ever before :)

Stay tuned for more to come!

10 komentarzy:

  1. Hi Maciej, Thanks for all your articles I have been learning a lot. However I've been stuck recently when I had to move the development from eclipse to kie workbench along with the kie-server as I have vertx-jca implemented in the application. when I deploy the containers from the controller and having vertx in cluster mode with hazelcast the Kie-server breaks or the container is disposed immediately if it is deployed on an unmanage kie-server. I was wondering if it is possible of joining the kie server on a cluster so I can communicate to other members?

    1. Byron,

      you would have to provide bit more context to be able to see what might be causing an issue. Looks like you have bunch of things integrated so without having proper background it's difficult to help. Feel free to use jbpm mailing lists as that might be easer to use.

    2. Hi Maciej, Thanks for your reply I will make sure to document the issue and send it to jbpm mailing list. Sorry for the ambiguity I thought since vertx uses hazelcast for High Availability as default it might cause problems inside the kie-server. I tried to use the JCA resource adapter since I'm deploying a vertx app within the kie-server on wildfly 10 with the bare minimum dependencies and a simple class implementing a listener interface. However the error log surpasses more than 1000 lines on my terminal looking like something forbidden. Thanks again.

    3. feel free to use pastern.com to share the log details maybe I will be able to spot anything "obvious"...

      I am not aware of any conflicting things on kie server with hazelcast, though it might be due to 3rd party libs both use...

  2. This is not working with kie-server 7.1.0.Final
    It doesn't understand protocol "ws"

    1. that's expected because it was added in 7.2

  3. Ten komentarz został usunięty przez autora.

  4. Hi Maciej,

    I have just tried to tie to KIE Server with websocket. I am getting this error:

    [org.kie.server.controller.websocket.client.WebsocketKieServerControllerImpl] (KieServer-ControllerConnect) Exception encountered while syncing with controller at ws://remotehostname:8080/kie-wb/websocket/controller/default-kieserver error hostname can't be null

    In the server.log there is this error:

    2017-10-06 16:25:47,995 INFO [org.jboss.as.server] (ServerService Thread Pool -- 38) WFLYSRV0010: Deployed "jbpm-console.war" (runtime-name : "jbpm-console.war")
    2017-10-06 16:25:47,996 INFO [org.jboss.as.server] (ServerService Thread Pool -- 38) WFLYSRV0010: Deployed "jbpm-casemgmt.war" (runtime-name : "jbpm-casemgmt.war")
    2017-10-06 16:25:47,996 INFO [org.jboss.as.server] (ServerService Thread Pool -- 38) WFLYSRV0010: Deployed "kie-server.war" (runtime-name : "kie-server.war")
    2017-10-06 16:25:48,548 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on
    2017-10-06 16:25:48,548 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on
    2017-10-06 16:25:48,548 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 10.1.0.Final (WildFly Core 2.2.0.Final) started in 88591ms - Started 1498 of 1761 services (478 services are lazy, passive or on-demand)
    2017-10-06 16:25:56,191 WARN [org.kie.server.controller.websocket.client.WebsocketKieServerControllerImpl] (KieServer-ControllerConnect) Exception encountered while syncing with controller at ws:// error Invalid response code 404


    1. this is because your workbench (that hosts controller) is on another context path - jbpm-console instead of kie-wb. so your controller url should be

      make sure you have it valid host name or ip address. does not seem to be valid to me