Kappa Bluemix

A single query to process historic and realtime data.

Kappa Bluemix is a prototype implementation of a Kappa Architecture using IBM Message Hub as the immutable log. Batch and streamed data are both processed identically, meaning developers only need write one query and set of logic. An Elastic Search-like query is posted, which returns a websocket, through which the current result and and any future updates will be sent.

Download the code or contribute on Github. Try or find out more about Message Hub on IBM Bluemix.

Kappa Bluemix assumes all data is stored in JSON format on a Message Hub queue. When a query is submitted a new consumer is created and it's position is set to the start of the queue (where the earliest message is). A URL to a websocket connection is returned to the client that submitted the query. Kappa Bluemix then processes all the messages in the queue in the order they were added and outputs the result to the websocket connection.

Once the process has reached the front of the queue (where the newest message is) it keeps the websocket connection open and waits for any new data to be added to the queue, automatically processing it and updating the client when it is.

Queries are shared across clients, so multiple clients with identical queries make use of the same consumer and processing. If the client closes the websocket connection and there are no other clients using the same query, Kappa Bluemix will stop waiting for more data to arrive and shutdown the consumer.

Kappa Bluemix was created by IBM Emerging Technology. Find out more about Emerging Technology, the other projects we work on, or get in touch - Darren.