XForms Push with MQTT over Websockets

The idea…

I’ve been looking recently at a project that creates pdfs via a ConTeXt typesetter. The front-end is XForms, feeding the backend queue of jobs that get done with a delay of perhaps a few minutes. The problem is how do I tell the web app, that a particular job has finished, because the user isn’t waiting; they’re on to another document.

What I need is some sort of Push Notification that I can integrate into the XForms world.

Of course within XForms it’s possible to do a sort of polling with native <xf:dispatch delay=”number”/> calling a <xf:submit/> to see if anything interesting has happened, but it really doesn’t scale. At that point you need a ‘push’ technology that will notify you when something you are interested happens instead. Handily in HTML5, there is scope for that in the use of WebSockets. These allow you to create a two-way link back to your server along which you can send/receive pretty much anything. Now there is nothing to say you can’t write your own system but  there are plenty of fine libraries out there that will do the heavy lifting for you.

In my case Pub/Sub was the way to go as a behaviour, and MQTT specifically as a protocol because it’s one of the best (if you don’t believe me ask FaceBook). Pub/Sub in brief is a message pattern (2lemetry have a great Nutshell). Publishers send data (of any sort) to one or more topics, e.g. house/temperature/lounge to a Broker (that’s the server bit). Subscribers, without any knowledge of who is doing the publishing, subscribe to the broker to receive data from one or more topics, either explicitly i.e. house/temperature/lounge or by wildcard (+ = 1 level, #= n level), i.e house/# or house/+/temperature.

The Broker mediates between the publishers and subscribers and makes sure messages get delivered. In my use-case, for instance, for a user Fred, working on a Case 00123, you might assume a topic tree something like fred/docs/00123.  When fred logged onto the web app, he’d subscribe to  fred/# and start receiving anything on that branch and down. The reason not to subscribe to say fred/docs/# is that it leaves the door open to using the same tech for something like system messages targeted at him via fred/messages or some way to lock a form via fred/locks/00123 or track form changes via fred/forms/00123 etc etc. Note: One of the nice benefits to doing it the MQTT way is that there’s nothing to stop another system subscribing to say +/docs/# to watch for problems or gather stats!

However, before any of those bells and whistles, perhaps I need to solve the basic problem of getting it to work at all via a small example. All code and runnable examples are as usual on GitHub.

Demo v1: Connect/Disconnect.

Untitled

click to run example.

First thing is a MQTT broker. The nice people at DC Square (@DCsquare)  have a public broker running at mqtt-dashboard.com we can play on based on HiveMQ which I’ll use in the example. Please be nice to their broker and use responsibly. Want to set up your own broker? The Mosquitto Open Source broker comes highly recommended. Second we need a Javascript library to talk to the broker. The Eclipse Paho project has one that is pretty good. To get it you’ll have to clone their repository like so:

git clone http://git.eclipse.org/gitroot/paho/org.eclipse.paho.mqtt.javascript.git

Lastly some code…. I’m going for the simplest xsltforms XForm with my MQTT connection details for the DC Square broker on it’s web socket port, and an initial topic to subscribe to (unused just yet) in the <xf:instance/>:

<xf:instance id="connection" xmlns=""> 
<data> 
<host>broker.mqttdashboard.com</host> 
<port>8000</port> 
<clientId></clientId> 
<initialtopic>xforms/chat</initialtopic> 
<status>disconnected</status>
 </data> 
</xf:instance>

Note the clientId is empty. I’m using a bit of xpath in the xforms-ready event to set this to a random value. This makes sure I don’t get kicked off the broker by someone connecting with the same id. (I’ll come back to this later). I’ve two trigger/buttons to connect and disconnect, which each call Javascript using the <xf:load><xf:resource/> method. Here’s the connect call as an example:

<xf:trigger id="mqttconnect">
  <xf:label>Connect</xf:label>
  <xf:load ev:event="DOMActivate">
    <xf:resource value="concat('javascript:mqttstart(&quot;',host,'&quot;,&quot;',
    port,'&quot;,&quot;',clientId,'&quot;)')" />
  </xf:load>
</xf:trigger>

Having to put in quotes around the params makes it a bit messy and it’s can be a pain. On bigger forms I tend to move this sort of JS stuff up into an <xf:event/> at the top of the form and call it, rather than spread it out through the code. Now I just need some JS to call. The main mqttstart() function sets up a Messaging Client with our parameters. The MQTT library does pretty much everything via callbacks, so most of the code is references to other functions.

function mqttstart(host,port,clientId) {
client = new Messaging.Client(host, Number(port), clientId);
client.onConnectionLost = onConnectionLost;
var options = {
timeout: 30,
onSuccess:onConnect,
onFailure:onFailure 
};
 client.connect(options);
};

For this version they’re pretty simple and I’m not interacting with the XForm the other way at all, i.e.

function onConnect() {
alert('Connected');
};
function onFailure() {
alert('Failed');
};
function onConnectionLost(responseObject) {
alert(client.clientId + " disconnected: " + responseObject.errorCode);
};

Only onFailure() is at all fancy and that’s just to get an error code. I’ll call this Version1 – here it is.

Version 2 : Feeding events back into the XForm.

Untitled

click to run example

Now I know the library works, I need to integrate the results from JS, back to my XForm. This is done by calling an <xf:event/> (note this call format is xsltforms specific) . I’ve got a generic JS function for this that takes the name of the event and a payload for the properties. The code is below, along with my new onConnect() function to call it.

Important! The JS side needs to know what the <xf:model/> id is for the call. I’ve hardcoded it here. Make sure it’s set on your XForm as well or it won’t work.

On the XForms side, I’ve now got the <xf:event/> that sets the connection status in my instance(). To add a visual cue, I’ve added a <xf:bind> to set the instance to readonly once connected  and reset if we disconnect:

function call_xform_event(xfevent,xfpayload) {
  var model=document.getElementById("modelid")
  XsltForms_xmlevents.dispatch(model,xfevent, null, null, null, null,xfpayload);
}
function onConnect() {
call_xform_event("mqtt_set_status",{status:  "connected"});
};
<xf:bind nodeset="instance('connection')" readonly="status = 'connected'" />
<xf:action ev:event="mqtt_set_status">
<xf:setvalue ref="status" value="event('status')" />
<xf:reset model="modelid" if="event('status') = 'disconnected'" />
</xf:action>

This is a much cleaner system than calling JS. I can set any number of properties on the JS side as a JSON string and they’ll appear as event properties on in the XForm.

Lastly, I need to add a subscription. In this case I’ve chosen xforms/chat.  To subscribe I have to wait to be connected and of course I could have done that at the JS level. However, this is about XForms so I’ve added a <xf:dispatch> to my ‘mqtt-set-status’ event to call another event to do it for me if I’m connected.

 <xf:dispatch name="mqtt_sub_first" targetid="modelid" if="event('status') = 'connected'" />

Next, I need to add support for messages arriving from this subscription. This is done by adding another callback into mqttstart() for onMessageArrived and then fleshing it out so that the message and it’s topic can be sent into my XForm:

function onMessageArrived(message) {
  dispatch_xforms("mqtt_message_arrived",{
    message: message.payloadString,
    topic: message.destinationName
    }); 
};

On the Xform side, I now need the event code. I’m going to drop all the messages into a new top-down “messagelog” instance. So it doesn’t escape off the screen, I’ve limited the log to 15 rows before I prune the bottom row. This simply uses a <xf:repeat> to output onto the screen.

<xf:action ev:event="mqtt_message">
  <xf:setvalue ref="message" value="event('message')" />
  <xf:insert nodeset="instance('messagelog')/event" at="1" position="before" />
  <xf:setvalue ref="instance('messagelog')/event[1]" value="concat(now(),' : ',event('topic'),' : ',event('message'))" />
  <xf:delete nodeset="instance('messagelog')/event[last()]" if="count(instance('messagelog')/event) &gt; 15" />
</xf:action>

You can try out v2 here. Once you connect, you’ll see, hopefully, the classic “Hello”. This is what is called a Retained Message and is sent to anyone subscribing as a default.  If you now go to the mqtt-dashboard.com site and use their Publish option, you should be able to publish anything you like to the xforms/chat topic and see it appear on the XForm – neat!

Version 3 : Publishing.

Untitled

Click to run example

We’re almost there. The last thing is how to publish a message to the broker. That’s actually the easiest bit of all. All I need is another instance to hold the topic and the message, so we can put it in the XForm, along with a <xf:trigger/> to call a JS function with the two values to send it. However, I can make it a little bit more interesting by talking about adding QoS and CleanSession.

QoS, or Quality of Service, is a flag that gets sent with a message or a subscription that sets how hard the broker and client work to handle your message.

At QoS 0, the default, the client deletes the message as soon as it’s sent to the broker. The broker in turn only sends it on to clients that are connected – it’s fire and forget.

At QoS  1, the client keeps the message stored until it knows the broker got it. In turn the broker keeps a copy until not only those connected get it, but those not connected currently but subscribed as well – you’ll get the message at least once, but might get a duplicate if the  broker/client aren’t sure.

At QoS 2 the same happens at QoS 1, with added actions by the broker/client to make sure you only get the message once.

So how does the broker keep track? It uses the clientId as the key to handle messages and subscriptions for a given client. When the client connects, there is a flag that tells the broker to either carry on or reset all messages and subscriptions for this client – that’s cleanSession and it’s true by default.

It’s worth knowing about since for a practical system you’d want cleanSession to be false and have a fixed clientId. Also, you’d probably end up with your messages being transferred at at least QoS 1. As they’re just flags it’s easy to add to the XForm for experimentation as version 3

Wrap Up.

It’s been fairly easy to integrate XForms with MQTT as it is with most JS libraries. Now I’ve a working example I should be able to generate my notification system fairly simply. Pub/Sub is an great technology and opens the way for some interesting form/form and form/server interactions.

Resources.

  • MQTT.org All things MQTT and where to go for libraries and brokers.
  • Eclipse Paho. M2M and IoT protocols. Home to the reference MQTT clients.
  • XSLTForms. Great client-side XForms engine.

 

Advertisements