It’s pretty interesting monitoring stuff at home and over time I’ve put together scripts to handle everything reliably so I can concentrate on the data. What it isn’t is pretty: there are several languages and libraries and some cron jobs to make it work, plus some stuff off the internet (I don’t fully understand). So, as part of a project with Apache ActiveMQ at work I was interested to find the sub-project Apache Camel: it’s a sort of integration engine, a switchboard between technologies that uses a ruleset to say what goes where.
Camel is intriguing because it opens the door to something, perhaps better, that can be more configured than built, but without being a dead-end if I need something particular. Now Camel is really designed as a framework using Java beans to specify the routing and manipulate the message contents. But, once built, with the appropriate connectivity, a Camel application can use just XML to describe the rules. So, in theory, I can take a Camel example, add the connectivity I want and then configure the system as needed. That last bit’s important to me; I don’t want to swap my scripts for a load of Java. The trade-off is that I may have to be more inventive to get it to do what I want via configuration.
The new world
The best place to start is with a test scenario:
- Get temperatures from the 1-wire system (exposed through OWFS as folders and files).
- Publish them onto a MQTT broker. I’ve blogged about doing this one already using Tcl, though I had to write the mqtt client code myself.
- Send them to MySQL for further analysis.
First steps is to get a copy of Camel and you’ll need Maven if you haven’t already got it installed.
- Go to the examples and find the ‘camel-example-console’.
- Open a terminal there – we’ll need it in a moment.
- At the console type mvn compile to set it up and then mvm exec:java to start it using the local Jetty.
It’s really simple: you can type some text in and get it back in uppercase – not too exciting but lets us know it works. We’re just using it as a base.
For the test, we’ll need to extend the abilities of the example with things we want, like SQL and MQTT. So, open pom.xml and add the following dependencies:
<dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-mqtt</artifactId> </dependency> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-script</artifactId> </dependency> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-jdbc</artifactId> </dependency> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-sql</artifactId> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.9</version> </dependency>
Call mvn compile again to recompile the example. From now on we’re just using configuration, so look in the directory target/classes/META-INF/spring for the file camel-context.xml. This is where the magic happens.
At the moment the only route there is this one :
<route>
<from uri="stream:in?promptMessage=Enter something: "/>
<transform>
<simple>${body.toUpperCase()}</simple>
</transform>
<to uri="stream:out"/>
</route>
It’s pretty clear: stream uri is the console, and we get an input message from it and output it back. In the middle we transform the content to uppercase using the “simple” language. Of course, <from> could be a lot of things: ftp, files, uri, rest, etc and <to> could be those same things or say twitter or sql or web sockets. Also, since Camel is a switchboard, we can route the same <from> to a series of <to>s in parallel, or series and if we want to, we can transform the message itself.
Getting File data in
Back at our test (but excited I hope by the possibilities), step 1 is to get regular readings from the “files” that are in OWFS: welcome to the File component.
The File component polls directories for files and sucks them up into the route. The component is really designed to consume files not re-read the same one, so we have to tell it a few specific things about the OWFS directory via the uri:
- noop=true. Don’t delete the file.
- readLock=none Don’t try to lock the file.
- idempotent=false Keep re-reading the same file (default is just once).
- delay=10000 Only read the directory every 10s.
- antInclude=*/temperature Use Ant syntax to just pick out temperature files.
- recursive=true Dive into sub directory.
This gives us the following <from> :
<from uri="file:/Users/mark/work/test?noop=true&readLock=none&idempotent=false&delay=10000&antInclude=*/temperature&recursive=true"/>
Ok, so we now have readings trooping into our route. However, it’s worth noting that the message is currently a file handle object wrapped in an envelope that contains various headers and properties. That means that for both an MQTT and SQL destinations, we’ll need to get hold of the actual contents. It also means we can use the headers to send meta-data along with the file.
Headers and message bodies
At the moment all we have are a stream of readings, but we also have the CamelFilePath header telling us where in OWFS they came i.e. lounge/temperature, office/temperature etc. If we get the first folder from the path, we can use it to create a topic on the fly in the <out> route later e.g. temperatures/lounge and as a key in SQL. Luckily, Camel lets us inject a script language to do the lifting:
<setHeader headerName="topic">
<javaScript>
request.getHeader('CamelFileName').split('/')[0].toLowerCase();
</javaScript>
</setHeader>
Here’s another one we’ll need later using the in-built <simple> (it really is) language:
<setHeader headerName="UTFDateTime">
<simple>
${date:now:yyyy-MM-dd hh:mm:ss}
</simple>
</setHeader>
Lastly, we need to make sure that the message body is the current reading as a string:
<transform>
<simple>${bodyAs(String)}</simple>
</transform>
So, we’ve the message body set up and the two ‘variables’ we’ll need later. Time to do some routing!
Sending to MQTT
First stop is the MQTT broker I’ve got running. I won’t go into the ins-and-outs of brokers; they’re very useful, you’ll just have to believe me.
There is a MQTT Component, so it should be as simple as this to talk to the local broker:
<to uri="mqtt:test:?publishTopic=temperatures.${header.topic}"/>
Sadly, that’s not the case. Camel doesn’t let you do this sort of thing in XML. All is not lost however. We can use the Recipient List from the Patterns routing recipes. Recipient List lets you send to a given list of outputs but more importantly, lets you use scripting to build them. It’s a bit of a bodge to make a list of one thing but it works.
Since it’s a bit more complicated now, we’ll split out the MQTT stuff into a sub-route (via the direct: uri) that we can call from the main one. Anyone who’s spent any time writing Ant scripts will recognise this game right away…
<route id="R1">
<from uri="direct:mqtt"/>
<recipientList ignoreInvalidEndpoints="true" >
<javaScript>'mqtt:test?publishTopicName=temperatures.' + request.getHeader('topic');
</javaScript>
</recipientList>
</route>
Sending to MySQL
We can use exactly the same idiom to send the data into SQL. We’ll use the SQL Component in the same way, but first we need a bit of setup to go in above the <routes> to tell Camel about the jdbc connection to MySQL (this apparently is standard Spring):
<bean id="myDS" class="org.springframework.jdbc.datasource.DriverManagerDataSource"> <property name="driverClassName" value="com.mysql.jdbc.Driver" /> <property name="url" value="jdbc:mysql://localhost:3306/1wire" /> <property name="username" value="user" /> <property name="password" value="password" /> </bean>
There are fancier ways to do this with pooling etc, but this will do for us. All that’s left is to make up an INSERT statement to drop readings into the 1wire database, where we’ve got these three fields in the readings table:
- reading – FLOAT.
- topic – VARCHAR(32).
- tdate – DATETIME.
Again we’ll use the subroutine approach to factor out this destination and to keep the xml in manageable chunks:
<route id="R2">
<from uri="direct:mysql"/>
<recipientList ignoreInvalidEndpoints="true" >
<javaScript>
"sql:insert into readings (reading,topic,tdate) values(" + request.body + ",'" + request.getHeader('topic') + "','"+ request.getHeader('UTFDateTime') + "')?dataSourceRef=myDS";
</javaScript>
</recipientList>
</route>
That’s it: call the named datasource with the statement. Now it’s time to put it all together.
Pipelines and Multicast
One gotcha from Camel is that multiple endpoints are chained together by default. So if you just list <to a> <to b> <to c> then the return from a is sent to b etc. This can lead to some very strange results, depending on your component! What we need instead is a <multicast>. This sends a copy of the message to each recipient. Knowing that, we can construct the whole route like so:
<!-- Poll for temperatures and route to mqtt and sql -->
<route id="GetTemperatures">
<from uri="file:/Users/mark/work/test?noop=true&readLock=none&idempotent=false&delay=10000&antInclude=*/temperature&recursive=true" />
<setHeader headerName="topic">
<javaScript>
request.getHeaders().get('CamelFileName').split('/')[0].toLowerCase();
</javaScript>
</setHeader>
<setHeader headerName="UTFDateTime">
<simple>${date:now:yyyy-MM-dd hh:mm:ss}</simple>
</setHeader>
<transform>
<simple>${bodyAs(String)}</simple>
</transform>
<multicast stopOnException="true">
<to uri="direct:mqtt"/>
<to uri="direct:mysql"/>
</multicast>
</route>
<!-- Push the message to MQTT using topic in header -->
<route id="Sub_1">
<from uri="direct:mqtt"/>
<recipientList ignoreInvalidEndpoints="false" >
<javaScript>
'mqtt:test?publishTopicName=temperatures.' + request.getHeader('topic');
</javaScript>
</recipientList>
</route>
<!-- Push the message to SQL using topic and date in header -->
<route id="Sub_2">
<from uri="direct:mysql"/>
<recipientList ignoreInvalidEndpoints="false" >
<javaScript>
"sql:insert into readings (reading,topic,tdate) values(" + request.body + ",'" + request.getHeader('topic') + "','"+ request.getHeader('UTFDateTime') + "')?dataSourceRef=myDS";
</javaScript>
</recipientList>
</route>
If you’ve got MQTT and MySQL, then you can add this route to the config file and restart the example to see it in action. If you don’t you can use the <log> or <stream> endpoints instead to just print out the results to the console.
Thoughts and ideas
So, 35’ish lines of Camel config gets us a polling file reader and outputs to MySQL and MQTT – that’s not bad. Also we now have a single place to do logging and a fairly simple idiom that can be expanded if other components are needed. Moreover, if as the base, we’d used something like the servlet-tomcat example to built a web app, we’d have the basis of a neat little switchboard with a web interface thrown in. Everything is rosy? not quite:
- It’s evident that the Camel docs lean heavily towards the java, beans, maven side rather than the config side. It can be difficult to get information. For example it was a few hours of internet wandering before I found out about Recipient List.
- The lack of variables or replacing properties in uri’s makes routing less straightforward than it should be. Luckily the user groups are pretty friendly.
Having said that it works, and if a Twitter output was needed or a websocket feed than it’s not much harder to add them.
Ok, so is this easier than using a scripting language I know well? Obviously, not at first but as someone who does a lot of XML anyway it’s just as viable. Now, if I didn’t know how to write scripts, then this idea becomes pretty useful.
I could see perhaps a pre-compiled version of Camel webapp with common routes set up being a really easy start point for the home measurement community (uncomment to us X etc) and perhaps more approachable than say ‘download ruby and then do…’.
Maybe what the Internet of Things needs is a good switchboard.
Another word for snow. 