Apache Camel for home monitoring.

It’s pretty interesting monitoring stuff at home and over time I’ve put together scripts to handle everything reliably so I can concentrate on the data. What it isn’t is pretty: there are several languages and libraries and some cron jobs to make it work, plus some stuff off the internet (I don’t fully understand). So, as part of a project with Apache ActiveMQ at work I was interested to find the sub-project Apache Camel: it’s a sort of integration engine, a switchboard between technologies that uses a ruleset to say what goes where.

Camel is intriguing because it opens the door to something, perhaps better, that can be more configured than built, but without being a dead-end if I need something particular. Now Camel is really designed as a framework using Java beans to specify the routing and manipulate the message contents. But, once built, with the appropriate connectivity, a Camel application can use just XML to describe the rules. So, in theory, I can take a Camel example, add the connectivity I want and then configure the system as needed. That last bit’s important to me; I don’t want to swap my scripts for a load of Java. The trade-off is that I may have to be more inventive to get it to do what I want via configuration.

The new world

The best place to start is with a test scenario:

  1. Get temperatures from the 1-wire system (exposed through OWFS as folders and files).
  2. Publish them onto a MQTT broker. I’ve blogged about doing this one already using Tcl, though I had to write the mqtt client code myself.
  3. Send them to MySQL for further analysis.

First steps is to get a copy of Camel and you’ll need Maven if you haven’t already got it installed.

  • Go to the examples and find the ‘camel-example-console’.
  • Open a terminal there – we’ll need it in a moment.
  • At the console type mvn compile to set it up and then mvm exec:java to start it using the local Jetty.

It’s really simple: you can type some text in and get it back in uppercase – not too exciting but lets us know it works. We’re just using it as a base.

For the test, we’ll need to extend the abilities of the example with things we want, like SQL and MQTT. So, open pom.xml and add the following dependencies:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-mqtt</artifactId>
</dependency>
<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-script</artifactId>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-jdbc</artifactId>
    </dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-sql</artifactId>
    </dependency>
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.9</version>
</dependency>

Call mvn compile again to recompile the example. From now on we’re just using configuration, so look in the directory target/classes/META-INF/spring for the file camel-context.xml.  This is where the magic happens.

At the moment the only route there is this one :

  <route>
      <from uri="stream:in?promptMessage=Enter something: "/>
      <transform>
        <simple>${body.toUpperCase()}</simple>
      </transform>
      <to uri="stream:out"/>
  </route>

It’s pretty clear: stream uri is the console, and we get an input message from it and output it back. In the middle we transform the content to uppercase using the “simple” language. Of course, <from> could be a lot of things: ftp, files, uri, rest, etc and <to> could be those same things or say twitter or sql or web sockets. Also, since Camel is a switchboard, we can route the same  <from> to a series of <to>s in parallel, or series and if we want to, we can transform the message itself.

Getting File data in

Back at our test (but excited I hope by the possibilities), step 1 is to get regular readings from the “files” that are in OWFS: welcome to the File component.

The File component polls directories for files and sucks them up into the route. The component is really designed to consume files not re-read the same one, so we have to tell it a few specific things about the OWFS directory via the uri:

  • noop=true. Don’t delete the file.
  • readLock=none Don’t try to lock the file.
  • idempotent=false Keep re-reading the same file (default is just once).
  • delay=10000 Only read the directory every 10s.
  • antInclude=*/temperature Use Ant syntax to just pick out temperature files.
  • recursive=true Dive into sub directory.

This gives us the following <from> :

<from uri="file:/Users/mark/work/test?noop=true&readLock=none&idempotent=false&delay=10000&antInclude=*/temperature&recursive=true"/>

Ok, so we now have readings trooping into our route. However, it’s worth noting that the message is currently a file handle object wrapped in an envelope that contains various headers and properties. That means that for both an MQTT and SQL destinations, we’ll need to get hold of the actual contents. It also means we can use the headers to send meta-data along with the file.

Headers and message bodies

At the moment all we have are a stream of readings, but we also have the CamelFilePath header telling us where in OWFS they came i.e. lounge/temperature, office/temperature etc. If we get the first folder from the path, we can use it to create a topic on the fly in the <out> route later e.g. temperatures/lounge and as a key in SQL. Luckily, Camel lets us inject a script language to do the lifting:

<setHeader headerName="topic">
<javaScript>
    request.getHeader('CamelFileName').split('/')[0].toLowerCase();
</javaScript>     
</setHeader>

Here’s another one we’ll need later using the in-built <simple> (it really is) language:

<setHeader headerName="UTFDateTime">
<simple>
    ${date:now:yyyy-MM-dd hh:mm:ss}
</simple>
</setHeader>

Lastly, we need to make sure that the message body is the current reading as a string:

<transform>
<simple>${bodyAs(String)}</simple>
</transform>

So, we’ve the message body set up and the two ‘variables’ we’ll need later. Time to do some routing!

Sending to MQTT

First stop is the MQTT broker I’ve got running. I won’t go into the ins-and-outs of brokers; they’re very useful, you’ll just have to believe me.

There is a MQTT Component, so it should be as simple as this to talk to the local broker:

<to uri="mqtt:test:?publishTopic=temperatures.${header.topic}"/>

Sadly, that’s not the case. Camel doesn’t let you do this sort of thing in XML. All is not lost however. We can use the Recipient List from the Patterns routing recipes. Recipient List lets you send to a given list of outputs but more importantly, lets you use scripting to build them. It’s a bit of a bodge to make a list of one thing but it works.

Since it’s a bit more complicated now, we’ll split out the MQTT stuff into a sub-route (via the direct: uri) that we can call from the main one. Anyone who’s spent any time writing Ant scripts will recognise this game right away…

<route id="R1">
    <from uri="direct:mqtt"/>
        <recipientList ignoreInvalidEndpoints="true" >
            <javaScript>'mqtt:test?publishTopicName=temperatures.' + request.getHeader('topic');
            </javaScript>                               
        </recipientList>
</route>

Sending to MySQL

We can use exactly the same idiom to send the data into SQL. We’ll use the SQL Component in the same way, but first we need a bit of setup to go in above the <routes> to tell Camel about the jdbc connection to MySQL (this apparently is standard Spring):

<bean id="myDS" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value="com.mysql.jdbc.Driver" />
    <property name="url" value="jdbc:mysql://localhost:3306/1wire" />
    <property name="username" value="user" />
    <property name="password" value="password" />
</bean>

There are fancier ways to do this with pooling etc, but this will do for us. All that’s left is to make up an INSERT statement to drop readings into the 1wire database, where we’ve got these three fields in the readings table:

  • reading  – FLOAT.
  • topic – VARCHAR(32).
  • tdate – DATETIME.

Again we’ll use the subroutine approach to factor out this destination and to keep the xml in manageable chunks:

<route id="R2">
       <from uri="direct:mysql"/>
             <recipientList ignoreInvalidEndpoints="true" >
                <javaScript>
                  "sql:insert into readings (reading,topic,tdate) values(" + request.body + ",'" + request.getHeader('topic') + "','"+ request.getHeader('UTFDateTime') + "')?dataSourceRef=myDS";  
                </javaScript>                                
            </recipientList>  
   </route>

That’s it: call the named datasource with the statement.  Now it’s time to put it all together.

Pipelines and Multicast

One gotcha from Camel is that multiple endpoints are chained together by default. So if you just list <to a> <to b> <to c> then the return from a is sent to b etc. This can lead to some very strange results, depending on your component! What we need instead is a <multicast>. This sends a copy of the message to each recipient. Knowing that, we can construct the whole route like so:

<!-- Poll for temperatures and route to mqtt and sql -->      
<route id="GetTemperatures">
<from uri="file:/Users/mark/work/test?noop=true&amp;readLock=none&amp;idempotent=false&amp;delay=10000&amp;antInclude=*/temperature&amp;recursive=true" />
<setHeader headerName="topic">
  <javaScript>
      request.getHeaders().get('CamelFileName').split('/')[0].toLowerCase();
  </javaScript>     
</setHeader>
<setHeader headerName="UTFDateTime">
    <simple>${date:now:yyyy-MM-dd hh:mm:ss}</simple>
</setHeader>
<transform>
    <simple>${bodyAs(String)}</simple>
</transform>
   <multicast stopOnException="true"> 
          <to uri="direct:mqtt"/>
          <to uri="direct:mysql"/>
   </multicast>        
</route>
<!-- Push the message to MQTT using topic in header -->
  <route id="Sub_1">
            <from uri="direct:mqtt"/>
            <recipientList ignoreInvalidEndpoints="false" >
                <javaScript>
                    'mqtt:test?publishTopicName=temperatures.' + request.getHeader('topic');
                 </javaScript>                               
            </recipientList>
   </route>

<!-- Push the message to SQL using topic and date in header --> 
   <route id="Sub_2">
       <from uri="direct:mysql"/>
             <recipientList ignoreInvalidEndpoints="false" >
                <javaScript>
                  "sql:insert into readings (reading,topic,tdate) values(" + request.body + ",'" + request.getHeader('topic') + "','"+ request.getHeader('UTFDateTime') + "')?dataSourceRef=myDS";  
                </javaScript>                                
            </recipientList>  
   </route>

If you’ve got MQTT and MySQL, then you can add this route to the config file and restart the example to see it in action. If you don’t you can use the <log> or <stream> endpoints instead to just print out the results to the console.

Thoughts and ideas

So, 35’ish lines of Camel config gets us a polling file reader and outputs to MySQL and MQTT – that’s not bad. Also we now have a single place to do logging and a fairly simple idiom that can be expanded if other components are needed. Moreover, if as the base, we’d used something like the servlet-tomcat example to built a web app, we’d have the basis of a neat little switchboard with a web interface thrown in. Everything is rosy? not quite:

  • It’s evident that the Camel docs lean heavily towards the java, beans, maven side rather than the config side. It can be difficult to get information. For example it was a few hours of internet wandering before I found out about Recipient List.
  • The lack of variables or replacing properties in uri’s makes routing less straightforward than it should be. Luckily the user groups are pretty friendly.

Having said that it works, and if a Twitter output was needed or a websocket feed than it’s not much harder to add them.

Ok, so is this easier than using a scripting language I know well? Obviously, not at first but as someone who does a lot of XML anyway it’s just as viable. Now, if I didn’t know how to write scripts, then this idea becomes pretty useful.

I could see perhaps a pre-compiled version of Camel webapp with common routes set up being a really easy start point for the home measurement community (uncomment to us X etc) and perhaps more approachable than say ‘download ruby and then do…’.

Maybe what the Internet of Things needs is a good switchboard.

Advertisements

1-wire with MQTT

Background

Over the last few weeks I’ve been experimenting with the Dallas 1-wire system for sensors around the house/garden. Although I originally thought of using Arduino/JeeNodes I couldn’t make the figures work. 1-wire may not be wireless, but it’s inexpensive, simple and robust. Basically, you need a run of 2/3-core wire with sensors on it where you need them: most people seem to use phone cable and RJ12 connectors. Each sensor has a unique address and for short’ish runs can all work on parasitic power from the bus – I got mine from HomeChip. They also do the bus controller that sits on one-end of the wire and has a usb connector so you can talk to the sensors on your server. By the way, it’s not all sensors, there’s A/D converters, switches etc and the bus is of course 2-way. See here for some idea of the things you can do.

Now this is great, and don’t get me wrong, of course you could run 1-wire sensors from an Arduino. What sold the more retro solution to me was the OWFS or 1-wire file system. This is a rather neat bit of software that turns your 1-wire bus sensor data into a file system. Neat! Now instead of an API etc, all the normal tools, scripts etc that work with files work with 1-wire and getting a temperature is as simple as reading the contents of

 /mnt/1wire/28.E1A6CC030000/temperature

Publishing to MQTT

A bit of installing and a modicum of soldering got me a working system on my Viglen MPC with a couple of sensors up and running. The common place to put the resulting data was into RDDTool, however it struck me that the filesystem idea in OWFS is a good match for the topic system in MQTT. I could simply push the file structure into MQTT and then subscribe to the topics that interest me, again using tools I already have and various cool ideas that I’ve read about on mqtt.org . And, as it’s two way, there’s nothing to stop me later using MQTT to publish data myself back to those same sensors or switches. Nothing of course stops me doing something else as well. In fact I’ve a cron job that spits everything into MySQL every 10 mins as well so I can analyse it in R stats later.

For the MQTT side, the only issue I had was to write a small Tcl library so that I could publish the data. That’s only because I like to use Tcl, there are already  libraries for Java, Perl and numerous others on Mqtt.org. Once I had a simple lib that could publish data at Qos 0, I just needed a simple script to find all the temperture sensors and publish their paths and values:

#!/usr/bin/env tclsh
lappend auto_path [pwd]
package require mqtt 1.0
#Using tcllib for 'cat', you could use open instead
package require fileutil
#Mount for 1-wire
set base "/mnt/1wire"
set mqtt_conn ""

#Get paths in this dir that are files
proc get_tree {d} {
 foreach s [glob -directory $d -types f *] {get_file $s}
 }
#Get contents of path
proc get_file {d} {
 global mqtt_conn
 set reading [string trim [fileutil::cat $d]]
 puts "Publishing $reading to $d"
 mqtt publish $mqtt_conn $d $reading
}

#Get all thermometers i.e. 28.xxxxxxxxxxx
proc get_thermometers {} {
global base
puts "Getting directories"
foreach d [exec find $base -maxdepth 1 -name 28.* ] {
#Get tree for each dir
 puts "Looking for $d"
 get_tree $d
 }
}

set mqtt_conn [mqtt connect localhost 1883 1Wire]
mqtt publish $mqtt_conn $base "[clock format [clock seconds]]"
get_thermometers
mqtt disconnect $mqtt_conn

With the RSMB broker running on the little Viglen MPC, which is called byron, which also has OWFS on, I could now publish my OWFS data and a test with stdoutsub tool on the laptop showed that I could see all the topics and data if I subscribed to /mnt/# .

./stdoutsub /mnt/# --host byron --qos 0

Fixing Aliases

One problem in the initial data was sensors are by default named by id and 28.2823FBE00300008A isn’t very semantic. Luckily, OWFS has an alias system that lets you link ids to human-readable names with a simple file. Once setup, it changes the OWFS file system and the folders magically get renamed to /office, /outside etc. Handily, if you’ve already got scripts talking sensor ids instead, they’ll still work as well!

All the Tcl script needed to use aliases was an initial look in /settings/alias/list for the list of sensors, rather than finding everything starting 28.* This gives a replacement get_thermometer() function like so:

proc get_thermometer {} {
global base
foreach d [exec cat "$base/settings/alias/list"] {
  set alias_dir "$base/[lindex [split $d =] 1]"
  get_tree $alias_dir
  }
}

If you run the script again you’ll get the more readable listing like so:

Publishing 28E1A6CC03000060 to /mnt/1wire/office/address
Publishing office to /mnt/1wire/office/alias
Publishing 60 to /mnt/1wire/office/crc8
Publishing 28 to /mnt/1wire/office/family
Publishing 21.5 to /mnt/1wire/office/fasttemp
Publishing E1A6CC030000 to /mnt/1wire/office/id
Publishing FFFFFFFFFFFFFFFF to /mnt/1wire/office/locator
Publishing 0 to /mnt/1wire/office/power
Publishing 60000003CCA6E128 to /mnt/1wire/office/r_address
Publishing 000003CCA6E1 to /mnt/1wire/office/r_id
Publishing FFFFFFFFFFFFFFFF to /mnt/1wire/office/r_locator
Publishing 21.4375 to /mnt/1wire/office/temperature
Publishing 21.5 to /mnt/1wire/office/temperature10
Publishing 21.5 to /mnt/1wire/office/temperature11
Publishing 21.4375 to /mnt/1wire/office/temperature12
Publishing 21.5 to /mnt/1wire/office/temperature9
Publishing 75 to /mnt/1wire/office/temphigh
Publishing 70 to /mnt/1wire/office/templow
Publishing DS18B20 to /mnt/1wire/office/type

As you can see, even a small temperature sensor puts out all sorts of data as well as the temperature in various resolutions. One of the beauties of the MQTT side is this complexity can be  simplified to just what’s required. So, by subscribing to /mnt/1wire/+/fasttemp for instance you get:

/mnt/1wire/outside/fasttemp 1
/mnt/1wire/office/fasttemp 21.5

Left to do.

There’s a few things left to do:

  • I’m only sending in Qos 0. It really needs to be Qos 1.
  • I need to set the Retained flag so last temperature is held.
  • I’m not sure I need everything from a sensor.
  • Subscribing is only the beginning – publishing is the end goal so I can interact with the 1-wire system.

But, it’s a start.

Resources.

If you’re interested in 1-wire and you’ve got Debian/Ubuntu, then be grateful to Cyril Lavier (@davromaniak) who has a neatly packaged repository otherwise get the sources from OWFS below and break out the build tools.

The Tcl script and simple MQTT lib at all here.

All the MQTT stuff is over at MQTT.org

All the OWFS stuff is at OWFS.org

I’ve since found out via Twitter that Nicholas Humphrey (@njh) has a fine daemon that does this directly. You can get his code from GitHub here.