Reading usb serial data with Apache Camel

I’ve done a couple of posts on using Apache Camel as part of my home-monitoring system, and why I think it’s a good fit as an IoT switchboard. However, one of the flaws in my master plan is that there isn’t a component for serial communications (Camel MINA does socket comms, but not serial). Note I’m not talking actual RS232 here but the version that comes in USB form. I’m using mostly JeeNodes, which talk RF12 between them, but even in this modern era, without a wi-fi shield or some similar way to get onto the LAN and TCP/IP, then at some point, you’re talking to a serial port on some box into which, in my case a controlling JeeLink, is plugged.

Now, I’ve got around this like I guess most people have, by creating a serial-tcp/ip bridge in code, and getting on with our lives. But, it isn’t, well elegant and it bugs the engineer in me. It would be nice to be able to link directly to Camel and use the considerable advantages it brings, like throttling and queues and all the other goodies. But sadly, my Java isn’t up to creating modules, and it’s a fairly niche use-case. So, up to now I’d pretty much given up on it.

What I hadn’t realised, was that the Stream component, not only lets you talk to the standard input and output streams but also files and urls. Reading further, Stream has support for reading a file continually, as if you were doing ‘tail -f’ on it. Now in Unix, famously “everything is a file”, so in theory, you should be able to read a serial device, like it was a file and if you could Stream should be able to as well. Cue ‘Road to Damascas’ moment.

For my test I quickly grabbed a JeeNode and plugged it into my MacBook. Then I wrote a short sketch that spat out a clock to the serial port:

#include <Stdio.h>
#include <Time.h>
char buffer[9] ="";
void setup()
{
   Serial.begin(9600); // Tried 19200 but didn't work!
   setTime(0,0,0,0,0,0); // reset starting time
}
void loop()
{
    digitalClockDisplay();
    delay(5000);
}
void digitalClockDisplay(){
    //Send a whole line at a time
    sprintf(buffer, "%02d:%02d:%02d",hour(),minute(),second());
    Serial.println(buffer);
 }
}

I could see it running in the Serial Monitor in Arduino, and I could see the device in /dev, but nothing if I tried “tail -f  /dev/cu.usbserial-A900acSz”

Turns out, the default baud rate is 9600 on Macs and also that you can’t alter them using Unix stty commands either!  After reading a whole lot of internet, it seemed that, at least for my demo, I’d just have to stick to 9600 or get pulled into a whole pile of Mac-mess. Also note, at least on a Mac I got /dev/cu.* and /dev/tty.*. Only the cu.* ones seemed to work. I think this is something to do with the tty.* ones expecting handshaking, but I’m happy to be corrected.

Once I’d altered the baud-rate every thing worked fine. I could tail -f onto my JeeNode device and it would happily spit out stuff like this:

00:00:20
00:00:25
00:00:30
00:00:35

On the Camel side, I made a copy of camel-example-console from the /examples folder and modified the camel-context.xml to hold my new routes. First a route to read from my JeeNode – nothing fancy, just get the data and put it in a SEDA queue as a string. Then a route to get it off the queue and send it to the screen via stdout:

 <route>
 <from uri="stream:file?fileName=/dev/cu.usbserial-A900acSz&amp;scanStream=true&amp;scanStreamDelay=1000"/>
 <convertBodyTo type="java.lang.String"/>
 <to uri="seda:myfeed"/>
 </route>
 <route>
 <from uri="seda:myfeed"/>
 <to uri="stream:out"/>
 </route>

Note: scanStream tells it to scan continuously (to get the tail -f effect) and the scanStreamDelay tells it to wait a second between scans.

A quick mvn clean compile exec:java later and it works! So, it seems that I can have my cake after all.

Now, there are caveats. For a start Camel gets pretty upset if you unplug the JeeNode and the device disappears. Also, it seems I’m stuck with 9600, at least on my Mac. But it does work and means a complete Camel solution is possible. Time for more experiments, but at least for the moment my inner engineer is quiet.

PS. It works the other way as well 🙂

<route>
 <from uri="stream:in?promptMessage=Enter something: "/>
 <transform>
 <simple>${body.toUpperCase()}</simple>
 </transform>
 <to uri="stream:file?fileName=/dev/tty.usbserial-A900acSz"/>
</route>

 

Advertisements

Camel on it’s own

In my last post, I looked at Apache Camel as a sort of switchboard, useful in a home monitoring situation; mostly because you could control it just by manipulating the configuration rather than coding. Originally, I was using Camel as a servlet by simply extending the Tomcat Servlet example, which neatly packaged up the end result into a war file, for me to drop onto Jetty. However, if you’ve no use for the web side and just want to load a console app it’s a bit more complicated.

Now there is a Console example: and full marks to the Camel team -it’s one of some 30-odd examples you get to play with. The problem is that mvn compile exec:java works fine, but you’re stuck in the examples folder. What’s needed is a way to make it a proper application with all the dependencies. Now I’m sure this is trivial for the java/maven savvy, but for the rest of us, here’s the recipe:

Step 1 – Add AppAssembler to your pom.

As I understand the Camel/Maven game, and honestly as an occasional java user I don’t really, what’s needed is a method to not only create a jar file and all the dependencies, but also generate a method to kick the whole thing off. A bit of research led me to AppAssembler from CodeHaus. What this does is create a directory for your app, all it’s dependencies and a couple of clever start-up scripts to kick the whole thing off. All you need to do is add the following to your pom file, after the exec-maven plugin (I’ve used the Console example here).

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<configuration>
<programs>
<program>
<mainClass>org.apache.camel.example.console.CamelConsoleMain</mainClass>
<name>camel-console</name>
</program>
</programs>
<repositoryName>lib</repositoryName>
<repositoryLayout>flat</repositoryLayout>
</configuration>
</plugin>

The are lots of things you can configure, but of note in this simple setup are:

  • Main-class. Set it to the one in the exec-maven plugin
  • Name. The bin scripts use this name so you get camel-console.bat etc.
  • RepositoryLayout. I’ve chosen flat which dumps all the jars in one directory. Default is to split them into packages.

Now if you run mvn clean compile appassembler:assemble you will find, after a bit of maven’ing, a new appassembler folder will appear under your target folder with /bin and /lib folders. If you fire up bin/camel-console (after chmod +x ing) or bin/camel-console.bat to suit then you should see the Console example running in all it’s glory. Moreover, you can move this directory anywhere and it will still run perfectly. Result.

However, there’s a problem – our camel-config.xml where the <route>s live is now buried in a jar somewhere. As the whole purpose here was to create a configurable camel, we need a small amount of modification so that we can have our xml file in a handy conf directory.

Step 2 – External config file.

First thing is to tell the code that our config file is external. So, open the CamelConsoleMain.java in src/main/java. In there you’ll find a line like so:

 main.setApplicationContextUri("META-INF/spring/camel-context.xml");

We want an external file so we’ll change this to:

main.setApplicationContextUri("file:conf/routes.xml");

Next, create a src/main/config and copy camel-context.xml into it (why will become clear in a mo’). Rename it to routes.xml. Note, the old config file etc will still get packaged up as well – that’s redundant, but I’m making the fewest mods possible here.

Lastly, we’ll leverage our new friend appassembler. Add the following lines to the pom.xml in the appassembler plugin you added previously, below <name> and run the pithy mvn clean compile appassembler:assemble again:

<!-- Set the target configuration directory -->
<configurationDirectory>conf</configurationDirectory>
<!-- Copy the contents from "/src/main/config" to the target -->
<copyConfigurationDirectory>true</copyConfigurationDirectory>
<!-- Include the target configuration directory in the classpath -->
<includeConfigurationDirectoryInClasspath>true</includeConfigurationDirectoryInClasspath>

UntitledThese properties tell it to copy anything in /src/main/config into /conf directory in the output. Notice I’ve also included the option not to add the /conf folder to the class path. Now, when our app runs it will find our config file in the right, relative, place. Of course, we could just create a /conf dir and drop our config into it once the assembly had happened. But, doing it this way gives us somewhere to put it in src.

Addendum.

So, it’s fairly simple to create a neat, deployable app that we can use on a variety of OS’s  to leverage the power of Camel. If you want to run on a server, a handy App Assembler option is to create Java Service Wrapper scripts so you can start the code as a daemon.

One thing I didn’t handle here is an external log file which is handy if you need to change logging level etc.  It turns out to be as simple as moving log4j.properties to our /src/main/config directory and setting the flag to add our /conf directory to the class path. Works like a charm.

Apache Camel for home monitoring.

It’s pretty interesting monitoring stuff at home and over time I’ve put together scripts to handle everything reliably so I can concentrate on the data. What it isn’t is pretty: there are several languages and libraries and some cron jobs to make it work, plus some stuff off the internet (I don’t fully understand). So, as part of a project with Apache ActiveMQ at work I was interested to find the sub-project Apache Camel: it’s a sort of integration engine, a switchboard between technologies that uses a ruleset to say what goes where.

Camel is intriguing because it opens the door to something, perhaps better, that can be more configured than built, but without being a dead-end if I need something particular. Now Camel is really designed as a framework using Java beans to specify the routing and manipulate the message contents. But, once built, with the appropriate connectivity, a Camel application can use just XML to describe the rules. So, in theory, I can take a Camel example, add the connectivity I want and then configure the system as needed. That last bit’s important to me; I don’t want to swap my scripts for a load of Java. The trade-off is that I may have to be more inventive to get it to do what I want via configuration.

The new world

The best place to start is with a test scenario:

  1. Get temperatures from the 1-wire system (exposed through OWFS as folders and files).
  2. Publish them onto a MQTT broker. I’ve blogged about doing this one already using Tcl, though I had to write the mqtt client code myself.
  3. Send them to MySQL for further analysis.

First steps is to get a copy of Camel and you’ll need Maven if you haven’t already got it installed.

  • Go to the examples and find the ‘camel-example-console’.
  • Open a terminal there – we’ll need it in a moment.
  • At the console type mvn compile to set it up and then mvm exec:java to start it using the local Jetty.

It’s really simple: you can type some text in and get it back in uppercase – not too exciting but lets us know it works. We’re just using it as a base.

For the test, we’ll need to extend the abilities of the example with things we want, like SQL and MQTT. So, open pom.xml and add the following dependencies:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-mqtt</artifactId>
</dependency>
<dependency>
  <groupId>org.apache.camel</groupId>
  <artifactId>camel-script</artifactId>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-jdbc</artifactId>
    </dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-sql</artifactId>
    </dependency>
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>5.1.9</version>
</dependency>

Call mvn compile again to recompile the example. From now on we’re just using configuration, so look in the directory target/classes/META-INF/spring for the file camel-context.xml.  This is where the magic happens.

At the moment the only route there is this one :

  <route>
      <from uri="stream:in?promptMessage=Enter something: "/>
      <transform>
        <simple>${body.toUpperCase()}</simple>
      </transform>
      <to uri="stream:out"/>
  </route>

It’s pretty clear: stream uri is the console, and we get an input message from it and output it back. In the middle we transform the content to uppercase using the “simple” language. Of course, <from> could be a lot of things: ftp, files, uri, rest, etc and <to> could be those same things or say twitter or sql or web sockets. Also, since Camel is a switchboard, we can route the same  <from> to a series of <to>s in parallel, or series and if we want to, we can transform the message itself.

Getting File data in

Back at our test (but excited I hope by the possibilities), step 1 is to get regular readings from the “files” that are in OWFS: welcome to the File component.

The File component polls directories for files and sucks them up into the route. The component is really designed to consume files not re-read the same one, so we have to tell it a few specific things about the OWFS directory via the uri:

  • noop=true. Don’t delete the file.
  • readLock=none Don’t try to lock the file.
  • idempotent=false Keep re-reading the same file (default is just once).
  • delay=10000 Only read the directory every 10s.
  • antInclude=*/temperature Use Ant syntax to just pick out temperature files.
  • recursive=true Dive into sub directory.

This gives us the following <from> :

<from uri="file:/Users/mark/work/test?noop=true&readLock=none&idempotent=false&delay=10000&antInclude=*/temperature&recursive=true"/>

Ok, so we now have readings trooping into our route. However, it’s worth noting that the message is currently a file handle object wrapped in an envelope that contains various headers and properties. That means that for both an MQTT and SQL destinations, we’ll need to get hold of the actual contents. It also means we can use the headers to send meta-data along with the file.

Headers and message bodies

At the moment all we have are a stream of readings, but we also have the CamelFilePath header telling us where in OWFS they came i.e. lounge/temperature, office/temperature etc. If we get the first folder from the path, we can use it to create a topic on the fly in the <out> route later e.g. temperatures/lounge and as a key in SQL. Luckily, Camel lets us inject a script language to do the lifting:

<setHeader headerName="topic">
<javaScript>
    request.getHeader('CamelFileName').split('/')[0].toLowerCase();
</javaScript>     
</setHeader>

Here’s another one we’ll need later using the in-built <simple> (it really is) language:

<setHeader headerName="UTFDateTime">
<simple>
    ${date:now:yyyy-MM-dd hh:mm:ss}
</simple>
</setHeader>

Lastly, we need to make sure that the message body is the current reading as a string:

<transform>
<simple>${bodyAs(String)}</simple>
</transform>

So, we’ve the message body set up and the two ‘variables’ we’ll need later. Time to do some routing!

Sending to MQTT

First stop is the MQTT broker I’ve got running. I won’t go into the ins-and-outs of brokers; they’re very useful, you’ll just have to believe me.

There is a MQTT Component, so it should be as simple as this to talk to the local broker:

<to uri="mqtt:test:?publishTopic=temperatures.${header.topic}"/>

Sadly, that’s not the case. Camel doesn’t let you do this sort of thing in XML. All is not lost however. We can use the Recipient List from the Patterns routing recipes. Recipient List lets you send to a given list of outputs but more importantly, lets you use scripting to build them. It’s a bit of a bodge to make a list of one thing but it works.

Since it’s a bit more complicated now, we’ll split out the MQTT stuff into a sub-route (via the direct: uri) that we can call from the main one. Anyone who’s spent any time writing Ant scripts will recognise this game right away…

<route id="R1">
    <from uri="direct:mqtt"/>
        <recipientList ignoreInvalidEndpoints="true" >
            <javaScript>'mqtt:test?publishTopicName=temperatures.' + request.getHeader('topic');
            </javaScript>                               
        </recipientList>
</route>

Sending to MySQL

We can use exactly the same idiom to send the data into SQL. We’ll use the SQL Component in the same way, but first we need a bit of setup to go in above the <routes> to tell Camel about the jdbc connection to MySQL (this apparently is standard Spring):

<bean id="myDS" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value="com.mysql.jdbc.Driver" />
    <property name="url" value="jdbc:mysql://localhost:3306/1wire" />
    <property name="username" value="user" />
    <property name="password" value="password" />
</bean>

There are fancier ways to do this with pooling etc, but this will do for us. All that’s left is to make up an INSERT statement to drop readings into the 1wire database, where we’ve got these three fields in the readings table:

  • reading  – FLOAT.
  • topic – VARCHAR(32).
  • tdate – DATETIME.

Again we’ll use the subroutine approach to factor out this destination and to keep the xml in manageable chunks:

<route id="R2">
       <from uri="direct:mysql"/>
             <recipientList ignoreInvalidEndpoints="true" >
                <javaScript>
                  "sql:insert into readings (reading,topic,tdate) values(" + request.body + ",'" + request.getHeader('topic') + "','"+ request.getHeader('UTFDateTime') + "')?dataSourceRef=myDS";  
                </javaScript>                                
            </recipientList>  
   </route>

That’s it: call the named datasource with the statement.  Now it’s time to put it all together.

Pipelines and Multicast

One gotcha from Camel is that multiple endpoints are chained together by default. So if you just list <to a> <to b> <to c> then the return from a is sent to b etc. This can lead to some very strange results, depending on your component! What we need instead is a <multicast>. This sends a copy of the message to each recipient. Knowing that, we can construct the whole route like so:

<!-- Poll for temperatures and route to mqtt and sql -->      
<route id="GetTemperatures">
<from uri="file:/Users/mark/work/test?noop=true&amp;readLock=none&amp;idempotent=false&amp;delay=10000&amp;antInclude=*/temperature&amp;recursive=true" />
<setHeader headerName="topic">
  <javaScript>
      request.getHeaders().get('CamelFileName').split('/')[0].toLowerCase();
  </javaScript>     
</setHeader>
<setHeader headerName="UTFDateTime">
    <simple>${date:now:yyyy-MM-dd hh:mm:ss}</simple>
</setHeader>
<transform>
    <simple>${bodyAs(String)}</simple>
</transform>
   <multicast stopOnException="true"> 
          <to uri="direct:mqtt"/>
          <to uri="direct:mysql"/>
   </multicast>        
</route>
<!-- Push the message to MQTT using topic in header -->
  <route id="Sub_1">
            <from uri="direct:mqtt"/>
            <recipientList ignoreInvalidEndpoints="false" >
                <javaScript>
                    'mqtt:test?publishTopicName=temperatures.' + request.getHeader('topic');
                 </javaScript>                               
            </recipientList>
   </route>

<!-- Push the message to SQL using topic and date in header --> 
   <route id="Sub_2">
       <from uri="direct:mysql"/>
             <recipientList ignoreInvalidEndpoints="false" >
                <javaScript>
                  "sql:insert into readings (reading,topic,tdate) values(" + request.body + ",'" + request.getHeader('topic') + "','"+ request.getHeader('UTFDateTime') + "')?dataSourceRef=myDS";  
                </javaScript>                                
            </recipientList>  
   </route>

If you’ve got MQTT and MySQL, then you can add this route to the config file and restart the example to see it in action. If you don’t you can use the <log> or <stream> endpoints instead to just print out the results to the console.

Thoughts and ideas

So, 35’ish lines of Camel config gets us a polling file reader and outputs to MySQL and MQTT – that’s not bad. Also we now have a single place to do logging and a fairly simple idiom that can be expanded if other components are needed. Moreover, if as the base, we’d used something like the servlet-tomcat example to built a web app, we’d have the basis of a neat little switchboard with a web interface thrown in. Everything is rosy? not quite:

  • It’s evident that the Camel docs lean heavily towards the java, beans, maven side rather than the config side. It can be difficult to get information. For example it was a few hours of internet wandering before I found out about Recipient List.
  • The lack of variables or replacing properties in uri’s makes routing less straightforward than it should be. Luckily the user groups are pretty friendly.

Having said that it works, and if a Twitter output was needed or a websocket feed than it’s not much harder to add them.

Ok, so is this easier than using a scripting language I know well? Obviously, not at first but as someone who does a lot of XML anyway it’s just as viable. Now, if I didn’t know how to write scripts, then this idea becomes pretty useful.

I could see perhaps a pre-compiled version of Camel webapp with common routes set up being a really easy start point for the home measurement community (uncomment to us X etc) and perhaps more approachable than say ‘download ruby and then do…’.

Maybe what the Internet of Things needs is a good switchboard.

Sensors updated.

Hmmm, this is harder than I thought. You take a mouse apart and get to the innards. The idea is that you can somehow get it to see the disk spinning around. However, the sensors come in pairs that face each other. Now with my tech head on, I’d unsolder a couple and make something up as loads of people have done already. However, with my “simple system” restrictions it’s more complicated. I played around with using internal reflection to rotate the sensor path through 90 degrees using a plastic fork. It didn’t work very well to say the least.

Down the road, I’m still stuck with how to use the PS/2 output 2oft away where the server is. Serial protocols are good as they work over a fair distance. With mice etc though you have to send initialization up to the device first, which is why you can’t plug a mouse in once a PC boots up. Basically it just gets harder.

This is why there are CurrentCost meters, custom electronics, X10 & ZigBee systems, WiFi stuff, Arduino boards in the mix: hi-tech custom solutions. More thought required, but I’m not giving up just yet.

Simple sensors

Thinking about adding a sensor to my electricity meter, I started looking at what others had done. There is some great stuff out there and a lot of  CurrentCost hookups. One thing that struck me was that the bar was set pretty high in terms of technology: hacking bits of hardware, Arduino boards, programming etc. That led me to pondering some sort of hookup that could use stuff people were likely to have and was low-tech enough to put together. What’s in the shed then?

  • Old phones. Everybody has at least one (or more).
  • Mice (PS/2 , serial not the squeaking sort).
  • Junk, toys, old radios and other bit and bobs.

So, first question – can you read a meter using a mouse but without unsoldering the ir sensors? Ive got an old wheel-mouse in the junk box. Time to experiment.