SimpleXist III
Last time, we finished with a working example on a small XQuery engine based on a simplified eXist install called, unimaginatively, SimpleXist. Now we could leave it there as it’s true that you can write a whole application with just the database and XQuery. However, once forms are involved, it’s so much easier to use XForms rather than native HTML which is of course also the front bit of the XRX architecture. In our case we use XSLTForms to do our XForms on the client side, although there are also good ways to do it server-side with something like BetterForms.
Note: I’m not getting into a XForms / XRX how-to just now. There are numerous resources such as Dan McCreary’s XRX Beginners Guide and there is an excellent WikiBooks XForms section to name but two you can read if unfamiliar with the topic.
Now with our initial project, we used a transform to render the output page at the end of each XQuery. This is a pretty useful idiom and we’d want to keep it going with our XForms. This means that our XSL style sheet needs modifying so that it can cope with two sorts of input, straight xml and xforms.
Simple XForm
Before we can do anything we need the XSLTForms software from the agenceXML site. Download it and put it in /ROOT/resources folder as this probably isn’t the only project that’s going to use it! They also have a simple HelloWord XForm on their site we’ll use as a template. Have a look at the source; what we’re going to do is make that work in SimpleXist. So, first off, create a hello.xql in /project1 and copy in the following:
xquery version "1.0";
declare namespace transform="http://exist-db.org/xquery/transform";
declare option exist:serialize "method=xhtml media-type=text/xml indent=yes";
let $input :=
<document xmlns:xf="http://www.w3.org/2002/xforms">
<header>
<title>Hello World in XForms</title>
<xf:model>
<xf:instance>
<data>
<PersonGivenName />
</data>
</xf:instance>
</xf:model>
</header>
<content>
<p>Type your first name in the input box.<br />
If you are running XForms, the output should be displayed in the output area.
</p>
<xf:input ref="PersonGivenName" incremental="true">
<xf:label>Please enter your first name: </xf:label>
</xf:input><br/>
<xf:output value="concat('Hello ', PersonGivenName, '. We hope you like XForms!')">
<xf:label>Output: </xf:label>
</xf:output>
</content>
</document>
return
transform:transform($input, "styles/style.xsl", ())
There are a couple of things to note here. First, we’re returning text/xml as the mime-type rather than text/xhtml as that’s what triggers the XForms XSL transform on the client. Next, we’ve got a namespace in our document section, and lastly, we’ve now got a separate <header> element in our XML as the <xf:model> parts of the form need to go in the HTML <head> element in the output.
Now if you try to run it at localhost:8080/project1/hello.xql , nothing interesting happens as we don’t have the XSLT to support it: that’s next.
XForms XSLT
First off, copy this style.xsl over the one in /project1/styles. It’s a small addition to find out if we’re in an XForm or not and send the right processing-instructions back. This is done by testing for xf:model in the header. Note that we’ve used /resources/xsltforms.xsl as we’re using the /ROOT/resources folder.
<xsl:if test="document/header/xf:model"> <xsl:processing-instruction name="xml-stylesheet"> href="/resources/xsltforms/xsltforms.xsl" type="text/xsl" </xsl:processing-instruction> <xsl:processing-instruction name="css-conversion"> no </xsl:processing-instruction> <xsl:processing-instruction name="xsltforms-options"> debug="no" </xsl:if>
We can also set other specific XSLTForms directives in here, such as setting support for ordinary css files and turning off debug. Using this code, the standard XQuery will be returned as before and the XForms version with appropriate processing instructions.
Now if you run the hello.xql file again you should get a working XForm!
Next steps: HerdBook
Now we’ve a set-up that’s simple, works with XQueries on the file system and supports XForms. Next stop a personal project to exercise the setup a bit more fully. I’ve been looking to scratch this itch for a while – I need a Herd Book. This is a database of all the llamas and alpacas we have, matings, medicines we’ve given them, shows they’ve been to and results. In short everything we have currently written in various calendars, notebooks and a set of index cards. It’s way overdue!
SimpleXist II
A couple of days ago I did a post on a simple eXist setup – lazily called SimpleXist . As I left it, it was a running system but there were a few shortcomings and complaints when it runs :- a) No JSP support, b) No log4j config, oh and the minor matter of no eXist client to talk to the database! So it’s time for a bit more configuration and then we’ll take it for a spin with a couple of simple examples.
As per last time if you want the version described here, you can just download it.
Errors & Warnings
This is what I get currently from the system on startup :
macbook:Test tingenek$ java -jar start.jar 2012-01-01 17:35:08.111:INFO::Logging to STDERR via org.mortbay.log.StdErrLog 2012-01-01 17:35:08.237:INFO::jetty-6.1.26 2012-01-01 17:35:08.503:INFO::NO JSP Support for, did not find org.apache.jasper.servlet.JspServlet log4j:WARN No such property [append] in org.apache.log4j.ConsoleAppender. 2012-01-01 17:35:10.531:INFO::Started SelectChannelConnector@0.0.0.0:8080
Now the JSP issue is easy, you can either add in the jsp-2.1 folder into Jetty /lib, or just ignore it – it does no harm. The logging just needs a bit more configuration.
The eXist software uses the log4j system to write sophisticated logs for various modules. In our case what we wanted was just simple logging to the console with everything else. It’s easy to setup. First create in /ROOT/WEB-INF a folder called /classes. In that folder put this file, log4j.xml. All it does is create an appender to pipe everything to the console. Note the use of the “priority” setting in the <root> element. This controls how much logging eXist does, at the moment it’s just errors but you can dial it up to “info”, “warn” or even “debug” if you really want to see everything as it happens.
eXist Client
It may not be obvious, but the eXist client is run from the start.jar in the .war file /lib directory. So, first step is to copy it, renaming it to client.jar and add it to SimpleXist. You should now have start.jar and client.jar in the same folder. If you try and run it, you’ll find it complains about not being able to find eXist root and bombs. What’s needed is a small shell file to fire it up, so make a file called called client.sh, and paste the following in:
#!/bin/bash java -Dexist.home=webapps/ROOT/WEB-INF -jar client.jar client
Now if you run this, it’ll complain about not finding lib files in various folders; that’s because we have a flat lib, but it’ll work and you’ll get the eXist client. Make sure you alter the client URL as we’ve used localhost:8080/xmlrpc rather than /exist/xmlrpc. Also, default .war database install doesn’t have an admin password – remember to set one once you’re in!
Note. I might get around to repackaging the client jar at some point so it doesn’t complain, but at the moment it’s not a priority.
Before we begin – housekeeping.
As with everything, it’s always good to have some sort of system in place; everyone’s happier when the expected occurs. Now, this is just the way I’ve set this up, feel free to do something different, the key is predictability. So, for SimpleXist there is:- A folder off from /ROOT for each project.
- A /resources folder off /ROOT holds common files, like XSLTForms, TinyMCE
- A /resources folder in each project holds resources for that project i.e. /css /scripts /images
- A folder called /styles holds any xslt files.
- The XQuery files are in each project root.
- In the eXist database a collection with the same name as the project off from a /projects collection.
Getting by without pipelining.
In anything but the most trivial system, we’ll need some sort of framing for the site: the header, footer, navigation etc that we don’t want to repeat per page. Now a common route is to use a pipeline with the XML being turned into HTML through Cocoon,XProc, URLRewriting etc. In SimpleXist this isn’t possible as we’ve got none of those things – though, we could have, but choose not to is more accurate. To get around that we’ll use the excellent transform module in eXist to convert our XQuery into HTML for us at the end of each XQuery using an XSL transform, like so:
First, create a new folder under /ROOT called project1. Now create an index.xql file with the following content:
xquery version "1.0"; declare namespace transform="http://exist-db.org/xquery/transform"; declare option exist:serialize "method=xhtml media-type=text/html indent=yes"; let $input := <document> <content> <h2><a name='h1'>Test</a></h2> <p>Hello from XSLT</p> </content> </document> return transform:transform($input, "styles/style.xsl", ())
This XQuery sets up a simple bit of XML with a mime-type of html and then calls a transform against it from our /styles folder, returning the results. To get this to work, create a /styles folder in /resources. In /styles put the following style.xsl file. There is also a tiny amount of css so you’ll also need /css and site.css. If you have a look at the XSL file, you’ll see that it simply pulls the document/content element out of the XQuery into the appropriate place in a simple piece of HTML. Now a quick look at localhost:8080/project1 should give you a plain’ish piece of html.
Now this might seem trivial but it just requires a bit more structure and CSS and you’ve the basics of a web site. Even better, if you look at the XQuery, you’ll see that that bit of XML could just as easily come from the eXist db. In fact, you could run a whole site from this bit of code, just by giving it the id of the bit of xml you want to drop in. We’ve got a bit of a WordPress vibe starting 
Using the database.
Fire up your eXist client (you created a password for admin didn’t you?) and go to File->Create Collection. Create a /projects collection and inside that a /project1 collection. In /project1 use File->Create Blank File, call it 1.xml and inside it copy :
<document> <content> <h2><a name='h1'>Page 1</a></h2> <p>Hello from page 1</p> </content> </document>
Make a couple more, excitingly called 2.xml and 3.xml. Now alter the index.xql file to pull the content from the database:
xquery version "1.0";
declare namespace transform="http://exist-db.org/xquery/transform";
declare option exist:serialize "method=xhtml media-type=text/html indent=yes";
let $data_collection := '/db/projects/project1/'
let $q := request:get-parameter('q', '1')
let $file := concat($data_collection,$q,'.xml')
let $error :=
<document>
<content>
<h2><a name='h1'>Oops</a></h2>
<p>I can't find {$file} in the database</p>
</content>
</document>
let $input := if(doc-available($file) = true())
then (doc($file)) else $error
return
transform:transform($input, "styles/style.xsl", ())
What happens here is that we look for a request parameter ‘q’. If it doesn’t exist, we set it to 1. Next we make up a path in the collection to from the root and the parameter to get a file path like /db/projects/project1/1.xml. Next, we check a document exists for that path, if it doesn’t we use an error doc, otherwise we fetch it. Lastly it gets transformed and returned.
If all goes well, you can now use localhost:8080/project1/index.xql?q=1 or the rather terser /project1/?q=1 to get document 1.xml etc.
This is one of the things I like about the XRX system: simple, clean XML data and small amounts of XQuery code give you (with a bit more work) the tools to put a site together!
What’s missing now is the XForms side. Next time..
SimpleXist
Over the last few weeks I’ve been looking at introducing the joys of the XRX world using eXist as we look to move away from Cocoon. One problem I’ve found is that eXist defaults to presenting, not only the eXist database, but a cornucopia of XML technologies. It’s an excellent showcase of things like XProc and not one, but two XForms implementations as well as concepts such as having the code and data in the database. But, if you’re used to putting code in a folder it can be tricky to know where to start and this gets in the way of the goal: learning about XQuery, XForms, and the database.
So, I’ve set up a simpler version called SimpleXist that either hides or removes most of the harder topics. The key things I’ve gone for are:
- Minimal servlets, only XQuery, REST (optional) and XML-RPC (for the client)
- XQueries run from the disk rather than the database.
- No URL-Rewriting.
- Can be locally deployed to a workstation but will work on a server.
I’ve written up the configuration below, but if you want to just download it then here you go. I’ve purposely not taken the route of repackaging jars or reducing their number. Everything is there, just not turned on. It’s all done with configuration and thus saves me embarrassing myself with my limited java skills.
Dear eXist gurus, I know there are other options, you can do a minimal recompile etc, etc. I’ve taken this route as it fits in with what we need, it’s easy to maintain and fits into the deployment and development methods we use.
Jetty Minimal
First step was a minimal Jetty. It’s not critical to use Jetty but it gives a nice small container to run eXist that works on a desktop. I’ve used a download of Jetty 6 (6.1.26) as the start point. Put Jetty somewhere; we only need a few files from it and then create a a folder for SimpleXist. In that folder make subfolders for /etc, /lib and /webapps. In the /webapps directory create a subfolder called /ROOT and in that create a folder /WEB-INF. Lastly in that folder create a folder /data.
Keep an eye on the folder structure shown on the left. This is what you’re heading towards.
Now, copy start.jar from Jetty to the SimpleXist folder. Next, Put jetty-6.1.26.jar, jetty-util-6.1.26.jar and servlet-api-2.5-20081211.jar into /lib. Lastly, in /etc put this jetty.xml file.
This is enough to set up a simple server on port 8080, which will run any webapp deployed in /webapps – in our case the default ROOT webapp. This is one of many reasons to love Jetty.
You can try it out with the command java -jar start.jar . Point your browser to http://localhost:8080. It won’t be interesting or pretty but it should work.
Now we need to set up a simplified eXist to run in our container.
eXist Install
As with Jetty, download a copy of eXist. In this case I used the setup jar so I had a working install of the full system as a start point. Next, in the eXist root directory, run build dist-webapp which will create a .war file in the eXist /dist directory. I’m using war format because a) I like all my lib files in one directory and b) because it makes the app easy to move to something like Tomcat, which we use for production, later on.
Open up the .war with your favorite unzip utility and from it’s WEB-INF folder copy conf.xml and the /lib folder into /ROOT/WEB-INF. Next, download this web.xml into WEB-INF
Web.xml is the meat of the system. It sets up:
- The eXist XQuery servlet to handle anything ending .xqy or .xql
- The XML-RPC servlet to handle the eXist client connection at /xmlrpc.
- The eXist Servlet to start the database and respond to the REST commands at /rest
- The default servlet to handle resources i.e. a catchall servlet for anything else. Note this is Jetty specific.
- Lastly it looks for index.xql for the welcome file if nothing is requested at all.
Finally, we’ll create the ubiquitous Hello World to test it all out. Make an index.xql file in /ROOT with the following content:
xquery version "1.0";
declare option exist:serialize "method=html media-type=text/html";
let $hello := 'Hello World!'
return
<html>
<body>{$hello}</body>
</html>
That’s it. You can now run java -jar start.jar and you should see the famous message at localhost:8080. The Jetty container will complain a bit about logging, there isn’t a client and a couple of things need tidying up, but basically we’re there. Next time for those bits and some development examples.
MQTT with Lotus Notes
Previously, I’ve used Apache Ant with MQTT Publish/Subscribe to keep track of overnight processes. Since then (2 years – amazing) The list of things talking to the sturdy RSMB broker has grown steadily. What it has lacked up ’til now was a means to get Lotus Notes messages from LotusScript agents.
One of the issues with Notes is that you can’t open network connections, so normal TCP/IP games are out. However, you can run java agents and they can load external libraries. The majority of agents are however written in Notes’ own scripting language, LotusScript, and that’s the situation here. I didn’t want to rewrite a load of stuff and I wanted something that could be dropped into an existing agent.
What I didn’t know about was the star of this show: LS2J the “LotusScript to Java Bridge” which allow you to interact with methods and properties of a java class from LS as if they were native. It’s a clever bit of kit, with a few caveats and the following bits of code show how it works for a simple MQTT class.
If you want to download it, I’ve wrapped all this up in a single NSF for convenience.
Java Class
So, first off you need an empty database and a new Java Script Library entry. I’ve called mine MQTT_Notes. You’ll need to use ‘Edit Project’ to pull in the wmqtt.jar file from the IA92 Java Client into the database.
I’ve kept the java side very simple. There are three methods: initialize, execute and disconnect to send a QoS 0 publication to a broker. You’ll also notice I’m not doing any try/catch error handling. This is because the errors are being sent up to the LS level above.
The code is listed below, any real java coders, please feel free to tut and shake your heads.
import com.ibm.mqtt.MqttClient;
import com.ibm.mqtt.MqttException;
public class MQTT_Notes {
private final static boolean CLEAN_START = true;
private final static boolean RETAINED = false;
private final static short KEEP_ALIVE = 30;
private final static short QOS = 0;
private String name="mqttnotes";
private String url="tcp://localhost:1882";
private String topic="mqttnotes";
private String payload="hello from notes";
private MqttClient client=null;
public void initialize(String strURL, String strName) throws MqttException {
name = strName;
url = strURL;
client = new MqttClient(url);
client.connect(name, CLEAN_START, KEEP_ALIVE );
}
public void execute(String strTopic, String strPayload) throws MqttException {
topic = strTopic;
payload = strPayload;
if (client != null) {
client.publish(topic,payload.getBytes(),QOS,RETAINED);
}
}
public void disconnect() throws MqttException {
if (client != null) {
client.disconnect();
}
}
}
LotusScript Class
Next up, you’ll need a new LotusScript Library. I’ve called mine Class_MQTT. This wraps up the calls into LS2J and our java class into a handy drop-in LotusScript class. The key lines are in the Options section:
- UseLSX “*javacon” which loads the LS2J bridge code.
- Use “Java_MQTT” which loads our java code from above.
The code is a standard LS Class, called MQTTSimple, with the addition of some new types.
- JavaSession is the link to the Java machine. It lets us find the classes we want to use. In this case “MQTT_Notes”.
- JavaClass is the handle to the class.
- JavaObject is the handle to the running object, using the class.
- JavaError lets us get errors from the underlying java into the LS On Error model.
Notice that there is an explicit Delete method in the code which is called when the LS object is destroyed to make sure the MQTT connection is closed. The code is listed below:
'Class_MQTT:
Option Public
Option Explicit
Uselsx "*javacon"
Use "Java_MQTT"
Class MQTTSimple
Private MQTTObj As JavaObject
Private MQTTError As JavaError
Private js As JavaSession
Sub New(url As String, title As String)
Set js = New JavaSession
Dim MQTTClass As JavaClass
On Error Goto ErrHandler
'Get our java class
Set MQTTClass = js.GetClass ("MQTT_Notes")
'Instantiate it.
Set MQTTObj = MQTTClass.CreateObject ()
'Call our initialize method to get a MQTT connection
'Takes connection url and connection handle
Call MQTTObj.Initialize (url, title)
Exit Sub
ErrHandler:
Set MQTTError = js.getLastJavaError()
Print MQTTError.ErrorMsg
Call js.ClearJavaError()
Resume Next
End Sub
Sub Delete
'Called as the object is destroyed.
'Makes sure MQTT connection is shut.
On Error Goto ErrHandler
Call MQTTObj.disconnect()
Exit Sub
ErrHandler:
Set MQTTError = js.getLastJavaError()
Print MQTTError.ErrorMsg
Call js.ClearJavaError()
Resume Next
End Sub
Sub Exec(topic As String, payload As String)
'Publish given payload to given topic
Call MQTTObj.execute(topic,payload)
Exit Sub
ErrHandler:
Set MQTTError = js.getLastJavaError()
Print MQTTError.ErrorMsg
Call js.ClearJavaError()
Resume Next
End Sub
End Class
This code gives us a new LS MQTTSimple class which takes a URL and connection name on creation.
It has one visible method, exec, which takes a topic and payload string.
Notice that the errorhandling in the code uses the standard LS error handler, but gets it’s error information from the underlying java.
It’s not quiet seamless though as you have to cancel the error yourself.
Test Script
Lastly, there is a test script to test it all out. To use the MQTT class, simply add it into the Options section. The little agent below opens the mail file of the user and publishes it’s size in Mb. Assuming you have a broker running and some sort of client subscribed, the following should work. I used the RSMB broker on my laptop on port 1882 and the IA92 client subscribed to the Notes topic.
Option Public
Option Explicit
Use "Class_MQTT"
Sub Initialize
Dim mqtt As MQTTSimple
Dim db As NotesDatabase
'Start the connection: URL and Connection string
Set mqtt = New MQTTSimple("tcp://localhost:1882","NotesAgent")
'Open user's mail file
Set db = New NotesDatabase( "", "" )
Call db.OpenMail
'Publish database size in Mb to MQTT in the Notes topic at Qos0
Call mqtt.Exec("Notes", "Mail database " & Format$(db.Size/1000000,"000,###")) )
End Sub
This is of course a bare bones system, but I can see a lot of uses: Mail-in databases, overnight agents, maintenance tasks etc. The best thing is that MQTT allows me to tie in all these disparate systems easily.
MQTT Ant task
Building an Ant Task
With an interest in publish-subscribe with MQTT , it seems a natural choice to try and get some other tools I use to talk to the microbroker we’ve got running. I chose Ant, partly because I use it for loads of things and because there is a handy java client library for MQTT to make things easier.
Sensors updated.
Hmmm, this is harder than I thought. You take a mouse apart and get to the innards. The idea is that you can somehow get it to see the disk spinning around. However, the sensors come in pairs that face each other. Now with my tech head on, I’d unsolder a couple and make something up as loads of people have done already. However, with my “simple system” restrictions it’s more complicated. I played around with using internal reflection to rotate the sensor path through 90 degrees using a plastic fork. It didn’t work very well to say the least.
Down the road, I’m still stuck with how to use the PS/2 output 2oft away where the server is. Serial protocols are good as they work over a fair distance. With mice etc though you have to send initialization up to the device first, which is why you can’t plug a mouse in once a PC boots up. Basically it just gets harder.
This is why there are CurrentCost meters, custom electronics, X10 & ZigBee systems, WiFi stuff, Arduino boards in the mix: hi-tech custom solutions. More thought required, but I’m not giving up just yet.
Simple sensors
Thinking about adding a sensor to my electricity meter, I started looking at what others had done. There is some great stuff out there and a lot of CurrentCost hookups. One thing that struck me was that the bar was set pretty high in terms of technology: hacking bits of hardware, Arduino boards, programming etc. That led me to pondering some sort of hookup that could use stuff people were likely to have and was low-tech enough to put together. What’s in the shed then?
- Old phones. Everybody has at least one (or more).
- Mice (PS/2 , serial not the squeaking sort).
- Junk, toys, old radios and other bit and bobs.
So, first question – can you read a meter using a mouse but without unsoldering the ir sensors? Ive got an old wheel-mouse in the junk box. Time to experiment.
