Friday, December 31, 2010

XML validation using a Schema

In earlier posts, I examined how we can use the built-in revXML library in LiveCode to validate XML data against a DTD, later refining it with a version check to match evolving requirements. Unfortunately, a Document Type Definition is quite a limited way of XML validation. So this time, we'll improve our defenses again, by incorporating XML Schemas.

Whereas a DTD is limited to defining the basic structure of the XML in terms of elements and attributes, XML Schemas allow you to define validation on the actual content of the elements and attributes. So you can be sure that an element defined as "xs:date" is actually a valid date, or that an attribute defines as "xs:positiveInteger" is actually a positive integer, etc. A full explanation of XML Schemas is beyond the scope of this post, you'll find plenty of information around the web - a good first stop is this W3Schools XML Schema tutorial.

This all sounds very good, but here's the rub: the revXML library offers no built-in support for XML Schemas. So yet again we turn to Java, with its built-in XML Validation API. We can easily execute Java code using LiveCode's shell function - so let's start by writing the XmlValidateSchema class:
import java.io.File;
import java.io.IOException;

import javax.xml.XMLConstants;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.Source;
import javax.xml.transform.dom.DOMSource;
import javax.xml.validation.Schema;
import javax.xml.validation.SchemaFactory;
import javax.xml.validation.Validator;

import org.w3c.dom.Document;
import org.xml.sax.SAXException;

public class XmlValidateSchema {
public static void main(String[] args) throws SAXException, ParserConfigurationException, IOException {
final File xmlFile = new File(args[0]);
final File xsdFile = new File(args[1]);
// Load the XML file
final DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
final DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();
final Document document = docBuilder.parse(xmlFile);
// Load the XSD file
final SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
final Schema schema = schemaFactory.newSchema(xsdFile);
final Validator validator = schema.newValidator();
// Validate the XML document against the XSL schema
final Source source = new DOMSource(document);
validator.validate(source);
}
}

In keeping with earlier Java examples, the code is a bit lazy when it comes to exception handling: if any exception is thrown, it will end up in the output of our shell function call. The only important thing to remember is that the first parameter is the XML file, and the second is the XML Schema Definition (XSD) file.

Let's go to LiveCode and create a new stack for the user interface.



As you can see, there's a field for the Schema text, a field for the XML text, and a button to Validate the XML against the Schema. Since it's perhaps a tad small, here's the content of the Schema field:
<?xml version="1.0"?>
<xs:schema
xmlns:xs="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://www.quartam.com"
xmlns="http://www.quartam.com"
elementFormDefault="qualified">

<xs:element name="RootNode" type="RootNode"/>

<xs:complexType name="RootNode">
<xs:sequence>
<xs:element name="BranchNode" type="BranchNode" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="SpecVersion" type="xs:string"/>
</xs:complexType>

<xs:complexType name="BranchNode">
<xs:sequence>
<xs:element name="LeafNode" type="xs:string" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>

</xs:schema>

And here's the content of the XML field:
<?xml version="1.0"?>
<RootNode
xmlns="http://www.quartam.com"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.quartam.com schema.xsd"
SpecVersion="1.0">
<BranchNode>
<LeafNode>The first leaf node</LeafNode>
<LeafNode>The second leaf node</LeafNode>
<LeafNode>The third leaf node</LeafNode>
</BranchNode>
<BranchNode>
<LeafNode>The fourth leaf node</LeafNode>
<LeafNode>The fifth leaf node</LeafNode>
</BranchNode>
</RootNode>

After saving the stack, we copy the compiled XmlValidateSchema.class file into the same directory as the stack. Now we can write the script for the 'Validate' button:
on mouseUp
--> write Schema and XML to temporary files
local tSchemaFile, tXmlFile
put the tempName into tSchemaFile
put field "Schema" into URL ("file:" & tSchemaFile)
put the tempName into tXmlFile
put field "XML" into URL ("file:" & tXmlFile)
--> assemble the shell command
local tShellCommand
put "java XmlValidateSchema" && \
ShellPath(tXmlFile) && \
ShellPath(tSchemaFile) \
into tShellCommand
--> execute the shell command
local tHideConsoleWindows, tDefaultFolder, tShellResult
put the hideConsoleWindows into tHideConsoleWindows
set the hideConsoleWindows to true
put the defaultFolder into tDefaultFolder
set the defaultFolder to AbsolutePathFromStack()
put shell(tShellCommand) into tShellResult
set the defaultFolder to tDefaultFolder
set the hideConsoleWindows to tHideConsoleWindows
--> cleanup the temporary files
delete file tSchemaFile
delete file tXmlFile
if tShellResult is not empty then
answer error tShellResult
end if
end mouseUp

function AbsolutePathFromStack pFileName
local tAbsolutePath
put the effective filename of this stack into tAbsolutePath
set the itemDelimiter to slash
if pFileName is not empty then
put pFileName into item -1 of tAbsolutePath
else
delete item -1 of tAbsolutePath
end if
return tAbsolutePath
end AbsolutePathFromStack

function ShellPath pPath
if the platform is "Win32" then
put quote & pPath & quote into pPath
else
replace space with backslash & space in pPath
end if
return pPath
end ShellPath

This time around, we didn't have to fiddle with the Java classpath, as the XML Validation API is built-in. However, we had to write the XML and Schema to temporary files, to avoid length limitations in the shell command. If we now make a deliberate mistake, say change one of the 'LeafNode' elements into a 'BeafNode' element, we see this error:



Again, the image is a bit small, so here's the content of the error:
ERROR:  'cvc-complex-type.2.4.a: Invalid content was found starting with element 'BeafNode'. One of '{"http://www.quartam.com":LeafNode}' is expected.'
Exception in thread "main" org.xml.sax.SAXParseException: cvc-complex-type.2.4.a: Invalid content was found starting with element 'BeafNode'. One of '{"http://www.quartam.com":LeafNode}' is expected.
at com.sun.org.apache.xerces.internal.jaxp.validation.Util.toSAXParseException(Util.java:109)
at com.sun.org.apache.xerces.internal.jaxp.validation.ErrorHandlerAdaptor.error(ErrorHandlerAdaptor.java:104)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:382)
at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:316)
at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(XMLSchemaValidator.java:429)
at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.reportSchemaError(XMLSchemaValidator.java:3185)
at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:1831)
at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(XMLSchemaValidator.java:705)
at com.sun.org.apache.xerces.internal.jaxp.validation.ValidatorHandlerImpl.startElement(ValidatorHandlerImpl.java:335)
at com.sun.org.apache.xml.internal.serializer.ToXMLSAXHandler.closeStartTag(ToXMLSAXHandler.java:205)
at com.sun.org.apache.xml.internal.serializer.ToXMLSAXHandler.characters(ToXMLSAXHandler.java:524)
at com.sun.org.apache.xml.internal.serializer.ToXMLSAXHandler.characters(ToXMLSAXHandler.java:467)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:229)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:215)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:215)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:215)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:121)
at com.sun.org.apache.xalan.internal.xsltc.trax.DOM2TO.parse(DOM2TO.java:85)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(TransformerImpl.java:615)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:661)
at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:300)
at com.sun.org.apache.xerces.internal.jaxp.validation.ValidatorImpl.process(ValidatorImpl.java:220)
at com.sun.org.apache.xerces.internal.jaxp.validation.ValidatorImpl.validate(ValidatorImpl.java:141)
at javax.xml.validation.Validator.validate(Validator.java:82)
at XmlValidateSchema.main(XmlValidateSchema.java:31)

Thanks to the combination of LiveCode and Java, we can develop cross-platform solution quickly, without having to give up the power of existing libraries. Unfortunately, loading Java every time for a shell call is not the optimal solution, so in another post, we'll investigate how we can run Java code using LiveCode's 'process' communication. Stay tuned...

Thursday, December 30, 2010

Stamping PDF files

In a previous post, I examined how we can use LiveCode and the Java-based iText library to concatenate a series of existing PDF files into a single PDF file. Now we will examine how we can 'stamp' a PDF file with an image using the same technique.

The first thing to code is the Java class that we will call using the shell function. Here's what I came up with:
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;

import com.lowagie.text.DocumentException;
import com.lowagie.text.Image;
import com.lowagie.text.Rectangle;
import com.lowagie.text.pdf.PdfContentByte;
import com.lowagie.text.pdf.PdfReader;
import com.lowagie.text.pdf.PdfStamper;

public class StampPdfFile {
public static void main(String[] args) throws IOException, DocumentException {
final String inputFile = args[0];
final String outputFile = args[1];
final String imageFile = args[2];
final String[] coords = args[3].split(",");
final PdfReader inputReader = new PdfReader(inputFile);
final OutputStream outputStream = new FileOutputStream(outputFile);
final PdfStamper outputStamper = new PdfStamper(inputReader, outputStream);
final int pageCount = inputReader.getNumberOfPages();
final Image image = Image.getInstance(imageFile);
final int left = Integer.parseInt(coords[0]);
final int top = Integer.parseInt(coords[1]);
final int right = Integer.parseInt(coords[2]);
final int bottom = Integer.parseInt(coords[3]);
final int height = bottom - top;
final int width = right - left;
image.scaleToFit(width, height);
for (int pageIndex = 1; pageIndex <= pageCount; pageIndex++) {
final PdfContentByte overContent = outputStamper.getOverContent(pageIndex);
final Rectangle pageSize = inputReader.getPageSize(pageIndex);
image.setAbsolutePosition(left, pageSize.getHeight() - bottom);
overContent.addImage(image);
}
outputStamper.close();
}
}

In a nutshell, the first parameter is the input file, the second the output file, the third the image file, and the fourth parameter is a comma-separated list of coordinates making up the target rectangle. As usual, the code is a tad lazy when it comes to faulty input parameters and exception handling - if there's a mistake you'll simply get the stacktrace as the output of the shell command.

The most important bit is in the loop over the pages, where we use the outputStamper.getOverContent() method to draw our image on top of the existing content. If you'd rather have the image in the back, as a watermark, you would use the outputStamper.getUnderContent() methopd instead. Also note that setting the image position coordinate system works from the bottomLeft of the page, so we have to use the original page height and subtract the bottom coordinate from it.

Now we can proceed with writing a LiveCode button script:
on mouseUp
--> determine the input, output and image files
local tInputFile, tOutputFile, tImageFile
put ShellPath(AbsolutePathFromStack("demo1.pdf")) \
into tInputFile
put ShellPath(AbsolutePathFromStack("stamp.pdf")) \
into tOutputFile
put ShellPath(AbsolutePathFromStack("Template.png")) \
into tImageFile
--> determine the image target rectangle
local tImageRect
put quote & "10,10,103,87" & quote \
into tImageRect
--> determine the class path
local tClassPath
if the platform is "Win32" then
put ".;iText-2.1.7.jar" into tClassPath
else
put ".:iText-2.1.7.jar" into tClassPath
end if
--> assemble the shell command
local tShellCommand
put "java -classpath" && tClassPath && \
"StampPdfFile" && \
tInputFile && \
tOutputFile && \
tImageFile && \
tImageRect \
into tShellCommand
--> execute the shell command
local tHideConsoleWindows, tDefaultFolder, tShellResult
put the hideConsoleWindows into tHideConsoleWindows
set the hideConsoleWindows to true
put the defaultFolder into tDefaultFolder
set the defaultFolder to AbsolutePathFromStack()
put shell(tShellCommand) into tShellResult
set the defaultFolder to tDefaultFolder
set the hideConsoleWindows to tHideConsoleWindows
if tShellResult is not empty then
answer error tShellResult
end if
end mouseUp

function AbsolutePathFromStack pFileName
local tAbsolutePath
put the effective filename of this stack into tAbsolutePath
set the itemDelimiter to slash
if pFileName is not empty then
put pFileName into item -1 of tAbsolutePath
else
delete item -1 of tAbsolutePath
end if
return tAbsolutePath
end AbsolutePathFromStack

function ShellPath pPath
if the platform is "Win32" then
put quote & pPath & quote into pPath
else
replace space with backslash & space in pPath
end if
return pPath
end ShellPath

Click the button, and it happily takes the existing PDF files (demo1.pdf), paints the image (Template.png) on top of all pages, and writes a new PDF file (stamp.pdf) in the same folder as our stack. There we have it, another example of using iText from within LiveCode.

Wednesday, December 29, 2010

Concatenating PDF files

Since the advent of LiveCode 4.5, developers have the ability to 'print' stack content directly to PDF files. And if you need pin-point control over what goes where, you can use Quartam PDF Library to generate PDF files from scripts. That's great if you are in full control of the content, but what if you need to work with existing PDF files? In the next few posts, we will examine how you can tap into the power of the Java-based iText library from LiveCode.
So let's start by downloading a copy of iText version 2.1.7 - do not use version 5.x as the API changed and the following example code won't work.

The first question is: how can we execute Java code from LiveCode? The simplest solution is the shell function: it allows you to execute DOS or Unix commands, as if you typed them in from the command line. Note that on Windows, using this function will show a DOS window, but you can control that by setting the hideConsoleWindows property before calling the shell function.
You can test it out by simply executing the following line from the message box:
  answer shell("java -version")

The second question is: what sort of Java code do we need to write? Well, I fired up a copy of Eclipse, started a new project, and created a new class 'ConcatPdfFiles' in the default package. Then I grabbed my paper copy of iText in action (first edition) and flipped to page 64 as this contains the examples for concatenating PDF files. A little bit of thinking, and I derived the following code:
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;

import com.lowagie.text.Document;
import com.lowagie.text.DocumentException;
import com.lowagie.text.pdf.PdfCopy;
import com.lowagie.text.pdf.PdfReader;

public class ConcatPdfFiles {
public static void main(String[] args) throws DocumentException, IOException {
final String outputFilePath = args[0];
final OutputStream outputStream = new FileOutputStream(outputFilePath);
final Document outputDocument = new Document();
final PdfCopy outputCopy = new PdfCopy(outputDocument, outputStream);
outputDocument.open();
for (int i = 1; i < args.length; i++) {
final PdfReader inputPdfReader = new PdfReader(args[i]);
final int pageCount = inputPdfReader.getNumberOfPages();
for (int pageIndex = 0; pageIndex < pageCount; pageIndex++) {
outputCopy.addPage(outputCopy.getImportedPage(inputPdfReader, pageIndex + 1));
}
}
outputDocument.close();
}
}

As you can see, the code is a bit lazy when it comes to exception handling: I just let the exceptions get thrown, and this will be the output of our shell call if something goes wrong. Note also that the first argument is the output file, followed by the input files that you want to concatenate into the output file.

More importantly, at this point in time, the code doesn't compile. The problem is, we haven't yet told Eclipse where that iText-2.1.7.jar library file is, so compilation fails. This is sometimes referred to as 'classpath hell' - you have to give Java a list of paths where it can find the necessary additional libraries, not just at compile time but also at runtime as we'll see later.
Because I like to keep everything together in my Java projects, I added a new 'lib' folder to my project, and copied the iText2.1.7.jar file into it. At that point, you can use the contextual menu on the iText.2.1.7.jar file, and add it to the Build Path. Now the code I showed earlier compiles just fine, and we can proceed to the next stage.

The third question is: how do we put everything together in LiveCode? We'll begin by putting all the necessary parts into a single folder: the iText-2.1.7.jar library file, the ConcatPdfFiles.class compiled file and two example PDF files (demo1.pdf and demo2.pdf). Then we fire up LiveCode, create a new stack 'ConcatPdfFiles' and save it in the same folder as the other files, naming it "ConcatPdfFiles.liveCode'. Now we can drop a button onto the stack and start scripting.

Now we need to determine the correct command to be executed by the shell function. It should look something like:
java -classpath <class-path> ConcatPdfFiles <output-file> <input-file-1> <input-file-2> ...

The java executable needs the correct classpath, and we need to pass in compatible file paths.

Let's start with the classpath. This is a list of places that java needs to look for its .class files - as separate files in folders, or stored together in a .jar file. And for extra fun, the separator character is a colon on Unix-based platforms, and a semicolon on Windows. You can have relative paths in this classpath, and '.' (period) is short for the current directory. So rather than building a long class path, we can circumvent the issue by setting the defaultFolder property to change the working directory before calling the shell function. Then our classpath can be as short as:
.:iText-2.1.7.jar
on MacOS X/Linux and
.;iText-2.1.7.jar
on Windows.

The next bit is compatible file paths. The good news: LiveCode uses a '/' (slash) as separator, regardless of the underlying platform, and Java is more than happy to accept '/' in a path, even when it's running on Windows. However, if there are spaces in the path, we need to save them by putting quotes around the path on Windows, and escaping the spaces with a backslash on Unix-based platforms.
And to determine the paths relative to the stack's location on your hard disk, we'll need a helper function that uses the effective filename property of our stack.

So finally, we have a button script as follows:
on mouseUp
--> determine the input and output files
local tInputFiles, tOutputFile
put ShellPath(AbsolutePathFromStack("demo1.pdf")) && \
ShellPath(AbsolutePathFromStack("demo2.pdf")) \
into tInputFiles
put ShellPath(AbsolutePathFromStack("output.pdf")) \
into tOutputFile
--> determine the class path
local tClassPath
if the platform is "Win32" then
put ".;iText-2.1.7.jar" into tClassPath
else
put ".:iText-2.1.7.jar" into tClassPath
end if
--> assemble the shell command
local tShellCommand
put "java -classpath" && tClassPath && \
"ConcatPdfFiles" && \
tOutputFile && tInputFiles \
into tShellCommand
--> execute the shell command
local tHideConsoleWindows, tDefaultFolder, tShellResult
put the hideConsoleWindows into tHideConsoleWindows
set the hideConsoleWindows to true
put the defaultFolder into tDefaultFolder
set the defaultFolder to AbsolutePathFromStack()
put shell(tShellCommand) into tShellResult
set the defaultFolder to tDefaultFolder
set the hideConsoleWindows to tHideConsoleWindows
if tShellResult is not empty then
answer error tShellResult
end if
end mouseUp

function AbsolutePathFromStack pFileName
local tAbsolutePath
put the effective filename of this stack into tAbsolutePath
set the itemDelimiter to slash
if pFileName is not empty then
put pFileName into item -1 of tAbsolutePath
else
delete item -1 of tAbsolutePath
end if
return tAbsolutePath
end AbsolutePathFromStack

function ShellPath pPath
if the platform is "Win32" then
put quote & pPath & quote into pPath
else
replace space with backslash & space in pPath
end if
return pPath
end ShellPath

Click the button, and it happily concatenates the two PDF files (demo1.pdf and demo2.pdf) into a single PDF file (output.pdf) in the same folder as our stack. There we have it, our first use of iText from within LiveCode.

Sunday, November 21, 2010

I went to the Devoxx 2010 conference and...

and I got the t-shirt to prove it


100% Javaholic was the motto of this year's Java developer conference, organised by the Belkgian JUG and held in Antwerp.
Of course, I wasn't there to collect the t-shirt - unlike some others who shall remain unnamed and went from booth to booth to fetch a t-shirt, even if they weren't interested in the product. You know who you are :-p

and I met Bruno Lowagie

For those who don't know Bruno Lowagie, he's the primary developer behind the iText project, an open-source library for producing PDF files using Java. There must be something in the water around here, as he lives about 30 miles from my house - two Belgians, both writing PDF libraries - what are the odds?
As he was busy signing copies of iText in Action (second edition) the conversation was rather short, but he signed my copy of his book with "To a fellow PDF developer" - what a compliment :-)

and I managed to squeeze LiveCode onto a whiteboard


Given that someone actually added 'HyperTalk :-)' to the board right behind my entry, I think I wasn't the only guy there who knows LiveCode. Next time, poke me and I'll buy you a drink.

and had a very good time overall

Plenty of interesting sessions, friendly environment and excellent organization. I'll definitely go back next time I get the chance.

Monday, September 20, 2010

Revolution is now LiveCode

Earlier today, RunRev announced the availability of LiveCode 4.5 - the new name and version for the development platform Revolution.

If you've come here and don't know what LiveCode/Revolution is: it is a cross-platform development tool for Windows, MacOSX and Linux, sporting a very-high-level-language that allows you to create working solutions in far less time than other tools and languages. The LiveCode Web browser plug-in works very similar to Flash and Silverlight. There are also pre-releases available of LiveCode Server (comparable to a PHP engine) and LiveCode for iOS (yep, IPhone, iPad and iPod Touch).

Whenever I get a chance, I'll use LiveCode to put together a quick tool for my own use, or a prototype/mockup for something that will eventually be developed in another language like Progress OpenEdge ABL or Java. I also offer Quartam Reports for LiveCode, the essential report generator, and Quartam PDF Library, for those occasions when you need to go beyond 'print to pdf file'.

Where Java is built around a relatively small platform-specific Virtual Machine implementation, with a huge bytecode library on top, LiveCode has a far more optimized yet surprisingly lean core implementation for each supported operating system, and a rich language that means you'll type fewer lines of code to accomplish the same result.

I like the name change and the clearer product line-up. And I like how this new version is faster than ever before. Cheers to the RunRev team for wrapping it up after the setback concerning the iPhone/iPad implementation. Combined with Apple's recent liberation of developer tools, the future looks decidedly LiveCode!

Sunday, September 12, 2010

XML validation using a DTD and Versions

In a previous post, I invetigated how we could validate an XML file in revTalk by using a DTD. As we all know, requirements evolve and our software needs to adapt likewise, especially at the integration end-points. So if we need to accept more data, we should (a) know that it's coming, and (b) make sure it's there before we process the incoming data.

How do we do that? Well, as we've seen beofre, we can check the structure of the XML file. And to make sure we check it correctly, we should introduce a specification version for our XML structure, and attach it to the root node of the document as an attribute. Then all we need to do is check the root node, extract its SpecificationVersion attribute, and we can apply the correct DTD validation.

Doesn't sound too complicated, does it? Let's expand our current stack design a bit so that it looks like this:


As you can see, the field "DTD" was renamed to "DTD 1.0" and another field "DTD 1.1" was added to hold the DTD for specification version 1.1; finally, I moved down the "Validate" button and modified its script:

constant kMaxVersion = 1.1

on mouseUp
-- Load the XML into a local variable
local tXmlText
put field "XML" into tXmlText
-- Parse the XML text into a Tree
local tXmlTree
put revCreateXmlTree(tXmlText, \
false, \
true, \
false) \
into tXmlTree
if tXmlTree is not an integer then
answer error \
"There is an error in the XML structure" & return & \
tXmlTree -- contains the full error message
else
-- Validate the root node
local tRootNode
put revXmlRootNode(tXmlTree) into tRootNode
if tRootNode is not "RootNode" then
answer error \
"The XML root node should be 'RootNode'"
else
-- Validate the SpecVersion
local tSpecVersion
put revXmlAttribute(tXmlTree, tRootNode, \
"SpecificationVersion") into tSpecVersion
if tSpecVersion begins with "xmlerr" then
answer error \
"The SpecificationVersion is missing"
else if tSpecVersion > kMaxVersion then
answer error \
"The SpecificationVersion " && tSpecVersion && \
"is newer than" && kMaxVersion
else
-- Load the corresponding DTD
local tDtdText
if tSpecVersion is 1.0 then
put field "DTD 1.0" into tDtdText
else
put field "DTD 1.1" into tDtdText
end if
-- Validate the XML against the DTD
local tValidationResult
put revXmlValidateDTD(tXmlTree, tDtdText) \
into tValidationResult
if tValidationResult is not empty then
answer error \
"The XML structure does not conform" & \
return & tValidationResult
else
answer information "The XML conforms to the DTD"
end if
end if
end if
-- Cleanup
revDeleteXmlTree tXmlTree
end if
end mouseUp


So how does this new version work?
- first, it parses the XML document
- next, it verifies the root node
- next, it checks the specification version
- next, it loads the appropriate DTD
- finally, it validates the XML against the DTD

If we test it, it correctly informs us that the XML document conforms to our specification version 1.1. What happens if we change the SpecificationVersion to 1.2?


Then we get this error message:


And finally, what happens if we change the SpecificationVersion to the original 1.0?


Then we get this error message:


This is a much safer way to check the incoming data in XML format. Unfortunately, a Document Type Definition is quite a limited way of XML validation. So next time, we'll improve our defenses again, by incorporating XML Schemas.

Tuesday, September 7, 2010

XML validation using a DTD

One of the positive aspects of the Extensible Markup Language XML is that it is a flexible way to structure data in a human-readable format, in a cross-platform and technology-independent way. No wonder it is widely used as a way to exchange data between applications, and forms the foundation for XML-RPC, SOAP and other Web Service methods.

But it would be naive to think that every XML document that we get is not only well-formed, but also in the format that we expect it to be, with the right elements and attributes. In this post, we'll examine a strategy to validate incoming XML data in our revTalk application, using a Document Type Definition - a.k.a. DTD.

Part of the XML specification since the very start, a DTD describes the structure of the XML elements and attributes. For more information, I advise you to study the excellent introductory tutorials on W3Schools.com. We're here to use it from revTalk, so let's start by creating a new stack for the user interface.


Simply drop two scrolling fields onto it, name them "XML" and "DTD" respectively, and then group each of them separately so we can put a nice group label on top (I have the memory of a goldfish so I might forget which-is-which ;-) ) Finally drop a button at the bottom of the stack and set its name to "Validate" - and now we're ready to start scripting the button.

First things first, we need to parse the XML text into an XML tree to use all the rev XML commands and functions.

on mouseUp
-- Load DTD and XML into local variables
local tDtdText, tXmlText
put field "DTD" into tDtdText
put field "XML" into tXmlText
-- Parse the XML text into a Tree
local tXmlTree
put revCreateXmlTree(tXmlText, \
false, \ -- must be well-formed
true, \ -- create a tree in memory
false) \ -- no SAX parser messages
into tXmlTree
if tXmlTree is not an integer then
answer error \
"There is an error in the XML structure" & return & \
tXmlTree -- contains the full error message
else
-- Clean up resources
revDeleteXmlTree tXmlTree
end if
end mouseUp

We use the revCreateXmlTree function to parse the XML text into a tree structure. If the XML test is not well-formed then we report the error, otherwise we know we have a valid XML tree structure at our disposal - which we need to cleanup after we're done, using the revDeleteXmlTree command. Now that we have the basics covered, we can add the DTD validation to our script.

on mouseUp
-- Load DTD and XML into local variables
local tDtdText, tXmlText
put field "DTD" into tDtdText
put field "XML" into tXmlText
-- Parse the XML text into a Tree
local tXmlTree
put revCreateXmlTree(tXmlText, \
false, \ -- must be well-formed
true, \ -- create a tree in memory
false) \ -- no SAX parser messages
into tXmlTree
if tXmlTree is not an integer then
answer error \
"There is an error in the XML structure" & return & \
tXmlTree -- contains the full error message
else
-- Validate the XML against the DTD
local tValidationResult
put revXmlValidateDTD(tXmlTree, tDtdText) \
into tValidationResult
if tValidationResult is not empty then
answer error \
"XML structure does not conform to the DTD" & return & \
tValidationResult -- contains the full error message
else
answer information "The XML conforms to the DTD"
end if
-- Clean up resources
revDeleteXmlTree tXmlTree
end if
end mouseUp

If the XML conforms to the DTD, the revXmlValidateDtd function will return empty, otherwise its output contains the validation error. Pretty straightforward, so let's test this with a simple XML and DTD:


When we click the 'Validate' button, we get the message that the XML conforms to the DTD. Exactly what we were hoping for. Now let's change the XML somewhat to see if it fails when our XML text clearly does not conform to the DTD.


And here's the error message that we get on our screen:


With very little scripting, we have added a first layer of defense against incoming XML data that is not up to our specifications. Next time, we'll elaborate on this example and bolster our defenses.

Wednesday, August 18, 2010

Fun with detailed files - and stat

In a previous post, I tried to find a way to convert Unix user and group ids into their corresponding names. After I reported my findings to the use-revolution mailing list, Andre Garzia stepped in, suggesting to take a look at the stat command.

Calling 'stat' through the shell function for every file would be costly and slow; using it to fetch the information for an entire folder in one go wouldn't be so bad, but if you already use the detailed files function built into revTalk, it would seem like a waste of cycles. So I decided to rewrite my work handlers, employing a caching mechanism for the id-to-name mapping, and using 'stat' to pick up the mapping information not available in the cache.
--> script-local 'static' variables

local sGroupIdMap, sUserIdMap

--> public functions and commands

command LoadIdMaps
p_LoadGroupIdMap
p_LoadUserIdMap
end LoadIdMaps

function GroupIdToName pGroupId, pFile
local tName, tShellCommand
put sGroupIdMap[pGroupId] into tName
if tName is empty and the platform is "MacOS" then
-- stat is only available on MacOS X
put p_GetFileStat(pFile, "%Sg") into tName
put tName into sGroupIdMap[pGroupId]
end if
return tName
end GroupIdToName

function UserIdToName pUserId, pFile
local tName, tShellCommand
put sUserIdMap[pUserId] into tName
if tName is empty and the platform is "MacOS" then
-- stat is only available on MacOS X
put p_GetFileStat(pFile, "%Su") into tName
put tName into sUserIdMap[pUserId]
end if
return tName
end UserIdToName

--> private functions and commands

private function p_GetFileStat pFile, pFormat
local tName, tShellCommand
-- make sure to escape spaces in the file name
replace space with backslash & space in pFile
put "stat -f" && pFormat && pFile into tShellCommand
put word 1 of shell(tShellCommand) into tName
return tName
end p_GetFileStat

private command p_LoadGroupIdMap
if the platform is "Win32" then exit p_LoadGroupIdMap
p_LoadFileIdMap "/etc/group", sGroupIdMap
end p_LoadGroupIdMap

private command p_LoadUserIdMap
if the platform is "Win32" then exit p_LoadUserIdMap
p_LoadFileIdMap "/etc/passwd", sUserIdMap
end p_LoadUserIdMap

private command p_LoadFileIdMap pFile, @pIdMap
local tData, tLine
put URL ("file:" & pFile) into tData
set the itemDelimiter to colon
repeat for each line tLine in tData
-- skip empty and comment lines
if tLine is empty \
or char 1 of tLine is "#" then next repeat
-- fill the id map
put item 1 of tLine into pIdMap[item 3 of tLine]
end repeat
end p_LoadFileIdMap

Armed with the above set of commands and functions, we can still use the detailed files function and obtain user and group names from a cache, calling into 'stat' as necessary. Let's update our sample button script:
on mouseUp
local tOldFolder, tNewFolder, tDetailedFiles
answer folder "Select a folder"
if the result is "Cancel" then exit mouseUp
put it into tNewFolder
--> extract the detailed files information
put the defaultFolder into tOldFolder
set the defaultFolder to tNewFolder
put the detailed files into tDetailedFiles
--> load the group and user id maps
LoadIdMaps
--> build the extended file information
local tDetailedFile, tExtendedFiles
local tFile, tUserId, tGroupId
repeat for each line tDetailedFile in tDetailedFiles
-- skip hidden files
if char 1 of tDetailedFile is "." then next repeat
-- fill the extended file entry
put urlDecode(item 1 of tDetailedFile) into tFile
put item 8 of tDetailedFile into tUserId
put item 9 of tDetailedFile into tGroupId
put tFileName & tab & \
UserIdToName(tUserId, tFile) & tab & \
GroupIdToName(tGroupId, tFile) & return \
after tExtendedFiles
end repeat
--> wrap things up
set the defaultFolder to tOldFolder
put char 1 to -2 of tExtendedFiles into field "Files"
end mouseUp

The more observant amongst you may think "If all you needed was a file's user and group name, why not just 'stat' and get it over with?" - well, the detailed files function returns a lot of useful information in one go, including the size of the resource fork of a file on MacOS systems.
Just one more thing: the 'stat' function was added in MacOS X 10.4 Tiger - so if your application still needs to support MacOS X 10.3 Panther, you're going to have to merge in the 'nireport' call from the previous post. Consider it an exercise for the reader.

On to the next challenge!

Tuesday, August 17, 2010

Fun with detailed files

Earlier today, the following question was posted on the use-revolution mailing list: file owner & group names. Rev veteran Monte Goulding was wondering how to convert file owner and group ids into actual user and group names. I happily logged into the Solaris server at work and dug around a bit, finally replying that he could parse this information from the following two files:
/etc/group
/etc/passwd

And what would such a parsing script look like in revTalk? I'm glad you asked:
command LoadGroupIdMap @pGroupIdMap
p_LoadFileIdMap "/etc/group", pGroupIdMap
end LoadGroupIdMap

command LoadUserIdMap @pUserIdMap
p_LoadFileIdMap "/etc/passwd", pUserIdMap
end LoadUserIdMap

private command p_LoadFileIdMap pFile, @pIdMap
local tData, tLine
put URL ("file:" & pFile) into tData
set the itemDelimiter to colon
repeat for each line tLine in tData
-- skip empty and comment lines
if tLine is empty \
or char 1 of tLine is "#" then next repeat
-- fill the id map
put item 1 of tLine into pIdMap[item 3 of tLine]
end repeat
end p_LoadFileIdMap

Given that both files use a colon as item delimiter, and have the name as first item and the id as third item, I extracted the parsing into a separate private command. Put the above handlers in your stack script, and then fetch each id-to-name-mapping before interpreting the detailed files.
Here's a simple button script that uses the above commands:
on mouseUp
local tOldFolder, tNewFolder, tDetailedFiles
answer folder "Select a folder"
if the result is "Cancel" then exit mouseUp
put it into tNewFolder
--> extract the detailed files information
put the defaultFolder into tOldFolder
set the defaultFolder to tNewFolder
put the detailed files into tDetailedFiles
set the defaultFolder to tOldFolder
--> load the group and user id maps
local tGroupIdMap, tUserIdMap
LoadGroupIdMap tGroupIdMap
LoadUserIdMap tUserIdMap
--> build the extended file information
local tDetailedFile, tExtendedFiles
repeat for each line tDetailedFile in tDetailedFiles
-- skip hidden files
if char 1 of tDetailedFile is "." then next repeat
-- fill the extended file entry
put urlDecode(item 1 of tDetailedFile) & tab & \
tUserIdMap[item 8 of tDetailedFile] & tab & \
tGroupIdMap[item 9 of tDetailedFile] & return \
after tExtendedFiles
end repeat
--> wrap things up
put char 1 to -2 of tExtendedFiles into field "Files"
end mouseUp

Once I got home, I wanted to verify this on my MacOS X machine, only to discover that it wasn't properly mapping the uid and gid to user and group names. Intrigued, I dug a bit deeper and found out that MacOS X actually relies on a DirectoryService when it's not running in single-user mode. Lovely, but how can we get what we need using revTalk?
Well, it turns out you can use the nireport command via the shell function to fetch a list of the group and user names with id. So let's add another helper command to do things the MacOS X way, and update our existing commands to use the correct approach, depending on the platform.
command LoadGroupIdMap @pGroupIdMap
if the platform is "MacOS" then
p_LoadNireportIdMap "/groups gid name", pGroupIdMap
else if the platform is "Linux" then
p_LoadFileIdMap "/etc/group", pGroupIdMap
end if
end LoadGroupIdMap

command LoadUserIdMap @pUserIdMap
if the platform is "MacOS" then
p_LoadNireportIdMap "/users uid name", pUserIdMap
else if the platform is "Linux" then
p_LoadFileIdMap "/etc/passwd", pUserIdMap
end if
end LoadUserIdMap

private command p_LoadFileIdMap pFile, @pIdMap
local tData, tLine
put URL ("file:" & pFile) into tData
set the itemDelimiter to colon
repeat for each line tLine in tData
-- skip empty and comment lines
if tLine is empty \
or char 1 of tLine is "#" then next repeat
-- fill the id map
put item 1 of tLine into pIdMap[item 3 of tLine]
end repeat
end p_LoadFileIdMap

private command p_LoadNireportIdMap pParams, @pIdMap
local tData, tLine
put shell ("nireport ." && pParams) into tData
set the itemDelimiter to tab
repeat for each line tLine in tData
put item 2 of tLine into pIdMap[item 1 of tLine]
end repeat
end p_LoadNireportIdMap

This seemed to do the trick on my iMac PowerPC G5 running MacOS X 10.4 Tiger - but when I tried it on my MacBook Pro Intel running MacOS X 10.6 Snow Leopard, it failed as there is no "nireport" command available. As it turns out, Apple dropped NetInfo when MacOS X 10.5 Leopard came out. So my solution is not quite complete, unfortunately.

To be continued...

Sunday, August 1, 2010

BorderLayout example - with resizing

In a previous post, I examined how you can mimic Java's BorderLayout behavior in revTalk. At the end of that post, I promised to explain how you can add resizers to the West and East panels. Just as a reminder, here's what we had accomplished in the previous installment:



Before we get to the meat of the resizers, let's do some preparation work. To clearly show the user that he can drag around the resizers, we'll make a local copy of the vertical divide resize cursor that the RunRev team ships with the IDE. Open the message box and type:

copy image "vdividecursor.gif" of card id 1030 of stack "revMacCursors" to this card

We'll be using that as our cursor later, but it doesn't have to be visible for that to work - so you can hide it now. Next, we add a rectangular button, carefully placing it over the touching edges of the West and Center panels. Set the button's name to "WestResizer" and its width to 5, and make sure its height is the same as the West and Center panel. Here's what it should look like:



Before we add the script, we'll also add two custom properties to the West graphic:
-set the uMinWidth custom property to 75
-set the uMaxWidth custom property to 150

Now that we have the visuals worked out, let's script the resizer button:

local sIsResizing
local sMinX, sMaxX

on mouseEnter
lock cursor
set the cursor to the short id of image "vdividecursor.gif"
end mouseEnter

on mouseLeave
unlock cursor
end mouseLeave

on mouseDown
local tMouseOffset
put item 1 of the mouseLoc - item 1 of the location of me \
into tMouseOffset
put the uMinWidth of graphic "West" + tMouseOffset into sMinX
put the uMaxWidth of graphic "West" + tMouseOffset into sMaxX
put true into sIsResizing
pass mouseDown
end mouseDown

on mouseUp
put false into sIsResizing
pass mouseUp
end mouseUp

on mouseRelease
put false into sIsResizing
pass mouseRelease
end mouseRelease

on mouseMove x,y
if sIsResizing is true then
local tNewX
put min(max(x, sMinX), sMaxX) into tNewX
lock screen
set the left of me to tNewX - 3
set the widthFromLeft of graphic "West" to tNewX
dispatch "resizeCenterPanel" to this card
unlock screen
end if
pass mouseMove
end mouseMove

While it's not too long a script, some explanation is in order:
- the mouseEnter and mouseLeave messages are handled to update the cursor
- the mouseDown message is handled to setup a few script local variables:
-> the sIsResizing variable is used as a flag to track the status
-> the sMinX and sMaxX variables are calculated once, based on the offset between the mouse and the resizer button, and taking into account the minimum and maximum width we set earlier
-> calculating those once helps cut down on the math later on
- the mouseUp and mouseRelease messages are handled to turn off the sIsResizing flag
- the mouseMove message is handled to resize the West and Center panels
-> but only if the sIsResizing flag is true
-> it uses the sMinX and sMaxX variables to make sure we don't go too far either way
-> it updates the resizer button itself, the West panel, and the Center panel.

To make this work, we obviously also have to add a handler for our custom 'resizeCenterPanel' event. We'll add that to the card script in a bit, after we've finished the resizer for the East panel. So duplicate the WestResizer button, name the copy EastResizer and move it over to the right, placing it over the touching edges of the East and Center panels.

Before we add the script, we'll also add two custom properties to the East graphic:
-set the uMinWidth custom property to 100
-set the uMaxWidth custom property to 200

Next, update its script to:

local sIsResizing
local sMinX, sMaxX

on mouseEnter
set the lockCursor to true
set the cursor to the short id of image "vdividecursor.gif"
end mouseEnter

on mouseLeave
set the lockCursor to false
end mouseLeave

on mouseDown
local tMouseOffset
put item 1 of the mouseLoc - item 1 of the location of me \
into tMouseOffset
put the width of this stack - \
the uMinWidth of graphic "East" + \
tMouseOffset into sMaxX
put the width of this stack - \
the uMaxWidth of graphic "East" + \
tMouseOffset into sMinX
put true into sIsResizing
pass mouseDown
end mouseDown

on mouseUp
put false into sIsResizing
pass mouseUp
end mouseUp

on mouseRelease
put false into sIsResizing
pass mouseRelease
end mouseRelease

on mouseMove x,y
if sIsResizing is true then
local tNewX
put min(max(x, sMinX), sMaxX) into tNewX
lock screen
set the left of me to tNewX - 3
set the left of graphic "East" to tNewX
set the widthFromLeft of graphic "East" to \
the width of this stack - tNewX
dispatch "resizeCenterPanel" to this card
unlock screen
end if
pass mouseMove
end mouseMove

The difference with the WestResizer is that the EastResizer also has to take into account the width of the stack, but that's not too complicated. Finally we'll update the card script, adding a handler for the 'resizeCenterPanel' custom event, and we'll also take care of updating the resizer's location and height when the stack is resized. The new card script looks like this:

on resizeStack pNewWidth, pNewHeight
local tCenterHeight, tCenterWidth
--
lock screen
-- update the north and south panels
set the widthFromLeft of graphic "North" to pNewWidth
set the widthFromLeft of graphic "South" to pNewWidth
set the bottom of graphic "South" to pNewHeight
put the top of graphic "South" - \
the bottom of graphic "North" \
into tCenterHeight
-- update the west and east panels
set the heightFromTop of graphic "West" to tCenterHeight
set the heightFromTop of graphic "East" to tCenterHeight
set the right of graphic "East" to pNewWidth
put the left of graphic "East" - \
the right of graphic "West" \
into tCenterWidth
-- update the west and east resizers
set the heightFromTop of button "WestResizer" to tCenterHeight
set the heightFromTop of button "EastResizer" to tCenterHeight
set the left of button "EastResizer" to \
the left of graphic "East" - 3
-- update the center panel
set the widthFromLeft of graphic "Center" to tCenterWidth
set the heightFromTop of graphic "Center" to tCenterHeight
--
unlock screen
pass resizeStack
end resizeStack

--> custom event handlers

on resizeCenterPanel
local tCenterWidth, tCenterHeight
put the top of graphic "South" - \
the bottom of graphic "North" \
into tCenterHeight
put the left of graphic "East" - \
the right of graphic "West" \
into tCenterWidth
lock screen
set the left of graphic "Center" to the right of graphic "West"
set the top of graphic "Center" to the bottom of graphic "North"
set the widthFromLeft of graphic "Center" to tCenterWidth
set the heightFromTop of graphic "Center" to tCenterHeight
unlock screen
pass resizeCenterPanel
end resizeCenterPanel

--> helper property setters

setProp widthFromLeft pNewWidth
local tRectangle
--
lock screen
put the rectangle of the target into tRectangle
put item 1 of tRectangle + pNewWidth into item 3 of tRectangle
set the rectangle of the target to tRectangle
unlock screen
end widthFromLeft

setProp heightFromTop pNewHeight
local tRectangle
--
lock screen
put the rectangle of the target into tRectangle
put item 2 of tRectangle + pNewHeight into item 4 of tRectangle
set the rectangle of the target to tRectangle
unlock screen
end heightFromTop

And there we have it: a BorderLayout with resizers for the West and East panels.



Once you're happy with the way the resizers work, you can change the style of the WestResizer and EastResizer button to transparent. They'll continue to work just fine, but won't be visible. With a little more work, you can also add resizers for the North and South panels. But I'll leave that as an exercise to the reader...

Monday, July 26, 2010

BorderLayout example script

As you may know, I spend quite a bit of my time in Java, not merely building back-end systems with message queues and raw socket communication, but also creating user interfaces in Swing. One of the most compelling features of Swing is its system of LayoutManagers - and certainly the easiest-to-use is the BorderLayout. Today, I'm going to explain how you can mimic this type of layout handling in revTalk.

The principle of the BorderLayout is quite simple: it divides the window into 5 panels: North, East, South, West and Center. As a picture says more than a thousand words, here's what this layout generally looks like:



The North panel is a good place to store a toolbar, the South panel usually plays host for a status area, while the West panel may show an outline, and the East panel a property grid, leaving the Center for the actual content.

For this example, I chose the easy route, creating a single rectangle graphic for each panel. I also picked distinct colors so you can more easily see the effect of resizing the window once the pieces of the puzzle fall into place. And it turns out it isn't that much work to implement, as I set the card script to:


on resizeStack pNewWidth, pNewHeight
local tCenterHeight, tCenterWidth
--
lock screen
-- update the north and south panels
set the widthFromLeft of graphic "North" to pNewWidth
set the widthFromLeft of graphic "South" to pNewWidth
set the bottom of graphic "South" to pNewHeight
put the top of graphic "South" - \
the bottom of graphic "North" \
into tCenterHeight
-- update the west and east panels
set the heightFromTop of graphic "West" to \
tCenterHeight
set the heightFromTop of graphic "East" to \
tCenterHeight
set the right of graphic "East" to pNewWidth
put the left of graphic "East" - \
the right of graphic "West" \
into tCenterWidth
-- update the center panel
set the widthFromLeft of graphic "Center" to \
tCenterWidth
set the heightFromTop of graphic "Center" to \
tCenterHeight
--
unlock screen
pass resizeStack
end resizeStack

--> helper property setters

setProp widthFromLeft pNewWidth
local tRectangle
--
lock screen
put the rectangle of the target into tRectangle
put item 1 of tRectangle + pNewWidth \
into item 3 of tRectangle
set the rectangle of the target to tRectangle
unlock screen
end widthFromLeft

setProp heightFromTop pNewHeight
local tRectangle
--
lock screen
put the rectangle of the target into tRectangle
put item 2 of tRectangle + pNewHeight \
into item 4 of tRectangle
set the rectangle of the target to tRectangle
unlock screen
end heightFromTop


Let's see what is happening here. The script handles the resizeStack event, which has 4 parameters (new width, new height, old width, old height) of which we only use the first two.

As the North and South panels use up the entire width, I can simply set their width to the new width of the stack. However, setting the width of a control in revTalk works from the location of the control (which is the center, not the topleft).
So I use a fake helper property setter 'widthFromLeft' to update the rectangle of the control instead. Finally, we move the South panel to the bottom edge of the window, and calculate the distance between the top of the south panel and the bottom of the north panel, as this is the new height for our West, East and Center panels.

Armed with this information, I follow a similar path for the West and East panels, updating their height using a fake helper property setter 'heightFromTop', and moving the East panel to the right edge of the window, calculating the distance between the east and west panels to determine the new width of the Center panel. And now it is dead-easy to update the Center panel with the calculated width and height.

That's all there is to it! Admittedly, this example merely resizes 5 graphics, but you can extend this logic to groups, handling the resizeControl message in your group scripts to automatically update the layout of the group's contents.

Here's what it looks like when stretched horizontally.



And here's what it looks like when stretched vertically.



Straightforward to implement with very little code. Next time, I'll add some resizers for the West and East panels.

Thursday, July 22, 2010

DataGrid Filter example - with sorting

The other day I posted an example of how you can add a filter to a DataGrid in Revolution. At the end of that blog post, I remarked how I had deliberately turned off column sorting. Fear not, it is straightforward to add support for this, again thanks to the DataGrid lessons available on the runrev.com website.

Just as a reminder, here's a screenshot of the example stack:



So what do we need to do to allow the user to click on the column header and apply the sort and filter at the same time? It's quite a simple bity of scripting, but before we get to that, you need to make some minor modifications to the datagrid columns.
- Switch to the 'pointer' tool, and select the datagrid, then open the Object Inspector, and switch to the 'Columns' panel
-> select the column 'index' and set its Sort by column type to 'Numeric'
-> select the column 'name' and make sure its Sort by column type is 'Text'

Now edit the script of the datagrid, and set it to:

local sOriginalData
local sIndexMap
local sFilter

setProp dgData pData
put pData into sOriginalData
RebuildMap
end dgData

getProp dgData
return sOriginalData
end dgData

setProp dgFilter pFilter
put pFilter into sFilter
RebuildMap
end dgFilter

getProp dgFilter
return sFilter
end dgFilter

--> datagrid callbacks

on GetDataForLine pLine, @pData
local tKey
-- retrieve the original line of data
put sIndexMap[pLine] into tKey
put sOriginalData[tKey] into pData
end GetDataForLine

on SortDataGridColumn pColumn
SortMap pColumn
HiliteAndStoreSortByColumn pColumn
dispatch "RefreshList" to me
end SortDataGridColumn

--> private commands and functions

private command RebuildMap
local tKeys, tKey, tIndex
put empty into sIndexMap
put 0 into tIndex
put the keys of sOriginalData into tKeys
sort tKeys numeric
repeat for each line tKey in tKeys
if sFilter is empty \
or sOriginalData[tKey]["name"] contains sFilter \
then
add 1 to tIndex
put tKey into sIndexMap[tIndex]
end if
end repeat
set the dgNumberOfRecords of me to tIndex
SortMap
dispatch "ResetList" to me
end RebuildMap

private command SortMap pColumn
local tSortDirection, tSortType, tSortCaseSensitive
local tSortCommand
local tKeys, tKey, tNewIndex, tNewIndexMap
-- works whether a column is specified or not
if pColumn is empty then
-- find the current sort column
put the dgProps["sort by column"] of me into pColumn
if pColumn is empty then
exit SortMap
end if
end if
-- assemble and do the sort command
put the dgColumnSortDirection[pColumn] of me \
into tSortDirection
put the dgColumnSortType[pColumn] of me \
into tSortType
put the dgColumnSortCaseSensitive[pColumn] of me \
into tSortCaseSensitive
put "sort lines of tKeys" && tSortDirection \
into tSortCommand
switch tSortType
case "numeric"
put " numeric" after tSortCommand
break
case "international"
put " international" after tSortCommand
break
case "datetime"
case "system datetime"'
put " dateTime" after tSortCommand
break
end switch
put " by ColumnValueForLine(pColumn, each)" \
after tSortCommand
set the caseSensitive to \
(tSortCaseSensitive is true)
set the useSystemDate to \
(tSortType is "system datetime")
put the keys of sIndexMap into tKeys
do tSortCommand
-- rebuild the index map
put 0 into tNewIndex
repeat for each line tKey in tKeys
add 1 to tNewIndex
put sIndexMap[tKey] into \
tNewIndexMap[tNewIndex]
end repeat
put tNewIndexMap into sIndexMap
end SortMap

private function ColumnValueForLine pColumn, pLine
local tKey
-- retrieve the original line of data
put sIndexMap[pLine] into tKey
return sOriginalData[tKey][pColumn]
end ColumnValueForLine

What's changed in comparison to the last implementation? Well, the bulk of the necessary work is done in the new private command SortMap, which reads the column sort properties and uses a custom sort function to get the filtered lines in the right order on the basis of the column value in the original lines of data. Then it builds a new IndexMap to replace the old one, and the regular refresh routines of the datagrid will "just work."

So what happens when we sort the data descending on the name, without applying a filter?



Looking good - does the filter still work? Let's type 'url' into the filter field.



You may have noticed that I also added getProp-handlers for dgData and dgFilter, making the above script easily reusable. With just a bit of lateral thinking and coding, you too can build just about anything, thanks to Revolution.

Monday, July 19, 2010

DataGrid Filter example

One of the powerful additions to Revolution 3.5 is the DataGrid - a custom control group that allows you to display any type of data with astonishing flexibility. Recently, I had to add filtering capabilities to a data grid for one of my projects. In this post, I will explain how I accomplished it, thanks to the DataGrid lessons available on the runrev.com website.

Here's a screenshot of the example stack:



Execute the following steps, if you want to follow along:
- Create a new stack
- Drag a button to the topleft of the card and set its name to "Fill"
- Drag a text field to the topright of the card
- Drag a label field next to the text field and set its content to "Filter:"
- Drag a data grid onto the card and configure it using the Object Inspector
-> In the 'Basic Properties' panel, turn off the checkbox "Allow Text Editing"
-> Switch to the 'Columns' panel, and add 2 columns "index" and "name"
-> Edit the script of the DataGrid, and set it to the following:

local sOriginalData
local sIndexMap
local sFilter

setProp dgData pData
put pData into sOriginalData
RebuildMap
end dgData

setProp dgFilter pFilter
put pFilter into sFilter
RebuildMap
end dgFilter

private command RebuildMap
local tKeys, tKey, tIndex
put 0 into tIndex
put the keys of sOriginalData into tKeys
sort tKeys numeric
repeat for each line tKey in tKeys
if sFilter is empty \
or sFilter is in sOriginalData[tKey]["name"] \
then
add 1 to tIndex
put tKey into sIndexMap[tIndex]
end if
end repeat
set the dgNumberOfRecords of me to tIndex
dispatch "ResetList" to me
end RebuildMap

on GetDataForLine pLine, @pData
local tKey
put sIndexMap[pLine] into tKey
put sOriginalData[tKey] into pData
end GetDataForLine

on SortDataGridColumn pColumn
-- prevent column sorting
end SortDataGridColumn

The above data grid script will take care of filtering the data. Now it's just a matter of wiring up the text field.
-> Edit the script of the text field and set it to the following:

local sUpdateMsg

on keyUp pKey
ScheduleUpdate
pass keyUp
end keyUp

on deleteKey
ScheduleUpdate
pass deleteKey
end deleteKey

on backspaceKey
ScheduleUpdate
pass backspaceKey
end backspaceKey

on pasteKey
ScheduleUpdate
pass pasteKey
end pasteKey

on cutKey
ScheduleUpdate
pass cutKey
end cutKey

command ScheduleUpdate
if sUpdateMsg is empty then
send "UpdateFilter" to me in 100 milliseconds
put the result into sUpdateMsg
end if
end ScheduleUpdate

on UpdateFilter
set the dgFilter of group "DataGrid 1" \
to the text of me
put empty into sUpdateMsg
end UpdateFilter

By trapping the different key events, we can update the data grid filter as we go along. Note that I used the ScheduleUpdate helper command to try and minimize the number of refreshes for quick typers. Play around with the interval until you find the sweet spot of responsive user interface.

Finally, it is time to fill the data gird with some data for testing, so edit the script of the button and set it to the following:

on mouseUp
local tData, tIndex, tName
put 0 into tIndex
repeat for each line tName in the functionNames
add 1 to tIndex
put tIndex into tData[tIndex]["index"]
put tName into tData[tIndex]["name"]
end repeat
set the dgData of group "DataGrid 1" to tData
end mouseUp

This simple script will fill our data grid with a list of the built-in functions that revTalk has to offer. Click the button, and type away in the filter text field to see the effect. Here's what happens when I type "cos":


Looks pretty good, doesn't it? It certainly was good enough for my use, but there are a few things to bear in mind about this implementation:
- I didn't have to edit the data in the grid.
- I didn't have to support sorting for my particular project.
Both of these limitations can be overcome, but I'll leave that for another time...

Sunday, July 18, 2010

Forums moved over to quartam.on-rev.com

Hi All,

After much deliberation, I have decided to move the forums over from ning.com to my new on-rev based website: http://quartam.on-rev.com/forums/index.php.
And here's an even shorter link: http://forums.quartam.com.

All posts were imported as plain text, and the topics were locked.
Unfortunately, you will have to re-register as a user on the new forum.

I apologize for any inconvenience, caused by this switch.

Best regards,

Jan Schenkel
--
Quartam Reports & PDF Library for Revolution
www.quartam.com

Sunday, March 14, 2010

But the waiting makes me curious

Be patient, is very good advice, but the waiting makes me curious.
It's a line from the song 'Very Good advice' written for the original Disney 'Alice in Wonderland' movie and recently remade by The Cure's Robert Smith for the 'Almost Alice' soundtrack. And I just couldn't think of a better introduction to this blog entry on the revTalk 'wait' command.

When your script is in a tight loop, screen updates my not happen until after the whole script is finished.
on mouseUp
repeat with i = 1 to 5000
put i into field "Output"
end repeat
end mouseUp

Rather than continuously updating the content of the field, the field may only get updated after the repeat loop. While that's good enough in this silly example, it's not suitable for keeping the user updated on the progress of a real-world data crunching routine.

You can give the engine some breathing room by inserting a call to the 'wait' command.
on mouseUp
repeat with i = 1 to 5000
put i into field "Output"
-- allow screen to redraw
wait 0 milliseconds
end repeat
end mouseUp

Now you'll see the content of the field refreshed regularly; but it also makes your loop much slower.
on mouseUp
constant kRefresh = 25
repeat with i = 1 to 5000
put i into field "Output"
if i mod kRefresh is 0 then
-- allow screen to redraw
wait 0 milliseconds
end if
end repeat
end mouseUp

By only redrawing the screen every 25th iteration, we can still refresh the screen with less impact on performance. Play with the 'kRefresh' constant value to find the right redraw/performance ratio.

But that's not all the 'wait' command can do: if used properly, you can give your users a way to 'cancel' out of a tight loop.
on mouseUp
constant kRefresh = 25
enable button "Cancel"
set the uCancel of me to false
repeat with i = 1 to 5000
put i into field "Output"
if i mod kRefresh is 0 then
-- allow screen to redraw and user to cancel
wait 0 milliseconds with messages
if the uCancel of me is true then
answer "Are you sure you want to cancel?" \
with "Yes" or "No"
if it is "Yes"
then exit repeat
else set the uCancel of me to false
end if
end if
end repeat
disable button "Cancel"
end mouseUp

Now we add a button "Cancel" and give it the following script:
on mouseUp
set the uCancel of button "TightLoop" to true
end mouseUp

And presto! Not only does the screen redraw, but we now allow the user to stop the time-consuming process in our tight loop. Of course you'll have to add your logic to 'revert' any changes you've made in the process, but the above provides you with a skeleton script for handling this sort of situation.

As with all good things in life, 'wait with messages' comes at a price: as the user can now click your button twice, or trigger other scripts, it is very important to disable those parts of your user interface that shouldn't be executed 'concurrently' - my favourite method is to switch to a 'limited' environment, disabling all controls, displaying a blended white overlay over them as well as the 'Cancel' button making it the only enabled control. After the loop, the stack returns to the 'unlimited' environment, hiding the overlay and enabling the controls.

Some of you may wonder what happens when a script issues 'wait 1 second with messages' and the user clicks on another button: the other button's script is executed first, and then the engine checks if the wait timeout has passed, and resumes the original script. This means that if your other button script takes 5 seconds to complete, your first button script will remain suspended - the revEngine is not multi-threaded. But with a little planning, you can provide a much better user experience.

Tuesday, January 12, 2010

In memoriam: Bill Marriott

As usual after the Holidays, I had some catching up to do with emails on the use-revolution mailing list. It was quite a shock when I read the news that William Marriott, marketing director at RunRev, had passed away.

To the rev developer community, he was the guy who worked his way up from the trenches, growing from a vocal advocate of quality, to the man who held surveys to figure out who we were and what we wanted, and helped in so many ways to reshape the image of the RunRev company. For the revSelect third-party add-on developers, he was the guy who not only gracefully hosted the webinars, but was in general someone you could easily work with to get results (a character trait shared at RunRev HQ, I might add).

At the RunRevLive '08 conference in Las Vegas, I had the pleasure of taking up an afternoon of his time to talk about how I could improve my existing products, and exchange thoughts on an assortment of ideas. His combined marketing and programming skills, plus his sense of humour, make him a hard act to follow.

My condolences and thoughts are with his family. May he rest in peace, or find a way to smuggle his laptop into heaven.