Data Integration Suite

This article was originally published in VSJ, which is now part of Developer Fusion.
The way we write applications changes, but the need to access data remains a challenge. DataDirect’s Data Integration Suite gives developers the tools to use data from virtually any data source in modern XML-based applications. The DataDirect Data Integration Suite has three main elements; there’s an XML application designer called Stylus Studio, an XQuery processor that you can use to query both XML and non-XML format data, and a set of XML Converters.

Figure 1
Figure 1: Creating an XML application using the Pipeline editor

Stylus Studio

Stylus Studio is the graphical designer where you can put together XML applications. Stylus Studio has modules that you can use to create or edit XML, XQuery, XSLT, XML Pipelines, XML reports, XML Schema, Web services, Java, DTD and relational data sources. Each module its own editor, and most have a choice of a graphical drag and drop type interface or the option of coding. In most cases you’ll build an application by starting with the pipeline editor. As the name suggests, this lets you link together the individual XML elements such as converters, queries and reports using pipelines that specify how the data from one element is linked to the next element. For example, to put together an application to take data from an order processing database and create a report showing sales for a particular product, you’d use a converter to turn the initial data into XML; an XQuery to find the data you want; and an XSLT tool to create an HTML report. The individual tools appear as icons that you drag and drop in the correct order, linking them with pipelines. While this all sounds nice and easy, you obviously need to define what the query does, what the report shows, what rules should be used to convert the data. To do this you create the elements using their appropriate modules, just as you would create queries, forms or reports in any standard application development package.

To design your XQueries there’s an XQuery design tool that has a text editor and graphical interfaces that you can again use to create a query either graphically or by writing the code directly. As you will know if you’ve encountered XQuery before, the heart of the query language for database users is the FLWOR expression, which is similar enough to a SQL Select expression for database designers to be able to use it without too much trouble. The name FLWOR comes from the five clauses you can use to build it – For, Let, Where, Order by, and Return. In Stylus Studio the graphical interface is a query mapper. You add source XML documents, specify the structure of the result you want the XQuery to return, then use the mapper canvas to define the conditions, functions, and operations that your query will carry out on the elements of the source document to find the values you want. These are then mapped to the target.

To use the mapper, you drop elements onto the canvas, usually starting with a FLWOR block. These have connectors for each element of the expression, and you can connect other query elements as the input to these connectors. So if you want to add a Where condition to your query, you choose a condition block such as an Equal. This has two input ports, so you could have a condition such as the Part ID from the Order data equals the Part ID from the Stock data. You can also enter values rather than using data elements, so you could connect one port to the Surname element from the Customer data, enter the value “Smith” for the other port, and connect the output port to the Where port on the FLWOR block to create an expression like:

where ($i/Surname) = "Smith"
The mapper provides a very easy way to create queries, and you can still enter the code directly if you want.

Figure 2
Figure 2: Adding a conditional block to an XQuery in the Mapper

From a database viewpoint the next important element in Stylus Studio is the report or Stylesheet designer. You can build reports in HTML+CSS or XSL:FO formats, using either XSLT or XQuery to define the report structure. The Stylus Studio XML Publisher has a drag and drop editor where you place report elements such as tables, groups, repeaters, and If conditions. You can drop data elements as text, images or aggregate operators such as Count and Sum.

There are a number of other designers in Stylus Studio; all provide either a visual drag and drop interface or some form of wizard to guide you through the steps of (for example) creating an EDI to XML converter. Overall, Stylus Studio is well designed and provides the tools you need to create an application. When you move away from the visual designers to the code behind the scenes, you get help in the form of automatic suggestions of XSLT or HTML tag and attribute names as you type, along with auto-complete and checking to ensure your XML is well formed. When working with XQuery you can debug using breakpoints and by monitoring variable values. A technique called backmapping shows the XQuery expression that created a result if you want to find out how a particular value was derived.

In addition to the IDE, the Data Integration Suite also includes an XQuery processor that you can use to query both XML and non-XML format data, and a set of XML Converters.

XQuery processor

The XQuery processor in the Data Integration Suite is called DataDirect XQuery. This is the same as the XQuery processor in Stylus Studio; however, this part of the suite is a processor that can be embedded into any Java program for runtime. You don’t need any other products, it doesn’t use a server, and it doesn’t need any additional product or application servers. You can use it to access and query XML, relational data, SOAP messages, EDI or other legacy data. The relational support includes the ability to update data, and your queries can be against a combination of data sources. It supports the XQuery for Java API (XQJ), which provides similar functionality to JDBC but using XQuery as its language. In your code you can either specify connection information explicitly:
DDXQDataSource ds = new DDXQDataSource();
ds.setJdbcUrl("jdbc:xquery:sqlserver://
	server1:1433;databaseName=orders");
…alternatively, you can use JNDI:
Context ctx = new InitialContext();
DDXQDataSource ds = (DDXQDataSource)ctx.
	lookup("stock_ds");
XQConnection conn = ds.getConnection(
	"myuserid", "mypswd");
If you need to work with a combination of XML, relational, and legacy data, the XQuery processor provides a comprehensive way to access the data.

Figure 3
Figure 3: Adding data fields to an XML report

XML Converters

Data Integration Suite comes with a set of file converters that provide bi-directional access to non-XML data from your Java and .NET applications. The converters support a range of interfaces including StAX, SAX, XMLReader, XMLWriter, DOM and I/O streaming, and can be embedded directly in your application if you need to translate between formats. You can also use them as part of a chain of programs including XSLT and XQuery, or inside your XML pipelines. The list of converters is extensive, but includes a number of EDI dialects – EANCOM, EDIFACT, Edig@s, HL7, IATA, and X12 – including HIPAA and ICD-10. Other formats include dBASE, HTML, CSV, JSON, and RTF, and you can create your own converters using Stylus Studio.

Web Services

One of the more common uses of XML is in Web services, and Data Integration Suite includes a framework that you can use to expose an XQuery as a Web service so you can handle an incoming request as XML, query the data on the middle tier, and produce the XML for the output.

Getting Started

There’s a lot to this package, so there’s a lot to learn when you first begin using it. DataDirect has chosen to show how to use the various elements using online videos, so you can choose to watch videos showing how to put elements together in a pipeline, how to create a report, find bottlenecks in a query – the list is extensive and more are being added all the time according to the online help. Those of us who prefer a more traditional approach can follow through the documentation in the help system.

Conclusion

This is a broad, sophisticated package that lets you create XML applications with very little coding. The pipeline editor and XQuery mapper are excellent designers and easy to use for application creation. The XQuery processor and converters let you use XQuery in your applications in a very simple way, and anyone used to JDBC will have no difficulty with that step.


Kay Ewbank is the editor of Server Management and is an experienced database developer and analyst.

DataDirect Technologies, +44 (0)1753 218 930

You might also like...

Comments

Contribute

Why not write for us? Or you could submit an event or a user group in your area. Alternatively just tell us what you think!

Our tools

We've got automatic conversion tools to convert C# to VB.NET, VB.NET to C#. Also you can compress javascript and compress css and generate sql connection strings.

“There are only 3 numbers of interest to a computer scientist: 1, 0 and infinity”