Although SOAs have been discussed now for several years, relatively few IT departments have fully implemented them across the organisation. However, those who have done so for real-time data-intensive systems realise that providing robust, scalable “Data Services” can be crucial to success. This article outlines what I’ve learned from my involvement with several business-critical projects.
SOA – more than Web Services
You often see Web Services mentioned in a discussion of SOA. However, Service Oriented Architectures are not limited to Web Services. To understand the distinction, consider both the motivations for moving to an SOA and the desired benefits.SOAs provide a flexible, reusable framework for construction and integration of many applications and application components. Shared application “services” enable rapid development of new applications. The commonly cited benefits of an SOA are:
- flexibility – enable developers to quickly build new applications
- reusability – employ investments across multiple applications
- loose coupling – eliminate and avoid the brittle nature of monolithic applications
Figure 1
Figure 1 shows multiple Web Services accessing the same persistent data. Every request requires database access. You can imagine that with many concurrent users, each service contends for the same data over limited connections, which can easily create a bottleneck.
This use of Web Services requires each application to duplicate key infrastructure services. Striving for the greatest re-use, consider key services all applications require. Such infrastructure services include object-relational mapping, caching, cache synchronisation, client notification, and all of the connection and transaction logic involved in data access.
Migration of legacy applications to an SOA can also expose data integrity issues. Traditional applications often have their own database “silo” which contains a copy of business reference data such as customer information, product information, and inventory levels. Typically, each database is synchronised only once a day, so each application often operates with different versions of the same data. When applications are re-distributed as enterprise services without integrating the data silos, these data inconsistencies can create unanticipated business errors.
Figure 2
Figure 2 illustrates the inconsistencies that can arise when such silo applications are exposed as services, each relying on a different source of truth. Imagine that an airline has two reservation systems, one that is manned by airline personnel and another that is a self-service Web portal. Both systems rely upon and update a database that tracks the seat inventory and makes price changes. The Reservation application receives inventory updates each hour, but the website only receives updates once a day. As you can see, a discrepancy in the number of seats available would cause immediate business problems.
Data Services provide the necessary infrastructure services to solve both of these issues. Data Services include transaction and connection management for domain-specific data classes that are available to multiple applications. Each application uses a subset of the classes managed by the data service, but the data service manages the relationships between the classes and ensures that each application is aware of all data changes, regardless of the source of change. Data Services can also provide notifications of important changes, allowing other applications or end-users to respond to these events.
Including Data Services as an infrastructure service increases reusability and flexibility, which shortens development and rollout time for new services. Caching in the middle tier alleviates the bottleneck that can easily occur when multiple applications access common data. By caching frequently requested data, more requests can be satisfied without querying the database, which improves performance and increases scalability and reliability. Fewer database connections are required to service the same number of requests. Data silos can be removed allowing more applications to access the same data and avoid data consistency issues.
Figure 3
Figure 3 shows how Data Services provides the infrastructure necessary to reap the full benefits of a Service Oriented Architecture.
Additional motivation for the Data Services approach
The logic to handle data access and updates accounts for a large percentage of enterprise application development efforts. A CIO of a Fortune 500 company recently told me “We have hundreds of applications each doing the same thing, only slightly different.” Each development team within his large IT group was implementing a unique data access component (through ADO.NET, ODBC, JDBC, or some other data access standard).He recognises that even when using standards, this low-level coding is tedious, error prone, and unnecessarily redundant. When each application contains an independent and unique data access layer, each will require its own specific tuning and corresponding maintenance. To meet expanding business needs, he must reduce this repetition. And, he’s far from alone. I hear this sentiment echoed by almost every architect or product manager with whom I speak.
Using the SOA paradigm, just as it is preferable to implement a single credit card authorisation services that can be reused by many applications, it makes sense to implement Data Services to be shared by a set of related applications. Applications with relatively simple requirements will reap the benefits described above: flexibility, reusability, and loose coupling. However, those with extremely demanding requirements for performance, scalability, and reliability may find that they cannot meet these needs without building Data Services into their system.
Real-World Data Services implementations
Systems in the financial and transportation industries put heavy demands on the infrastructure, but they may have different requirements. For example, at peak hours, a trading system may need to service thousands of users concurrently. In contrast, a logistics system for transportation may have fewer users, but provide instant access to thousands of related pieces of data. Both are excellent candidates for Data Services.Trading system example
A leader in the financial industry gained a competitive advantage by delivering superior services. They had very demanding requirements for reliability, performance, and scalability, processing 5,000 transactions per second at peak volume. By carefully choosing a data services technology, they are able to deliver immediate response to the end-user, to scale to meet the needs of their business, and to ensure the system is available at all times.To enable these rich services, they implemented more than 40 applications that utilise the same data services. Since the applications are related and share common data model and common data, the data services ensure that each server and client application has up-to-date information to enable accurate business decisions.
Realising the effort and complexity involved, they chose to buy rather than build. After considering available alternatives, they settled on a full-lifecycle Data Services technology that offered caching, optimised updates, distributed cache synchronisation, load balancing, failover, and client notification. If each application implemented its own data access, they would still be building their system. By amortising the investment in a data services technology over 40+ applications, they were able to deliver on time to meet business needs.
Logistics example
A major fractional jet ownership corporation has distinguished itself in the marketplace with the ability to deliver an aircraft to an owner almost anywhere in the world within a few hours of receiving a customer’s call. If the company fails to send the right airplane and crew to the right place at just the right time, they are forced to hire an expensive third-party charter company to fill the customer’s needs.The data required to perform such scheduling consists of accessing hundreds of related bits of information, from customer preferences, to the availability of catering and crew.
Because they chose a Data Services infrastructure that allowed them to easily optimise performance, application architects found scalability to be nearly linear across all servers. Performance and infrastructure are so well integrated now, that they’re actually non-issues. Data Services with middle-tier caching provides instant access to the details of a trip, while the customer is on the phone, not three hours later when the owners are already packing their bags. They moved from a manual system to an automated one that takes about two seconds to book, synchronises itself, and delivers an entire set of workflows with vital customer data to everyone who needs to know- just in time for it to be useful.
Conclusion
When done right, SOAs represent a great leap forward in system design. The real power of an SOA can be realised when it clearly separates infrastructure services from business services. High-performing, scalable Data Services are a critical component to SOAs with stringent requirements for real-time access to persistent data.Simply wrapping existing applications with Web Services can create bottlenecks that limit performance and scalability. The integration of legacy applications can also create issues with data consistency. Data Services can address these problems by including middle-tier caching and synchronisation.
A Data Services technology should deliver benefits across the project by reducing development efforts and ensuring the performance and scalability necessary to meet business needs. Innovative companies are already reaping these benefits from systems that give them a competitive advantage and fully justify the investment in an SOA.
Derek Henninger is the Director of Professional Services for the ObjectStore division of Progress Software, having been Vice President of Worldwide Field Operations for Persistence. He holds a BSc degree in Computer Science and Mathematics and a BA degree in Economics, both from University of California at Davis.
Comments