by Alexis Henry, Chief Technology Officer at Blu Age
While modernizing customer mainframes, our team at Blu Age discovered that Batch can be a very complex aspect of a mainframe migration to Amazon Web Services [AWS]. It often dictates whether a mainframe migration is successful or not. To succeed in a transition to microservices, it is critical to design your AWS architecture to account for the key Batch stringent performance requirements such as intensive I/Os, large datasets, and short durations.
In this post, I will describe how to migrate mainframe Batch to AWS microservices using Blu Age automated transformation technology.
Customers choose microservices aiming for more agility, innovation, quality, scalability, and availability. Despite all these advantages, a microservices approach introduces operational complexity. AWS has a number of offerings that address important challenges of microservices architectures: Managed Services, service orientation, programming languages polyglot, on-demand resources, infrastructure as code, and continuous delivery, among others.
Experience is still growing on microservices topics. Much of the existing technical literature describe microservices in the context of new applications or peeling monoliths, such as transitioning Java or client/server applications. However, most of the existing worldwide IT relies on mainframe monoliths. Many corporations and public agencies are looking for strategies to migrate their mainframe to cloud microservices, minimizing project risk, duration, and cost.
Batch processing usually involves bulk processing of data that could not be processed in real-time due to the limited capabilities of transactional engines at the time of their initial design and implementation. Batch software design was predicated on the constraints and assumptions of the mainframe environment, such as high CPU power for mono thread application, locking of I/O to data storage—which prevents concurrent processing of Batch and transaction—and higher Total Cost of Ownership (TCO) for provisioning nightly CPU peaks. Those constraints are still directly influencing Million Instructions Per Second (MIPS) estimation, cost, and operational model.
A more efficient and cost optimized architecture is now available with AWS. It can be achieved by transforming legacy Batch processes to real-time microservices, leveraging Amazon Kinesis for data streaming, Amazon API Gateway for service invocation, and AWS Lambda and serverless computing for compute and storage.
In the following example, I will explain how to transition a typical retail banking mainframe application from the dual Online/Batch model toward real-time microservices combining AWS services and Blu Age modernization technology.
In Figure 1, we use an example mainframe Batch architecture that we will transform into a microservice architecture in later sections. This typical scenario shows a mainframe legacy system using z/OS CICS, JCL, Cobol, DB2, VSAM files, GDG files, and tapes.
The Batch programs have been designed to avoid multiple user locking and waiting for transaction responses. During the day, loan request transactions append transaction data to a temporary file, with one file per physical office. At night, CICS transactions that share data with Batches are switched off to avoid concurrency, locks, and consistency issues.
Blu Age technology accelerates legacy application modernization with automation for both reverse-engineering of the legacy procedural mainframe applications (code + data) as well as forward-engineering to new microservice-ready object-oriented applications.
When modernizing from mainframe monoliths toward AWS, both the transformation and the definition of the target architecture are automated and standardized for AWS by Blu Age Velocity transformation technology. This execution environment is available off-the-shelf and relies upon two components:
Our recommendation for a successful mainframe to microservices project is to separate the technical stack transformation phase from the business split transformation phase in order to keep the microservices transformation complexity manageable for each phase and minimize project risks.
With such an approach, the first technical stack transformation phase focuses on the application code and data isofunctional migration, keeping the same application model with mostly infrastructure teams from both the mainframe and the AWS sides. The later business split transformation phase focuses on creating domain model boundaries for each microservice and will not involve a mainframe team. It does require participants from the Line of Business with an understanding of the business functions and processes.
For the technical stack microservice transformation phase, the mainframe Batch architecture is automatically refactored with Blu Age Velocity in the following way:
For the business split transformation phase, a Domain-Driven Design approach is recommended to identify each microservice scope with a Bounded Context. Blu Age Analyzer automates domain discovery by analyzing data and call dependencies between legacy programs. Domain decomposition refactoring using functional input is supported as well by Blu Age Analyzer. Decomposition strategy produced by Blu Age Analyzer is then driving the modernization transformations.
To learn more about this approach, see details about Blu Age Analyzer, Martin Fowler Domain Driven Design, Bounded Context, and Wikipedia Domain Driven Design. Once the microservices scope and Bounded Context have been defined, Blu Age automation can quickly re-factor the application code to separate and create the new microservices application packages.
Getting back to our mainframe legacy Batch example, application owners decide to modernize the mainframe with two main goals in mind:
For this purpose, executives decide to transform their mainframe Batch leveraging Blu Age technology as described in the preceding section. We now detail the resulting real-time microservices architecture on AWS. This example microservices Bounded Contexts split is as follows:
We now describe components of this real-time microservices architecture.
Angular enriches user experience while ensuring users of the legacy mainframe application do not require retraining for using the modernized system. All surface behaviors of the former mainframe system are mimicked with user experience related processes and screens transformed into a portal.
This architecture automatically distributes user connections to different containers and regions for better availability and reliability. It distributes the local versions of retail application in various countries and offices as well (because of various languages and legislation). As soon as a user submits a request, the Angular single-page application emits data into Kinesis to route and process this request in real-time.
Kinesis is the unified data hub for the real-time system, also called the data transport layer. Requests are pushed into Kinesis, which acts as a central, fast, and robust streaming hub. In addition to basic message queuing, Kinesis continuously sends data in the stream and allows data replay and broadcasting.
Kinesis is fully managed, which means you do not have to manage the infrastructure nor the configuration to adapt to burst or capacity variations. Even though data will be processed on the fly, Kinesis provides a buffer if needed. This is beneficial for mainframe Batch when there is a need to replay or reject prior data processing. Furthermore, Kinesis is used to support the Database per Service design pattern as per Martin Fowler’s microservices description.
API Gateway identifies which API or service to map to the incoming Kinesis data. It is the perfect fit as the central hub to access all your microservices, whatever their underlying technology and locations. Moreover, API Gateway serves as a service locator to call appropriate microservices, and enables service versioning and Canary deployment strategy. This allows reducing the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody.
Another reason for using API Gateway as a Canary strategy is when different business versions exist because large banks typically operate in many countries with different regulations. Using microservices through API Gateway, both solve multichannel and multi regulation issues.
API Gateway uses Lambda as a proxy to call microservices. Lambda initiates context and parameter values injected into the remote service. In the legacy system, JCL receives parameter values from a scheduler or programmatically defined variables. In such context, those parameters are used to set up the Batch runtime environment with the name of the dataset, version of deployed programs, execute Batch in production or test partition.
In the new paradigm, Lambda is used to keep this runtime parameter capability, which the client should not do for decoupling reasons, and triggers the appropriate Groovy script (which replaces z/OS JCL after being transformed with Blu Age). Groovy is preferred to Java to be modified without compilation while sharing the same JVM as the Java classes to be run.
Therefore, mainframe Batch job steps or run units may be reproduced, and modification can be done to Batch setup without compilation. Groovy and Java classes are called via REST and may be either synchronous or asynchronous. Asynchronous is preferred in case of long processing time. Indeed, Lambda lifetime must be kept below 500 seconds, and in such case detached services is the right pattern.
One specificity of mainframe system is I/O capabilities provided both by the underlying file system and non-relational databases built on top of it. Among those, VSAM relies on indexed sequential data store for which modern databases (RDBMS, graph database, column database) do not provide equivalent indexing, at least not preserving performance for all features. Blu Age BluSam uses ElastiCache (Redis implementation) in order to bring equivalent performance while supporting necessary I/O capabilities features:
Aurora is the preferred target AWS database for mainframe modernization because of its performance and equivalence. Legacy mainframe applications rely mostly on VSAM and SQL I/O capabilities, and changing to another data access type such as put/get of a document could be risky and time-consuming as it involves a major rewrite of the application logic disconnecting it from the data access APIs. However, using Aurora with BluSam and ElastiCache allows preserving transformation automation and performance with no need for refactoring for both VSAM’s like indexes (permanent storage in Aurora; live indexes in ElastiCache), and native SQL support.
Scalability is also important when modernizing mainframe because legacy application usage varies over time. For example, payroll systems or tax payment systems have a peak of activity every month/quarter. Aurora is a managed database with storage that can automatically grow to 64 TB per instance.
One challenge with microservices is the Database per Service design pattern. This pattern creates a need for data synchronization within the constraints of the CAP theorem because data is distributed. While specific patterns exist to handle the trade-off between eventual consistency, rollback mechanism, and transactional delay, they were all designed for online transactions. Each of these suffer, however, from network and transactional delay which do not fit with the mainframe Batch latency requirements. This includes a commit time within one milliseconds, whereas patterns such as Saga or API Composition introduce up to 100 milliseconds delay.
Aurora Serverless opens new strategies to achieve high performance and behaves like the regular Aurora service but automatically scales up or down based on your application’s needs while preserving ACID transactions. Because of the rapid scaling, Aurora Serverless is a cost-effective solution for Batch, burst, bulk data transfers, data consolidation, reducing elapse time for long processes such as payroll Batches.
Aurora brings a simple yet effective capability for data synchronization with native Lambda integration. Whenever a record is modified, then a Lambda is triggered. The Lambda is used to stream into Kinesis which delivers to API Gateway. Kinesis allows having multiple subscribers to propagate the data change to their local data store, while API Gateway allows doing API management for each domain. Then, all domains are synchronized simultaneously while each implements its private synchronization APIs based on its private choice of languages and databases
In the mainframe system, long-term data storage was handled at the application level, with several Batch jobs being responsible for archiving, back up, and pruning. Furthermore, the mainframe archival is costly and complex because it relies on generational GDG files on mainframe itself, and on tapes shipped to off-site storage.
Preserving the overall defined architecture, Kinesis and API Gateway remain the central data hub. AWS Batch is used to schedule data storage actions through a Lambda function. The Lambda function copies Aurora data into Amazon S3 with a command similar to the following:
SELECT INTO OUTFILE S3 <sql statement> , where sql statement selects data to be stored externally to the application database.
Amazon S3 functionalities replace z/OS GDG features, local copies of data, audit trails, medium term archiving and pruning of data, and extra backups. For long-term data archival and to satisfy regulatory requirements, the data is later moved from Amazon S3 into Amazon Glacier based on a Lifecycle Rule.
Blu Age Velocity is a ready-to-use solution that accelerates the migration of mainframe Batch to AWS microservices. Because it is a packaged solution, it minimizes project risk and costs. There are savings coming from transitioning from a mainframe MIPS cost structure to pay-as-you-go AWS Managed Services. Such savings typically finance the modernization project in a short time period and allow for a quicker return on investment.
The target architecture uses AWS Managed Services and serverless technology. As such, each microservice is elastic and minimizes system administrator tasks. It adapts to the client demand, automatically ensuring availability and performance of services while only paying for what you use.
From a design perspective, mainframe Batch applications are migrated to real-time, which improves the customer experience and satisfaction. Batch applications are also transformed into microservices that benefit from more flexibility, increased agility, independent business domains, deployment automation, and safe deployment strategy. In short: better, faster and safer capability to deliver and implement new features.
Blu Age Velocity can be used by any customer in any industry, for any mainframe executing languages such as COBOL (including most of its various flavors), JCL, and subsystems such as CICS, IMS, and VSAM. Blu Age Velocity accelerates both the code automated modernization and the target architecture definition.
Blu Age also facilitates the necessary activities from legacy code base inventory and analysis to control of like-for-like business logic testing and compliance with the latest development standards. Blu Age recommends performing a Proof of Concept with the technical Batch jobs. This proves the technology robustness and minimizes risk for the other jobs or programs.
Also available on AWS Partner Network (APN) Blog.