Batch Processing in Enterprise world

Batch processing is very common in Mainframe based solutions. Batch approach is specifically helpful when you need to process bulk data, and may require processing of flat files.

Batch processing in J2EE world has a weak story, as J2EE containers are designed to develop server side logic. J2EE specification addressed the needs of flat file processing using JCA (Java Connector Architecture).

In IBM suite of products WebSphere Adapter for Flat files, is one such solution to process the flat files. In open source ServiceMix has a component using JBI file binding. I have not personally experienced on how far these two technologies will scale to real needs in bulk processing of flat files.

The custom build solution would like below.







The custom resource adapter developed using Java Connector Architecture would poll for files and process (do necessary data transformation). The RA (resource adapter) would then call the respective business process either using a JMS protocol or simple EJB call. The core business can be implemented in EJB or BPEL.


The above solution would suite for J2EE world, but resource adapter would be able to use J2EE API's or transactions in the J2EE sense, or data sources defined in container. The management of threads resources, transactions are still with the RA developer, and makes more difficulties than giving a simple straight forward solution. Often programmers tend to write File IO in EJB's to give simple workable solution (though file IO not recommended in EJB's by J2EE Spec).

The simple implementation is to delegate, file processing to an external product (like WebSphere Transformation Extender, or custom application using MQ/Messaging), and invoke the business process in asynchronous fashion. The low level file processing can be well handled and wired to business process using standards like JMS.

Enterprise Schedulers in SOA environment

In my current assignment, I have been asked to write a small design/approach document to use a Enterprise Scheduler for the current project. This project being developed using SOA principles. The current environment consists of J2EE application servers and BPEL runtime engine, and Enterprise database servers and miscellaneous COTS (Commercial Off The Shelf), products.

Most the of work driven in batch mode, handling lot of flat files which consists of transactions (some in EDI format). It's tough to envision, how schedulers would fit to SOA based solutions. Since most the SOA based solutions (for that matter J2EE based solutions), are modeled as event driven.

Schedulers would work on time based, e.g. Run xyz process at 8PM everyday. In order to have a smooth play in SOA environment, we have to modify scheduler as a service. Two things here 1) Enterprise components accessing scheduler to add/update a task Meta data (when to run, credentials, log information, additional restart logic), or 2) Scheduler accessing Enterprise components to initiate a business process.

In my opinion we shall emphasis more on point 2, i.e., How to initiate a business process using a Enterprise Scheduler.

Thanks to modern schedulers, most of the schedulers (BMC Control-M, Autosys or open source scheduler Quartz), supports Web Services. If you expose your business process as a web service, schedulers can call that service using standard WSDL interface.

Blog experience

This is a test blog; I would like to write my own experience on SOA/ESB and some management practices observed from my own experience. I will try not to quote any company name or location/person, if you find any such material please let me know I will remove immediately or alter the content.

Regards
Arun