Table of Contents
Preliminary note | |
---|---|
For concision, "Summer.Batch." expression may be abbreviated on some occasions using the "S.B." expression, when writing fully qualified names of classes or interfaces;
e.g.:
|
Reading and writing flat files is a very common batch; flat files are still largely used to share information between components of the same system or to integrate data coming from outer systems.
A flat file reader implementation is directly available within Summer Batch, and covers typical needs. The class to use is Summer.Batch.Infrastructure.Item.File.FlatFileItemReader<T>
.
The template object T represents business object that will be filled by the records read from flat file. Consequently, some mapping has to be done between the records and their properties of target business object: this is achieved using a line mapper that must be provided at initialization time.
A line mapper is a class implementing Summer.Batch.Infrastructure.Item.File.ILineMapper<out T>
interface.
A line mapper should implement
T MapLine(string line, int lineNumber)
generic method; this method returns a T object given a line (and its line number within file, which might be relevant).
A default implementation is provided :
Summer.Batch.Infrastructure.Item.File.Mapping.DefaultLineMapper<T>
The mapping is done in a two phases process:
Summer.Batch.Infrastructure.Item.File.Transform.ILineTokenizer
interface ). Two implementations are being provided to cover the most typical needs:
Summer.Batch.Infrastructure.Item.File.Transform.FixedLengthTokenizer
: for lines with a fixed-length format. The fields are being specified using ranges (see Summer.Batch.Infrastructure.Item.File.Transform.Range
);Summer.Batch.Infrastructure.Item.File.Transform.DelimitedLineTokenizer
: for lines holding separated fields, e.g. CSV files (the separator string is configurable and defaults to comma).The result of first phase is a field set.
Note | |
---|---|
see
|
Its fields will be mapped to a business object properties using a field set mapper ( class that implements
Summer.Batch.Infrastructure.Item.File.Mapping.IFieldSetMapper
Field set mappers are bound to your business model; each target business object (intended to be filled by records read from flat file) should have an available mapper.
Now let's see a sample. First, the XML job configuration.
Example 6.1. FlatFileItemReader
declaration in the job XML file
FlatFileBO
business object.
Here is sample flat file data, which we'll be using:
Example 6.2. Sample delimited flat file data
1;FlatFile1 ; FlatFile1 ;20100101 2;FlatFile2 ; FlatFile2 ;20100101
Example 6.3. Sample flat file target business object
using System;
namespace Com.Netfective.Bluage.Business.Batch.Flatfile.Bos
{
/// <summary>
/// Entity FlatFileBO.
/// </summary>
[Serializable]
public class FlatFileBO
{
/// <summary>
/// Property Code.
/// </summary>
public int? Code { get; set; }
/// <summary>
/// Property Name.
/// </summary>
public string Name { get; set; }
/// <summary>
/// Property Description.
/// </summary>
public string Description { get; set; }
/// <summary>
/// Property Date.
/// </summary>
public DateTime? Date { get; set; }
}
}
IFieldSetMapper
Example 6.4. Sample flat file target business object field set mapper
using Summer.Batch.Extra; using Summer.Batch.Infrastructure.Item.File.Mapping; using Summer.Batch.Infrastructure.Item.File.Transform; namespace Com.Netfective.Bluage.Business.Batch.Flatfile.Bos.Mappers { /// <summary> /// Implementation of <see cref="IFieldSetMapper{T}" /> that creates /// instances of <see cref="FlatFileBO" />. /// </summary> public class FlatFileMapper : IFieldSetMapper<FlatFileBO> { private IDateParser _dateParser = new DateParser(); /// <summary> /// Parser for date columns. /// </summary> private IDateParser DateParser { set { _dateParser = value; } } /// <summary> /// Maps a <see cref="IFieldSet"/> to a <see cref="FlatFileBO" />. /// <param name="fieldSet">the field set to map</param> /// <returns>the corresponding item</returns> /// </summary> public FlatFileBO MapFieldSet(IFieldSet fieldSet) { // Create a new instance of the current mapped object return new FlatFileBO { Code = fieldSet.ReadInt(0), Name = fieldSet.ReadRawString(1), Description = fieldSet.ReadRawString(2), Date = _dateParser.Decode(fieldSet.ReadString(3)), }; } } }
Note | |
---|---|
Note the use of a dedicated helper ( |
Example 6.5. Delimited flat file reader - sample unity configuration
/// <summary> /// Registers the artifacts required for step FlatFileReader. /// </summary> /// <param name="container">the unity container to use for registrations</param> private void RegisterFlatFileReader(IUnityContainer container) { // Reader - FlatFileReader/FlatFileReader container.StepScopeRegistration<IItemReader<FlatFileBO>, FlatFileItemReader<FlatFileBO>>("FlatFileReader/FlatFileReader") .Property("Resource") .Resource("#{settings['BA_FLATFILE_READER.FlatFileReader.FILENAME_IN']}") .Property("Encoding").Value(Encoding.GetEncoding("UTF-8")) .Property("LineMapper") .Reference<ILineMapper<FlatFileBO>>("FlatFileReader/FlatFileReader/LineMapper") .Register(); // Line mapper container.StepScopeRegistration<ILineMapper<FlatFileBO>, DefaultLineMapper<FlatFileBO>>("FlatFileReader/FlatFileReader/LineMapper") .Property("Tokenizer") .Reference<ILineTokenizer>("FlatFileReader/FlatFileReader/Tokenizer") .Property("FieldSetMapper") .Reference<IFieldSetMapper<FlatFileBO>> ("FlatFileReader/FlatFileReader/FieldSetMapper") .Register(); // Tokenizer container .StepScopeRegistration<ILineTokenizer, DelimitedLineTokenizer>("FlatFileReader/FlatFileReader/Tokenizer") .Property("Delimiter").Value(";") .Register(); // Field set mapper container .StepScopeRegistration<IFieldSetMapper<FlatFileBO>, FlatFileMapper>("FlatFileReader/FlatFileReader/FieldSetMapper") .Register(); // ... -- processor and writer registration is not being shown here -- }
Note | |
---|---|
|
Provided implementation is in
Summer.Batch.Infrastructure.Item.File.FlatFileItemWriter<T>
class, where T
is the type of business object
that will be "dumped" into flat file. FlatFileItemWriter
uses the following properties:
Resource
: resource to be written to;LineAggregator
: a class implementing
Summer.Batch.Infrastructure.Item.File.Transform.ILineAggregator<in T>
interface
LineSeparator
: line separator for the lines to write in flat file; defaults to System.Environment.NewLine
;Transactional
: a flag to specify if the writer should take part in the active transaction (meaning that data will be effectively written at commit time); defaults to true; AutoFlush
: a flag to specify if the writer buffer should be flushed after each write; defaults to false; SaveState
: a flag to specify if the state of the item writer should be saved in the execution context when Update method is called; defaults to true;AppendAllowed
: a flag to specify if an existing resource should be written to in append mode; defaults to false;DeleteIfExists
: a flag to specify if an existing resource should be deleted; if AppendAllowed is set to true, this flag is IGNORED; defaults to false;DeleteIfEmpty
: a flag to specify if an empty target resource (no lines were written) should be deleted; defaults to false;HeaderWriter
: a header writer (class implementing the
Summer.Batch.Infrastructure.Item.File.IHeaderWriter
interface); Used to write the header of file; No default value;
FooterWriter
: a footer writer (class implementing the
Summer.Batch.Infrastructure.Item.File.IFooterWriter
interface); Used to write the footer of file; No default value;
The writing process takes a business object as input, transforms it into a string using LineAggregator
and append the string to target resource.
Now let's review an example; first, the job XML configuration :
Example 6.6. FlatFileItemWriter
declaration in the job XML file
LineAggregator
; Summer Batch comes with several ILineAggregator
implementations :
Summer.Batch.Infrastructure.Item.File.Transform.DelimitedLineAggregator<T>
: transforms an object properties into a delimited list of strings. Default delimiter is comma, but can set to any arbitrary string; This is the natural choice to write CSV files;Summer.Batch.Infrastructure.Item.File.Transform.FormatterLineAggregator<T>
: transforms an object properties into a string, using a provided format (the computed string is the result of a call to string.Format
method, using provided format.);Summer.Batch.Infrastructure.Item.File.Transform.PassThroughLineAggregator<T>
: transforms an object to a string by simply calling ToString
method of the object;FormatterLineAggregator
, providing a format string through unity configuration;
To fully understand unity configuration that follows, used business object EmployeeDetailBO
must be shown:
Example 6.7. Sample flat file writer input business object
using System;
namespace Com.Netfective.Bluage.Business.Batch.Flatfilewriter.Bo
{
/// <summary>
/// Entity EmployeeDetailBO.
/// </summary>
[Serializable]
public class EmployeeDetailBO
{
/// <summary>
/// Property EmpId.
/// </summary>
public int? EmpId { get; set; }
/// <summary>
/// Property EmpName.
/// </summary>
public string EmpName { get; set; }
/// <summary>
/// Property Name.
/// </summary>
public string Name { get; set; }
/// <summary>
/// Property EmpDob.
/// </summary>
public DateTime? EmpDob { get; set; }
/// <summary>
/// Property EmpSalary.
/// </summary>
public decimal? EmpSalary { get; set; }
/// <summary>
/// Property EmailId.
/// </summary>
public string EmailId { get; set; }
/// <summary>
/// Property BuildingNo.
/// </summary>
public int? BuildingNo { get; set; }
/// <summary>
/// Property StreetName.
/// </summary>
public string StreetName { get; set; }
/// <summary>
/// Property City.
/// </summary>
public string City { get; set; }
/// <summary>
/// Property State.
/// </summary>
public string State { get; set; }
}
}
Example 6.8. Formatted flat file writer - sample unity configuration
/// <summary> /// Registers the artifacts required for step step1. /// </summary> /// <param name="container">the unity container to use for registrations</param> private void RegisterStep1(IUnityContainer container) { //... -- reader and processor registration is not being shown here -- // step1/FlatFileListWriter/Delegate Writer container.StepScopeRegistration<IItemWriter<EmployeeDetailBO>, FlatFileItemWriter<EmployeeDetailBO>>("step1/FlatFileWriter") .Property("Resource") .Resource("#{settings['BA_FLAT_FILE_WRITER.step1.FlatFileWriter.FILENAME_OUT']}") .Property("Encoding").Value(Encoding.GetEncoding("UTF-8")) .Property("LineAggregator") .Reference<ILineAggregator<EmployeeDetailBO>>("step1/FlatFileWriter/LineAggregator") .Register(); // Line aggregator container.StepScopeRegistration<ILineAggregator<EmployeeDetailBO>, FormatterLineAggregator<EmployeeDetailBO>>("step1/FlatFileWriter/LineAggregator") .Property("Format") .Value("{0},{1},{2},{3:yyyy-MM-dd},{4},{5},{6},{7},{8},{9}") .Property("FieldExtractor") .Reference<IFieldExtractor<EmployeeDetailBO>>("step1/FlatFileWriter/FieldsExtractor") .Register(); // Fields Extractor container.StepScopeRegistration<IFieldExtractor<EmployeeDetailBO>, PropertyFieldExtractor<EmployeeDetailBO>>("step1/FlatFileWriter/FieldsExtractor") .Property("Names").LateBinding<string[]>("EmpId,EmpName,Name,EmpDob,EmpSalary," + "EmailId,BuildingNo,StreetName,City,State") .Register(); }