Chapter 6. Using Basic Features
Prev		Next

Chapter 6. Using Basic Features

Table of Contents

Preliminary note

	Preliminary note
For the sake of concision, "Summer.Batch." expression may be abbreviated on some occasions using the "S.B." expression, when writing fully qualified names of classes or interfaces; e.g.: `Summer.Batch.Infrastructure.Item.File.FlatFileItemReader<T>` may be abbreviated in `S.B.Infrastructure.Item.File.FlatFileItemReader<T>`

For the sake of concision, "Summer.Batch." expression may be abbreviated on some occasions using the "S.B." expression, when writing fully qualified names of classes or interfaces;

e.g.:

Summer.Batch.Infrastructure.Item.File.FlatFileItemReader<T>

may be abbreviated in

S.B.Infrastructure.Item.File.FlatFileItemReader<T>

Reading and writing flat files

Reading and writing flat files is a very common batch; flat files are still largely used to share information between components of the same system or to integrate data coming from outer systems.

Using a flat file reader

A flat file reader implementation is directly available within Summer Batch, and should cover typical needs in that matter. The class to use is Summer.Batch.Infrastructure.Item.File.FlatFileItemReader<T>. The template object T represents the business object that will be filled by the records read from the flat file. As a consequence, some mapping has to be done between the records and the properties of the target business object: this is achieved using a line mapper that must be provided at initialization time. A line mapper is a class implementing the Summer.Batch.Infrastructure.Item.File.ILineMapper<out T> interface.

A line mapper has to implement the

 T MapLine(string line, int lineNumber)

generic method; this method returns a T object given a line (and its line number within the file, which can be relevant).

A default implementation is provided :

Summer.Batch.Infrastructure.Item.File.Mapping.DefaultLineMapper<T>

The mapping is done in a two phases process:

The read line from the flat file is split into fields, using a tokenizer that must be specified at initialization time (a class that implements the Summer.Batch.Infrastructure.Item.File.Transform.ILineTokenizer interface ). Two implementations are being provided to cover the most typical needs :
- Summer.Batch.Infrastructure.Item.File.Transform.FixedLengthTokenizer: for lines with a fixed-lenght format. The fields are being specified using ranges (see Summer.Batch.Infrastructure.Item.File.Transform.Range);
- Summer.Batch.Infrastructure.Item.File.Transform.DelimitedLineTokenizer: for lines holding separated fields -- e.g. CSV files -- (the separator string is configurable and defaults to comma).
The result of that first phase is a field set.
Note
see the
Summer.Batch.Infrastructure.Item.File.Transform.IFieldSet
interface and the default implementation
Summer.Batch.Infrastructure.Item.File.Transform.DefaultFieldSet
Its fields will be mapped to a business object properties using a field set mapper (a class that implement the
Summer.Batch.Infrastructure.Item.File.Mapping.IFieldSetMapper
interface).
Field set mappers are bound to your business model; each target business object (intended to be filled by records read from the flat file) should have an available mapper.

Now let's see a sample. First, the -- trivial -- xml job configuration.

Example 6.1. FlatFileItemReader declaration in the job xml file

	<step id="FlatFileReader">
		<chunk item-count="1000">
			<reader ref="FlatFileReader/FlatFileReader" />
			...
		</chunk>
	</step>

Wiring the unity configuration is a bit more complex. Our sample makes uses of a semicolon (";") separated flat file whose records will be mapped to the FlatFileBO business object. Here is the sample flat file data we'll be using :

Example 6.2. Sample delimited flat file data

1;FlatFile1 ; FlatFile1 ;20100101
2;FlatFile2 ; FlatFile2 ;20100101

Example 6.3. Sample flat file target business object

using System;

namespace Com.Netfective.Bluage.Business.Batch.Flatfile.Bos
{
    /// <summary>
    /// Entity FlatFileBO.
    /// </summary>
    [Serializable]
    public class FlatFileBO
    {
        /// <summary>
        /// Property Code.
        /// </summary>
        public int? Code { get; set; }

        /// <summary>
        /// Property Name.
        /// </summary>
        public string Name { get; set; }

        /// <summary>
        /// Property Description.
        /// </summary>
        public string Description { get; set; }

        /// <summary>
        /// Property Date.
        /// </summary>
        public DateTime? Date { get; set; }

    }
}

The mapping between the data and the target business object is achieved using the following IFieldSetMapper

Example 6.4. Sample flat file target business object field set mapper

using Summer.Batch.Extra;
using Summer.Batch.Infrastructure.Item.File.Mapping;
using Summer.Batch.Infrastructure.Item.File.Transform;

namespace Com.Netfective.Bluage.Business.Batch.Flatfile.Bos.Mappers
{
    /// <summary>
    /// Implementation of <see cref="IFieldSetMapper{T}" /> that creates
    /// instances of <see cref="FlatFileBO" />.
    /// </summary>
    public class FlatFileMapper : IFieldSetMapper<FlatFileBO>
    {
        private IDateParser _dateParser = new DateParser();

        /// <summary>
        /// Parser for date columns.
        /// </summary>
        private IDateParser DateParser { set { _dateParser = value; } }

        /// <summary>
        /// Maps a <see cref="IFieldSet"/> to a <see cref="FlatFileBO" />.
        /// <param name="fieldSet">the field set to map</param>
        /// <returns>the corresponding item</returns>
        /// </summary>
        public FlatFileBO MapFieldSet(IFieldSet fieldSet)
        {
            // Create a new instance of the current mapped object
            return new FlatFileBO
            {
                Code = fieldSet.ReadInt(0),
                Name = fieldSet.ReadRawString(1),
                Description = fieldSet.ReadRawString(2),
                Date = _dateParser.Decode(fieldSet.ReadString(3)),
            };
        }
    }
}

	Note
	Note the use of a dedicated helper (`Summer.Batch.Extra.IDateParser`) to handle the DateTime handling. The `Summer.Batch.Extra.DateParser` is provided, but you can provide your own implementation to cover more specific needs. Please review the corresponding API doc to see what services are provided by the `DateParser`.

Now that all bricks are set, let's build the unity configuration.

Example 6.5. Delimited flat file reader - sample unity configuration

/// <summary>
/// Registers the artifacts required for step FlatFileReader.
/// </summary>
/// <param name="container">the unity container to use for registrations</param>
private void RegisterFlatFileReader(IUnityContainer container)
{
    // Reader - FlatFileReader/FlatFileReader
    container.StepScopeRegistration<IItemReader<FlatFileBO>,
    			FlatFileItemReader<FlatFileBO>>("FlatFileReader/FlatFileReader")
        .Property("Resource")
        	.Resource("#{settings['BA_FLATFILE_READER.FlatFileReader.FILENAME_IN']}")
        .Property("Encoding").Value(Encoding.GetEncoding("UTF-8"))
        .Property("LineMapper")
        	.Reference<ILineMapper<FlatFileBO>>("FlatFileReader/FlatFileReader/LineMapper")
        .Register();

    // Line mapper
    container.StepScopeRegistration<ILineMapper<FlatFileBO>,
    			DefaultLineMapper<FlatFileBO>>("FlatFileReader/FlatFileReader/LineMapper")
        .Property("Tokenizer")
        	.Reference<ILineTokenizer>("FlatFileReader/FlatFileReader/Tokenizer")
        .Property("FieldSetMapper")
        	.Reference<IFieldSetMapper<FlatFileBO>>
        		("FlatFileReader/FlatFileReader/FieldSetMapper")
        .Register();

    // Tokenizer
    container
    	.StepScopeRegistration<ILineTokenizer,
    		DelimitedLineTokenizer>("FlatFileReader/FlatFileReader/Tokenizer")
        .Property("Delimiter").Value(";")
        .Register();

    // Field set mapper
    container
    	.StepScopeRegistration<IFieldSetMapper<FlatFileBO>,
    		FlatFileMapper>("FlatFileReader/FlatFileReader/FieldSetMapper")
        .Register();

    // ... -- processor and writer registration is not being shown here --
}

Note

All registrations within unity container are made using the Step Scope (container.StepScopeRegistration);
In addition to the mandatory LineMapper property, the FlatFileItemReader uses :
- An optional encoding to specify the flat file encoding; Use the Encoding.GetEncoding static methods family to provide a proper encoding;
- Other optional properties not shown in the sample :
  - LinesToSkip : if set, the given number of lines will be skipped at the start of the resource;
  - Strict : flag for the strict mode; in strict mode, an exception will be thrown if the resource to be read does not exist (vs. a simple warn logged is non-strict mode);

Using a flat file writer

Provided implementation is the

Summer.Batch.Infrastructure.Item.File.FlatFileItemWriter<T> class, where T is the type of the business object that will be "dumped" into the flat file. The FlatFileItemWriter uses the following properties:

Mandatory properties (to be set at initialization time):

Resource : the resource to be written to;
LineAggregator: a class implementing the
Summer.Batch.Infrastructure.Item.File.Transform.ILineAggregator<in T> interface
; this class is responsible of aggregating the business object properties into a single string that can be used to write a line into the target flat file.

Optional properties (some having default values):
- LineSeparator : the line separator for the lines to write in the flat file;defaults to System.Environment.NewLine;
- Transactional : a flag to tell whether the writer should take part in the active transaction (meaning that data will be effectively written at commit time); defaults to true;
- AutoFlush : a flag to tell whether the writer buffer should be flushed after each write; defaults to false;
- SaveState: a flag to tell if the state of the item writer should be saved in the execution context when the Update method is called; defaults to true;
- AppendAllowed: a flag to tell if an existing resource should be written to in append mode; defaults to false;
- DeleteIfExists: a flag to tell if an existing resource should be deleted; if AppendAllowed is set to true, this flag is IGNORED; defaults to false;
- DeleteIfEmpty: a flag to tell if an empty target resource (no lines were written) should be deleted; defaults to false;
- HeaderWriter: a header writer (class implementing the
  Summer.Batch.Infrastructure.Item.File.IHeaderWriter interface); Used to write the header of the file, if this makes sense; No default value;
- FooterWriter: a footer writer (class implementing the
  Summer.Batch.Infrastructure.Item.File.IFooterWriter interface); Used to write the footer of the file, if this makes sense; No default value;

The writing process takes a business object as input, transforms it into a string using the LineAggregator and append the string to the target resource.

Now let's review an example; first, the job xml configuration :

Example 6.6. FlatFileItemWriter declaration in the job xml file

<step id="step1">
	<chunk item-count="1000">
		...
		<writer ref="step1/FlatFileWriter" />
	</chunk>
</step>

The crucial part is the LineAggregator ; Summer Batch comes with several ILineAggregator implementations :

Summer.Batch.Infrastructure.Item.File.Transform.DelimitedLineAggregator<T>: transforms an object properties into a delimited list of strings. Default delimiter is comma, but can set to any arbitrary string; This is the natural choice to write CSV files;
Summer.Batch.Infrastructure.Item.File.Transform.FormatterLineAggregator<T>: transforms an object properties into a string, using a provided format (the computed string is the result of a call to string.Format method, using the provided format.);
Summer.Batch.Infrastructure.Item.File.Transform.PassThroughLineAggregator<T>: transforms an object to a string by simply calling the ToString method of the object;

Our example uses the FormatterLineAggregator, providing a format string through the unity configuration; To fully understand the unity configuration that follows, the used business object EmployeeDetailBO must be shown :

Example 6.7. Sample flat file writer input business object

using System;

namespace Com.Netfective.Bluage.Business.Batch.Flatfilewriter.Bo
{
    /// <summary>
    /// Entity EmployeeDetailBO.
    /// </summary>
    [Serializable]
    public class EmployeeDetailBO
    {
        /// <summary>
        /// Property EmpId.
        /// </summary>
        public int? EmpId { get; set; }

        /// <summary>
        /// Property EmpName.
        /// </summary>
        public string EmpName { get; set; }

        /// <summary>
        /// Property Name.
        /// </summary>
        public string Name { get; set; }

        /// <summary>
        /// Property EmpDob.
        /// </summary>
        public DateTime? EmpDob { get; set; }

        /// <summary>
        /// Property EmpSalary.
        /// </summary>
        public decimal? EmpSalary { get; set; }

        /// <summary>
        /// Property EmailId.
        /// </summary>
        public string EmailId { get; set; }

        /// <summary>
        /// Property BuildingNo.
        /// </summary>
        public int? BuildingNo { get; set; }

        /// <summary>
        /// Property StreetName.
        /// </summary>
        public string StreetName { get; set; }

        /// <summary>
        /// Property City.
        /// </summary>
        public string City { get; set; }

        /// <summary>
        /// Property State.
        /// </summary>
        public string State { get; set; }

    }
}

Eventually, here comes the unity configuration :

Example 6.8. Formatted flat file writer - sample unity configuration

/// <summary>
/// Registers the artifacts required for step step1.
/// </summary>
/// <param name="container">the unity container to use for registrations</param>
private void RegisterStep1(IUnityContainer container)
{
 	   //... -- reader and processor registration is not being shown here --
       // step1/FlatFileListWriter/Delegate Writer
       container.StepScopeRegistration<IItemWriter<EmployeeDetailBO>,
       		FlatFileItemWriter<EmployeeDetailBO>>("step1/FlatFileWriter")
           .Property("Resource")
           	.Resource("#{settings['BA_FLAT_FILE_WRITER.step1.FlatFileWriter.FILENAME_OUT']}")
           .Property("Encoding").Value(Encoding.GetEncoding("UTF-8"))
           .Property("LineAggregator")
           	.Reference<ILineAggregator<EmployeeDetailBO>>("step1/FlatFileWriter/LineAggregator")
           .Register();

       // Line aggregator
       container.StepScopeRegistration<ILineAggregator<EmployeeDetailBO>,
       		FormatterLineAggregator<EmployeeDetailBO>>("step1/FlatFileWriter/LineAggregator")
           .Property("Format")
           	.Value("{0},{1},{2},{3:yyyy-MM-dd},{4},{5},{6},{7},{8},{9}")
           .Property("FieldExtractor")
           	.Reference<IFieldExtractor<EmployeeDetailBO>>("step1/FlatFileWriter/FieldsExtractor")
           .Register();

       // Fields Extractor
       container.StepScopeRegistration<IFieldExtractor<EmployeeDetailBO>,
       		PropertyFieldExtractor<EmployeeDetailBO>>("step1/FlatFileWriter/FieldsExtractor")
				.Property("Names").LateBinding<string[]>("EmpId,EmpName,Name,EmpDob,EmpSalary," +
                    "EmailId,BuildingNo,StreetName,City,State")
           .Register();
}

Note

The FormatterLineAggregator requires a Summer.Batch.Infrastructure.Item.File.Transform.IFieldExtractor<in T> implementation to be provided at initialization time; The IFieldExtractor is in charge of converting a business object into an array of its parts (array of values, build using the object properties); Summer Batch provides several implementations :
- S.B.Infrastructure.Item.File.Transform.PropertyFieldExtractor<T>: this is the implementation being used in the example; it retrieves values from the property names (using the Names property); Examining the line :
```
.Property("Names").LateBinding<string[]>("EmpId,EmpName,Name,EmpDob,EmpSalary,"
							+"EmailId,BuildingNo,StreetName,City,State")
```
  we see that ALL the EmployeeDetailBO properties are being selected to be written to the target flat file; The order is significant. The properties values will be passed in that order to the string.Format method used by the FormattedLineAggregator. The format being used
```
.Property("Format").Value("{0},{1},{2},{3:yyyy-MM-dd},{4},{5},{6},{7},{8},{9}")
```
  indicates that all selected properties will be effectively written to the flat file.
- S.B.Infrastructure.Item.File.Transform.PassThroughFieldExtractor<object>: this implementation returns the business object as an array; see dedicated api doc for details;