Preliminary note

For concision, "Summer.Batch." expression may be abbreviated on some occasions using the "S.B." expression, when writing fully qualified names of classes or interfaces;

e.g.:

Summer.Batch.Infrastructure.Item.File.FlatFileItemReader<T>

may be abbreviated as

S.B.Infrastructure.Item.File.FlatFileItemReader<T>

Reading and writing flat files

Reading and writing flat files is a very common batch; flat files are still largely used to share information between components of the same system or to integrate data coming from outer systems.

Using a flat file reader

A flat file reader implementation is directly available within Summer Batch, and covers typical needs. The class to use is Summer.Batch.Infrastructure.Item.File.FlatFileItemReader<T>. The template object T represents business object that will be filled by the records read from flat file. Consequently, some mapping has to be done between the records and their properties of target business object: this is achieved using a line mapper that must be provided at initialization time. A line mapper is a class implementing Summer.Batch.Infrastructure.Item.File.ILineMapper<out T> interface.

A line mapper should implement

T MapLine(string line, int lineNumber)

generic method; this method returns a T object given a line (and its line number within file, which might be relevant).

A default implementation is provided :

Summer.Batch.Infrastructure.Item.File.Mapping.DefaultLineMapper<T>

The mapping is done in a two phases process:

• Read line from the flat file is split into fields, using a tokenizer that must be specified at initialization time ( class that implements Summer.Batch.Infrastructure.Item.File.Transform.ILineTokenizer interface ). Two implementations are being provided to cover the most typical needs:

  • Summer.Batch.Infrastructure.Item.File.Transform.FixedLengthTokenizer: for lines with a fixed-length format. The fields are being specified using ranges (see Summer.Batch.Infrastructure.Item.File.Transform.Range);

  • Summer.Batch.Infrastructure.Item.File.Transform.DelimitedLineTokenizer: for lines holding separated fields, e.g. CSV files (the separator string is configurable and defaults to comma).

• The result of first phase is a field set.

  Note

  see

  Summer.Batch.Infrastructure.Item.File.Transform.IFieldSet

  interface and default implementation

  Summer.Batch.Infrastructure.Item.File.Transform.DefaultFieldSet

          Its fields will be mapped to a business object properties using a field set mapper ( class           that implements Summer.Batch.Infrastructure.Item.File.Mapping.IFieldSetMapper           interface).

          Field set mappers are bound to your business model; each target business object           (intended to be filled by records read from flat file) should have an available mapper.

Now let's see a sample. First, the XML job configuration.

Example 6.1. FlatFileItemReader declaration in the job XML file

<step id="FlatFileReader">
    <chunk item-count="1000">
        <reader ref="FlatFileReader/FlatFileReader" />
        ...
    </chunk>flat file writer
</step>

Wiring unity configuration is a bit more complex. Our sample makes uses of a semicolon (";") separated flat file whose records will be mapped to the FlatFileBO business object. Here is sample flat file data, which we'll be using:

Example 6.2. Sample delimited flat file data

1;FlatFile1 ; FlatFile1 ;20100101

2;FlatFile2 ; FlatFile2 ;20100101

Example 6.3. Sample flat file target business object

using System;

namespace Com.Netfective.Bluage.Business.Batch.Flatfile.Bos
{
    /// <summary>
    /// Entity FlatFileBO.
    /// </summary>
    [Serializable]
    public class FlatFileBO
    {
        /// <summary>
        /// Property Code.
        /// </summary>
        public int? Code { get; set; }

        /// <summary>
        /// Property Name.
        /// </summary>
        public string Name { get; set; }

        /// <summary>
        /// Property Description.
        /// </summary>
        public string Description { get; set; }

        /// <summary>
        /// Property Date.
        /// </summary>
        public DateTime? Date { get; set; }

    }
}

The mapping between data and targeted business object is achieved using IFieldSetMapper

Example 6.4. Sample flat file target business object field set mapper

using Summer.Batch.Extra;
using Summer.Batch.Infrastructure.Item.File.Mapping;
using Summer.Batch.Infrastructure.Item.File.Transform;

namespace Com.Netfective.Bluage.Business.Batch.Flatfile.Bos.Mappers
{
    /// <summary>
    /// Implementation of <see cref="IFieldSetMapper{T}" /> that creates
    /// instances of <see cref="FlatFileBO" />.
    /// </summary>
    public class FlatFileMapper : IFieldSetMapper<FlatFileBO>
    {
        private IDateParser _dateParser = new DateParser();

        /// <summary>
        /// Parser for date columns.
        /// </summary>
        private IDateParser DateParser { set { _dateParser = value; } }

        /// <summary>
        /// Maps a <see cref="IFieldSet"/> to a <see cref="FlatFileBO" />.
        /// <param name="fieldSet">the field set to map</param>
        /// <returns>the corresponding item</returns>
        /// </summary>
        public FlatFileBO MapFieldSet(IFieldSet fieldSet)
        {
            // Create a new instance of the current mapped object
            return new FlatFileBO
            {
                Code = fieldSet.ReadInt(0),
                Name = fieldSet.ReadRawString(1),
                Description = fieldSet.ReadRawString(2),
                Date = _dateParser.Decode(fieldSet.ReadString(3)),
            };
        }
    }
}

Note

Note the use of a dedicated helper (Summer.Batch.Extra.IDateParser) to handle DateTime. By default Summer.Batch.Extra.DateParser is provided, but you can provide your own implementation to cover more specific needs. Please review API doc to see what services are provided by DateParser.

Now that all bricks are set, let's build the unity configuration.

Example 6.5. Delimited flat file reader - sample unity configuration

/// <summary>
/// Registers the artifacts required for step FlatFileReader.
/// </summary>
/// <param name="container">the unity container to use for registrations</param>
private void RegisterFlatFileReader(IUnityContainer container)
{
    // Reader - FlatFileReader/FlatFileReader
    container.StepScopeRegistration<IItemReader<FlatFileBO>,
                FlatFileItemReader<FlatFileBO>>("FlatFileReader/FlatFileReader")
        .Property("Resource")
            .Resource("#{settings['BA_FLATFILE_READER.FlatFileReader.FILENAME_IN']}")
        .Property("Encoding").Value(Encoding.GetEncoding("UTF-8"))
        .Property("LineMapper")
            .Reference<ILineMapper<FlatFileBO>>("FlatFileReader/FlatFileReader/LineMapper")
        .Register();

    // Line mapper
    container.StepScopeRegistration<ILineMapper<FlatFileBO>,
                DefaultLineMapper<FlatFileBO>>("FlatFileReader/FlatFileReader/LineMapper")
        .Property("Tokenizer")
            .Reference<ILineTokenizer>("FlatFileReader/FlatFileReader/Tokenizer")
        .Property("FieldSetMapper")
            .Reference<IFieldSetMapper<FlatFileBO>>
                ("FlatFileReader/FlatFileReader/FieldSetMapper")
        .Register();

    // Tokenizer
    container
        .StepScopeRegistration<ILineTokenizer,
            DelimitedLineTokenizer>("FlatFileReader/FlatFileReader/Tokenizer")
        .Property("Delimiter").Value(";")
        .Register();

    // Field set mapper
    container
        .StepScopeRegistration<IFieldSetMapper<FlatFileBO>,
            FlatFileMapper>("FlatFileReader/FlatFileReader/FieldSetMapper")
        .Register();

    // ... -- processor and writer registration is not being shown here --
}

Note

• All registrations within unity container are made using Step Scope (container.StepScopeRegistration);
• In addition to the mandatory LineMapper property, FlatFileItemReader uses :
  • A resource to be read (= the flat file); In the sample, resource path is read from the Settings.config file using a key;
  • To specify flat file encoding; Use the Encoding.GetEncoding static methods family to provide a proper encoding (Optional);
  • Other optional properties not shown in the sample :
    • LinesToSkip : given number of lines will be skipped at the start of resource;
    • Strict : flag for the strict mode; in strict mode, an exception will be thrown if specified resource does not exist (vs. a simple warn logged in non-strict mode);

Using a flat file writer

Provided implementation is in

Summer.Batch.Infrastructure.Item.File.FlatFileItemWriter<T> class, where T is the type of business object that will be "dumped" into flat file. FlatFileItemWriter uses the following properties:

• Mandatory properties (to be set at initialization time):

  • Resource : resource to be written to;

  • LineAggregator: a class implementing

    Summer.Batch.Infrastructure.Item.File.Transform.ILineAggregator<in T> interface; this class is responsible for aggregating the business object properties into a single string that can be used to write a line into target flat file.

• Optional properties (some having default values):

  • LineSeparator : line separator for the lines to write in flat file; defaults to System.Environment.NewLine;

  • Transactional : a flag to specify if the writer should take part in the active transaction (meaning that data will be effectively written at commit time); defaults to true;

  • AutoFlush : a flag to specify if the writer buffer should be flushed after each write; defaults to false;

  • SaveState : a flag to specify if the state of the item writer should be saved in the execution context when Update method is called; defaults to true;

  • AppendAllowed : a flag to specify if an existing resource should be written to in append mode; defaults to false;

  • DeleteIfExists : a flag to specify if an existing resource should be deleted; if AppendAllowed is set to true, this flag is IGNORED; defaults to false;

  • DeleteIfEmpty : a flag to specify if an empty target resource (no lines were written) should be deleted; defaults to false;

  • HeaderWriter : a header writer (class implementing the

    Summer.Batch.Infrastructure.Item.File.IHeaderWriter interface); Used to write the header of file; No default value;

  • FooterWriter: a footer writer (class implementing the

    Summer.Batch.Infrastructure.Item.File.IFooterWriter interface); Used to write the footer of file; No default value;

The writing process takes a business object as input, transforms it into a string using LineAggregator and append the string to target resource.

Now let's review an example; first, the job XML configuration :

Example 6.6. FlatFileItemWriter declaration in the job XML file

<step id="step1">
    <chunk item-count="1000">
        ...
        <writer ref="step1/FlatFileWriter" />
    </chunk>
</step>

The crucial part is LineAggregator; Summer Batch comes with several ILineAggregator implementations :

• Summer.Batch.Infrastructure.Item.File.Transform.DelimitedLineAggregator<T>: transforms an object properties into a delimited list of strings. Default delimiter is comma, but can set to any arbitrary string; This is the natural choice to write CSV files;

• Summer.Batch.Infrastructure.Item.File.Transform.FormatterLineAggregator<T>: transforms an object properties into a string, using a provided format (the computed string is the result of a call to string.Format method, using provided format.);

• Summer.Batch.Infrastructure.Item.File.Transform.PassThroughLineAggregator<T>: transforms an object to a string by simply calling ToString method of the object;

Our example uses FormatterLineAggregator, providing a format string through unity configuration; To fully understand unity configuration that follows, used business object EmployeeDetailBO must be shown:

Example 6.7. Sample flat file writer input business object

using System;

namespace Com.Netfective.Bluage.Business.Batch.Flatfilewriter.Bo
{
    /// <summary>
    /// Entity EmployeeDetailBO.
    /// </summary>
    [Serializable]
    public class EmployeeDetailBO
    {
        /// <summary>
        /// Property EmpId.
        /// </summary>
        public int? EmpId { get; set; }

        /// <summary>
        /// Property EmpName.
        /// </summary>
        public string EmpName { get; set; }

        /// <summary>
        /// Property Name.
        /// </summary>
        public string Name { get; set; }

        /// <summary>
        /// Property EmpDob.
        /// </summary>
        public DateTime? EmpDob { get; set; }

        /// <summary>
        /// Property EmpSalary.
        /// </summary>
        public decimal? EmpSalary { get; set; }

        /// <summary>
        /// Property EmailId.
        /// </summary>
        public string EmailId { get; set; }

        /// <summary>
        /// Property BuildingNo.
        /// </summary>
        public int? BuildingNo { get; set; }

        /// <summary>
        /// Property StreetName.
        /// </summary>
        public string StreetName { get; set; }

        /// <summary>
        /// Property City.
        /// </summary>
        public string City { get; set; }

        /// <summary>
        /// Property State.
        /// </summary>
        public string State { get; set; }

    }
}

Unity configuration:

Example 6.8. Formatted flat file writer - sample unity configuration

/// <summary>
/// Registers the artifacts required for step step1.
/// </summary>
/// <param name="container">the unity container to use for registrations</param>
private void RegisterStep1(IUnityContainer container)
{
       //... -- reader and processor registration is not being shown here --
       // step1/FlatFileListWriter/Delegate Writer
       container.StepScopeRegistration<IItemWriter<EmployeeDetailBO>,
                FlatFileItemWriter<EmployeeDetailBO>>("step1/FlatFileWriter")
           .Property("Resource")
                .Resource("#{settings['BA_FLAT_FILE_WRITER.step1.FlatFileWriter.FILENAME_OUT']}")
           .Property("Encoding").Value(Encoding.GetEncoding("UTF-8"))
           .Property("LineAggregator")
                .Reference<ILineAggregator<EmployeeDetailBO>>("step1/FlatFileWriter/LineAggregator")
           .Register();

       // Line aggregator
       container.StepScopeRegistration<ILineAggregator<EmployeeDetailBO>,
                FormatterLineAggregator<EmployeeDetailBO>>("step1/FlatFileWriter/LineAggregator")
           .Property("Format")
                .Value("{0},{1},{2},{3:yyyy-MM-dd},{4},{5},{6},{7},{8},{9}")
           .Property("FieldExtractor")
                .Reference<IFieldExtractor<EmployeeDetailBO>>("step1/FlatFileWriter/FieldsExtractor")
           .Register();

       // Fields Extractor
       container.StepScopeRegistration<IFieldExtractor<EmployeeDetailBO>,
            PropertyFieldExtractor<EmployeeDetailBO>>("step1/FlatFileWriter/FieldsExtractor")
                .Property("Names").LateBinding<string[]>("EmpId,EmpName,Name,EmpDob,EmpSalary," +
                    "EmailId,BuildingNo,StreetName,City,State")
           .Register();
}

Note

• FormatterLineAggregator requires a Summer.Batch.Infrastructure.Item.File.Transform.IFieldExtractor<in T> implementation at initialization time; The IFieldExtractor is in charge of converting a business object into an array of its parts (array of values, build using the object properties); Summer Batch provides several implementations:

  • S.B.Infrastructure.Item.File.Transform.PropertyFieldExtractor<T>: this is the implementation being used in the example; it retrieves values from property names (using Names property); Examining the line :

    .Property("Names").LateBinding<string[]>("EmpId,EmpName,Name,EmpDob,EmpSalary,"
                        +"EmailId,BuildingNo,StreetName,City,State")
    

    we see that all the EmployeeDetailBO properties are being selected to be written to target flat file; The order is significant.

    These properties values will be passed in that order to string.Format method which is used by FormattedLineAggregator. The format being used

    .Property("Format").Value("{0},{1},{2},{3:yyyy-MM-dd},{4},{5},{6},{7},{8},{9}")
    

    indicates that all selected properties will be effectively written to targeted flat file.

  • S.B.Infrastructure.Item.File.Transform.PassThroughFieldExtractor<object>: this implementation returns the business object as an array; see api doc for details;