I'm reading a datafile that's being generated from a data source that's denormalizing data into a flat file. As an example:
References,Reference_Count,Reference_1,Reference_2"abc,def",2,abc,defabc,1,abc,
This would be easy to load in, except that at some point, we're going to get an entry that has a third reference, which will then include a Reference_3
column in the input file, and I want to be able to process that column automatically, without having to re-write the Transformer.
The References
column does contain all the data, concatenated from the different columns; but it's inconsistent - there are columns that use comma separators; some with newlines; and some with no separator at all, which means it can't be used for this.
I've tried using a loop in the Transformer stage, using the Reference_Count
field. This gives the correct number of iterations, but I can't figure out how to set up the appropriate column as a loop variable. Using:
Link."Reference_" : @ITERATIION
generates a syntax error; while:
"Reference_": @ITERATION
passed Reference_1
- the name of the column, and not its value.
Is what I'm trying to do possible in DataStage? I can't see much in the documentation about using dynamically-named columns.