Quantcast
Channel: User andrewsi - Stack Overflow
Viewing all articles
Browse latest Browse all 51

Transformer stage and dynamic files structure

$
0
0

I'm reading a datafile that's being generated from a data source that's denormalizing data into a flat file. As an example:

References,Reference_Count,Reference_1,Reference_2"abc,def",2,abc,defabc,1,abc,

This would be easy to load in, except that at some point, we're going to get an entry that has a third reference, which will then include a Reference_3 column in the input file, and I want to be able to process that column automatically, without having to re-write the Transformer.

The References column does contain all the data, concatenated from the different columns; but it's inconsistent - there are columns that use comma separators; some with newlines; and some with no separator at all, which means it can't be used for this.

I've tried using a loop in the Transformer stage, using the Reference_Count field. This gives the correct number of iterations, but I can't figure out how to set up the appropriate column as a loop variable. Using:

Link."Reference_" : @ITERATIION

generates a syntax error; while:

"Reference_": @ITERATION

passed Reference_1 - the name of the column, and not its value.

Is what I'm trying to do possible in DataStage? I can't see much in the documentation about using dynamically-named columns.


Viewing all articles
Browse latest Browse all 51

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>