Reading variable length files without RDW

[jpdev-sb]

As I understand, in order to read variable length files without RDWs, you need to implement a custom record headers parser and use the record_header_parser option. Is that right? Is there an example implementation you could point me to?

In my files, the record length is indicated in the first field:

08 REC-LNGTH-CNT          PIC S9(5) COMP-3.

Thanks!

[yruslan]

The custom record parser is a possibility. But fortunately your use case seems much simpler:

.option("record_length_field", "REC-LNGTH-CNT")

It should work for variable record length files as a replacement of

.option("is_record_sequence", "true")

(this option should not be used if record_length_field is specified)

I see that option is lacking documentation and, perhaps, an example. Will keep this issue open to add one.

Please, let me know if it worked for you. A custom record header parser example can be found in this test: Test11CustomRDWParser. But hopefully, you won't need it.

[jpdev-sb]

Thanks! I'm glad this option exists. I tried it. But, it seems I'm still missing something. Let me elaborate my situation.

My copybook includes multiple "members", since the datafile contains records of different types. For example:

        01 JP-REC.
           02 CARR-CLM-REC.
             04 CARR-CLM-FIX-GRP.
               06 CLM-REC-IDENT-GRP.
                 08 REC-LNGTH-CNT          PIC S9(5) COMP-3.
...
            02 DMERC-CLM-REC.
              04 DMERC-CLM-FIX-GRP.
                06 CLM-REC-IDENT-GRP.
                  08 REC-LNGTH-CNT          PIC S9(5) COMP-3.
...

The schema that Cobrix prints out is like this:

19/07/30 09:21:01 INFO CobolSchema: Layout positions:
-------- FIELD LEVEL/NAME --------- --ATTRIBS--    FLD  START     END  LENGTH

JP_REC                                                       1 136717 136717
  2 CARR_CLM_REC                                    249      1  24193  24193
    4 CARR_CLM_FIX_GRP                              109      1   1592   1592
      6 CLM_REC_IDENT_GRP                             8      1      8      8
        8 REC_LNGTH_CNT                               4      1      3      3
...
  2 DMERC_CLM_REC                                   462  24194  43120  18927
    4 DMERC_CLM_FIX_GRP                             349  24194  25251   1058
      6 CLM_REC_IDENT_GRP                           257  24194  24201      8
        8 REC_LNGTH_CNT                             253  24194  24196      3
...

So, what should I use as the value of record_length_field? Is it the "fully qualified" name - e.g. JP_REC.CARR_CLM_REC.CLM_REC_IDENT_GRP.REC_LNGTH_CT? If so, this would be problematic, since each record will have a different name for that field. Is it the name that is used in the copybook - i.e. without dashes rather than underscores?

[yruslan]

You have a couple of options here. But all involve tuning the copybook slightly.
But in all cases DMERC_CLM_REC should redefine CARR_CLM_REC since each segment should redefine each other. Ensure other segments, if any, redefine DMERC_CLM_REC pr each other as well.
The options are:

1. Move REC_LNGTH_CNT to level 2 making it a top field independent of all the segments.
   Or
2. Use any of the fully qualified record length fields. Since segments redefine each other the record length fields should have the same position anyway. I'd prefer the first one (JP_REC.CARR_CLM_REC.CLM_REC_IDENT_GRP.REC_LNGTH_CNT).

Sometimes a record length field does not correspond to the actual length. It might or might not include itself as a part of the length. I'm working on adding the ability to specify record length offset via rdw_adjustment option. So if you encounter a situation where specifying the length field doesn't work, please let me know.

[chandrasekaravr]

1. If i adopt Option#2, from above, can i use any wildcard (*) when i specify the record_length_field like so JP_REC.*.*.REC_LNGTH_CNT or level index?

2. If JP_REC. DMERC_CLM_REC and JP_REC.CARR_CLM_REC are two different VLR types in the merged copybook, is it required to have REDEFINE for it to identify the previous segment?

[yruslan]

When you redefine GROUPs that represent segment-specific parts of your data you should end up with the layout like this:

-------- FIELD LEVEL/NAME --------- --ATTRIBS--    FLD  START     END  LENGTH

JP_REC                                                       1 136717 136717
  2 CARR_CLM_REC                                    249      1  24193  24193
    4 CARR_CLM_FIX_GRP                              109      1   1592   1592
      6 CLM_REC_IDENT_GRP                             8      1      8      8
        8 REC_LNGTH_CNT                               4      1      3      3
...
  2 DMERC_CLM_REC                                   462      1  18927  18927
    4 DMERC_CLM_FIX_GRP                             349      1   1058   1058
      6 CLM_REC_IDENT_GRP                           257      1      8      8
        8 REC_LNGTH_CNT                             253      1      3      3
...

(e.g. each segment record starts from position 1)

Here are the answers to your questions:

1. No, you can't use wildcards to specify record length fields. you need to specify a single field that has a stable position across all segments. Since the length of the record is determined before classifying which segment a record belongs to.
2. Yes, the REDEFINE is required since a record can belong only to a one segment type.

Basically, you can think of it this way. Cobrix does not aware of segments in the data since copybooks do not provide this information. So in order to make Cobrix parse multiple record types properly, the copybook should match any of the segments of the interest. The way to achieve this is to make segment-specific GROUPs redefine the same field so they will share the same space in the data.