PIPEFIX

PURPOSE   OPERATION   OPTIONS   COMMAND LINES   PARAMETER FILE   RELATED PROGRAMS


Author: Dan Mares, dmares @ maresware . com
Portions Copyright © 1998-2016 by Dan Mares and Mares and Company, LLC
Phone: 678-427-3275
Last update: August 08, 2010

top

PURPOSE

The PIPEFIX program is designed to work with files that contain variable length records containing pipes as field delimiters. This type of file is most often a result of an unload of a database. Most notably, ACCESS, INFORMIX, ORACLE, and DBASE can create this type of output.

If the user knows what the record schema is, (record layout), he can use pipefix to make the records a fixed length, with each field also being a fixed length. Then the output file can be processed as any other fixed length record.

The input records, MUST end with some sort of a carriage return indicator.(CR/LF, CR, or LF).

The user can also trick the program into making files with carriage returns (variable length records) look as if each record contained a single field, and cause pipefix to make the output record, one fixed length field. Thus ending up with fixed length records.


top

OPERATION

To run the program, a (text) parameter file is created. Each line in the paramater file is a number representing the final length of that field in the record. Depending on the length of the field asked for in the parameter file, the input record field can either be truncated, expanded, or eliminated completely from the output record.

The output fields can also be left, or right justified, and 0 (zero) or blank filled depending on the modifier added to each parameter line.

The program first reads the parameter file, which contains information regarding how to process each field of the record.

It then reads the input record to each pipe which represents a corresponding field of the parameter file. The program then either truncates, expands or eliminates that field while adding the proper padding characters as necessary to create a fixed length field in the output record.

It is also, not uncommon to find about 1% bad records. These may not exactly be bad records, but what they actually are, are records that already contain a pipe in one of the character fields. This extra pipe causes the program to think it found a record with too many pipes and thus drops it as a bad record. At this point there is no fix for this situation.

NOTE:

Different databases may output the pipes in unusual seguences at the beginning and end of the records. For instance, one database places a pipe prior to each record, while another may place a pipe at the end of the record and just before the carriage return. If this is the case, you may have to trick the program into thinking there is another field in the record, and use a 0 length parameter line to bypass this extra field.

In other words, become very familiar with the input record before setting up your parameter file. It is suggested that you use a small subset of the input file and test your process prior to running the entire input file.


top

COMMAND LINE

c:>pipefix  input_filename  output_filename  parameter_file_name  [options -APpd ]


top

PARAMETER FILE

Parameter file contents are as follows:

One item per line, created with an ascii text editor. (edit, notepad, or a real editor).

Each line contains one or two items.

The first item (which is mandatory) is the final size you wish this field to be in the output file. Use a 0 (zero) to completely eliminate this field from the output record. This 0 is also used in the situation noted above, where data bases add extra pipes before or after records. These are “fields” that must be eliminated.

The second item, (which is optional,) and immediately follows the first (no spaces between) the first, is an alpha code ( see below ) to justify each field.

Alpha Codes:

No alpha code results in a Default of ‘L’ which Left Justifies the field with a Blank Fill to Right.

An L = same as the default. Left justification, Blank fill to right.   |DATA     |

An l = Left Justification with Zero Fill to the right.   |DATA000|

An R = Right Justification with Blank Fill to the left.  |     DATA|

An r = Right Justification with Zero Fill to the left.  |000DATA|

An M = This field to be treated as a date, and you are certain the format is M/D/Y, like 2/12/2010 and convert to YYYY/MM/DD and Right justify.   |   10/02/12|

An m = This field to be treated as a date, and you are certain the format is M/D/Y, like 2/2/2010 and convert to YYYY/MM/DD and Left justify.   |10/02/2012   |

A Y = This field to be treated as a date, and you are certain the format is Y/M/D 2010/2/15 and convert to YYYY/MM/DD and Right justify.   |   2010/02/12|

A y = This field to be treated as a date, and you are certain the format is Y/M/D 2010/2/15 and convert to YYYY/MM/DD and Left justify.   |99/02/12   |

Note: for the MmYy codes above, the date can have either 2 digit or single digit M, MM, D, DD, and either 2 or 4 digit year 98, 1998, 10, 2010. The if you have a 10 for 2010, etc, the program makes a best guess at the format

A D = This field to be treated as a date, like 2/12/2010 and convert to YYYY/MM/DD and Right justify.   |   2010/02/12|

A d = This field to be treated as a date, like 2/2/2010 and convert to YYYY/MM/DD and Left justify.   |2010/02/12   |

Note: for the dD codes above, the program makes a best guess at the date format and works accordingly. If some of the date formats are not consistant with general style, the program may abort.

Sample file to work with:

This would be the input data file with appropriate pipes delimeting fields.
|one|two|three|four|five|six
|Dan|Mares|1234|anycity|georgia|4049866955

Sample parameter file:

0    /* there is a dummy first field prior to the initial | that is to be eliminated */
2    /* make this 2 characters long. will truncate extra characters*/
3L   /* make this 3 characters, with left justify, blank fill */
0    /* eliminate this field */
4r   /* 4 characters, with right justify, zero fill */
6l   /* 6 characters, with left justify, zero fill */
8R   /* 8 characters, with right justify, blank fill */
12m  /* the field is a m/d/y field and convert to YYYY/MM/DD, left justify, blank fill */
12y  /* the field is a y/m/d field and convert to YYYY/MM/DD, left justify, blank fill */

The overall final record length will be 23 plus 2 for the carriage return and line feed.

Ouput would be:

ontwofourfive00     six
DaMaranycgeorgi40498659


top

OPTIONS

-A:  create/append to default accounting output file called "ACCT-ING"

-P:  maintain pipes in output file

-d + delimiter:  If the input file is pipe delimited, then the -d option is not needed. If the input file is not pipe delimited, indicate what delimiter to use in one of the following formats.

Enter the delimiter as a single character. It may be necessary to quote “|” the character if the character has a special meaning to DOS. As DOS may not pass it directly to the program. If the character is not a special DOS character merely enter it. Be especially careful if the file is “,” comma delimited. As some fields may also have imbedded commas which will give bad results.

If you need to enter a delimiter that is not printable, say for instance the tab character (hex 09) is used as a delimiter then you can enter the ascii value of the delimiter. This must be entered as either a 2 or 3 digit number. So single digit numbers like 9 (tab) must have a leading 0. This would look like -d 09.


RELATED PROGRAMS

CSV2PIPE : convert a "," csv file to pipe delimeted.

top