Re: Limits of charts from Ascii data files?

[Sanford Morton <smorton@xxxxxxxxx>]

On Thu, 6 Feb 2003, Sanford Morton wrote:

> I am trying to load a one minute chart (ES and/or NQ) with one minute
> bars from an ascii data file in TS2000i. I can load only a certain
> number of days.... Every bar is flat, O=H=L=C, and volume is always 1. 

A couple of folks wrote to suggest that the most likely cause of the
problem was duplicate or out of order lines within the ascii data
file, a common problem with cut and paste operations. They were
exactly right, thank you. The day on which my chart hiccupped had a
single duplicate line/bar. As it happened a couple of earlier days
were duplicated in their entirety as well.

My data was from www.anfutures.com (a reasonable deal at $5.90 for a
year of one minute ES or NQ mini data--I've informed them of the
problem). I wrote a Perl script, tailored to their format, to check
for duplicate or out of order bars.  I append it below, though with
other data, you may need to alter the script for different formats.

Thanks again to those who offered advice.

Regards,
Sanford Morton

===================================================================
    # check ANfutures.com data for duplicate or out of
    # order bars
    # usage: perl Check.pl inputfile.csv 

    $priordatetime = '190000000000'; # format yyyy mm dd hr mi
    $lineNum = 0;

    while (<>) {
        $lineNum++;
        
        ($symbol,$interval,$d,$t,$o,$h,$l,$c,$v,$cont) = split(/,/);
        $d =~ m/^(\d\d)(\d\d)(\d\d)$/ or die;
        $yy = $1; $mm = $2; $dd = $3;
        $yy = ($yy < 20 ? '20'.$yy : '19'.$yy); 
        
        $t =~ /^(\d\d):(\d\d)$/ or die;
        $hr = $1; $min = $2;

        $datetime = "$yy$mm$dd$hr$min";
        print "Line Num: $lineNum, $yy/$mm/$dd $hr:$min\n" 
                unless $datetime > $priordatetime;
        $priordatetime = $datetime;
    }

Sample output:

     C:\Perl\ANfutures>perl check.pl nq0212mg.csv
     Line Num: 3028, 2002/09/18 00:01
     Line Num: 10351, 2002/10/01 00:17
     Line Num: 23707, 2002/10/22 15:14