But you
need to filter the ticks, Dennis—hence the need to implement it in the DLL.
Per TJ, his
IQFeed plugin “has ATR-based filter
(user-configurable) that filters out most of bad ticks from 1-minute bars and
up”.
Could be
some complications in that the 1.90 DLL dealt with pre tick-based IQFeed, but
should still be a slam dunk if he has full access to the eSignal source.
Bob
-----Original
Message-----
From: amibroker@xxxxxxxxxxxxxxx
[mailto:amibroker@xxxxxxxxxxxxxxx]On Behalf
Of Dennis Brown
Sent: Sunday, August 05, 2007 8:31
PM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Second
thoughts on Static arrays --second problem set
Bob,
Thanks for
your comments to Tomasz on my behalf.
I have
experimented with a number of bad tick data filters so far, and none is perfect
(at the 5 second level). The data is simply not available to determine
the right answer. It is a tricky and error prone process even at the tick
level, as has been discussed here before. I would certainly welcome an
accurate built-in filter as that would mean fewer data errors that I had to
remove by other means. However, I can not see how to do a good job at the
5 second level without being able to take a suspect bar, request the actual
tick data one bar before and including that bar, then process the tick values
to make a final determination, then correct the H or L of the 5 second bar.
As I said before, it is not practical to run a tic database for the high volume
ETFs that I trade.
It is
actually my ATR function (that I base many other indicators on) that suffers
from bad ticks the most. If I were to use the ATR function to clip the
bad ticks (bad ticks increase the ATR value) to make the ATR value smaller,
more ticks would be called bad, creating an interesting paradox.
Depending on where you start in the data stream, you could get different
answers. Over clipping drives the ATR value down causing more over
clipping. Under clipping increases the ATR value causing more under clipping.
I took a
different approach. I used Median(H,4) or Median(L,4) as key values.
I first used logic to trim back a H or L that was also the O or C to the
key values. Then I did not let any bar exceed the key values + the height
of a bar to a key value. This works good as a first pass filter (the
after hours stuff gets a little hard to clean up).
I found that
I had to switch between no filtering and two different types of filtering
depending on the nature of the bad data and also turn 24 hour mode on and off
to see how much filtering works best for the ATR function at any point in time.
Opening gaps cause me to need pre-market data. Not perfect, but
good enough.
Also don't
forget that part of my problem is that I need to be able to select the bar
volume from the AFL. I did not know I could override the user selections
for intraday preferences from AFL. Have I missed something here?
Dennis
On Aug 5,
2007, at 8:39 PM, Bob Jagow wrote:
Tomasz,
I’d
think that the solution to Dennis’s “Second
Problem Set” [which is merely bad-tick filtering] is already at hand in the ATR
filter you wrote for the IQFeed 1.9.0 plugin.
Since
it appears that you didn’t provide C++ source for either IQFeed or eSignal,
looks like you’re da only man who could add the filter to a custom eSignal DLL.
Bob
-----Original Message-----
From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx]On Behalf Of Tomasz
Janeczko
Sent: Sunday, August 05, 2007 2:18 AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Second thoughts on
Static arrays --whole problem set
Hello,
Regarding ATC bar requirements: that's true because ATC is
designed to create composites and
for consistency it always requests all bars so composite is
complete (generated from all quotes, not just from "viewing" area).
This behaviour however can be turned off using
SetBarsRequired() placed at the *end* of the formula
(that will overwrite any AB's own calculations). See http://finance.groups.yahoo.com/group/amibroker/message/113635
for more details.
From what I read below, you want "static array"
to be used ONLY when BarCount does not change and symbol does not change.
Well - this could work without synchronization and all
"bookkeeping" I mentioned below and I guess it should solve all your
problems
because it would be roughly as fast as other arrays,
BUT....I have only one problem with this idea - that I will get endless
"bug reports" from people not reading the docs even
if I write in big red letters that static arrays do not perform any checking/
aligninig/ padding
the way foreign() works and that the user is responsible
for using it wisely. I am not speaking about you or anyone else experienced in
AB, because I am sure you know
what you are doing, but I mean newbies :-)
Maybe make it "hidden" functionality???
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message
-----
Sent: Sunday,
August 05, 2007 3:53 AM
Subject: Re:
[amibroker] Second thoughts on Static arrays --whole problem set
Tomasz,
First let me thank you for your patients in explaining this
one more time. It makes a very good post for the benefit of others.
Yes, I am aware of everything you explained because you had
explained it before and I listened and tried out the ideas.
I sent you a direct email through support #39911 on May 16
for which I received no reply (and none was technically required). I
will quote it:
"I am taking this subject off the
yahoo board now because I don't think anyone can help but you.
It can go back on the board when there is a
solution that will work.
I took your advice and made all the
changes you suggested to speed up my AFL.
The functionality of the AFL is working
nicely, behaving just as it should.
However, instead of running 10 times
faster, now it runs twice as slow as before.
I have narrowed the problem down to the
very thing that was supposed to speed it up.
1. When I use the ATC & Foreign
functions the number of bars required goes from 3518, 0 to 1000000, 1000000.
This looks like it turns off fast AFL
and slows all normal processing by requiring all the bars. I have 77,000 data
bars total.
I do not know if this is intentional or a
bug.
2. The ATC function is slow, but I
only do it once per bar (1 minute).
3. The foreign function which is done
every time is slow also (relative to normal arrays).
It takes .04 seconds each and I need dozens
to replace the missing static array functionality.
Perhaps this is also due to the 1000000
bars.
Without being able to have persistent
arrays that are as fast as normal arrays I will not be able to achieve my
trading goals for very many years until computers speed up by 10x. That
is the same as never to me. I wish I could just declare some variables to
be Persistent or Static and be done with it. It is easy for me to say,
but perhaps it is not so easy for you to do. However, being able to
successfully use the methods discussed on the board opens up a lot of extra
possibilities due to the extra speed possible with AFL.
I appreciate any help you can give in this
matter."
As I want a workable solution to my problem and want to use
AB for my project (I would be trading this already using my AB system if I had
a solution), I will state the fundamental problems I face clearly together
here (instead of disjointed posts like before) and then perhaps
a solution might occur to you or somebody else. Please forgive
the length and complexity of this post.
Background:
My trading system is very simple in principle. AFL
calculates a complex indicator, I draw manual trend lines, and I place manual
trades on a single stock or ETF at a time. It is day trading, so I only
have a few seconds to trigger my trade when the conditions are right. My
AFL allows rapid manual selection of different timeframes via a
parameter, and selects the proper volume bar settings dynamically for different
ETFs or stocks. The program automatically runs through many timeframes to
select trade trigger possibilities for me to consider.
1. There is a heavy compute environment --several
seconds of AFL.
2. The environment is highly interactive --drawing
trend lines.
3. Seconds count for placing trades.
First Problem set:
Under heavy compute loads (when AFL time is greater than
refresh rate), AB stops responding to the UI.
Workarounds:
The only way I can get some UI functionality back is to set
the refresh rate to one second longer than the AFL compute time. Then I
can get UI functionality back, but with many delays.
I also have a ParamToggle() to turn off most indicator
processing so that I can interact with the UI better at times (but I lose my
indicator plot references).
This is a partially effective workaround that I could limp
along with.
Requested solution:
Realizing that the root problem is that the UI functions only
run (or initiate) in between AFL passes, I looked for a way to do less
processing on each pass. The AFL load could be much less most of the time
because I only need to calculate everything when a new bar is added.
I thought of two possibilities:
1. Ask for an AFL function that is essentially
DoUserInterface(). Then put AFL into an infinite loop and put these
commands at appropriate places. Since this is backwards from the way the
AFL appears to work, I did not make this suggestion.
2. Ask for arrays that keep their data between AFL
passes. I tried using ATC for this. It functioned like it was
supposed to, but was too slow to be a practical solution. I understand
the reason they are so slow, and that is because they are a true static
permanent array. However, what I wanted was only for a normal array to be
saved between passes. They only need to be identified by which pane they
are associated with (could be part of the internal name). They are only
valid between new bars (same array size). Changes to symbol, timeframe,
parameters, etc., can be detected in the AFL and the array updated. I
know this can be done, because I did it for the ATC version I wrote and
everything worked perfectly (just not faster). Because AFL is so fast
working with arrays, It seems that it should be possible to just save an array
in memory in a high speed way, then recall it to repopulate the array instead
of recalculate the array if the AFL determines that the data would not have
changed since the last pass.
There may be another possible solution that I have not
thought of and I would love to hear it.
Because this is a realtime trading system, decisions have to
be made from raw data which contains bad ticks that may fool the system.
Volume bars are used in the system, and must be able to
change volume per bar on different AFL passes.
Built-in timeframe or volume settings do not allow the
flexibility to address these
Workarounds:
A tick database is too large to have enough history for the
indicators. Running a database of 5 seconds for a basic 1 minute bar
(or equivalent for volume bars) allows for reasonably effective bad
tick removal and volume bar generation. History can be just 60K bars.
Bad ticks are removed and volume bar settings are determined using the 5
second bars. Arrays are built up using TimeFrameSet() to get the actual
bars used for indicator calculations and graph plots. This approach works
really well, however, there is one big problem with it.
Even though I can display all my price and indicators in the
timeframe I desire, I can not draw studies that align to that timeframe.
All studies are drawn in the pane relative to 5 second bars. This
workaround is not acceptable for trading.
Requested
solution:
#1117 Make an AFL way to set the timeframe of the drawing
studies to match the OHLC array created by TimeFrameSet().
Of course implementing this has a general benefit
of giving an almost infinite number of time/volume/range bars settings via AFL
and parameters.
Please consider the problems (not just proposed solutions)
and advise if other possibilities come to mind for solutions.
Thank you very much for your patients and consideration,
Dennis Brown
On Aug 4, 2007, at 6:16 AM, Tomasz Janeczko wrote:
Dennis,
I explained it many
times:
1. You can not simply
"KEEP" the array from previous execution and reference it
directy becase:
a) previous execution
may be on different symbol/different time frame/different zoom level leading
to DIFFERENT NUMBER OF
ELEMENTS in the array and different TIMESTAMPS of array bars.
2. Because of (1) the
only way to implement "static array" is to:
a) store not only
array BUT also TIMESTAMPS with every bar
b) during reading
perform timestamp checking and synchronization so referenced array data is
aligned
with current execution
data
If those two were not
implemented you would NOT be able to use "static arrays" at all.
3. Functionality already
exists
a) Point 2a - storing
arrays with timestamps is exactly what AddToComposite ALREADY DOES
b) Point 2b - reading
arrays with timestamps and aligning is exactly what Foreign ALREADY DOES
4. Implementing
dedicated "static" array functions would
a) give NO SPEED
ADVANTAGE over AddToComposite/Foreign
b) give only "the
impression" that it is something else and "faster" than
ATC/Foreign while it would
be FALSE impression.
Summary:
a) functionality already
exists
b) no speed advantage
over existing functionality
So here are the
functions you are asking for:
function StaticVarArraySet( Varname, array )
{
AddToComposite( array, "~SA_"+VarName, "C", atcFlagDefaults | atcFlagEnableInBacktest | atcFlagEnableInExplore | atcFlagEnableInIndicator | atcFlagEnableInPortfolio );
}
function StaticVarArrayGet( VarName );
{
return Foreign( "~SA_"+VarName, "C" );
}
Best regards,
Tomasz Janeczko
amibroker.com
----- Original Message
-----
Sent: Saturday,
August 04, 2007 5:26 AM
Subject: [amibroker]
Second thoughts on Static arrays
Hi,
The more I have thought about static arrays for
my AFL needs, the more I see that what I am looking for is a quite
limited implementation.
Static arrays would have low overhead and give
the speed needed just like regular arrays. The primary need is to
avoid re-computing complex things all the time that only need to be computed
under certain conditions or perhaps once per new bar. This allows far
richer AFLs to be developed for real time trading, while improving the
responsiveness of the UI interaction under heavy compute conditions.
The life of a static array is fleeting.
Static arrays are not permanent like ATC, and do
not have the big overhead. Static arrays as I envision them, would have
to be updated via the user AFL program whenever a change in environment caused
the data to become invalid --like changing the symbol or the timeframe --or
even adding a new bar (which changes the array size). Having static
arrays like this would allow breaking the AFL into two parts --the fast part
that needs to run for every second, and the part that needs to run once per
compressed timeframe, volume, or range bar.
I am sure there are technical issues involved, or
TJ would have already provided them since they are so useful for speeding up
AFL. First rule of making programs fast is pre-compute everything possible and
not inside a loop. AFL is one big loop.
Perhaps what I am looking for are not technically
"static" arrays in the normal sense. Maybe I need to call them
by a different name for the suggestion box. Perhaps there is an even
easier way to address this need with a new AFL syntax: keep(myArray);
Comments anyone?
Dennis Brown