Tomasz,
I’d think
that the solution to Dennis’s “Second
Problem Set” [which is merely bad-tick filtering] is already at hand in the ATR
filter you wrote for the IQFeed 1.9.0 plugin.
Since it
appears that you didn’t provide C++ source for either IQFeed or eSignal, looks
like you’re da only man who could add the filter to a custom eSignal DLL.
Bob
-----Original
Message-----
From: amibroker@xxxxxxxxxxxxxxx
[mailto:amibroker@xxxxxxxxxxxxxxx]On Behalf
Of Tomasz Janeczko
Sent: Sunday, August 05, 2007 2:18
AM
To: amibroker@xxxxxxxxxxxxxxx
Subject: Re: [amibroker] Second
thoughts on Static arrays --whole problem set
Hello,
Regarding ATC
bar requirements: that's true because ATC is designed to create composites and
for
consistency it always requests all bars so composite is complete (generated
from all quotes, not just from "viewing" area).
This
behaviour however can be turned off using SetBarsRequired() placed at the *end*
of the formula
(that will
overwrite any AB's own calculations). See http://finance.groups.yahoo.com/group/amibroker/message/113635
for more
details.
From what I
read below, you want "static array" to be used ONLY when BarCount
does not change and symbol does not change.
Well - this
could work without synchronization and all "bookkeeping" I mentioned
below and I guess it should solve all your problems
because it
would be roughly as fast as other arrays, BUT....I have only one problem with
this idea - that I will get endless
"bug
reports" from people not reading the docs even if I write in big red
letters that static arrays do not perform any checking/ aligninig/ padding
the way
foreign() works and that the user is responsible for using it wisely. I am not
speaking about you or anyone else experienced in AB, because I am sure you know
what you are
doing, but I mean newbies :-)
Maybe make it
"hidden" functionality???
Best regards,
Tomasz Janeczko
amibroker.com
-----
Original Message -----
To: amibroker@xxxxxxxxxxxxxxx
Sent: Sunday, August 05, 2007
3:53 AM
Subject: Re:
[amibroker] Second thoughts on Static arrays --whole problem set
Tomasz,
First let me
thank you for your patients in explaining this one more time. It makes a
very good post for the benefit of others.
Yes, I am
aware of everything you explained because you had explained it before and I
listened and tried out the ideas.
I sent you a
direct email through support #39911 on May 16 for which I received no reply
(and none was technically required). I will quote it:
"I
am taking this subject off the yahoo board now because I don't think anyone can
help but you.
It can
go back on the board when there is a solution that will work.
I took
your advice and made all the changes you suggested to speed up my AFL.
The
functionality of the AFL is working nicely, behaving just as it should.
However,
instead of running 10 times faster, now it runs twice as slow as before.
I have
narrowed the problem down to the very thing that was supposed to speed it up.
1.
When I use the ATC & Foreign functions the number of bars required goes
from 3518, 0 to 1000000, 1000000.
This
looks like it turns off fast AFL and slows all normal processing by requiring
all the bars. I have 77,000 data bars total.
I do
not know if this is intentional or a bug.
2.
The ATC function is slow, but I only do it once per bar (1 minute).
3.
The foreign function which is done every time is slow also (relative to normal
arrays).
It
takes .04 seconds each and I need dozens to replace the missing static array
functionality.
Perhaps
this is also due to the 1000000 bars.
Without
being able to have persistent arrays that are as fast as normal arrays I will
not be able to achieve my trading goals for very many years until computers
speed up by 10x. That is the same as never to me. I wish I could
just declare some variables to be Persistent or Static and be done with it.
It is easy for me to say, but perhaps it is not so easy for you to do.
However, being able to successfully use the methods discussed on the board
opens up a lot of extra possibilities due to the extra speed possible with AFL.
I
appreciate any help you can give in this matter."
As I want a workable solution to my
problem and want to use AB for my project (I would be trading this already
using my AB system if I had a solution), I will state the fundamental problems
I face clearly together here (instead of disjointed posts like before) and
then perhaps a solution might occur to you or somebody else.
Please forgive the length and complexity of this post.
Background:
My trading system is very simple
in principle. AFL calculates a complex indicator, I draw manual
trend lines, and I place manual trades on a single stock or ETF at a time.
It is day trading, so I only have a few seconds to trigger my trade when
the conditions are right. My AFL allows rapid manual selection of different timeframes
via a parameter, and selects the proper volume bar settings dynamically for
different ETFs or stocks. The program automatically runs through many
timeframes to select trade trigger possibilities for me to consider.
1. There is a heavy compute environment --several
seconds of AFL.
2. The environment is highly
interactive --drawing trend lines.
3. Seconds count for placing trades.
First Problem set:
Under heavy compute loads (when AFL time
is greater than refresh rate), AB stops responding to the UI.
Workarounds:
The only way I can get some UI
functionality back is to set the refresh rate to one second longer than the AFL
compute time. Then I can get UI functionality back, but with many delays.
I also have a ParamToggle() to turn off
most indicator processing so that I can interact with the UI better at times
(but I lose my indicator plot references).
This is a partially effective workaround
that I could limp along with.
Requested solution:
Realizing that the root problem is that
the UI functions only run (or initiate) in between AFL passes, I looked for a
way to do less processing on each pass. The AFL load could be much less
most of the time because I only need to calculate everything when a new bar is added.
I thought of two possibilities:
1. Ask for an AFL function that is
essentially DoUserInterface(). Then put AFL into an infinite loop and put
these commands at appropriate places. Since this is backwards from the
way the AFL appears to work, I did not make this suggestion.
2. Ask for arrays that keep their
data between AFL passes. I tried using ATC for this. It functioned
like it was supposed to, but was too slow to be a practical solution. I
understand the reason they are so slow, and that is because they are a true
static permanent array. However, what I wanted was only for a normal
array to be saved between passes. They only need to be identified by
which pane they are associated with (could be part of the internal name).
They are only valid between new bars (same array size). Changes to
symbol, timeframe, parameters, etc., can be detected in the AFL and the array
updated. I know this can be done, because I did it for the ATC version I
wrote and everything worked perfectly (just not faster). Because AFL is
so fast working with arrays, It seems that it should be possible to just save
an array in memory in a high speed way, then recall it to repopulate the array
instead of recalculate the array if the AFL determines that the data would not
have changed since the last pass.
There may be another possible solution
that I have not thought of and I would love to hear it.
http://www.amibroker.com/bin/ADK.exe
Because this is a realtime trading system,
decisions have to be made from raw data which contains bad ticks that may fool
the system.
Volume bars are used in the system, and
must be able to change volume per bar on different AFL passes.
Built-in timeframe or volume settings do
not allow the flexibility to address these
Workarounds:
A tick database is too large to have
enough history for the indicators. Running a database of 5 seconds for a
basic 1 minute bar (or equivalent for volume bars) allows for
reasonably effective bad tick removal and volume bar generation. History
can be just 60K bars. Bad ticks are removed and volume bar settings are
determined using the 5 second bars. Arrays are built up using
TimeFrameSet() to get the actual bars used for indicator calculations and graph
plots. This approach works really well, however, there is one big problem
with it.
Even though I can display all my price and
indicators in the timeframe I desire, I can not draw studies that align to that
timeframe. All studies are drawn in the pane relative to 5 second bars.
This workaround is not acceptable for trading.
Requested solution:
#1117 Make an AFL way to set the timeframe
of the drawing studies to match the OHLC array created by TimeFrameSet().
Of course implementing this has
a general benefit of giving an almost infinite number of time/volume/range bars
settings via AFL and parameters.
Please consider the
problems (not just proposed solutions) and advise if other possibilities come
to mind for solutions.
Thank you very much for your patients and
consideration,
Dennis Brown
On Aug 4, 2007, at 6:16 AM, Tomasz
Janeczko wrote:
Dennis,
I
explained it many times:
1.
You can not simply "KEEP" the array from previous execution and
reference it directy becase:
a)
previous execution may be on different symbol/different time frame/different
zoom level leading
to
DIFFERENT NUMBER OF ELEMENTS in the array and different TIMESTAMPS of array
bars.
2.
Because of (1) the only way to implement "static array" is to:
a) store
not only array BUT also TIMESTAMPS with every bar
b)
during reading perform timestamp checking and synchronization so referenced
array data is aligned
with
current execution data
If
those two were not implemented you would NOT be able to use "static
arrays" at all.
3.
Functionality already exists
a)
Point 2a - storing arrays with timestamps is exactly what AddToComposite
ALREADY DOES
b)
Point 2b - reading arrays with timestamps and aligning is exactly what Foreign
ALREADY DOES
4.
Implementing dedicated "static" array functions would
a)
give NO SPEED ADVANTAGE over AddToComposite/Foreign
b)
give only "the impression" that it is something else and
"faster" than ATC/Foreign while it would
be
FALSE impression.
Summary:
a)
functionality already exists
b)
no speed advantage over existing functionality
So
here are the functions you are asking for:
function StaticVarArraySet( Varname, array )
{
AddToComposite( array, "~SA_"+VarName, "C", atcFlagDefaults | atcFlagEnableInBacktest | atcFlagEnableInExplore | atcFlagEnableInIndicator | atcFlagEnableInPortfolio );
}
function StaticVarArrayGet( VarName );
{
return Foreign( "~SA_"+VarName, "C" );
}
Best regards,
Tomasz Janeczko
amibroker.com
-----
Original Message -----
To: amibroker@xxxxxxxxxxxxxxx
Sent: Saturday, August 04,
2007 5:26 AM
Subject: [amibroker] Second
thoughts on Static arrays
Hi,
The
more I have thought about static arrays for my AFL needs, the more I see that
what I am looking for is a quite limited implementation.
Static arrays would
have low overhead and give the speed needed just like regular
arrays. The primary need is to avoid re-computing complex things all
the time that only need to be computed under certain conditions or perhaps once
per new bar. This allows far richer AFLs to be developed for real time
trading, while improving the responsiveness of the UI interaction under
heavy compute conditions. The life of a static array is fleeting.
Static arrays are not
permanent like ATC, and do not have the big overhead. Static arrays as I
envision them, would have to be updated via the user AFL program whenever a
change in environment caused the data to become invalid --like changing the
symbol or the timeframe --or even adding a new bar (which changes the array
size). Having static arrays like this would allow breaking the AFL into
two parts --the fast part that needs to run for every second, and the part that
needs to run once per compressed timeframe, volume, or range bar.
I am sure there are
technical issues involved, or TJ would have already provided them since they
are so useful for speeding up AFL. First rule of making programs fast is
pre-compute everything possible and not inside a loop. AFL is one
big loop.
Perhaps what I am
looking for are not technically "static" arrays in the normal sense.
Maybe I need to call them by a different name for the suggestion box.
Perhaps there is an even easier way to address this need with a new AFL
syntax: keep(myArray);
Comments anyone?
Dennis Brown