|
PureBytes Links
Trading Reference Links
|
Dennis,
I believe what you are asking for is different from the ATC and
Foreign functionality since you do not want the persistent disk
storage that ATC provides. I think that point might have been missed
in previous discussions. You said:
"Because AFL is so fast working with arrays, It seems that it should
be possible to just save an array in memory in a high speed way, then
recall it to repopulate the array instead of recalculate the array if
the AFL determines that the data would not have changed since the last
pass."
So you want in-memory arrays that continue to exist outside of a
single AFL script execution, but are not persistent to disk. In other
words, if you close the AB session and reopen, then you do not expect
your static arrays to have survived. I think the term "static arrays"
is appropriate, analogous to "static variables" that are also not
persistent. I'd be happier if static variables also could just store
arrays as well as non-array values.
It does seem that this form of static array memory should be nearly as
fast as the temporary arrays that live only within a single script
execution. They should merely occupy a different part of the AB
memory space. I agree this would be very valuable for me as well,
provided they were fast.
Maybe ATC and Foreign functions effectively work this way by not
saving to disk unless you actually save them explicitly (or implicitly
if you enable that). If so, then it is not clear where the time goes.
Maybe there is a way to optimize the use of these arrays to avoid
time-frame translation, for example.
Daniel LaLiberte
--- In amibroker@xxxxxxxxxxxxxxx, Dennis Brown <see3d@xxx> wrote:
>
> Tomasz,
>
> First let me thank you for your patients in explaining this one more
> time. It makes a very good post for the benefit of others.
> Yes, I am aware of everything you explained because you had explained
> it before and I listened and tried out the ideas.
>
> I sent you a direct email through support #39911 on May 16 for which
> I received no reply (and none was technically required). I will
> quote it:
>
> "I am taking this subject off the yahoo board now because I don't
> think anyone can help but you.
> It can go back on the board when there is a solution that will work.
>
> I took your advice and made all the changes you suggested to speed up
> my AFL.
> The functionality of the AFL is working nicely, behaving just as it
> should.
>
> However, instead of running 10 times faster, now it runs twice as
> slow as before.
>
> I have narrowed the problem down to the very thing that was supposed
> to speed it up.
>
> 1. When I use the ATC & Foreign functions the number of bars
> required goes from 3518, 0 to 1000000, 1000000.
> This looks like it turns off fast AFL and slows all normal
> processing by requiring all the bars. I have 77,000 data bars total.
> I do not know if this is intentional or a bug.
>
> 2. The ATC function is slow, but I only do it once per bar (1 minute).
>
> 3. The foreign function which is done every time is slow also
> (relative to normal arrays).
> It takes .04 seconds each and I need dozens to replace the missing
> static array functionality.
> Perhaps this is also due to the 1000000 bars.
>
> Without being able to have persistent arrays that are as fast as
> normal arrays I will not be able to achieve my trading goals for very
> many years until computers speed up by 10x. That is the same as
> never to me. I wish I could just declare some variables to be
> Persistent or Static and be done with it. It is easy for me to say,
> but perhaps it is not so easy for you to do. However, being able to
> successfully use the methods discussed on the board opens up a lot of
> extra possibilities due to the extra speed possible with AFL.
>
> I appreciate any help you can give in this matter."
>
> As I want a workable solution to my problem and want to use AB for my
> project (I would be trading this already using my AB system if I had
> a solution), I will state the fundamental problems I face clearly
> together here (instead of disjointed posts like before) and then
> perhaps a solution might occur to you or somebody else. Please
> forgive the length and complexity of this post.
>
> Background:
>
> My trading system is very simple in principle. AFL calculates a
> complex indicator, I draw manual trend lines, and I place manual
> trades on a single stock or ETF at a time. It is day trading, so I
> only have a few seconds to trigger my trade when the conditions are
> right. My AFL allows rapid manual selection of different timeframes
> via a parameter, and selects the proper volume bar settings
> dynamically for different ETFs or stocks. The program automatically
> runs through many timeframes to select trade trigger possibilities
> for me to consider.
> 1. There is a heavy compute environment --several seconds of AFL.
> 2. The environment is highly interactive --drawing trend lines.
> 3. Seconds count for placing trades.
>
> First Problem set:
>
> Under heavy compute loads (when AFL time is greater than refresh
> rate), AB stops responding to the UI.
>
> Workarounds:
>
> The only way I can get some UI functionality back is to set the
> refresh rate to one second longer than the AFL compute time. Then I
> can get UI functionality back, but with many delays.
> I also have a ParamToggle() to turn off most indicator processing so
> that I can interact with the UI better at times (but I lose my
> indicator plot references).
> This is a partially effective workaround that I could limp along with.
>
> Requested solution:
>
> Realizing that the root problem is that the UI functions only run (or
> initiate) in between AFL passes, I looked for a way to do less
> processing on each pass. The AFL load could be much less most of the
> time because I only need to calculate everything when a new bar is
> added.
> I thought of two possibilities:
>
> 1. Ask for an AFL function that is essentially DoUserInterface().
> Then put AFL into an infinite loop and put these commands at
> appropriate places. Since this is backwards from the way the AFL
> appears to work, I did not make this suggestion.
>
> 2. Ask for arrays that keep their data between AFL passes. I tried
> using ATC for this. It functioned like it was supposed to, but was
> too slow to be a practical solution. I understand the reason they
> are so slow, and that is because they are a true static permanent
> array. However, what I wanted was only for a normal array to be
> saved between passes. They only need to be identified by which pane
> they are associated with (could be part of the internal name). They
> are only valid between new bars (same array size). Changes to
> symbol, timeframe, parameters, etc., can be detected in the AFL and
> the array updated. I know this can be done, because I did it for the
> ATC version I wrote and everything worked perfectly (just not
> faster). Because AFL is so fast working with arrays, It seems that
> it should be possible to just save an array in memory in a high speed
> way, then recall it to repopulate the array instead of recalculate
> the array if the AFL determines that the data would not have changed
> since the last pass.
>
> There may be another possible solution that I have not thought of and
> I would love to hear it.
>
> Second Problem set:
>
> Because this is a realtime trading system, decisions have to be made
> from raw data which contains bad ticks that may fool the system.
> Volume bars are used in the system, and must be able to change volume
> per bar on different AFL passes.
> Built-in timeframe or volume settings do not allow the flexibility to
> address these
>
> Workarounds:
>
> A tick database is too large to have enough history for the
> indicators. Running a database of 5 seconds for a basic 1 minute bar
> (or equivalent for volume bars) allows for reasonably effective bad
> tick removal and volume bar generation. History can be just 60K
> bars. Bad ticks are removed and volume bar settings are determined
> using the 5 second bars. Arrays are built up using TimeFrameSet() to
> get the actual bars used for indicator calculations and graph plots.
> This approach works really well, however, there is one big problem
> with it.
>
> Even though I can display all my price and indicators in the
> timeframe I desire, I can not draw studies that align to that
> timeframe. All studies are drawn in the pane relative to 5 second
> bars. This workaround is not acceptable for trading.
>
> Requested solution:
>
> #1117 Make an AFL way to set the timeframe of the drawing studies to
> match the OHLC array created by TimeFrameSet().
>
> Of course implementing this has a general benefit of giving an almost
> infinite number of time/volume/range bars settings via AFL and
> parameters.
>
> Please consider the problems (not just proposed solutions) and advise
> if other possibilities come to mind for solutions.
>
> Thank you very much for your patients and consideration,
>
> Dennis Brown
>
>
> On Aug 4, 2007, at 6:16 AM, Tomasz Janeczko wrote:
>
> > Dennis,
> >
> > I explained it many times:
> >
> > 1. You can not simply "KEEP" the array from previous execution and
> > reference it directy becase:
> > a) previous execution may be on different symbol/different time
> > frame/different zoom level leading
> > to DIFFERENT NUMBER OF ELEMENTS in the array and different
> > TIMESTAMPS of array bars.
> >
> > 2. Because of (1) the only way to implement "static array" is to:
> > a) store not only array BUT also TIMESTAMPS with every bar
> > b) during reading perform timestamp checking and synchronization so
> > referenced array data is aligned
> > with current execution data
> >
> > If those two were not implemented you would NOT be able to use
> > "static arrays" at all.
> >
> > 3. Functionality already exists
> > a) Point 2a - storing arrays with timestamps is exactly what
> > AddToComposite ALREADY DOES
> > b) Point 2b - reading arrays with timestamps and aligning is
> > exactly what Foreign ALREADY DOES
> >
> > 4. Implementing dedicated "static" array functions would
> > a) give NO SPEED ADVANTAGE over AddToComposite/Foreign
> > b) give only "the impression" that it is something else and
> > "faster" than ATC/Foreign while it would
> > be FALSE impression.
> >
> > Summary:
> > a) functionality already exists
> > b) no speed advantage over existing functionality
> >
> > So here are the functions you are asking for:
> >
> > function StaticVarArraySet( Varname, array )
> > {
> > AddToComposite( array, "~SA_"+VarName, "C", atcFlagDefaults |
> > atcFlagEnableInBacktest | atcFlagEnableInExplore |
> > atcFlagEnableInIndicator | atcFlagEnableInPortfolio );
> > }
> >
> > function StaticVarArrayGet( VarName );
> > {
> > return Foreign( "~SA_"+VarName, "C" );
> > }
> >
> > Best regards,
> > Tomasz Janeczko
> > amibroker.com
> > ----- Original Message -----
> > From: Dennis Brown
> > To: amibroker@xxxxxxxxxxxxxxx
> > Sent: Saturday, August 04, 2007 5:26 AM
> > Subject: [amibroker] Second thoughts on Static arrays
> >
> > Hi,
> >
> > The more I have thought about static arrays for my AFL needs, the
> > more I see that what I am looking for is a quite limited
> > implementation.
> >
> > Static arrays would have low overhead and give the speed needed
> > just like regular arrays. The primary need is to avoid re-
> > computing complex things all the time that only need to be computed
> > under certain conditions or perhaps once per new bar. This allows
> > far richer AFLs to be developed for real time trading, while
> > improving the responsiveness of the UI interaction under heavy
> > compute conditions. The life of a static array is fleeting.
> >
> > Static arrays are not permanent like ATC, and do not have the big
> > overhead. Static arrays as I envision them, would have to be
> > updated via the user AFL program whenever a change in environment
> > caused the data to become invalid --like changing the symbol or the
> > timeframe --or even adding a new bar (which changes the array
> > size). Having static arrays like this would allow breaking the AFL
> > into two parts --the fast part that needs to run for every second,
> > and the part that needs to run once per compressed timeframe,
> > volume, or range bar.
> >
> > I am sure there are technical issues involved, or TJ would have
> > already provided them since they are so useful for speeding up AFL.
> > First rule of making programs fast is pre-compute everything
> > possible and not inside a loop. AFL is one big loop.
> >
> > Perhaps what I am looking for are not technically "static" arrays
> > in the normal sense. Maybe I need to call them by a different name
> > for the suggestion box. Perhaps there is an even easier way to
> > address this need with a new AFL syntax: keep(myArray);
> >
> > Comments anyone?
> >
> > Dennis Brown
> >
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|