|
PureBytes Links
Trading Reference Links
|
Tomasz,
I just found problem with the SetBarsRequired() approach while trying
to test it on my real AFL application data now.
I am running a 5 second database to generate my clean bars for volume
based aggregation and using the TimeFrameSet() to create a smaller
2000 bar OHLC "working" array. However, I still need ALL the 5
second bars loaded initially to create the smaller OHLC array. The
smaller arrays will not actually be smaller though --just not fully
populated. Saving them using ATC will not result in a fast save of a
small number of bars. So even though the speed of saving 2000 bars
shown below would be useful, I would have to save a 20K-40K size
array and the speed would not be practical.
So I think that when the user selects a timeframe, say 1 minute with
a 5 second database, you aggregate the data into a smaller number of
bars first, then the AFL runs with only these smaller arrays. In
essence I am trying to do the same thing, but all from AFL and I see
that it is not designed to support it this way.
This may sound crazy, but perhaps I can run the 5 second bars and
make my clean volume or time based OHLC bars in one tiny pane and
save them with ATC to a symbol. Then use Foreign to read a smaller
array into my main pane that runs with a SetBarsRequired(2000,0) at
the end.
I will experiment with this concept. Though it seems like I am
jumping through hoops trying to overcome these problems, I am happy
to do it if it is the only way to proceed in the short term to give
me a tradable system solution. However, it takes a lot of time and
energy to do this, and I don't want to go down a fruitless path.
I would appreciate your comments.
Best regards,
Dennis Brown
On Aug 6, 2007, at 8:25 AM, Tomasz Janeczko wrote:
> Hello,
>
> Thank you for your comments.
> And I just realized one thing that largely affects the results.
> That Foreign() funciton will process all bars anyway regardless of
> SetBarsRequired()
> because in principle it was "general-purpose", not "chart-only" and
> does
> not really pay any attention "required bars".
> Something to think about in the future releases....
>
> Best regards,
> Tomasz Janeczko
> amibroker.com
> ----- Original Message -----
> From: "Dennis Brown" <see3d@xxxxxxxxxxx>
> To: <amibroker@xxxxxxxxxxxxxxx>
> Sent: Monday, August 06, 2007 1:55 PM
> Subject: Re: [amibroker] Re: Second thoughts on Static arrays --ATC/
> Foreign times
>
>
>> Tomasz,
>>
>> Yes, I suspected it might be the case that only pointers were being
>> copied for internal arrays. That is why I tried both in my code.
>> for the read mode:
>> myArray = C;
>> and for the write mode:
>> myArray = C + i; (i instead of 1 just in case you did some
>> optimizing with constants)
>>
>> At the resolution of my results, the slowest results for either case
>> were between 1ms and 2ms.
>> This means that the overhead of running the short AFL code shown was
>> dominating the execution time and the array processing time was not
>> even significant enough to noticeably affect the results. So in
>> reality, the array processing is likely faster than I claimed.
>> That is also why I also said that "static arrays" would would be very
>> fast even if they take 10 times as long to process as internal
>> arrays.
>>
>> I was astounded at the efficiency of your code for the internal
>> arrays. You have done a very good job.
>>
>> Best regards,
>> Dennis Brown
>>
>> On Aug 6, 2007, at 6:43 AM, Tomasz Janeczko wrote:
>>
>>> Dennis,
>>>
>>> This is NOT surprising, because internal array does NOT use memory
>>> copy at all!
>>>
>>> Internal arrays are so fast because referencing them does nothing
>>> except
>>> referncing via pointer (4 bytes always, regardless of how long is
>>> the array),
>>> while Foreign has to actually COPY memory (create new array).
>>>
>>> So if you do assignment like this:
>>> myArray = C;
>>>
>>> It effectivelly copies only 4 bytes (the pointer) and increases
>>> "reference counter".
>>> This is what makes it hyper-fast.
>>>
>>> But... as soon as you create NEW array, say by doing this:
>>>
>>> myArray = C + 1;
>>>
>>> Then timing would be much worse because of requried memory copy.
>>> Testing this would give you true estimate of how fast "static
>>> arrays" could be.
>>>
>>> Best regards,
>>> Tomasz Janeczko
>>> amibroker.com
>>> ----- Original Message -----
>>> From: "Dennis Brown" <see3d@xxxxxxxxxxx>
>>> To: <amibroker@xxxxxxxxxxxxxxx>
>>> Sent: Monday, August 06, 2007 5:03 AM
>>> Subject: Re: [amibroker] Re: Second thoughts on Static arrays --ATC/
>>> Foreign times
>>>
>>>
>>>> Tomasz,
>>>>
>>>> I ran a timing test to see what the difference would be for
>>>> using the
>>>> ATC/Foreign functions vs internal arrays with practical values. I
>>>> selected an array size of 2000 and generated 100 uniqueATC symbols.
>>>> Here are the AFL execution times on my machine:
>>>>
>>>> ATC Write: 780ms
>>>> Foreign Read: 310ms
>>>> Internal arrays: 1ms
>>>>
>>>> This is compared to an ATC time of 23000ms before I added the
>>>> SetBarsRequired(2000,0) at the END of the AFL --a 30x improvement
>>>> because there are about 40K bars in the 5 second database.
>>>>
>>>> So the bottom line is that internal arrays seem to be at least 300
>>>> times as fast as ATC/Foreign.
>>>> The potential improvements in speed seem very significant --even if
>>>> static arrays as proposed were 10 times as slow as internal arrays.
>>>> Perhaps it would be possible to just add another mode to ATC/
>>>> Foreign
>>>> that bypassed most overhead for fast static arrays.
>>>>
>>>> Regardless of what you decide to do though, the 30x improvement
>>>> using
>>>> SetBarsRequired(2000,0) at the END is enough to make it worthwhile
>>>> for me to save some of my more time consuming arrays between AFL
>>>> passes using ATC/Foreign to help with the UI responsiveness.
>>>>
>>>> If I had another 30x improvement in "static" array speed (which
>>>> seems
>>>> quite possible), I would save a lot more results and gain
>>>> significantly more speed. In either case I believe I can now move
>>>> forward from the first problem set and it only leaves the second
>>>> problem set to enable me to begin trading this system.
>>>>
>>>> Here is the simple AFL I used for the test. I just turned on
>>>> the AFL
>>>> chart timing for the results.
>>>>
>>>> Thank you,
>>>> Dennis Brown
>>>>
>>>> function StaticArraySet(VarName,array){AddToComposite
>>>> (array ,"~SA_"+VarName ,"C",atcFlagEnableInIndicator|
>>>> atcFlagDefaults);}
>>>> function StaticArrayGet(VarName){return Foreign
>>>> ("~SA_"+VarName,"C");}
>>>>
>>>> TestName = ParamStr("Test Name", "Test");
>>>> Cycles = Param("Cycles",1,1,100,1);
>>>> UseATC = ParamToggle("ATC mode","No|Yes",0);
>>>> Writemode = ParamToggle("R/W mode","Read|Write",1);
>>>>
>>>> for(i=0;i<Cycles;i++)
>>>> {
>>>> if(UseATC & Writemode){StaticArraySet(TestName+i,C);}
>>>> if(UseATC & NOT Writemode){myArray = StaticArrayGet(TestName+i);}
>>>> if(NOT UseATC & Writemode){myArray = C+i;}
>>>> if(NOT UseATC & NOT Writemode){myArray = C;}
>>>> }
>>>> SetBarsRequired(2000,0);
>>>>
>>>> On Aug 5, 2007, at 5:38 PM, Dennis Brown wrote:
>>>>
>>>>> Tomasz and Daniel,
>>>>>
>>>>> Excellent points about the advantages of ATC vs hidden memory
>>>>> arrays.
>>>>>
>>>>> On the other side, when the purpose of the hidden arrays is
>>>>> just to
>>>>> hold a temporary array value between AFL passes that have a short
>>>>> life span of perhaps a minute, and if you are creating dozens of
>>>>> them, not having a "file system" copy of data to read is of no
>>>>> more
>>>>> consequence than not having a filesystem copy of any other AFL
>>>>> array
>>>>> that is destroyed every pass. There could be some debug value as
>>>>> confirmation that you are actually creating them.
>>>>>
>>>>> With the memory only "hidden" arrays, there might need to be a
>>>>> simple
>>>>> way to "flush" all the saved arrays for a "pane" from memory,
>>>>> either
>>>>> as a logical and automatic consequence of AB, and/or via a single
>>>>> explicit AFL command.
>>>>>
>>>>> The purpose of the proposal is to speed up AFL by saving results
>>>>> that
>>>>> do not need to be recalculated. ATC/Foreign have all the required
>>>>> functionality to do this. With the hint by Tomasz (thank you)
>>>>> about
>>>>> SetBarsRequired() at the END of the AFL, part of the speed
>>>>> problems
>>>>> is solved. The only reason to do a more limited version of ATC/
>>>>> Foreign functionality (in some form) is to cut down on the extra
>>>>> overhead to get the maximum speed improvement. It is only worth
>>>>> doing if it can be done in a way that will make the this
>>>>> functionality at least twice as fast as regular ATC/Foreign. I
>>>>> need
>>>>> to test that actual difference in generating and reading a normal
>>>>> array and ATC/Foreign now to see how much could be gained. I will
>>>>> report it soon.
>>>>>
>>>>> Best regards,
>>>>> Dennis Brown
>>>>>
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Foreign() is hand optimized and generally very fast - generally
>>>>>> speaking it's speed is
>>>>>> 70%..80% of memcpy() so I am afraid "signatures" would not
>>>>>> give any
>>>>>> advantage considering
>>>>>> the fact that you need to add special bookkeeping for them.
>>>>>>
>>>>>> And your comment regarding "hidden" nature of static variables is
>>>>>> very true and that
>>>>>> is one of my concerns.
>>>>>>
>>>>>> Best regards,
>>>>>> Tomasz Janeczko
>>>>>> amibroker.com
>>>>>> ----- Original Message -----
>>>>>> From: "liberte721" <daniel.laliberte@xxxxxxxxx>
>>>>>> To: <amibroker@xxxxxxxxxxxxxxx>
>>>>>> Sent: Sunday, August 05, 2007 4:36 PM
>>>>>> Subject: [amibroker] Re: Second thoughts on Static arrays --whole
>>>>>> problem set
>>>>>>
>>>>>>
>>>>>>> Tomasz,
>>>>>>>
>>>>>>> Thanks for your reply and explanation. I have a couple other
>>>>>>> ideas,
>>>>>>> and then I'll resume my usual silence. :)
>>>>>>>
>>>>>>> The use of Foreign to import an array might be optimized by a
>>>>>>> quick
>>>>>>> comparison of the "signature" of the array with that of the
>>>>>>> current
>>>>>>> symbol. If they are identical, then no translation would be
>>>>>>> necessary. By the "signature", I mean something like a
>>>>>>> combination of
>>>>>>> the barcount, date/time range and timeframe, but since internal
>>>>>>> bars
>>>>>>> might be missing, there should be a way to normalize the data
>>>>>>> and
>>>>>>> mark
>>>>>>> it as normalized. This might be more complex than it is worth.
>>>>>>>
>>>>>>> The additional "symbol" entries that ATC creates provide one
>>>>>>> other
>>>>>>> advantage for users - these symbols can be viewed or removed
>>>>>>> manually.
>>>>>>> In contrast, runtime memory would tend to fill up with invisible
>>>>>>> static arrays requiring some other way to view and remove them.
>>>>>>>
>>>>>>> Daniel LaLiberte
>>>>>>>
>>>>>>>
>>>>>>> --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko" <groups@xxx>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Daniel,
>>>>>>>>
>>>>>>>> ATC are NOT saved to disk unless cache needs to be flushed
>>>>>>>> to the
>>>>>>>> disk,
>>>>>>>> so disk access has nothing to do with the problem.
>>>>>>>>
>>>>>>>> There are two areas that affect speed
>>>>>>>> 1) the most important - ATC by default requires ALL bars to be
>>>>>>>> used
>>>>>>>> so your formula is not executed in QuickAFL mode
>>>>>>>> 2) Foreign needs to be done to sychronize length/date stamps
>>>>>>>> with
>>>>>>> currently selected symbol
>>>>>>>>
>>>>>>>> First point can be easily resolved by adding
>>>>>>>> SetBarsRequired at the END of the formula.
>>>>>>>> Doing so, overrides any automatically calculated bar
>>>>>>>> requirements
>>>>>>>> and
>>>>>>>> you will be able to shorten the number of bars used and turn on
>>>>>>> QuickAFL
>>>>>>>> even if ATC is used.
>>>>>>>>
>>>>>>>> For example adding
>>>>>>>> SetBarsRequired( 1000, 0 );
>>>>>>>> at the end will make that the formula will require 1000 prior
>>>>>>>> bars
>>>>>>> and 0 future bars.
>>>>>>>>
>>>>>>>> This should give large boost in speed if you are using many
>>>>>>> thousands of bars.
>>>>>>>>
>>>>>>>> Second point is unevitable - see my previous response.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Tomasz Janeczko
>>>>>>>> amibroker.com
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "liberte721" <daniel.laliberte@xxx>
>>>>>>>> To: <amibroker@xxxxxxxxxxxxxxx>
>>>>>>>> Sent: Sunday, August 05, 2007 8:09 AM
>>>>>>>> Subject: [amibroker] Re: Second thoughts on Static arrays --
>>>>>>>> whole
>>>>>>> problem set
>>>>>>>>
>>>>>>>>
>>>>>>>>> Dennis,
>>>>>>>>>
>>>>>>>>> I believe what you are asking for is different from the ATC
>>>>>>>>> and
>>>>>>>>> Foreign functionality since you do not want the persistent
>>>>>>>>> disk
>>>>>>>>> storage that ATC provides. I think that point might have been
>>>>>>>>> missed
>>>>>>>>> in previous discussions. You said:
>>>>>>>>>
>>>>>>>>> "Because AFL is so fast working with arrays, It seems that it
>>>>>>>>> should
>>>>>>>>> be possible to just save an array in memory in a high speed
>>>>>>>>> way,
>>>>>>>>> then
>>>>>>>>> recall it to repopulate the array instead of recalculate the
>>>>>>>>> array if
>>>>>>>>> the AFL determines that the data would not have changed since
>>>>>>>>> the last
>>>>>>>>> pass."
>>>>>>>>>
>>>>>>>>> So you want in-memory arrays that continue to exist outside
>>>>>>>>> of a
>>>>>>>>> single AFL script execution, but are not persistent to
>>>>>>>>> disk. In
>>>>>>>>> other
>>>>>>>>> words, if you close the AB session and reopen, then you do not
>>>>>>>>> expect
>>>>>>>>> your static arrays to have survived. I think the term "static
>>>>>>>>> arrays"
>>>>>>>>> is appropriate, analogous to "static variables" that are also
>>>>>>>>> not
>>>>>>>>> persistent. I'd be happier if static variables also could
>>>>>>>>> just
>>>>>>>>> store
>>>>>>>>> arrays as well as non-array values.
>>>>>>>>>
>>>>>>>>> It does seem that this form of static array memory should be
>>>>>>>>> nearly as
>>>>>>>>> fast as the temporary arrays that live only within a single
>>>>>>>>> script
>>>>>>>>> execution. They should merely occupy a different part of
>>>>>>>>> the AB
>>>>>>>>> memory space. I agree this would be very valuable for me as
>>>>>>>>> well,
>>>>>>>>> provided they were fast.
>>>>>>>>>
>>>>>>>>> Maybe ATC and Foreign functions effectively work this way
>>>>>>>>> by not
>>>>>>>>> saving to disk unless you actually save them explicitly (or
>>>>>>>>> implicitly
>>>>>>>>> if you enable that). If so, then it is not clear where the
>>>>>>>>> time
>>>>>>>>> goes.
>>>>>>>>> Maybe there is a way to optimize the use of these arrays to
>>>>>>>>> avoid
>>>>>>>>> time-frame translation, for example.
>>>>>>>>>
>>>>>>>>> Daniel LaLiberte
>>>>>>>>>
>>>>>>>>> --- In amibroker@xxxxxxxxxxxxxxx, Dennis Brown <see3d@> wrote:
>>>>>>>>>>
>>>>>>>>>> Tomasz,
>>>>>>>>>>
>>>>>>>>>> First let me thank you for your patients in explaining
>>>>>>>>>> this one
>>>>>>> more
>>>>>>>>>> time. It makes a very good post for the benefit of others.
>>>>>>>>>> Yes, I am aware of everything you explained because you had
>>>>>>> explained
>>>>>>>>>> it before and I listened and tried out the ideas.
>>>>>>>>>>
>>>>>>>>>> I sent you a direct email through support #39911 on May 16
>>>>>>>>>> for
>>>>>>> which
>>>>>>>>>> I received no reply (and none was technically required). I
>>>>>>>>>> will
>>>>>>>>>> quote it:
>>>>>>>>>>
>>>>>>>>>> "I am taking this subject off the yahoo board now because I
>>>>>>>>>> don't
>>>>>>>>>> think anyone can help but you.
>>>>>>>>>> It can go back on the board when there is a solution that
>>>>>>>>>> will
>>>>>>>>>> work.
>>>>>>>>>>
>>>>>>>>>> I took your advice and made all the changes you suggested to
>>>>>>> speed up
>>>>>>>>>> my AFL.
>>>>>>>>>> The functionality of the AFL is working nicely, behaving just
>>>>>>>>>> as it
>>>>>>>>>> should.
>>>>>>>>>>
>>>>>>>>>> However, instead of running 10 times faster, now it runs
>>>>>>>>>> twice as
>>>>>>>>>> slow as before.
>>>>>>>>>>
>>>>>>>>>> I have narrowed the problem down to the very thing that was
>>>>>>> supposed
>>>>>>>>>> to speed it up.
>>>>>>>>>>
>>>>>>>>>> 1. When I use the ATC & Foreign functions the number of bars
>>>>>>>>>> required goes from 3518, 0 to 1000000, 1000000.
>>>>>>>>>> This looks like it turns off fast AFL and slows all normal
>>>>>>>>>> processing by requiring all the bars. I have 77,000 data bars
>>>>>>>>>> total.
>>>>>>>>>> I do not know if this is intentional or a bug.
>>>>>>>>>>
>>>>>>>>>> 2. The ATC function is slow, but I only do it once per
>>>>>>>>>> bar (1
>>>>>>> minute).
>>>>>>>>>>
>>>>>>>>>> 3. The foreign function which is done every time is slow
>>>>>>>>>> also
>>>>>>>>>> (relative to normal arrays).
>>>>>>>>>> It takes .04 seconds each and I need dozens to replace the
>>>>>>>>>> missing
>>>>>>>>>> static array functionality.
>>>>>>>>>> Perhaps this is also due to the 1000000 bars.
>>>>>>>>>>
>>>>>>>>>> Without being able to have persistent arrays that are as
>>>>>>>>>> fast as
>>>>>>>>>> normal arrays I will not be able to achieve my trading goals
>>>>>>>>>> for
>>>>>>> very
>>>>>>>>>> many years until computers speed up by 10x. That is the
>>>>>>>>>> same as
>>>>>>>>>> never to me. I wish I could just declare some variables
>>>>>>>>>> to be
>>>>>>>>>> Persistent or Static and be done with it. It is easy for
>>>>>>>>>> me to
>>>>>>> say,
>>>>>>>>>> but perhaps it is not so easy for you to do. However, being
>>>>>>>>>> able
>>>>>>> to
>>>>>>>>>> successfully use the methods discussed on the board opens
>>>>>>>>>> up a
>>>>>>> lot of
>>>>>>>>>> extra possibilities due to the extra speed possible with AFL.
>>>>>>>>>>
>>>>>>>>>> I appreciate any help you can give in this matter."
>>>>>>>>>>
>>>>>>>>>> As I want a workable solution to my problem and want to
>>>>>>>>>> use AB
>>>>>>> for my
>>>>>>>>>> project (I would be trading this already using my AB system
>>>>>>>>>> if I
>>>>>>> had
>>>>>>>>>> a solution), I will state the fundamental problems I face
>>>>>>>>>> clearly
>>>>>>>>>> together here (instead of disjointed posts like before) and
>>>>>>>>>> then
>>>>>>>>>> perhaps a solution might occur to you or somebody else.
>>>>>>>>>> Please
>>>>>>>>>> forgive the length and complexity of this post.
>>>>>>>>>>
>>>>>>>>>> Background:
>>>>>>>>>>
>>>>>>>>>> My trading system is very simple in principle. AFL
>>>>>>>>>> calculates a
>>>>>>>>>> complex indicator, I draw manual trend lines, and I place
>>>>>>>>>> manual
>>>>>>>>>> trades on a single stock or ETF at a time. It is day
>>>>>>>>>> trading,
>>>>>>>>>> so I
>>>>>>>>>> only have a few seconds to trigger my trade when the
>>>>>>>>>> conditions
>>>>>>>>>> are
>>>>>>>>>> right. My AFL allows rapid manual selection of different
>>>>>>> timeframes
>>>>>>>>>> via a parameter, and selects the proper volume bar settings
>>>>>>>>>> dynamically for different ETFs or stocks. The program
>>>>>>> automatically
>>>>>>>>>> runs through many timeframes to select trade trigger
>>>>>>>>>> possibilities
>>>>>>>>>> for me to consider.
>>>>>>>>>> 1. There is a heavy compute environment --several seconds of
>>>>>>>>>> AFL.
>>>>>>>>>> 2. The environment is highly interactive --drawing trend
>>>>>>>>>> lines.
>>>>>>>>>> 3. Seconds count for placing trades.
>>>>>>>>>>
>>>>>>>>>> First Problem set:
>>>>>>>>>>
>>>>>>>>>> Under heavy compute loads (when AFL time is greater than
>>>>>>>>>> refresh
>>>>>>>>>> rate), AB stops responding to the UI.
>>>>>>>>>>
>>>>>>>>>> Workarounds:
>>>>>>>>>>
>>>>>>>>>> The only way I can get some UI functionality back is to
>>>>>>>>>> set the
>>>>>>>>>> refresh rate to one second longer than the AFL compute time.
>>>>>>> Then I
>>>>>>>>>> can get UI functionality back, but with many delays.
>>>>>>>>>> I also have a ParamToggle() to turn off most indicator
>>>>>>>>>> processing
>>>>>>> so
>>>>>>>>>> that I can interact with the UI better at times (but I
>>>>>>>>>> lose my
>>>>>>>>>> indicator plot references).
>>>>>>>>>> This is a partially effective workaround that I could limp
>>>>>>>>>> along
>>>>>>> with.
>>>>>>>>>>
>>>>>>>>>> Requested solution:
>>>>>>>>>>
>>>>>>>>>> Realizing that the root problem is that the UI functions
>>>>>>>>>> only run
>>>>>>> (or
>>>>>>>>>> initiate) in between AFL passes, I looked for a way to do
>>>>>>>>>> less
>>>>>>>>>> processing on each pass. The AFL load could be much less
>>>>>>>>>> most of
>>>>>>> the
>>>>>>>>>> time because I only need to calculate everything when a new
>>>>>>>>>> bar is
>>>>>>>>>> added.
>>>>>>>>>> I thought of two possibilities:
>>>>>>>>>>
>>>>>>>>>> 1. Ask for an AFL function that is essentially
>>>>>>>>>> DoUserInterface
>>>>>>>>>> ().
>>>>>>>>>> Then put AFL into an infinite loop and put these commands at
>>>>>>>>>> appropriate places. Since this is backwards from the way
>>>>>>>>>> the AFL
>>>>>>>>>> appears to work, I did not make this suggestion.
>>>>>>>>>>
>>>>>>>>>> 2. Ask for arrays that keep their data between AFL
>>>>>>>>>> passes. I
>>>>>>> tried
>>>>>>>>>> using ATC for this. It functioned like it was supposed
>>>>>>>>>> to, but
>>>>>>>>>> was
>>>>>>>>>> too slow to be a practical solution. I understand the reason
>>>>>>>>>> they
>>>>>>>>>> are so slow, and that is because they are a true static
>>>>>>>>>> permanent
>>>>>>>>>> array. However, what I wanted was only for a normal array
>>>>>>>>>> to be
>>>>>>>>>> saved between passes. They only need to be identified by
>>>>>>>>>> which
>>>>>>> pane
>>>>>>>>>> they are associated with (could be part of the internal
>>>>>>>>>> name).
>>>>>>> They
>>>>>>>>>> are only valid between new bars (same array size).
>>>>>>>>>> Changes to
>>>>>>>>>> symbol, timeframe, parameters, etc., can be detected in
>>>>>>>>>> the AFL
>>>>>>>>>> and
>>>>>>>>>> the array updated. I know this can be done, because I did
>>>>>>>>>> it for
>>>>>>> the
>>>>>>>>>> ATC version I wrote and everything worked perfectly (just not
>>>>>>>>>> faster). Because AFL is so fast working with arrays, It
>>>>>>>>>> seems
>>>>>>>>>> that
>>>>>>>>>> it should be possible to just save an array in memory in a
>>>>>>>>>> high
>>>>>>> speed
>>>>>>>>>> way, then recall it to repopulate the array instead of
>>>>>>>>>> recalculate
>>>>>>>>>> the array if the AFL determines that the data would not have
>>>>>>> changed
>>>>>>>>>> since the last pass.
>>>>>>>>>>
>>>>>>>>>> There may be another possible solution that I have not
>>>>>>>>>> thought of
>>>>>>> and
>>>>>>>>>> I would love to hear it.
>>>>>>>>>>
>>>>>>>>>> Second Problem set:
>>>>>>>>>>
>>>>>>>>>> Because this is a realtime trading system, decisions have
>>>>>>>>>> to be
>>>>>>> made
>>>>>>>>>> from raw data which contains bad ticks that may fool the
>>>>>>>>>> system.
>>>>>>>>>> Volume bars are used in the system, and must be able to
>>>>>>>>>> change
>>>>>>> volume
>>>>>>>>>> per bar on different AFL passes.
>>>>>>>>>> Built-in timeframe or volume settings do not allow the
>>>>>>> flexibility to
>>>>>>>>>> address these
>>>>>>>>>>
>>>>>>>>>> Workarounds:
>>>>>>>>>>
>>>>>>>>>> A tick database is too large to have enough history for the
>>>>>>>>>> indicators. Running a database of 5 seconds for a basic 1
>>>>>>>>>> minute
>>>>>>> bar
>>>>>>>>>> (or equivalent for volume bars) allows for reasonably
>>>>>>>>>> effective
>>>>>>>>>> bad
>>>>>>>>>> tick removal and volume bar generation. History can be just
>>>>>>>>>> 60K
>>>>>>>>>> bars. Bad ticks are removed and volume bar settings are
>>>>>>>>>> determined
>>>>>>>>>> using the 5 second bars. Arrays are built up using
>>>>>>> TimeFrameSet() to
>>>>>>>>>> get the actual bars used for indicator calculations and graph
>>>>>>> plots.
>>>>>>>>>> This approach works really well, however, there is one big
>>>>>>>>>> problem
>>>>>>>>>> with it.
>>>>>>>>>>
>>>>>>>>>> Even though I can display all my price and indicators in the
>>>>>>>>>> timeframe I desire, I can not draw studies that align to that
>>>>>>>>>> timeframe. All studies are drawn in the pane relative to 5
>>>>>>>>>> second
>>>>>>>>>> bars. This workaround is not acceptable for trading.
>>>>>>>>>>
>>>>>>>>>> Requested solution:
>>>>>>>>>>
>>>>>>>>>> #1117 Make an AFL way to set the timeframe of the drawing
>>>>>>>>>> studies
>>>>>>> to
>>>>>>>>>> match the OHLC array created by TimeFrameSet().
>>>>>>>>>>
>>>>>>>>>> Of course implementing this has a general benefit of
>>>>>>>>>> giving an
>>>>>>> almost
>>>>>>>>>> infinite number of time/volume/range bars settings via AFL
>>>>>>>>>> and
>>>>>>>>>> parameters.
>>>>>>>>>>
>>>>>>>>>> Please consider the problems (not just proposed solutions)
>>>>>>>>>> and
>>>>>>> advise
>>>>>>>>>> if other possibilities come to mind for solutions.
>>>>>>>>>>
>>>>>>>>>> Thank you very much for your patients and consideration,
>>>>>>>>>>
>>>>>>>>>> Dennis Brown
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Aug 4, 2007, at 6:16 AM, Tomasz Janeczko wrote:
>>>>>>>>>>
>>>>>>>>>>> Dennis,
>>>>>>>>>>>
>>>>>>>>>>> I explained it many times:
>>>>>>>>>>>
>>>>>>>>>>> 1. You can not simply "KEEP" the array from previous
>>>>>>>>>>> execution
>>>>>>> and
>>>>>>>>>>> reference it directy becase:
>>>>>>>>>>> a) previous execution may be on different symbol/different
>>>>>>>>>>> time
>>>>>>>>>>> frame/different zoom level leading
>>>>>>>>>>> to DIFFERENT NUMBER OF ELEMENTS in the array and different
>>>>>>>>>>> TIMESTAMPS of array bars.
>>>>>>>>>>>
>>>>>>>>>>> 2. Because of (1) the only way to implement "static
>>>>>>>>>>> array" is
>>>>>>>>>>> to:
>>>>>>>>>>> a) store not only array BUT also TIMESTAMPS with every bar
>>>>>>>>>>> b) during reading perform timestamp checking and
>>>>>>> synchronization so
>>>>>>>>>>> referenced array data is aligned
>>>>>>>>>>> with current execution data
>>>>>>>>>>>
>>>>>>>>>>> If those two were not implemented you would NOT be able
>>>>>>>>>>> to use
>>>>>>>>>>> "static arrays" at all.
>>>>>>>>>>>
>>>>>>>>>>> 3. Functionality already exists
>>>>>>>>>>> a) Point 2a - storing arrays with timestamps is exactly what
>>>>>>>>>>> AddToComposite ALREADY DOES
>>>>>>>>>>> b) Point 2b - reading arrays with timestamps and aligning is
>>>>>>>>>>> exactly what Foreign ALREADY DOES
>>>>>>>>>>>
>>>>>>>>>>> 4. Implementing dedicated "static" array functions would
>>>>>>>>>>> a) give NO SPEED ADVANTAGE over AddToComposite/Foreign
>>>>>>>>>>> b) give only "the impression" that it is something else and
>>>>>>>>>>> "faster" than ATC/Foreign while it would
>>>>>>>>>>> be FALSE impression.
>>>>>>>>>>>
>>>>>>>>>>> Summary:
>>>>>>>>>>> a) functionality already exists
>>>>>>>>>>> b) no speed advantage over existing functionality
>>>>>>>>>>>
>>>>>>>>>>> So here are the functions you are asking for:
>>>>>>>>>>>
>>>>>>>>>>> function StaticVarArraySet( Varname, array )
>>>>>>>>>>> {
>>>>>>>>>>> AddToComposite( array, "~SA_"+VarName, "C",
>>>>>>>>>>> atcFlagDefaults |
>>>>>>>>>>> atcFlagEnableInBacktest | atcFlagEnableInExplore |
>>>>>>>>>>> atcFlagEnableInIndicator | atcFlagEnableInPortfolio );
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> function StaticVarArrayGet( VarName );
>>>>>>>>>>> {
>>>>>>>>>>> return Foreign( "~SA_"+VarName, "C" );
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> Tomasz Janeczko
>>>>>>>>>>> amibroker.com
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: Dennis Brown
>>>>>>>>>>> To: amibroker@xxxxxxxxxxxxxxx
>>>>>>>>>>> Sent: Saturday, August 04, 2007 5:26 AM
>>>>>>>>>>> Subject: [amibroker] Second thoughts on Static arrays
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> The more I have thought about static arrays for my AFL
>>>>>>>>>>> needs,
>>>>>>>>>>> the
>>>>>>>>>>> more I see that what I am looking for is a quite limited
>>>>>>>>>>> implementation.
>>>>>>>>>>>
>>>>>>>>>>> Static arrays would have low overhead and give the speed
>>>>>>>>>>> needed
>>>>>>>>>>> just like regular arrays. The primary need is to avoid re-
>>>>>>>>>>> computing complex things all the time that only need to be
>>>>>>> computed
>>>>>>>>>>> under certain conditions or perhaps once per new bar. This
>>>>>>> allows
>>>>>>>>>>> far richer AFLs to be developed for real time trading, while
>>>>>>>>>>> improving the responsiveness of the UI interaction under
>>>>>>>>>>> heavy
>>>>>>>>>>> compute conditions. The life of a static array is fleeting.
>>>>>>>>>>>
>>>>>>>>>>> Static arrays are not permanent like ATC, and do not have
>>>>>>>>>>> the
>>>>>>>>>>> big
>>>>>>>>>>> overhead. Static arrays as I envision them, would have
>>>>>>>>>>> to be
>>>>>>>>>>> updated via the user AFL program whenever a change in
>>>>>>>>>>> environment
>>>>>>>>>>> caused the data to become invalid --like changing the
>>>>>>>>>>> symbol or
>>>>>>> the
>>>>>>>>>>> timeframe --or even adding a new bar (which changes the
>>>>>>>>>>> array
>>>>>>>>>>> size). Having static arrays like this would allow breaking
>>>>>>>>>>> the
>>>>>>> AFL
>>>>>>>>>>> into two parts --the fast part that needs to run for every
>>>>>>> second,
>>>>>>>>>>> and the part that needs to run once per compressed
>>>>>>>>>>> timeframe,
>>>>>>>>>>> volume, or range bar.
>>>>>>>>>>>
>>>>>>>>>>> I am sure there are technical issues involved, or TJ would
>>>>>>>>>>> have
>>>>>>>>>>> already provided them since they are so useful for
>>>>>>>>>>> speeding up
>>>>>>> AFL.
>>>>>>>>>>> First rule of making programs fast is pre-compute everything
>>>>>>>>>>> possible and not inside a loop. AFL is one big loop.
>>>>>>>>>>>
>>>>>>>>>>> Perhaps what I am looking for are not technically "static"
>>>>>>>>>>> arrays
>>>>>>>>>>> in the normal sense. Maybe I need to call them by a
>>>>>>>>>>> different
>>>>>>> name
>>>>>>>>>>> for the suggestion box. Perhaps there is an even easier
>>>>>>>>>>> way to
>>>>>>>>>>> address this need with a new AFL syntax: keep(myArray);
>>>>>>>>>>>
>>>>>>>>>>> Comments anyone?
>>>>>>>>>>>
>>>>>>>>>>> Dennis Brown
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please note that this group is for discussion between users
>>>>>>>>> only.
>>>>>>>>>
>>>>>>>>> To get support from AmiBroker please send an e-mail
>>>>>>>>> directly to
>>>>>>>>> SUPPORT {at} amibroker.com
>>>>>>>>>
>>>>>>>>> For NEW RELEASE ANNOUNCEMENTS and other news always check
>>>>>>>>> DEVLOG:
>>>>>>>>> http://www.amibroker.com/devlog/
>>>>>>>>>
>>>>>>>>> For other support material please check also:
>>>>>>>>> http://www.amibroker.com/support.html
>>>>>>>>>
>>>>>>>>> Yahoo! Groups Links
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Please note that this group is for discussion between users
>>>>>>> only.
>>>>>>>
>>>>>>> To get support from AmiBroker please send an e-mail directly to
>>>>>>> SUPPORT {at} amibroker.com
>>>>>>>
>>>>>>> For NEW RELEASE ANNOUNCEMENTS and other news always check
>>>>>>> DEVLOG:
>>>>>>> http://www.amibroker.com/devlog/
>>>>>>>
>>>>>>> For other support material please check also:
>>>>>>> http://www.amibroker.com/support.html
>>>>>>>
>>>>>>> Yahoo! Groups Links
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Please note that this group is for discussion between users only.
>>>>>>
>>>>>> To get support from AmiBroker please send an e-mail directly to
>>>>>> SUPPORT {at} amibroker.com
>>>>>>
>>>>>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>>>>>> http://www.amibroker.com/devlog/
>>>>>>
>>>>>> For other support material please check also:
>>>>>> http://www.amibroker.com/support.html
>>>>>>
>>>>>> Yahoo! Groups Links
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Please note that this group is for discussion between users only.
>>>>>
>>>>> To get support from AmiBroker please send an e-mail directly to
>>>>> SUPPORT {at} amibroker.com
>>>>>
>>>>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>>>>> http://www.amibroker.com/devlog/
>>>>>
>>>>> For other support material please check also:
>>>>> http://www.amibroker.com/support.html
>>>>>
>>>>> Yahoo! Groups Links
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> Please note that this group is for discussion between users only.
>>>>
>>>> To get support from AmiBroker please send an e-mail directly to
>>>> SUPPORT {at} amibroker.com
>>>>
>>>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>>>> http://www.amibroker.com/devlog/
>>>>
>>>> For other support material please check also:
>>>> http://www.amibroker.com/support.html
>>>>
>>>> Yahoo! Groups Links
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> Please note that this group is for discussion between users only.
>>>
>>> To get support from AmiBroker please send an e-mail directly to
>>> SUPPORT {at} amibroker.com
>>>
>>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>>> http://www.amibroker.com/devlog/
>>>
>>> For other support material please check also:
>>> http://www.amibroker.com/support.html
>>>
>>> Yahoo! Groups Links
>>>
>>>
>>>
>>
>>
>>
>> Please note that this group is for discussion between users only.
>>
>> To get support from AmiBroker please send an e-mail directly to
>> SUPPORT {at} amibroker.com
>>
>> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
>> http://www.amibroker.com/devlog/
>>
>> For other support material please check also:
>> http://www.amibroker.com/support.html
>>
>> Yahoo! Groups Links
>>
>>
>>
>>
>>
>
>
> Please note that this group is for discussion between users only.
>
> To get support from AmiBroker please send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> For other support material please check also:
> http://www.amibroker.com/support.html
>
> Yahoo! Groups Links
>
>
>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|