|
PureBytes Links
Trading Reference Links
|
Tomasz,
I thought I could trick AB into doing what I want, but it did not
quite work as I had hoped.
I made a chart running on the native 5 second bars.
In that AFL I ran my bad tick filter and aggregated the bars into one
minute bars using TimeFrameSet(60), then I saved the OHLCV as a
single ATC symbol (~SA_Clean).
Next I ran another chart with the ~SA_Clean symbol selected and a
SetBarsRequired(2000,0) at the end of the AFL. The 1 minute price
chart was displayed, but only if I selected 5 seconds as the time
base --which just puts me back in the same circle I am trying to get
out of. I want the 1 minute chart to be displayed correctly with a 1
minute time base selection so I can draw trend lines, but that
setting aggregates the bars 12 to 1. Of course I also want volume
bars displayed correctly, but one step at a time.
It looks like the ATC ignores the time base of the bars I was running
in (60 seconds due to the TimeFrameSet command) and instead
substituted the time stamp of the 5 second database when it saved out
the data. I thought it would use TimeNum() for the time stamps on
the bars. Perhaps it could be an ATC option to use TimeNum() for the
time stamps.
Is there a way to get around this problem and get the TimeNum() time
stamp on the bars saved with ATC?
I am trying my best to think of creative solutions to get my trading
on line with AB soon --while longer term elegant AB solutions are
devised for future releases.
I can send this problem through support if you prefer, but I thought
that the problem and solutions would be of more general interest and
would make useful posts, and hopefully my working through these
problems will make it easier for creating solutions in future
releases of AB.
Thank you,
Dennis Brown
On Aug 6, 2007, at 2:20 PM, Dennis Brown wrote:
> Tomasz,
>
> I just found problem with the SetBarsRequired() approach while trying
> to test it on my real AFL application data now.
>
> I am running a 5 second database to generate my clean bars for volume
> based aggregation and using the TimeFrameSet() to create a smaller
> 2000 bar OHLC "working" array. However, I still need ALL the 5
> second bars loaded initially to create the smaller OHLC array. The
> smaller arrays will not actually be smaller though --just not fully
> populated. Saving them using ATC will not result in a fast save of a
> small number of bars. So even though the speed of saving 2000 bars
> shown below would be useful, I would have to save a 20K-40K size
> array and the speed would not be practical.
>
> So I think that when the user selects a timeframe, say 1 minute with
> a 5 second database, you aggregate the data into a smaller number of
> bars first, then the AFL runs with only these smaller arrays. In
> essence I am trying to do the same thing, but all from AFL and I see
> that it is not designed to support it this way.
>
> This may sound crazy, but perhaps I can run the 5 second bars and
> make my clean volume or time based OHLC bars in one tiny pane and
> save them with ATC to a symbol. Then use Foreign to read a smaller
> array into my main pane that runs with a SetBarsRequired(2000,0) at
> the end.
>
> I will experiment with this concept. Though it seems like I am
> jumping through hoops trying to overcome these problems, I am happy
> to do it if it is the only way to proceed in the short term to give
> me a tradable system solution. However, it takes a lot of time and
> energy to do this, and I don't want to go down a fruitless path.
>
> I would appreciate your comments.
>
> Best regards,
> Dennis Brown
>
>
> On Aug 6, 2007, at 8:25 AM, Tomasz Janeczko wrote:
>
>> Hello,
>>
>> Thank you for your comments.
>> And I just realized one thing that largely affects the results.
>> That Foreign() funciton will process all bars anyway regardless of
>> SetBarsRequired()
>> because in principle it was "general-purpose", not "chart-only" and
>> does
>> not really pay any attention "required bars".
>> Something to think about in the future releases....
>>
>> Best regards,
>> Tomasz Janeczko
>> amibroker.com
>> ----- Original Message -----
>> From: "Dennis Brown" <see3d@xxxxxxxxxxx>
>> To: <amibroker@xxxxxxxxxxxxxxx>
>> Sent: Monday, August 06, 2007 1:55 PM
>> Subject: Re: [amibroker] Re: Second thoughts on Static arrays --ATC/
>> Foreign times
>>
>>
>>> Tomasz,
>>>
>>> Yes, I suspected it might be the case that only pointers were being
>>> copied for internal arrays. That is why I tried both in my code.
>>> for the read mode:
>>> myArray = C;
>>> and for the write mode:
>>> myArray = C + i; (i instead of 1 just in case you did some
>>> optimizing with constants)
>>>
>>> At the resolution of my results, the slowest results for either case
>>> were between 1ms and 2ms.
>>> This means that the overhead of running the short AFL code shown was
>>> dominating the execution time and the array processing time was not
>>> even significant enough to noticeably affect the results. So in
>>> reality, the array processing is likely faster than I claimed.
>>> That is also why I also said that "static arrays" would would be
>>> very
>>> fast even if they take 10 times as long to process as internal
>>> arrays.
>>>
>>> I was astounded at the efficiency of your code for the internal
>>> arrays. You have done a very good job.
>>>
>>> Best regards,
>>> Dennis Brown
>>>
>>> On Aug 6, 2007, at 6:43 AM, Tomasz Janeczko wrote:
>>>
>>>> Dennis,
>>>>
>>>> This is NOT surprising, because internal array does NOT use memory
>>>> copy at all!
>>>>
>>>> Internal arrays are so fast because referencing them does nothing
>>>> except
>>>> referncing via pointer (4 bytes always, regardless of how long is
>>>> the array),
>>>> while Foreign has to actually COPY memory (create new array).
>>>>
>>>> So if you do assignment like this:
>>>> myArray = C;
>>>>
>>>> It effectivelly copies only 4 bytes (the pointer) and increases
>>>> "reference counter".
>>>> This is what makes it hyper-fast.
>>>>
>>>> But... as soon as you create NEW array, say by doing this:
>>>>
>>>> myArray = C + 1;
>>>>
>>>> Then timing would be much worse because of requried memory copy.
>>>> Testing this would give you true estimate of how fast "static
>>>> arrays" could be.
>>>>
>>>> Best regards,
>>>> Tomasz Janeczko
>>>> amibroker.com
>>>> ----- Original Message -----
>>>> From: "Dennis Brown" <see3d@xxxxxxxxxxx>
>>>> To: <amibroker@xxxxxxxxxxxxxxx>
>>>> Sent: Monday, August 06, 2007 5:03 AM
>>>> Subject: Re: [amibroker] Re: Second thoughts on Static arrays --
>>>> ATC/
>>>> Foreign times
>>>>
>>>>
>>>>> Tomasz,
>>>>>
>>>>> I ran a timing test to see what the difference would be for
>>>>> using the
>>>>> ATC/Foreign functions vs internal arrays with practical values. I
>>>>> selected an array size of 2000 and generated 100 uniqueATC
>>>>> symbols.
>>>>> Here are the AFL execution times on my machine:
>>>>>
>>>>> ATC Write: 780ms
>>>>> Foreign Read: 310ms
>>>>> Internal arrays: 1ms
>>>>>
>>>>> This is compared to an ATC time of 23000ms before I added the
>>>>> SetBarsRequired(2000,0) at the END of the AFL --a 30x improvement
>>>>> because there are about 40K bars in the 5 second database.
>>>>>
>>>>> So the bottom line is that internal arrays seem to be at least 300
>>>>> times as fast as ATC/Foreign.
>>>>> The potential improvements in speed seem very significant --
>>>>> even if
>>>>> static arrays as proposed were 10 times as slow as internal
>>>>> arrays.
>>>>> Perhaps it would be possible to just add another mode to ATC/
>>>>> Foreign
>>>>> that bypassed most overhead for fast static arrays.
>>>>>
>>>>> Regardless of what you decide to do though, the 30x improvement
>>>>> using
>>>>> SetBarsRequired(2000,0) at the END is enough to make it worthwhile
>>>>> for me to save some of my more time consuming arrays between AFL
>>>>> passes using ATC/Foreign to help with the UI responsiveness.
>>>>>
>>>>> If I had another 30x improvement in "static" array speed (which
>>>>> seems
>>>>> quite possible), I would save a lot more results and gain
>>>>> significantly more speed. In either case I believe I can now move
>>>>> forward from the first problem set and it only leaves the second
>>>>> problem set to enable me to begin trading this system.
>>>>>
>>>>> Here is the simple AFL I used for the test. I just turned on
>>>>> the AFL
>>>>> chart timing for the results.
>>>>>
>>>>> Thank you,
>>>>> Dennis Brown
>>>>>
>>>>> function StaticArraySet(VarName,array){AddToComposite
>>>>> (array ,"~SA_"+VarName ,"C",atcFlagEnableInIndicator|
>>>>> atcFlagDefaults);}
>>>>> function StaticArrayGet(VarName){return Foreign
>>>>> ("~SA_"+VarName,"C");}
>>>>>
>>>>> TestName = ParamStr("Test Name", "Test");
>>>>> Cycles = Param("Cycles",1,1,100,1);
>>>>> UseATC = ParamToggle("ATC mode","No|Yes",0);
>>>>> Writemode = ParamToggle("R/W mode","Read|Write",1);
>>>>>
>>>>> for(i=0;i<Cycles;i++)
>>>>> {
>>>>> if(UseATC & Writemode){StaticArraySet(TestName+i,C);}
>>>>> if(UseATC & NOT Writemode){myArray = StaticArrayGet(TestName+i);}
>>>>> if(NOT UseATC & Writemode){myArray = C+i;}
>>>>> if(NOT UseATC & NOT Writemode){myArray = C;}
>>>>> }
>>>>> SetBarsRequired(2000,0);
>>>>>
>>>>> On Aug 5, 2007, at 5:38 PM, Dennis Brown wrote:
>>>>>
>>>>>> Tomasz and Daniel,
>>>>>>
>>>>>> Excellent points about the advantages of ATC vs hidden memory
>>>>>> arrays.
>>>>>>
>>>>>> On the other side, when the purpose of the hidden arrays is
>>>>>> just to
>>>>>> hold a temporary array value between AFL passes that have a short
>>>>>> life span of perhaps a minute, and if you are creating dozens of
>>>>>> them, not having a "file system" copy of data to read is of no
>>>>>> more
>>>>>> consequence than not having a filesystem copy of any other AFL
>>>>>> array
>>>>>> that is destroyed every pass. There could be some debug value as
>>>>>> confirmation that you are actually creating them.
>>>>>>
>>>>>> With the memory only "hidden" arrays, there might need to be a
>>>>>> simple
>>>>>> way to "flush" all the saved arrays for a "pane" from memory,
>>>>>> either
>>>>>> as a logical and automatic consequence of AB, and/or via a single
>>>>>> explicit AFL command.
>>>>>>
>>>>>> The purpose of the proposal is to speed up AFL by saving results
>>>>>> that
>>>>>> do not need to be recalculated. ATC/Foreign have all the
>>>>>> required
>>>>>> functionality to do this. With the hint by Tomasz (thank you)
>>>>>> about
>>>>>> SetBarsRequired() at the END of the AFL, part of the speed
>>>>>> problems
>>>>>> is solved. The only reason to do a more limited version of ATC/
>>>>>> Foreign functionality (in some form) is to cut down on the extra
>>>>>> overhead to get the maximum speed improvement. It is only worth
>>>>>> doing if it can be done in a way that will make the this
>>>>>> functionality at least twice as fast as regular ATC/Foreign. I
>>>>>> need
>>>>>> to test that actual difference in generating and reading a normal
>>>>>> array and ATC/Foreign now to see how much could be gained. I
>>>>>> will
>>>>>> report it soon.
>>>>>>
>>>>>> Best regards,
>>>>>> Dennis Brown
>>>>>>
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Foreign() is hand optimized and generally very fast - generally
>>>>>>> speaking it's speed is
>>>>>>> 70%..80% of memcpy() so I am afraid "signatures" would not
>>>>>>> give any
>>>>>>> advantage considering
>>>>>>> the fact that you need to add special bookkeeping for them.
>>>>>>>
>>>>>>> And your comment regarding "hidden" nature of static
>>>>>>> variables is
>>>>>>> very true and that
>>>>>>> is one of my concerns.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Tomasz Janeczko
>>>>>>> amibroker.com
>>>>>>> ----- Original Message -----
>>>>>>> From: "liberte721" <daniel.laliberte@xxxxxxxxx>
>>>>>>> To: <amibroker@xxxxxxxxxxxxxxx>
>>>>>>> Sent: Sunday, August 05, 2007 4:36 PM
>>>>>>> Subject: [amibroker] Re: Second thoughts on Static arrays --
>>>>>>> whole
>>>>>>> problem set
>>>>>>>
>>>>>>>
>>>>>>>> Tomasz,
>>>>>>>>
>>>>>>>> Thanks for your reply and explanation. I have a couple other
>>>>>>>> ideas,
>>>>>>>> and then I'll resume my usual silence. :)
>>>>>>>>
>>>>>>>> The use of Foreign to import an array might be optimized by a
>>>>>>>> quick
>>>>>>>> comparison of the "signature" of the array with that of the
>>>>>>>> current
>>>>>>>> symbol. If they are identical, then no translation would be
>>>>>>>> necessary. By the "signature", I mean something like a
>>>>>>>> combination of
>>>>>>>> the barcount, date/time range and timeframe, but since internal
>>>>>>>> bars
>>>>>>>> might be missing, there should be a way to normalize the data
>>>>>>>> and
>>>>>>>> mark
>>>>>>>> it as normalized. This might be more complex than it is worth.
>>>>>>>>
>>>>>>>> The additional "symbol" entries that ATC creates provide one
>>>>>>>> other
>>>>>>>> advantage for users - these symbols can be viewed or removed
>>>>>>>> manually.
>>>>>>>> In contrast, runtime memory would tend to fill up with
>>>>>>>> invisible
>>>>>>>> static arrays requiring some other way to view and remove them.
>>>>>>>>
>>>>>>>> Daniel LaLiberte
>>>>>>>>
>>>>>>>>
>>>>>>>> --- In amibroker@xxxxxxxxxxxxxxx, "Tomasz Janeczko"
>>>>>>>> <groups@xxx>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Daniel,
>>>>>>>>>
>>>>>>>>> ATC are NOT saved to disk unless cache needs to be flushed
>>>>>>>>> to the
>>>>>>>>> disk,
>>>>>>>>> so disk access has nothing to do with the problem.
>>>>>>>>>
>>>>>>>>> There are two areas that affect speed
>>>>>>>>> 1) the most important - ATC by default requires ALL bars to be
>>>>>>>>> used
>>>>>>>>> so your formula is not executed in QuickAFL mode
>>>>>>>>> 2) Foreign needs to be done to sychronize length/date stamps
>>>>>>>>> with
>>>>>>>> currently selected symbol
>>>>>>>>>
>>>>>>>>> First point can be easily resolved by adding
>>>>>>>>> SetBarsRequired at the END of the formula.
>>>>>>>>> Doing so, overrides any automatically calculated bar
>>>>>>>>> requirements
>>>>>>>>> and
>>>>>>>>> you will be able to shorten the number of bars used and
>>>>>>>>> turn on
>>>>>>>> QuickAFL
>>>>>>>>> even if ATC is used.
>>>>>>>>>
>>>>>>>>> For example adding
>>>>>>>>> SetBarsRequired( 1000, 0 );
>>>>>>>>> at the end will make that the formula will require 1000 prior
>>>>>>>>> bars
>>>>>>>> and 0 future bars.
>>>>>>>>>
>>>>>>>>> This should give large boost in speed if you are using many
>>>>>>>> thousands of bars.
>>>>>>>>>
>>>>>>>>> Second point is unevitable - see my previous response.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Tomasz Janeczko
>>>>>>>>> amibroker.com
>>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "liberte721" <daniel.laliberte@xxx>
>>>>>>>>> To: <amibroker@xxxxxxxxxxxxxxx>
>>>>>>>>> Sent: Sunday, August 05, 2007 8:09 AM
>>>>>>>>> Subject: [amibroker] Re: Second thoughts on Static arrays --
>>>>>>>>> whole
>>>>>>>> problem set
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Dennis,
>>>>>>>>>>
>>>>>>>>>> I believe what you are asking for is different from the ATC
>>>>>>>>>> and
>>>>>>>>>> Foreign functionality since you do not want the persistent
>>>>>>>>>> disk
>>>>>>>>>> storage that ATC provides. I think that point might have
>>>>>>>>>> been
>>>>>>>>>> missed
>>>>>>>>>> in previous discussions. You said:
>>>>>>>>>>
>>>>>>>>>> "Because AFL is so fast working with arrays, It seems that it
>>>>>>>>>> should
>>>>>>>>>> be possible to just save an array in memory in a high speed
>>>>>>>>>> way,
>>>>>>>>>> then
>>>>>>>>>> recall it to repopulate the array instead of recalculate the
>>>>>>>>>> array if
>>>>>>>>>> the AFL determines that the data would not have changed since
>>>>>>>>>> the last
>>>>>>>>>> pass."
>>>>>>>>>>
>>>>>>>>>> So you want in-memory arrays that continue to exist outside
>>>>>>>>>> of a
>>>>>>>>>> single AFL script execution, but are not persistent to
>>>>>>>>>> disk. In
>>>>>>>>>> other
>>>>>>>>>> words, if you close the AB session and reopen, then you do
>>>>>>>>>> not
>>>>>>>>>> expect
>>>>>>>>>> your static arrays to have survived. I think the term
>>>>>>>>>> "static
>>>>>>>>>> arrays"
>>>>>>>>>> is appropriate, analogous to "static variables" that are also
>>>>>>>>>> not
>>>>>>>>>> persistent. I'd be happier if static variables also could
>>>>>>>>>> just
>>>>>>>>>> store
>>>>>>>>>> arrays as well as non-array values.
>>>>>>>>>>
>>>>>>>>>> It does seem that this form of static array memory should be
>>>>>>>>>> nearly as
>>>>>>>>>> fast as the temporary arrays that live only within a single
>>>>>>>>>> script
>>>>>>>>>> execution. They should merely occupy a different part of
>>>>>>>>>> the AB
>>>>>>>>>> memory space. I agree this would be very valuable for me as
>>>>>>>>>> well,
>>>>>>>>>> provided they were fast.
>>>>>>>>>>
>>>>>>>>>> Maybe ATC and Foreign functions effectively work this way
>>>>>>>>>> by not
>>>>>>>>>> saving to disk unless you actually save them explicitly (or
>>>>>>>>>> implicitly
>>>>>>>>>> if you enable that). If so, then it is not clear where the
>>>>>>>>>> time
>>>>>>>>>> goes.
>>>>>>>>>> Maybe there is a way to optimize the use of these arrays to
>>>>>>>>>> avoid
>>>>>>>>>> time-frame translation, for example.
>>>>>>>>>>
>>>>>>>>>> Daniel LaLiberte
>>>>>>>>>>
>>>>>>>>>> --- In amibroker@xxxxxxxxxxxxxxx, Dennis Brown <see3d@>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Tomasz,
>>>>>>>>>>>
>>>>>>>>>>> First let me thank you for your patients in explaining
>>>>>>>>>>> this one
>>>>>>>> more
>>>>>>>>>>> time. It makes a very good post for the benefit of others.
>>>>>>>>>>> Yes, I am aware of everything you explained because you had
>>>>>>>> explained
>>>>>>>>>>> it before and I listened and tried out the ideas.
>>>>>>>>>>>
>>>>>>>>>>> I sent you a direct email through support #39911 on May 16
>>>>>>>>>>> for
>>>>>>>> which
>>>>>>>>>>> I received no reply (and none was technically required). I
>>>>>>>>>>> will
>>>>>>>>>>> quote it:
>>>>>>>>>>>
>>>>>>>>>>> "I am taking this subject off the yahoo board now because I
>>>>>>>>>>> don't
>>>>>>>>>>> think anyone can help but you.
>>>>>>>>>>> It can go back on the board when there is a solution that
>>>>>>>>>>> will
>>>>>>>>>>> work.
>>>>>>>>>>>
>>>>>>>>>>> I took your advice and made all the changes you suggested to
>>>>>>>> speed up
>>>>>>>>>>> my AFL.
>>>>>>>>>>> The functionality of the AFL is working nicely, behaving
>>>>>>>>>>> just
>>>>>>>>>>> as it
>>>>>>>>>>> should.
>>>>>>>>>>>
>>>>>>>>>>> However, instead of running 10 times faster, now it runs
>>>>>>>>>>> twice as
>>>>>>>>>>> slow as before.
>>>>>>>>>>>
>>>>>>>>>>> I have narrowed the problem down to the very thing that was
>>>>>>>> supposed
>>>>>>>>>>> to speed it up.
>>>>>>>>>>>
>>>>>>>>>>> 1. When I use the ATC & Foreign functions the number of
>>>>>>>>>>> bars
>>>>>>>>>>> required goes from 3518, 0 to 1000000, 1000000.
>>>>>>>>>>> This looks like it turns off fast AFL and slows all normal
>>>>>>>>>>> processing by requiring all the bars. I have 77,000 data
>>>>>>>>>>> bars
>>>>>>>>>>> total.
>>>>>>>>>>> I do not know if this is intentional or a bug.
>>>>>>>>>>>
>>>>>>>>>>> 2. The ATC function is slow, but I only do it once per
>>>>>>>>>>> bar (1
>>>>>>>> minute).
>>>>>>>>>>>
>>>>>>>>>>> 3. The foreign function which is done every time is slow
>>>>>>>>>>> also
>>>>>>>>>>> (relative to normal arrays).
>>>>>>>>>>> It takes .04 seconds each and I need dozens to replace the
>>>>>>>>>>> missing
>>>>>>>>>>> static array functionality.
>>>>>>>>>>> Perhaps this is also due to the 1000000 bars.
>>>>>>>>>>>
>>>>>>>>>>> Without being able to have persistent arrays that are as
>>>>>>>>>>> fast as
>>>>>>>>>>> normal arrays I will not be able to achieve my trading goals
>>>>>>>>>>> for
>>>>>>>> very
>>>>>>>>>>> many years until computers speed up by 10x. That is the
>>>>>>>>>>> same as
>>>>>>>>>>> never to me. I wish I could just declare some variables
>>>>>>>>>>> to be
>>>>>>>>>>> Persistent or Static and be done with it. It is easy for
>>>>>>>>>>> me to
>>>>>>>> say,
>>>>>>>>>>> but perhaps it is not so easy for you to do. However, being
>>>>>>>>>>> able
>>>>>>>> to
>>>>>>>>>>> successfully use the methods discussed on the board opens
>>>>>>>>>>> up a
>>>>>>>> lot of
>>>>>>>>>>> extra possibilities due to the extra speed possible with
>>>>>>>>>>> AFL.
>>>>>>>>>>>
>>>>>>>>>>> I appreciate any help you can give in this matter."
>>>>>>>>>>>
>>>>>>>>>>> As I want a workable solution to my problem and want to
>>>>>>>>>>> use AB
>>>>>>>> for my
>>>>>>>>>>> project (I would be trading this already using my AB system
>>>>>>>>>>> if I
>>>>>>>> had
>>>>>>>>>>> a solution), I will state the fundamental problems I face
>>>>>>>>>>> clearly
>>>>>>>>>>> together here (instead of disjointed posts like before) and
>>>>>>>>>>> then
>>>>>>>>>>> perhaps a solution might occur to you or somebody else.
>>>>>>>>>>> Please
>>>>>>>>>>> forgive the length and complexity of this post.
>>>>>>>>>>>
>>>>>>>>>>> Background:
>>>>>>>>>>>
>>>>>>>>>>> My trading system is very simple in principle. AFL
>>>>>>>>>>> calculates a
>>>>>>>>>>> complex indicator, I draw manual trend lines, and I place
>>>>>>>>>>> manual
>>>>>>>>>>> trades on a single stock or ETF at a time. It is day
>>>>>>>>>>> trading,
>>>>>>>>>>> so I
>>>>>>>>>>> only have a few seconds to trigger my trade when the
>>>>>>>>>>> conditions
>>>>>>>>>>> are
>>>>>>>>>>> right. My AFL allows rapid manual selection of different
>>>>>>>> timeframes
>>>>>>>>>>> via a parameter, and selects the proper volume bar settings
>>>>>>>>>>> dynamically for different ETFs or stocks. The program
>>>>>>>> automatically
>>>>>>>>>>> runs through many timeframes to select trade trigger
>>>>>>>>>>> possibilities
>>>>>>>>>>> for me to consider.
>>>>>>>>>>> 1. There is a heavy compute environment --several seconds of
>>>>>>>>>>> AFL.
>>>>>>>>>>> 2. The environment is highly interactive --drawing trend
>>>>>>>>>>> lines.
>>>>>>>>>>> 3. Seconds count for placing trades.
>>>>>>>>>>>
>>>>>>>>>>> First Problem set:
>>>>>>>>>>>
>>>>>>>>>>> Under heavy compute loads (when AFL time is greater than
>>>>>>>>>>> refresh
>>>>>>>>>>> rate), AB stops responding to the UI.
>>>>>>>>>>>
>>>>>>>>>>> Workarounds:
>>>>>>>>>>>
>>>>>>>>>>> The only way I can get some UI functionality back is to
>>>>>>>>>>> set the
>>>>>>>>>>> refresh rate to one second longer than the AFL compute time.
>>>>>>>> Then I
>>>>>>>>>>> can get UI functionality back, but with many delays.
>>>>>>>>>>> I also have a ParamToggle() to turn off most indicator
>>>>>>>>>>> processing
>>>>>>>> so
>>>>>>>>>>> that I can interact with the UI better at times (but I
>>>>>>>>>>> lose my
>>>>>>>>>>> indicator plot references).
>>>>>>>>>>> This is a partially effective workaround that I could limp
>>>>>>>>>>> along
>>>>>>>> with.
>>>>>>>>>>>
>>>>>>>>>>> Requested solution:
>>>>>>>>>>>
>>>>>>>>>>> Realizing that the root problem is that the UI functions
>>>>>>>>>>> only run
>>>>>>>> (or
>>>>>>>>>>> initiate) in between AFL passes, I looked for a way to do
>>>>>>>>>>> less
>>>>>>>>>>> processing on each pass. The AFL load could be much less
>>>>>>>>>>> most of
>>>>>>>> the
>>>>>>>>>>> time because I only need to calculate everything when a new
>>>>>>>>>>> bar is
>>>>>>>>>>> added.
>>>>>>>>>>> I thought of two possibilities:
>>>>>>>>>>>
>>>>>>>>>>> 1. Ask for an AFL function that is essentially
>>>>>>>>>>> DoUserInterface
>>>>>>>>>>> ().
>>>>>>>>>>> Then put AFL into an infinite loop and put these commands at
>>>>>>>>>>> appropriate places. Since this is backwards from the way
>>>>>>>>>>> the AFL
>>>>>>>>>>> appears to work, I did not make this suggestion.
>>>>>>>>>>>
>>>>>>>>>>> 2. Ask for arrays that keep their data between AFL
>>>>>>>>>>> passes. I
>>>>>>>> tried
>>>>>>>>>>> using ATC for this. It functioned like it was supposed
>>>>>>>>>>> to, but
>>>>>>>>>>> was
>>>>>>>>>>> too slow to be a practical solution. I understand the
>>>>>>>>>>> reason
>>>>>>>>>>> they
>>>>>>>>>>> are so slow, and that is because they are a true static
>>>>>>>>>>> permanent
>>>>>>>>>>> array. However, what I wanted was only for a normal array
>>>>>>>>>>> to be
>>>>>>>>>>> saved between passes. They only need to be identified by
>>>>>>>>>>> which
>>>>>>>> pane
>>>>>>>>>>> they are associated with (could be part of the internal
>>>>>>>>>>> name).
>>>>>>>> They
>>>>>>>>>>> are only valid between new bars (same array size).
>>>>>>>>>>> Changes to
>>>>>>>>>>> symbol, timeframe, parameters, etc., can be detected in
>>>>>>>>>>> the AFL
>>>>>>>>>>> and
>>>>>>>>>>> the array updated. I know this can be done, because I did
>>>>>>>>>>> it for
>>>>>>>> the
>>>>>>>>>>> ATC version I wrote and everything worked perfectly (just
>>>>>>>>>>> not
>>>>>>>>>>> faster). Because AFL is so fast working with arrays, It
>>>>>>>>>>> seems
>>>>>>>>>>> that
>>>>>>>>>>> it should be possible to just save an array in memory in a
>>>>>>>>>>> high
>>>>>>>> speed
>>>>>>>>>>> way, then recall it to repopulate the array instead of
>>>>>>>>>>> recalculate
>>>>>>>>>>> the array if the AFL determines that the data would not have
>>>>>>>> changed
>>>>>>>>>>> since the last pass.
>>>>>>>>>>>
>>>>>>>>>>> There may be another possible solution that I have not
>>>>>>>>>>> thought of
>>>>>>>> and
>>>>>>>>>>> I would love to hear it.
>>>>>>>>>>>
>>>>>>>>>>> Second Problem set:
>>>>>>>>>>>
>>>>>>>>>>> Because this is a realtime trading system, decisions have
>>>>>>>>>>> to be
>>>>>>>> made
>>>>>>>>>>> from raw data which contains bad ticks that may fool the
>>>>>>>>>>> system.
>>>>>>>>>>> Volume bars are used in the system, and must be able to
>>>>>>>>>>> change
>>>>>>>> volume
>>>>>>>>>>> per bar on different AFL passes.
>>>>>>>>>>> Built-in timeframe or volume settings do not allow the
>>>>>>>> flexibility to
>>>>>>>>>>> address these
>>>>>>>>>>>
>>>>>>>>>>> Workarounds:
>>>>>>>>>>>
>>>>>>>>>>> A tick database is too large to have enough history for the
>>>>>>>>>>> indicators. Running a database of 5 seconds for a basic 1
>>>>>>>>>>> minute
>>>>>>>> bar
>>>>>>>>>>> (or equivalent for volume bars) allows for reasonably
>>>>>>>>>>> effective
>>>>>>>>>>> bad
>>>>>>>>>>> tick removal and volume bar generation. History can be just
>>>>>>>>>>> 60K
>>>>>>>>>>> bars. Bad ticks are removed and volume bar settings are
>>>>>>>>>>> determined
>>>>>>>>>>> using the 5 second bars. Arrays are built up using
>>>>>>>> TimeFrameSet() to
>>>>>>>>>>> get the actual bars used for indicator calculations and
>>>>>>>>>>> graph
>>>>>>>> plots.
>>>>>>>>>>> This approach works really well, however, there is one big
>>>>>>>>>>> problem
>>>>>>>>>>> with it.
>>>>>>>>>>>
>>>>>>>>>>> Even though I can display all my price and indicators in the
>>>>>>>>>>> timeframe I desire, I can not draw studies that align to
>>>>>>>>>>> that
>>>>>>>>>>> timeframe. All studies are drawn in the pane relative to 5
>>>>>>>>>>> second
>>>>>>>>>>> bars. This workaround is not acceptable for trading.
>>>>>>>>>>>
>>>>>>>>>>> Requested solution:
>>>>>>>>>>>
>>>>>>>>>>> #1117 Make an AFL way to set the timeframe of the drawing
>>>>>>>>>>> studies
>>>>>>>> to
>>>>>>>>>>> match the OHLC array created by TimeFrameSet().
>>>>>>>>>>>
>>>>>>>>>>> Of course implementing this has a general benefit of
>>>>>>>>>>> giving an
>>>>>>>> almost
>>>>>>>>>>> infinite number of time/volume/range bars settings via AFL
>>>>>>>>>>> and
>>>>>>>>>>> parameters.
>>>>>>>>>>>
>>>>>>>>>>> Please consider the problems (not just proposed solutions)
>>>>>>>>>>> and
>>>>>>>> advise
>>>>>>>>>>> if other possibilities come to mind for solutions.
>>>>>>>>>>>
>>>>>>>>>>> Thank you very much for your patients and consideration,
>>>>>>>>>>>
>>>>>>>>>>> Dennis Brown
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Aug 4, 2007, at 6:16 AM, Tomasz Janeczko wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Dennis,
>>>>>>>>>>>>
>>>>>>>>>>>> I explained it many times:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. You can not simply "KEEP" the array from previous
>>>>>>>>>>>> execution
>>>>>>>> and
>>>>>>>>>>>> reference it directy becase:
>>>>>>>>>>>> a) previous execution may be on different symbol/different
>>>>>>>>>>>> time
>>>>>>>>>>>> frame/different zoom level leading
>>>>>>>>>>>> to DIFFERENT NUMBER OF ELEMENTS in the array and different
>>>>>>>>>>>> TIMESTAMPS of array bars.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Because of (1) the only way to implement "static
>>>>>>>>>>>> array" is
>>>>>>>>>>>> to:
>>>>>>>>>>>> a) store not only array BUT also TIMESTAMPS with every bar
>>>>>>>>>>>> b) during reading perform timestamp checking and
>>>>>>>> synchronization so
>>>>>>>>>>>> referenced array data is aligned
>>>>>>>>>>>> with current execution data
>>>>>>>>>>>>
>>>>>>>>>>>> If those two were not implemented you would NOT be able
>>>>>>>>>>>> to use
>>>>>>>>>>>> "static arrays" at all.
>>>>>>>>>>>>
>>>>>>>>>>>> 3. Functionality already exists
>>>>>>>>>>>> a) Point 2a - storing arrays with timestamps is exactly
>>>>>>>>>>>> what
>>>>>>>>>>>> AddToComposite ALREADY DOES
>>>>>>>>>>>> b) Point 2b - reading arrays with timestamps and
>>>>>>>>>>>> aligning is
>>>>>>>>>>>> exactly what Foreign ALREADY DOES
>>>>>>>>>>>>
>>>>>>>>>>>> 4. Implementing dedicated "static" array functions would
>>>>>>>>>>>> a) give NO SPEED ADVANTAGE over AddToComposite/Foreign
>>>>>>>>>>>> b) give only "the impression" that it is something else and
>>>>>>>>>>>> "faster" than ATC/Foreign while it would
>>>>>>>>>>>> be FALSE impression.
>>>>>>>>>>>>
>>>>>>>>>>>> Summary:
>>>>>>>>>>>> a) functionality already exists
>>>>>>>>>>>> b) no speed advantage over existing functionality
>>>>>>>>>>>>
>>>>>>>>>>>> So here are the functions you are asking for:
>>>>>>>>>>>>
>>>>>>>>>>>> function StaticVarArraySet( Varname, array )
>>>>>>>>>>>> {
>>>>>>>>>>>> AddToComposite( array, "~SA_"+VarName, "C",
>>>>>>>>>>>> atcFlagDefaults |
>>>>>>>>>>>> atcFlagEnableInBacktest | atcFlagEnableInExplore |
>>>>>>>>>>>> atcFlagEnableInIndicator | atcFlagEnableInPortfolio );
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> function StaticVarArrayGet( VarName );
>>>>>>>>>>>> {
>>>>>>>>>>>> return Foreign( "~SA_"+VarName, "C" );
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Tomasz Janeczko
>>>>>>>>>>>> amibroker.com
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: Dennis Brown
>>>>>>>>>>>> To: amibroker@xxxxxxxxxxxxxxx
>>>>>>>>>>>> Sent: Saturday, August 04, 2007 5:26 AM
>>>>>>>>>>>> Subject: [amibroker] Second thoughts on Static arrays
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> The more I have thought about static arrays for my AFL
>>>>>>>>>>>> needs,
>>>>>>>>>>>> the
>>>>>>>>>>>> more I see that what I am looking for is a quite limited
>>>>>>>>>>>> implementation.
>>>>>>>>>>>>
>>>>>>>>>>>> Static arrays would have low overhead and give the speed
>>>>>>>>>>>> needed
>>>>>>>>>>>> just like regular arrays. The primary need is to avoid re-
>>>>>>>>>>>> computing complex things all the time that only need to be
>>>>>>>> computed
>>>>>>>>>>>> under certain conditions or perhaps once per new bar. This
>>>>>>>> allows
>>>>>>>>>>>> far richer AFLs to be developed for real time trading,
>>>>>>>>>>>> while
>>>>>>>>>>>> improving the responsiveness of the UI interaction under
>>>>>>>>>>>> heavy
>>>>>>>>>>>> compute conditions. The life of a static array is
>>>>>>>>>>>> fleeting.
>>>>>>>>>>>>
>>>>>>>>>>>> Static arrays are not permanent like ATC, and do not have
>>>>>>>>>>>> the
>>>>>>>>>>>> big
>>>>>>>>>>>> overhead. Static arrays as I envision them, would have
>>>>>>>>>>>> to be
>>>>>>>>>>>> updated via the user AFL program whenever a change in
>>>>>>>>>>>> environment
>>>>>>>>>>>> caused the data to become invalid --like changing the
>>>>>>>>>>>> symbol or
>>>>>>>> the
>>>>>>>>>>>> timeframe --or even adding a new bar (which changes the
>>>>>>>>>>>> array
>>>>>>>>>>>> size). Having static arrays like this would allow breaking
>>>>>>>>>>>> the
>>>>>>>> AFL
>>>>>>>>>>>> into two parts --the fast part that needs to run for every
>>>>>>>> second,
>>>>>>>>>>>> and the part that needs to run once per compressed
>>>>>>>>>>>> timeframe,
>>>>>>>>>>>> volume, or range bar.
>>>>>>>>>>>>
>>>>>>>>>>>> I am sure there are technical issues involved, or TJ would
>>>>>>>>>>>> have
>>>>>>>>>>>> already provided them since they are so useful for
>>>>>>>>>>>> speeding up
>>>>>>>> AFL.
>>>>>>>>>>>> First rule of making programs fast is pre-compute
>>>>>>>>>>>> everything
>>>>>>>>>>>> possible and not inside a loop. AFL is one big loop.
>>>>>>>>>>>>
>>>>>>>>>>>> Perhaps what I am looking for are not technically "static"
>>>>>>>>>>>> arrays
>>>>>>>>>>>> in the normal sense. Maybe I need to call them by a
>>>>>>>>>>>> different
>>>>>>>> name
>>>>>>>>>>>> for the suggestion box. Perhaps there is an even easier
>>>>>>>>>>>> way to
>>>>>>>>>>>> address this need with a new AFL syntax: keep(myArray);
>>>>>>>>>>>>
>>>>>>>>>>>> Comments anyone?
>>>>>>>>>>>>
>>>>>>>>>>>> Dennis Brown
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
Please note that this group is for discussion between users only.
To get support from AmiBroker please send an e-mail directly to
SUPPORT {at} amibroker.com
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|