[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: What is your Largest AFL file?



PureBytes Links

Trading Reference Links

Paul,

Initially I had been doing it manually. It hasn't been that bad (e.g. 
copying and pasting the html output of individual walk forward 
results into single unified file).

However, I have very recently begun experimenting with scripts to 
automate the process and stitch the results back togeather.

I have a few ideas that I'm exploring, but am not yet able to 
discuss 'till I've had a chance to further organize my thinking and 
do a proof of concept.

It is still early in my efforts, and I have limited time (nights 
after work and once the family is in bed), but first results look 
promising.

In any event, I will gladly make available whatever I come up with 
once it's reached a minimal level of functionality. So, if your spare 
equiptment is not causing you too much hardship, it may be worth 
hanging on to it a little longer.

Mike

--- In amibroker@xxxxxxxxxxxxxxx, "Paul Ho" <paul.tsho@xxx> wrote:
>
> Mike,
> Are you spreading your load over several machines manually, or 
you're using
> programs/scripts to automate that process?
> I have a few old machines that I was thinking about throwing out, 
may be I
> could utilise them if there was a practical way to share the load.
> /Paul.
> 
> 
>   _____  
> 
> From: amibroker@xxxxxxxxxxxxxxx [mailto:amibroker@xxxxxxxxxxxxxxx] 
On Behalf
> Of Mike
> Sent: Wednesday, 9 July 2008 6:26 PM
> To: amibroker@xxxxxxxxxxxxxxx
> Subject: [amibroker] Re: What is your Largest AFL file?
> 
> 
> 
> Ken,
> 
> For about a year now, I have been running a nightly exploration 
over 
> all symbols of AMEX, NASDAQ, NYSE combined. The script itself 
applies 
> logic to reject low liquidity stock, etc. and is applied to all 
> stocks, rather than preprocessing a watchlist. The script is 
> currently under 100 lines and completes in around 10 minutes (don't 
> remember exactly, I stopped paying attention a long time ago).
> 
> Optimizing that same script over 108 iterations takes anywhere from 
> 9 - 13 hours depending on the machine/memory (strictly low end 
> equiptment - Pentium 4's). Doing a walk forward of any meaningful 
> length would take days, except that I split the load over multiple 
> machines.
> 
> In an earlier incarnation, I had been writing out to composite 
> symbols in order to retain a sorted rank of PositionScore based on 
> all symbols scanned, not just ones that resulted in trade signals. 
I 
> then made reference to those rankings within my custom backtest 
code 
> to filter out signals that would not have occured due to no limit 
> order actually having been placed
> (http://finance.
> <http://finance.groups.yahoo.com/group/amibroker/message/114739>
> groups.yahoo.com/group/amibroker/message/114739). 
> 
> Ultimately, that proved to be prohibitively expensive (at least as 
> implemented, Thomasz had an alternate approach that was much faster 
> at the cost of altering the strategy logic), taking over 30 minutes 
> for an exploration. Optimizations were frustrating and impractical, 
> leading to my eventual purchase of many cheap low end machines to 
> farm out the load. Walk forward would have been out of the question.
> 
> Fortunately, I was able to change the money management aspect of 
the 
> system and move away from fixed number of positions. It's been 
> trading profitably ever since :)
> 
> Building on Herman's excellent suggestions, I would add that low 
end 
> machines (1.6 Ghz) can be had for under $150 these days (I got 
lucky 
> on eBay, paying $65 each for 7 of them!). Having a small farm 
allows 
> you to spread the load of a single walk forward (i.e. each machine 
> handling a different time period) or to backtest/optimize multiple 
> different versions of your strategy at the same time. 
> 
> Mike
> 
> --- In amibroker@xxxxxxxxx <mailto:amibroker%40yahoogroups.com> 
ps.com, "Ken
> Close" <ken45140@> wrote:
> >
> > A side question is this: what is the largest practical AFL file 
> ever created
> > that runs and is used regularly?
> > 
> > Obviously a hard question to answer (what is yours in terms of 
> number of
> > lines or number of symbols operated on). We talk about memory and 
> multiple
> > cores, and speed, etc, I will bet primarily in terms of 
> optimization speed.
> > But what about just a large number of symbols and a large number 
of
> > calculations and variables? That is what my question is aimed at.
> > 
> > If you are wondering why this question, here is a brief 
> background: all of
> > my recent postings have been because I am trying to work my way 
> towards
> > implementation of what seems like a very large system. I want to 
> calculate,
> > for each and every symbol, a ranking indicator or value really. I 
> want to
> > use the "ordinal value" approach that I have recently been asking 
> about.
> > Creating these ordinal values consumes a LOT of processing time 
and
> > calculations etc. I would like to do this on 1000s of symbols 
> (maybe the
> > entire database) but may have to settle for some subset, perhaps 
> only in the
> > hundreds. But that is only the first step.
> > 
> > After having these calculated values, I thought I would save them 
> into the
> > OI field of each symbol. Then, separately, I would create an 
> elaborate
> > rotational trading system, with a variety of parameters, and 
which 
> would use
> > the ordinal ranking parameter I had previously calculated and 
saved 
> into the
> > OI field. The rotational trading system, I am estimating, would 
> take a lot
> > of statements and need to keep track of a lot of information, and 
> that would
> > be before attempting any opimization of the trading parameters. I 
> am
> > guessing that the rawbacktester mode is required.
> > 
> > If separating the ranking calculation and saving it into the OI 
> field is a
> > problem (because of the interplay between local storage vs plugin 
> access
> > that I asked about in another message), then I could consider 
> putting
> > everything (I mean everything) into one humongous AFL file. 
> Calculate
> > ordinal rankings, keep them in their variables (by symbol) and 
then 
> go right
> > into a rotational trading routine, all in the same code file. I 
> suspect it
> > would either overload memory, or would take days of continual 
> running to
> > complete.
> > 
> > That is why I asked what is the largest code file you have ever 
> made. Maybe
> > my quest is a futile one, but I am still plowing along.
> > 
> > Thanks for listening.
> > 
> > Ken
> >
>



------------------------------------

Please note that this group is for discussion between users only.

To get support from AmiBroker please send an e-mail directly to 
SUPPORT {at} amibroker.com

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

For other support material please check also:
http://www.amibroker.com/support.html
Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    mailto:amibroker-digest@xxxxxxxxxxxxxxx 
    mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/