Re: [amibroker] Re: Paul Ho: Memory Challenges with Great Ranking Tool, AmiBroker Email List Archive

Hi Ara,

Thank you, but I know I can speak better! ;-)

Point 1 is correct.

Point 2 is incorrect. The backtest is only for... backtesting. I want to develop and make the system better before trading it. I don't want to backtest in real-time; I want to backtest my one-yeat 1-minute bar data right now, or every night. This is only backtesting. But I need to be sure that what I am backtesting can happen in real-time trading.

In live trading, this will be easy. I will run the scan each evening and get the best 500 tickers (e.g. those closest to their hhv 52 wks) out of my list of 8000 tickers and will simply add the result to a watchlist which will be what I will trade the next day. This is easy.

But BEFORE going to live trading, I must build the code and make it better and test it and retest it. Before putting my money there, I need to be sure it works. And right now I found no way to backtest it that way. I am not sure I need to go with OLE (anyway I am not even sure what it is). I am pretty confident it can be done with AFL.

Here what the AFL code should have:

A ranking feature and a loop meaning that for each daily-bar (I would compress to daily bar) the ranking would be done and the best 500 symbols would be chosen and used the next day in intraday. This would simply be part of a buy condition: to get a buy signal there would need to have a buying signal intraday AND to be part of the "chosen 500".

Can't this be done with AFL? And if it can be done, would time be a matter? I mean: if for each daily bar the ranking as to be done, would this be a problem, like it would take 10 hours for a simple backtest?

If it's not possible, I'm so screwed. I worked on something for the last 4-5 months only to discover it wouldn't work because there is a limit of 500 streaming symbols. I need to find a way to fix this and so far I didn't find any way to backtest this. It was easy to get 50% a year scanning 8,000 tickers, but it's far more difficult when I am limited to 500...

As for point #3, I don't believe in emotional trading. My gf brother-in-law is a successful trader which makes a lot of money and well enough for him and his wife and three children. But stress makes his life difficult. I know how to trade emotionally, but that's precisely what I am trying not to do. If I am building a system, it is to buy when it says buy, to sell when it says sell, and to jump off the bridge if it says so. I will have 100% faith in my system and for that to be possible and need to be able to backtest it as I would do in live-trading.

Thanks again for your help. I appreciate a lot.

Louis

2008/7/13 Ara Kaloustian <ara1@xxxxxxxcom>:

Louis,

You English seems quite good ...

First lets separate the functions you want to perform.

1. You want to get the best 500 stocks from EOD data every night. Correct?

2. You want to run a backtest in real time of the 500 stocks.

    - Every 1minute? every 5 minutes?

    - Using 1 minute data ??

3. Somehow compare these backtest results to something what can be acheived in live trading ... or assume     that the backtest results are the best than can be acheived in live trading.

#1. You need to do this step with an exploration as I indicated earlier. You can automate it with OLE, but save the effort and see if the concept works for you. Manually doing this is very simply.

#2. Running backtest of 500 in real time is a real problem. Graham mentioned the issues involved. You may need to take the real time data off line and do your backtest at end of day for that 1 day and get a best approximation. At lleast everything will be stable, which is not true when freesh data is coming in continuously.

#3. Remember that a very significant component of live trading is emotional. No amount of simulation will address that part.

Perhaps a better approach might be to generate trading signals (alerts) in real time and use them to paper trade manually. This has the benefit of getting you involved to some extent even though clearly paper trading does not generate the emotional response anywhere near live real money trading.

IB and other brokers have paper trading facilities that are not perfect, but very very useful.

----- Original Message -----

From: Louis Préfontaine

To: amibroker@xxxxxxxxxps.com

Sent: Sunday, July 13, 2008 5:11 PM

Subject: Re: [amibroker] Re: Paul Ho: Memory Challenges with Great Ranking Tool

Hi Ara,

Of course, this is what I will do when, and only when, I will be ready to go in live trading...

I know my english is poor, but I don't know how I can explain what I need better. I will try again; I hope this time it will be clearer.

I need to backtest AS IF I was in live trading, which mean I CAN NOT backtest 8000 symbols if in live trading I have only 500. This is why I need my backtesting code to automatically select the 500 best results EACH DAY to be sure that the results of the backtest will be as close as possible to what can be acieved in live trading...

Thanks,

Louis

2008/7/13 Ara Kaloustian <ara1@xxxxxxxcom>:

Looks like you need to run an exploration evry night, not a becktest. The exloration and ranking will give you the list o pick thr top 500 from. It can be done manually from AA very simply.

Actually you don't even need to rank them .. you can manually sort the exploration results with just a click

or make it more sophisticated and automatically update watchlist using more complex code.

Suggest keep things simple.

----- Original Message -----

From: Louis Préfontaine

To: amibroker@xxxxxxxxxps.com

Sent: Sunday, July 13, 2008 4:44 PM

Subject: Re: [amibroker] Re: Paul Ho: Memory Challenges with Great Ranking Tool

Hi Graham,

Thank you for your response.

It seems to me there can be a little misunderstanding. Maybe I am not clear enough in what I want to do.

I do NOT want to use this ranking during the day on intraday live data. I want to use this ranking at night when I have the EOD data and can select the 500 symbols which will be put in the watchlist for tomorrow's trade.

But this is the easy part (well, it seems easier than what's next).

What I want is to reproduce this in a backtest. I want to be able to run a backtest that would select each day's 500 tickers and use my 1-minute bar code on them each day. I want to be able to run the backtest on my 8000 symbols and know that each day a different set of 500 symbols will be selected for the code to be applied on them.

This is very important, because if I can't do that then everything I could build in a backtest could not be done in real time data. Do you understand what I mean?   I need to be able to click the backtest button and have the script select the 500 tickers closest to their 52 weeks high on day 1 and then apply the 1-minute code for the next day, and then select the 500 tickers for day 2, etc..

Can this be done?

Louis

2008/7/13 Graham <kavemanperth@gmail.com>:

Thinking about using ranking during the day on intraday live data
Running raking through 500 symbols would be very time consuming,
taking a long time to do calculations as each symbol ran through every
other 500 symbols. While this is running the data would also be
updating and your results could very well be nonsense.

My suggestion would be to run this differently using script and the
OLE control. You could run a straight exploration sorting on the
required column, export to a text file, then read off the top ranked
number of symbols on that list. From this just place them into a
watchlist, and run whatever other afl calculations etc to get your
trade signals. I would think that the script could be run every X
minutes from windows, must be some sort of tool that can do this.
I feel this would be much faster than the alternative ranking as you
would only be calculating the indicator values once in a single pass.
Provided your afl where reasonably straightforward your run time could
be less than a minute or 2, or less.

Now as you suggest you only want the previous daily values, then
simply put you run the ranking afl nightly and save the best ranked
symbols to watchlists. Or you sould create composite for each symbol
with the ranked value results in the arrays. Just run your intraday
scans referencing the composites
If you need to automate this, then again use OLE script and set up
windows to run the script at a certain time each day.

--
Cheers
Graham Kav
AFL Writing Service
http://www.aflwriting.com

2008/7/14 Louis Préfontaine <rockprog80@xxxxxxcom>:

> Hi,
>
> Thanks to you two for your responses.
>
> @Ara: I think I could achieve this (well I may need some help but this is
> something I am close to be able to do by myself), but the point 3 is the
> problem: I need this to be automatic. I need to be able to run a backtest
> using only the selected 500 symbols. E.g. If I use a script that backtests
> 8000 symbols and then can only act on 500 in real-time trading, then there
> is a problem. I need to find a code that would automatically select the 500
> "best" results (e.g. the closest to HHV 52 weeks) for the backtest to go
> on. How would you do that? Thanks!
>
> @Graham: I plan to use EOD ranking to decide which 500 symbols to trade
> intraday the next day. I can run the code at night based on the previous
> day Close, and in the morning I can manually select the 500 stocks which are
> closest to 52 weeks high, and those will be the one I will trade. As I
> wrote to Ara, I need to find a way to make this process automatic, to
> include it in the backtesting so as not to distort the results.
>
> Why do I use that much symbols? Because I never know in advance what I will
> trade.
>
> Are you serious when you say you trade 10 symbols at a time? This seem like
> so little data to me! Or maybe you trade ticks? I prefer to trade swing
> trading but with better entry points based on 1-minute data... Is this
> possible?
>
> Thanks,
>
> Louis
>
> 2008/7/13 Graham <kavemanperth@gmail.com>:
>>
>> Louis there would be no reasonable way to use intraday ranking values
>> for live trading on 500 symbols. The time to run the ranking would
>> mean the market moved on.
>> I personally cannot see any reason to use 500 symbols intraday unless
>> your system only gives 1 or 2 signals per day over all the stocks,
>> surely 10 symbols would be sufficient for live intraday trading
>>
>> --
>> Cheers
>> Graham Kav
>> AFL Writing Service
>> http://www.aflwriting.com
>>
>> 2008/7/13 Louis Préfontaine <rockprog80@xxxxxxcom>:
>> > Hi Graham,
>> >
>> > What exactly is a multi pass method and how can I use composite to get
>> > exactly the 500 tickers I want to trade each day?
>> >
>> > I'm really confused about this, but this seem essential to me. I wasted
>> > dozens of hours on a system that is working but that would not be
>> > working in
>> > RT because of the 500 symbols Real-time limitation. I must absolutely
>> > find
>> > a way to reproduce this limitation in a backtest. Hence the idea of
>> > getting
>> > a ranking of the 500 tickers. I understand that there can be problems,
>> > and
>> > it is surely possible to add a limitation that would not consider
>> > tickers
>> > that had major holes, as an example.
>> >
>> > But where to start? I tried the code that is posted in this thread and 2
>> > hours later AA is still running and nothing is happening. Is there a
>> > code
>> > that would help me? I must not be the only one with this problem!
>> >
>> > Thanks,
>> >
>> > Louis
>> >
>> > 2008/7/12 Graham <kavemanperth@gmail.com>:
>> >>
>> >> Some of the things I have found in the past running afl similar to the
>> >> one provided here for ranking
>> >> The ranking works great if all the symbols contain exactly the same
>> >> data dates or datetimes
>> >> Where it can cause problems with values is when some contain less
>> >> history, have data holes, or stopped trrading some time in the past
>> >> The indicator values are all calculated based on the bars of the
>> >> current symbol, so data holes in other symbols being referenced with
>> >> foreign function have padded or ignored bar information. This may not
>> >> affect all indicators but does with some.
>> >> eg base has holes, so the last 10 days may, in fuller symbols actually
>> >> be 11 days, thus 1 day is ignored. worse if the current symbol is a
>> >> very low traded symbol with many holes
>> >> Consider a simple HHV(H,100). In a fully traded symbol this would be,
>> >> say, 20 weeks of data. But in a symbol with holes it could represent
>> >> 25 weeks. Thus the full symbols are being calculated over 25 weeks not
>> >> 20, and the holes in base symbol may actually coincide with the
>> >> relevant indicator highs in fuller symbols
>> >> Of course you can get around this by using padding to reference symbol
>> >> in analsyis window. This can overcome some problems, but can also
>> >> introduce others in hole filled symbols depending on the indicator you
>> >> are using.
>> >>
>> >> Consider the effect of hole padding when your base symbol has full
>> >> trading. Your calculations will use the padded data for the weaker
>> >> symbols being referenced as foreign.
>> >>
>> >> A further problem can be that short history symbols being padded will
>> >> have the un-traded padded part at the start and give results of zero
>> >> or null. If zero then if your indicator has positive and negative
>> >> numbers (eg ROC) then the zero will be included in the ranking, even
>> >> though the symbol never actually existed at that earleir time. Same
>> >> can occur at the end of data if a symbol has not traded for some time
>> >> before the most recent date. it will provide results even though it
>> >> may have ceased trading on the exchange
>> >>
>> >> These are just some thoughts for you to consider
>> >>
>> >> I recommend that unless you are ranking over symbols that are
>> >> comparable in data that you use a multi pass method, utilising the
>> >> power of composites for storing the indicator values bgefore ranking
>> >> them. This can overcome some of the problems.
>> >>
>> >> --
>> >> Cheers
>> >> Graham Kav
>> >> AFL Writing Service
>> >> http://www.aflwriting.com
>> >>