[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Trends, random series



PureBytes Links

Trading Reference Links


 
<BLOCKQUOTE 
>
----- Original Message ----- 
<DIV 
>From: 
Richard 
Alford 
To: <A title=amibroker@xxxxxxxxxx 
href="">amibroker@xxxxxxxxxxxxxxx 
Sent: Thursday, June 20, 2002 1:26 
PM
Subject: Re: [amibroker] Trends, random 
series

Why wouldn't one simply design on now - 6 months, and 
confirm on the previous 6 months?  The suggestion of designing to today, 
then testing for the next 6 months doesn't appear to be any different  - 
if one is absolutely certain they did not use information from the preceding 6 
months...  Of course, selecting systems that perform well on the last 6 
out of sample data does use that data - tis a puzzlement.  Certainlyonly 
trying a new system every 6 months would constrain losses.
 
Using a portion of existing data for 
system development and the balance for out of data testing is the standard 
procedure, thereby having the ultimate Scrambler.  However, one of the 
pitfalls in evaluations of this nature is being able to ensure that the period 
used for development includes a variety of market conditions, which usually 
requires a fair amount of data.  Otherwise, one probably only has a 
system that is good for bull/bear/flat markets, etc.  

 
wrt Al's point - I will let him manage that one.  I am 
marketing bullet proof boots, btw.
 
Cheers,
 
Richard
 
 
<BLOCKQUOTE 
>
----- Original Message ----- 
<DIV 
>From: 
Ken Close 

To: <A title=amibroker@xxxxxxxxxxxx 
href="">amibroker@xxxxxxxxxxxxxxx
Sent: Thursday, June 20, 2002 11:35 
AM
Subject: RE: [amibroker] Trends,random 
series

<SPAN 
class=170473116-20062002>William:
<SPAN 
class=170473116-20062002> 
I think the 
point Al was making with his idea of using the Simulator is that you save 
TIME.  If you optimize (bad word--deduce a system) based on the last 
two years up until yesterday, and then you wish to test "out-of-sample"for 
6 months, you will have to sit and collect the next six months worth ofdata 
before you can draw a conclusion about the worth of the system you have 
developed.  The idea behind the simulator as I understand it is to 
generate the next six months worth of data NOW and if results are 
satisfactory, THEN begin trading tomorrow.
<SPAN 
class=170473116-20062002> 
The purpose 
is to save time.
<SPAN 
class=170473116-20062002> 
However, you 
shoot yourself in the foot if the conclusion is that the simulated datais 
worthless as an out-of-sample test.
<SPAN 
class=170473116-20062002> 
I think I 
have Al's purpose stated correctly.
<SPAN 
class=170473116-20062002> 
Ken 

<FONT face=Tahoma 
size=2>-----Original Message-----From: wpeters_1 
[mailto:wpeters_1@xxxx]Sent: Thursday, June 20, 20021:17 
PMTo: amibroker@xxxxxxxxxxxxxxxSubject: RE: 
[amibroker] Trends, random series
<FONT face=Arial color=#0000ff 
size=2>Thank you Al for that..i'm not sure thatyou saw 
this follow up of mine: 
<FONT face=Arial color=#0000ff 
size=2> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002>"You then have to go on and ask what would 
be the value of studying biases introduced in a series of randomised data by 
mathematicians which does not occur in the real world market 
place.
<FONT 
face=Arial color=#0000ff size=2> 
<FONT 
face=Arial>Surely what would be valuable 
is to study and root out biases already in the market due to human 
behaviour or whatever. And get to learn about human behaviour and the effect 
on the market place.<SPAN 
class=875075116-20062002>"
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002>Also-
<FONT 
face=Arial>I still 
don't see the value in this approach over using real data history. 
If at the core of your argument you are saying there is some possibility of 
this system reflecting the future, what might happen in the futureetc, 
then so could the data from the 
past..
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial>For me 
it always comes back to 'where is the value' over straight ticker history, 
which we know is as real as its going to get. Humanistic patterns have been 
more or less the same over many years at a base level and they are 
represented in history of the chart for us to study. Any tinkering 
with those patterns and you get some artificial result which will never 
happen in the real world market....and again what is the worth of studying 
that data over the real?
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002>Appreciate your 
courage
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002>William
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial color=#0000ff size=2> 

<FONT face=Tahoma 
size=2>-----Original Message-----From: Avcinci 
[mailto:avcinci@xxxx]Sent: Wednesday 19 June, 2002 07:12 
PMTo: amibroker@xxxxxxxxxxxxxxxSubject: Re: 
[amibroker] Trends, random series

William (and Richard),
 
Let me try to explain Leo’s 
scrambler. It was developed by Tushar Chande from his book Beyond 
Technical Analysis, 2nd ed. Chande randomly rearranges the data of a 
ticker to create new sequences. He does this as follows: he observes the 
relationship between the O, H, L, and C of the 2nd bar by using 
the C of the 1st bar as reference. So, he writes the 
relationship as: 
 
DeltaO = O – ref(C,-1);
DeltaH = H – ref(C,-1);
DeltaL = L – ref(L,-1);
DeltaC = C – ref(C,-1);
 
He samples with replacement 
with these formulas and creates patterns that bear the market’s signature 
as defined by relative price relationships. The next step is to use a 
random number generator to scramble the bars. When you have a new 
sequence, you need a starting point, which is usually the prior C. The new 
bar is derived from the prior C as follows (where the Syn prefix stands 
for the new synthetic values): 
 
Syn-C = ref(C,-1) + 
deltaC;
Syn-H = ref(C,-1) + 
deltaH;
Syn-L = ref(C,-1) 
+deltaL;
Syn-O=ref(C,-1) + 
deltaO;
 
He calculates the interbar 
relationships as defined above for a ticker of your choice. He uses a 
random number generator to pick a number from 1 to x (x determined bythe 
no. of bars you want to scramble). That number is the next bar of the 
sequence. Suppose on the 10th pick, you pick bar 5. Then the 
original bar 5 becomes bar 10 of the new sequence. The bars may repeat. 
You can generate as long a sequence as desired. You use the synthetic 
values determined by the equations above to establish the next bar’s 
appearance. Thus, what you wind up doing is encapsulating the market 
behavior in the original bar 10 and reproducing it in another sequence to 
create new synthetic data. You can generate a variety of chart patterns of 
any length using data scrambling. According to Chande, you can generate 
100 years of data and test your system against a variety of market 
conditions. Since these are the types of patterns you are likely to see in 
the future, this is the most rigorous out-of-sample testing you can 
achieve. 
 
The above commentary was 
partly plagiarized from Chande’s book, and it is merely a synopsis.To get 
a clearer understanding of the methodology with spreadsheet examples,you 
should read his chapter devoted to scrambling. 
 
I have attached 2 gifs 
showing what scrambled data look like on 2 different tickers. Note that 
the patterns are not any more unusual than those of normal chart patterns, 
and you can see definite trends developing. In fact, if you look at recent 
real charts of the market indices, you will see many instances of V-tops 
and V-bottoms, which is what William was concerned about earlier today. 
Hope this helps. Many thanks to Leo Timmermans for programming Chande’s 
scrambler for Amibroker. I asked him recently if he could program it in 
such a way to create synthetic data for an entire watch list at one time. 
That might be tricky, but he said he would look into it. 
 
Al V.
<BLOCKQUOTE 
>
----- Original Message ----- 
<DIV 
>From: 
<A title=wpeters_1@xxxx 
href="">wpeters_1 
To: <A 
title=amibroker@xxxxxxxxxxxxxxx 
href="">amibroker@xxxxxxxxxxxxxxx 

Sent: Wednesday, June 19, 2002 1:22 
PM
Subject: RE: [amibroker] Trends, 
random series, etc : was MetaStock and AmiBroker

<FONT face=Arial color=#0000ff 
size=2>AL,
<FONT face=Arial color=#0000ff 
size=2> 
<FONT face=Arial color=#0000ff 
size=2>With randomised data isn't there a chance that the so-called 
'trend' can reverse on the next tick 'more-so' than data based on human 
intervention. And that includes mechnical systems (ie. if enough traders 
started using a certain indicator).
<FONT face=Arial color=#0000ff 
size=2>The obvious predictability in human behaviour is what can give 
you the little advantage and this is not in totally randomised 
data.
<FONT face=Arial color=#0000ff 
size=2> 
<FONT face=Arial color=#0000ff 
size=2>Surely if so your 'randomised' data 'trend' does not have an 
equal chance of reversing on each and every tick then its not 
randomised.
<FONT face=Arial color=#0000ff 
size=2> 
<FONT face=Arial color=#0000ff 
size=2>Very interesting topic.
 
<FONT face=Arial color=#0000ff 
size=2>William

<FONT face=Tahoma 
size=2>-----Original Message-----From: Al Venosa 
[mailto:avcinci@xxxx]Sent: Wednesday 19 June, 2002 
10:54 AMTo: amibroker@xxxxxxxxxxxxxxxSubject: 
Re: [amibroker] Trends, random series, etc : was MetaStock and 
AmiBroker


Richard,
I'm glad you took the liberty of changing the subject line. I don't 
think you are being a pessimistic, grumpy old man (sorry, don't know 
how old you are). Let me throw in another 2 cents into the discussion 
of trends. You said in order to have a trend, there must exist some 
information in the immediate past that would cause the trend to 
persist for a long enough time to profit from the move. I cannot agree 
more. That is absolutely true. A bad earnings report comes out on 
INTC, and the market immediately reacts by selling. This is further 
fueled by some nitwit analyst who changes his buy recommendation to 
hold or sell (of course, always after the fact!). Then, all the other 
individual "investors" climb aboard and do more selling, driving the 
prices still further down. Then, the company foretells that over the 
next several quarters there will be more of the same in regards to 
diminished sales, inducing further pessimism that drives the prices 
still further. All of this information results in a downward trend. 
This does not nor cannot happen with random events like coin tosses. 
So, technically speaking, you are perfectly correct in your assertions 
. 
Now, the trend follower comes in. He couldn't care less what is 
driving the prices down. In fact, he purposefully never pays attention 
to news events and announcements that may drive prices in one 
direction or another. He doesn't care about the cause of the trend. 
All he does is climb aboard the resulting trend, using whatever entry 
he has fashioned to enable him to take advantage of this 'breakout' 
and whatever exit that enables him to exit when the trend falters. Who 
was taking the other side of the market when Neeson was loading up on 
Nikkei futures, driving the Barings Bank into bankruptcy? The trend 
followers who recognized what was happening and capitalized. WhatI'm 
saying is, the trend follower buys or sells all breakouts (however you 
want to define the term 'breakout') and hopes that the resulting price 
behavior continues long into the future. With the scrambler, these 
'trends' can indeed take place even with randomized data, as you have 
already pointed out. What I'm suggesting is that your system of 
entries and exits doesn't have a clue what caused the breakout to 
occur because it's nothing more than a mathematical algorithm that 
trades based on certain signals programmed by the user when certain 
price behavior occurs. I contend you can test your system on such 
price behavior just to learn how well the system reacts to price 
breakouts and to see if, indeed, it is effective in finding trends in 
the data, albeit randomized data. By the way, these comments are not 
limited to trend following systems. I chose that trading approach 
merely as an example to make my point. 
Richard, I'm not trying to be argumentative on this topic. I'mjust 
pointing out why I think there might be value in testing a fully 
optimized trading model on randomized data. Thanks for the provocative 
discussion. 
Al V.





>From: "Richard Alford" 
>Reply-To: amibroker@xxxxxxxxxxxxxxx 
>To: 
>Subject: [amibroker] Trends, random series, etc : was 
MetaStock and AmiBroker 
>Date: Wed, 19 Jun 2002 08:35:15 -0500 
> 
>I agree on the oxymoronic nature of so called random 
series. The problem results from using very particular distributions 
for the random numbers which allows the mathematicians to write books 
and teach classes. The Gaussian functions are one of the most 
well-behaved functions around. 
> 
>wrt trends in scrambler and other "random" time series 
- simply stated if they are truly random there is absolutely no 
information about the past in the next event. Your coin toss 
observation is a great example: given the 1/1000 case of 10 headsin a 
row the odds of another head are precisely 50% for the next toss. 
> 
>In order to discuss a "trend" there must be some 
information that persists. One can generally, perhaps always, see 
trends in a random series - in hindsight - however it contains no 
predictive information. There are systems that seek to identify 
trending vs. trading range behavior (I think Ehlers addresses this 
issue - can't say with how much success), however, they are inaccurate 
at the changes. A moving average on the coin toss series will identify 
trends - but the lag in the moving average that we all know of and try 
to remove is the "hindsight" problem. 
> 
>Trend scale is still another issue. The apparent 
fractal behavior of the stock market suggests that one can find 
trends, and/or search for trends, over any time scale desired. (The 
inflationary or perhaps evolving nature of economies have placed a 
long term upward bias on the trends that can confuse the issue.) Once 
again a moving average of any length on a random series will display 
periods of upward and downward "trends". 
> 
>Sorry to be such a pessimistic old grump. 
> 
>Cordially, 
> 
>Richard 
> 
> ----- Original Message ----- 
> From: Avcinci 
> To: amibroker@xxxxxxxxxxxxxxx 
> Sent: Tuesday, June 18, 2002 8:19 PM 
> Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> 
> 
> Richard, 
> 
> >>distribution of the trends is predictablefor 
a random series >> This sounds oxymoronic to me, i.e., 
predictable distribution of random trends. But, I'll take your word 
for it. :-)) 
> 
> Regarding the scale when speaking of trends, I was 
referring to weeks to months, perhaps even a year or two at most 
(stocks, not futures), but not decades. This brings us back to 
discussing the Scrambler. You said you don't hold much if any value in 
it, but can you elaborate just a little more on what you mean in terms 
of the scale I am referring to, because in a period of 130 trading 
days, I noticed some pretty decent trends develop that lasted long 
enough to be of value in system testing? Thanks, Richard. By the way, 
I agree that this discussion, interesting as it is, has absolutely 
nothing to do with Metastock!! 
> 
> AV 
> 
> ----- Original Message ----- 
> From: Richard Alford 
> To: amibroker@xxxxxxxxxxxxxxx 
> Sent: Tuesday, June 18, 2002 3:55 PM 
> Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> 
> 
> wrt rsi/stochastics/cmo/etc... my point was that they 
attempt to identify a phenomenon that has been observed as an 
indication of overbought/oversold. There is a reasonable belief that 
the arithmetic created to identify that behavior can be a useful 
indictor of that condition. The comment offered for Elliot wave and 
Fibinacci behavior also points to a large community that believesthat 
there is a hidden structure to the market and human behavior - I am 
not a devotee of that belief, but I also have my doubts about Madam 
Cleo.... 
> 
> wrt trends in random numbers: there is, of course, a 
trend between any two non=equal numbers. Given a suitable sample, the 
distribution of the trends is predictable for a random series - 
probably also random and if the normal Gaussian (pardon the pun) 
distribution is used, as the derivative (or instantaneous slope/trend) 
is a skewed Gaussian. Interesting to the statisticians but not very 
useful to the average investor. 
> 
> wrt: trending 30% - that really has to be a matter of 
scale. If you chose a suitably long moving average you can probably 
even consider today as part of a long time upward trend - not 
particularly useful - although even my meager portfolio looks better 
today than it did in 1970 - not saying much.... 
> 
> Interesting discussion - don't have the faintest idea 
what it has to do with MetaStock :) 
> 
> Cheers, 
> 
> Richard 
> 
> 
> ----- Original Message ----- 
> From: Al Venosa 
> To: amibroker@xxxxxxxxxxxxxxx 
> Sent: Tuesday, June 18, 2002 2:28 PM 
> Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> 
> 
> Thanks, Richard. I always enjoy your mathematical 
insights. I hope you didn't get the impression I was suggesting 
'optimizing' on random bars. I was merely suggesting 'testing' your 
already optimized system on the scrambled data, seeing if it can 
detect those occasional occurrences of non-random behavior of random 
numbers, that's all. Certainly if a trend-following system can detect 
trends in random price bars, it ought to be able to detect them in 
real, non-random price bars, too, don't you think? Markets trend only 
about 30% of the time, as I am told, so wouldn't you consider 
non-trending markets (i.e., sideways markets) somewhat representative 
of random price behavior? 
> 
> I also get the impression you are not an advocateof 
overbought/oversold oscillators, right? 
> 
> Al Venosa 
> avcinci@xxxx 
> >From: "Richard Alford" 
> >Reply-To: amibroker@xxxxxxxxxxxxxxx 
> >To: 
> >Subject: Re: [amibroker] Re: MetaStock and 
AmiBroker 
> >Date: Tue, 18 Jun 2002 12:55:33 -0500 
> > 
> >The non-random appearance of random numbers is a 
very well know phenomenon - probably accounts for the continued 
enthusiasm in slot machines in Vegas... 
> > 
> >I stand by my comments. In particular, the various 
oversold/bought indicators rely on the observation that the close 
tends to be higher in the l-h range in the overbought condition -the 
persistence of such a trend is interpreted as buy-sell behavior. 
> > 
> >I personally see no use in random data. Thereis 
no information in random data and no expectation of the past affecting 
the future - the only reason for technical analysis (other than too 
much time on one's hands) is to glean information about the near 
future. There are, of course, degenerate uses for random numbers in 
testing, but to test, build, or, heaven forbid - optimize, using 
random time series is pointless at best. 
> > 
> >wrt coin tosses: the coin is a Markov processof 
length 0 - no memory. If the market is the same, we may as well head 
for Vegas and get free drinks as we go broke. (OK - the market MAY be 
a fair game, but I doubt it.) 
> > 
> >Just some thoughts... 
> > 
> >Richard 
> > ----- Original Message ----- 
> > From: Al Venosa 
> > To: amibroker@xxxxxxxxxxxxxxx 
> > Sent: Tuesday, June 18, 2002 9:57 AM 
> > Subject: Re: [amibroker] Re: MetaStock and 
AmiBroker 
> > 
> > 
> > Well, Richard, to be honest, I haven't yet done 
any back (er, I mean forward) testing of the random time bars created 
by Scrambler. However, I have looked at the results of randomly 
scrambling the bars and watching them form on the screen, and youcan 
see definite trends develop that can last several months, much like in 
real life. So, at least theoretically, it seems to me that, if you are 
testing a short-term or intermediate-term trend-following system,it 
ought to be able to pick up on those short, seemingly non-random 
trends (even though they were developed from a random number 
generator). Remember, if you flip an honest coin 1000 times, there is 
a low but finite probability that it will come up heads 10 or 12 or 15 
times in a row. 
> > 
> > AV 
> > 
> > 
> > >From: "Richard Alford" 
> > 
> > >Reply-To: amibroker@xxxxxxxxxxxxxxx 
> > >To: 
> > >Subject: Re: [amibroker] Re: MetaStock and 
AmiBroker 
> > >Date: Tue, 18 Jun 2002 09:48:35 -0500 
> > > 
> > >I understood that the underlying premiseof 
technical analysis was that there IS information in the price-volume 
behavior. If you test against random data the results had better 
result in random behavior - perhaps mean behavior is more accurate. 
> > > 
> > >Perhaps I am delusional, however, I would 
appreciate insight into the value of random time bars? 
> > > 
> > >Cordially, 
> > >Richard 
> > > ----- Original Message ----- 
> > > From: Al Venosa 
> > > To: amibroker@xxxxxxxxxxxxxxx 
> > > Sent: Tuesday, June 18, 2002 9:23 AM 
> > > Subject: Re: [amibroker] Re: MetaStock and 
AmiBroker 
> > > 
> > > 
> > > Sorry, Dimitris, I thought you knew about 
it. Go to post 19331 and download it, following Leo Timmerman's 
instructions. What he has done is created a VBscript tool based on 
Tuchar Chande's price bar scrambler that scrambles all the bars of a 
given ticker from the previous x bars (bars and ticker set by the 
user) and then re-orders those bars randomly into the future, thus 
creating a ticker that you can use in forward-testing. If you setthe 
no. of bars to, say, 130, you can generate 6 months worth of new data 
for each ticker you do this on. Very, very cool. It differs from 
William's Simulator in that it actually generates new data ratherthan 
repeating existing data from the last x bars. Try it, you'll likeit. 
> > > 
> > > Al Venosa 
> > > 
> > > >From: "dtsokakis" 
> > > 
> > > >Reply-To: amibroker@xxxxxxxxxxxxxxx 
> > > >To: amibroker@xxxxxxxxxxxxxxx 
> > > >Subject: [amibroker] Re: MetaStock and 
AmiBroker 
> > > >Date: Tue, 18 Jun 2002 13:49:30 -0000 
> > > > 
> > > >What is Leo's scrambler ?? 
> > > >DT 
> > > >--- In amibroker@xxxx, "Al Venosa" 
wrote: 
> > > > > 
> > > > 
> > > 
> > > 
> > 
>------------------------------------------------------------------------------ 

> > > Chat with friends online, try MSN Messenger: 
Click Here 
> > > 
> > > Your use of Yahoo! Groups is subject tothe 
Yahoo! Terms of Service. 
> > > 
> > 
> > 
> 
>------------------------------------------------------------------------------ 

> > Get your FREE download of MSN Explorer at 
http://explorer.msn.com. 
> > 
> > Your use of Yahoo! Groups is subject to the 
Yahoo! Terms of Service. 
> > 
> 
> 
>-------------------------------------------------------------------------- 

> Join the world's largest e-mail service with MSN 
Hotmail. Click Here 
> 
> 
> Your use of Yahoo! Groups is subject to the Yahoo! 
Terms of Service. 
> 
> 
> 
> Your use of Yahoo! Groups is subject to the Yahoo! 
Terms of Service. 
> 
> 
> Yahoo! Groups Sponsor 
> ADVERTISEMENT 
> 
> 
> 
> 
> Your use of Yahoo! Groups is subject to the Yahoo! 
Terms of Service. 
> 


Send and receive Hotmail on your mobile device: <A 
href="">Click HereYour 
use of Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of 
Service. Your use of Yahoo!Groups 
is subject to the Yahoo! 
Terms of Service. Your use of Yahoo! 
Groups is subject to the <A 
href="">Yahoo! Terms of Service. 
Your use of Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of Service. 
Your use of Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of Service. 
Your use of Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of Service.