[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [amibroker] Trends, random series



PureBytes Links

Trading Reference Links


<SPAN 
class=170473116-20062002>William:
<SPAN 
class=170473116-20062002> 
I think the point 
Al was making with his idea of using the Simulator is that you save TIME.  
If you optimize (bad word--deduce a system) based on the last two years up until 
yesterday, and then you wish to test "out-of-sample" for 6 months, you willhave 
to sit and collect the next six months worth of data before you can draw a 
conclusion about the worth of the system you have developed.  The idea 
behind the simulator as I understand it is to generate the next six months worth 
of data NOW and if results are satisfactory, THEN begin trading 
tomorrow.
<SPAN 
class=170473116-20062002> 
The purpose is to 
save time.
<SPAN 
class=170473116-20062002> 
However, you 
shoot yourself in the foot if the conclusion is that the simulated data is 
worthless as an out-of-sample test.
<SPAN 
class=170473116-20062002> 
I think I have 
Al's purpose stated correctly.
<SPAN 
class=170473116-20062002> 
Ken 

<FONT face=Tahoma 
size=2>-----Original Message-----From: wpeters_1 
[mailto:wpeters_1@xxxx]Sent: Thursday, June 20, 2002 1:17 
PMTo: amibroker@xxxxxxxxxxxxxxxSubject: RE: [amibroker] 
Trends, random series
<FONT 
color=#000000>Thank you Al for that..i'm not sure that you saw this follow up of 
mine: 
<FONT color=#0000ff face=Arial 
size=2> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002>"You then have to go on and ask what would be 
the value of studying biases introduced in a series of randomised data by 
mathematicians which does not occur in the real world market 
place.
<FONT 
color=#0000ff face=Arial size=2> 
<FONT 
face=Arial>Surely what would be valuable 
is to study and root out biases already in the market due to human 
behaviour or whatever. And get to learn about human behaviour and the effect on 
the market place.<SPAN 
class=875075116-20062002>"
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002>Also-
<FONT 
face=Arial>I still 
don't see the value in this approach over using real data history. If 
at the core of your argument you are saying there is some possibility of this 
system reflecting the future, what might happen in the future etc, then so 
could the data from the past..
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial>For me it 
always comes back to 'where is the value' over straight ticker history, which we 
know is as real as its going to get. Humanistic patterns have been more or less 
the same over many years at a base level and they are represented 
in history of the chart for us to study. Any tinkering with those 
patterns and you get some artificial result which will never happen in the real 
world market....and again what is the worth of studying that data over the 
real?
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial>Appreciate 
your courage
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002>William
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
face=Arial><SPAN 
class=875075116-20062002> 
<FONT 
color=#0000ff face=Arial size=2> 

<FONT face=Tahoma 
size=2>-----Original Message-----From: Avcinci 
[mailto:avcinci@xxxx]Sent: Wednesday 19 June, 2002 07:12 
PMTo: amibroker@xxxxxxxxxxxxxxxSubject: Re: [amibroker] 
Trends, random series

William (and Richard),
 
Let me try to explain Leo&#8217;s 
scrambler. It was developed by Tushar Chande from his book Beyond Technical 
Analysis, 2nd ed. Chande randomly rearranges the data of a ticker to create 
new sequences. He does this as follows: he observes the relationship between 
the O, H, L, and C of the 2nd bar by using the C of the 
1st bar as reference. So, he writes the relationship as: 
 
DeltaO = O &#8211; ref(C,-1);
DeltaH = H &#8211; ref(C,-1);
DeltaL = L &#8211; ref(L,-1);
DeltaC = C &#8211; ref(C,-1);
 
He samples with replacement 
with these formulas and creates patterns that bear the market&#8217;s signature as 
defined by relative price relationships. The next step is to use a random 
number generator to scramble the bars. When you have a new sequence, you need 
a starting point, which is usually the prior C. The new bar is derived from 
the prior C as follows (where the Syn prefix stands for the new synthetic 
values): 
 
Syn-C = ref(C,-1) + deltaC;
Syn-H = ref(C,-1) + deltaH;
Syn-L = ref(C,-1) +deltaL;
Syn-O=ref(C,-1) + deltaO;
 
He calculates the interbar 
relationships as defined above for a ticker of your choice. He uses a random 
number generator to pick a number from 1 to x (x determined by the no. ofbars 
you want to scramble). That number is the next bar of the sequence. Suppose on 
the 10th pick, you pick bar 5. Then the original bar 5 becomesbar 
10 of the new sequence. The bars may repeat. You can generate as long a 
sequence as desired. You use the synthetic values determined by the equations 
above to establish the next bar&#8217;s appearance. Thus, what you wind up doing is 
encapsulating the market behavior in the original bar 10 and reproducing it in 
another sequence to create new synthetic data. You can generate a varietyof 
chart patterns of any length using data scrambling. According to Chande, you 
can generate 100 years of data and test your system against a variety of 
market conditions. Since these are the types of patterns you are likely to see 
in the future, this is the most rigorous out-of-sample testing you can 
achieve. 
 
The above commentary was partly 
plagiarized from Chande&#8217;s book, and it is merely a synopsis. To geta clearer 
understanding of the methodology with spreadsheet examples, you should read 
his chapter devoted to scrambling. 
 
I have attached 2 gifsshowing 
what scrambled data look like on 2 different tickers. Note that the patterns 
are not any more unusual than those of normal chart patterns, and you cansee 
definite trends developing. In fact, if you look at recent real charts ofthe 
market indices, you will see many instances of V-tops and V-bottoms, which is 
what William was concerned about earlier today. Hope this helps. Many thanks 
to Leo Timmermans for programming Chande&#8217;s scrambler for Amibroker.I asked 
him recently if he could program it in such a way to create synthetic data for 
an entire watch list at one time. That might be tricky, but he said he would 
look into it. 
 
Al V.
<BLOCKQUOTE 
>
----- Original Message ----- 
<DIV 
>From: 
<A href="" 
title=wpeters_1@xxxx>wpeters_1 
To: <A 
href="" 
title=amibroker@xxxxxxxxxxxxxxx>amibroker@xxxxxxxxxxxxxxx 
Sent: Wednesday, June 19, 2002 1:22 
PM
Subject: RE: [amibroker] Trends,random 
series, etc : was MetaStock and AmiBroker

<FONT color=#0000ff face=Arial 
size=2>AL,
<FONT color=#0000ff face=Arial 
size=2> 
<FONT color=#0000ff face=Arial 
size=2>With randomised data isn't there a chance that the so-called 'trend' 
can reverse on the next tick 'more-so' than data based on human 
intervention. And that includes mechnical systems (ie. if enough traders 
started using a certain indicator).
<FONT color=#0000ff face=Arial 
size=2>The obvious predictability in human behaviour is what can giveyou 
the little advantage and this is not in totally randomised 
data.
<FONT color=#0000ff face=Arial 
size=2> 
<FONT color=#0000ff face=Arial 
size=2>Surely if so your 'randomised' data 'trend' does not have an equal 
chance of reversing on each and every tick then its not 
randomised.
<FONT color=#0000ff face=Arial 
size=2> 
<FONT color=#0000ff face=Arial 
size=2>Very interesting topic.
 
<FONT color=#0000ff face=Arial 
size=2>William

<FONT face=Tahoma 
size=2>-----Original Message-----From: Al Venosa 
[mailto:avcinci@xxxx]Sent: Wednesday 19 June, 2002 10:54 
AMTo: amibroker@xxxxxxxxxxxxxxxSubject: Re: 
[amibroker] Trends, random series, etc : was MetaStock and 
AmiBroker


Richard,
I'm glad you took the liberty of changing the subject line. I don't 
think you are being a pessimistic, grumpy old man (sorry, don't know how 
old you are). Let me throw in another 2 cents into the discussion of 
trends. You said in order to have a trend, there must exist some 
information in the immediate past that would cause the trend to persist 
for a long enough time to profit from the move. I cannot agree more. That 
is absolutely true. A bad earnings report comes out on INTC, and the 
market immediately reacts by selling. This is further fueled by some 
nitwit analyst who changes his buy recommendation to hold or sell (of 
course, always after the fact!). Then, all the other individual 
"investors" climb aboard and do more selling, driving the prices still 
further down. Then, the company foretells that over the next several 
quarters there will be more of the same in regards to diminished sales, 
inducing further pessimism that drives the prices still further. All of 
this information results in a downward trend. This does not nor cannot 
happen with random events like coin tosses. So, technically speaking,you 
are perfectly correct in your assertions . 
Now, the trend follower comes in. He couldn't care less what is driving 
the prices down. In fact, he purposefully never pays attention to news 
events and announcements that may drive prices in one direction or 
another. He doesn't care about the cause of the trend. All he does is 
climb aboard the resulting trend, using whatever entry he has fashioned to 
enable him to take advantage of this 'breakout' and whatever exit that 
enables him to exit when the trend falters. Who was taking the other side 
of the market when Neeson was loading up on Nikkei futures, driving the 
Barings Bank into bankruptcy? The trend followers who recognized whatwas 
happening and capitalized. What I'm saying is, the trend follower buys or 
sells all breakouts (however you want to define the term 'breakout') and 
hopes that the resulting price behavior continues long into the future. 
With the scrambler, these 'trends' can indeed take place even with 
randomized data, as you have already pointed out. What I'm suggestingis 
that your system of entries and exits doesn't have a clue what causedthe 
breakout to occur because it's nothing more than a mathematical algorithm 
that trades based on certain signals programmed by the user when certain 
price behavior occurs. I contend you can test your system on such price 
behavior just to learn how well the system reacts to price breakouts and 
to see if, indeed, it is effective in finding trends in the data, albeit 
randomized data. By the way, these comments are not limited to trend 
following systems. I chose that trading approach merely as an exampleto 
make my point. 
Richard, I'm not trying to be argumentative on this topic. I'm just 
pointing out why I think there might be value in testing a fully optimized 
trading model on randomized data. Thanks for the provocative discussion. 

Al V.





>From: "Richard Alford" 
>Reply-To: amibroker@xxxxxxxxxxxxxxx 
>To: 
>Subject: [amibroker] Trends, random series, etc : was 
MetaStock and AmiBroker 
>Date: Wed, 19 Jun 2002 08:35:15 -0500 
> 
>I agree on the oxymoronic nature of so called random 
series. The problem results from using very particular distributions for 
the random numbers which allows the mathematicians to write books and 
teach classes. The Gaussian functions are one of the most well-behaved 
functions around. 
> 
>wrt trends in scrambler and other "random" time series- 
simply stated if they are truly random there is absolutely no information 
about the past in the next event. Your coin toss observation is a great 
example: given the 1/1000 case of 10 heads in a row the odds of another 
head are precisely 50% for the next toss. 
> 
>In order to discuss a "trend" there must be some 
information that persists. One can generally, perhaps always, see trends 
in a random series - in hindsight - however it contains no predictive 
information. There are systems that seek to identify trending vs. trading 
range behavior (I think Ehlers addresses this issue - can't say with how 
much success), however, they are inaccurate at the changes. A moving 
average on the coin toss series will identify trends - but the lag inthe 
moving average that we all know of and try to remove is the "hindsight" 
problem. 
> 
>Trend scale is still another issue. The apparent fractal 
behavior of the stock market suggests that one can find trends, and/or 
search for trends, over any time scale desired. (The inflationary or 
perhaps evolving nature of economies have placed a long term upward bias 
on the trends that can confuse the issue.) Once again a moving average of 
any length on a random series will display periods of upward and downward 
"trends". 
> 
>Sorry to be such a pessimistic old grump. 
> 
>Cordially, 
> 
>Richard 
> 
> ----- Original Message ----- 
> From: Avcinci 
> To: amibroker@xxxxxxxxxxxxxxx 
> Sent: Tuesday, June 18, 2002 8:19 PM 
> Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> 
> 
> Richard, 
> 
> >>distribution of the trends is predictable fora 
random series >> This sounds oxymoronic to me, i.e., predictable 
distribution of random trends. But, I'll take your word for it. :-)) 
> 
> Regarding the scale when speaking of trends, I was 
referring to weeks to months, perhaps even a year or two at most (stocks, 
not futures), but not decades. This brings us back to discussing the 
Scrambler. You said you don't hold much if any value in it, but can you 
elaborate just a little more on what you mean in terms of the scale Iam 
referring to, because in a period of 130 trading days, I noticed some 
pretty decent trends develop that lasted long enough to be of value in 
system testing? Thanks, Richard. By the way, I agree that this discussion, 
interesting as it is, has absolutely nothing to do with Metastock!! 
> 
> AV 
> 
> ----- Original Message ----- 
> From: Richard Alford 
> To: amibroker@xxxxxxxxxxxxxxx 
> Sent: Tuesday, June 18, 2002 3:55 PM 
> Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> 
> 
> wrt rsi/stochastics/cmo/etc... my point was that they 
attempt to identify a phenomenon that has been observed as an indication 
of overbought/oversold. There is a reasonable belief that the arithmetic 
created to identify that behavior can be a useful indictor of that 
condition. The comment offered for Elliot wave and Fibinacci behavioralso 
points to a large community that believes that there is a hidden structure 
to the market and human behavior - I am not a devotee of that belief,but 
I also have my doubts about Madam Cleo.... 
> 
> wrt trends in random numbers: there is, of course, a trend 
between any two non=equal numbers. Given a suitable sample, the 
distribution of the trends is predictable for a random series - probably 
also random and if the normal Gaussian (pardon the pun) distribution is 
used, as the derivative (or instantaneous slope/trend) is a skewed 
Gaussian. Interesting to the statisticians but not very useful to the 
average investor. 
> 
> wrt: trending 30% - that really has to be a matter of 
scale. If you chose a suitably long moving average you can probably even 
consider today as part of a long time upward trend - not particularly 
useful - although even my meager portfolio looks better today than itdid 
in 1970 - not saying much.... 
> 
> Interesting discussion - don't have the faintest ideawhat 
it has to do with MetaStock :) 
> 
> Cheers, 
> 
> Richard 
> 
> 
> ----- Original Message ----- 
> From: Al Venosa 
> To: amibroker@xxxxxxxxxxxxxxx 
> Sent: Tuesday, June 18, 2002 2:28 PM 
> Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> 
> 
> Thanks, Richard. I always enjoy your mathematical 
insights. I hope you didn't get the impression I was suggesting 
'optimizing' on random bars. I was merely suggesting 'testing' your 
already optimized system on the scrambled data, seeing if it can detect 
those occasional occurrences of non-random behavior of random numbers, 
that's all. Certainly if a trend-following system can detect trends in 
random price bars, it ought to be able to detect them in real, non-random 
price bars, too, don't you think? Markets trend only about 30% of the 
time, as I am told, so wouldn't you consider non-trending markets (i.e., 
sideways markets) somewhat representative of random price behavior? 
> 
> I also get the impression you are not an advocate of 
overbought/oversold oscillators, right? 
> 
> Al Venosa 
> avcinci@xxxx 
> >From: "Richard Alford" 
> >Reply-To: amibroker@xxxxxxxxxxxxxxx 
> >To: 
> >Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> >Date: Tue, 18 Jun 2002 12:55:33 -0500 
> > 
> >The non-random appearance of random numbers is a very 
well know phenomenon - probably accounts for the continued enthusiasmin 
slot machines in Vegas... 
> > 
> >I stand by my comments. In particular, the various 
oversold/bought indicators rely on the observation that the close tends to 
be higher in the l-h range in the overbought condition - the persistence 
of such a trend is interpreted as buy-sell behavior. 
> > 
> >I personally see no use in random data. There is no 
information in random data and no expectation of the past affecting the 
future - the only reason for technical analysis (other than too much time 
on one's hands) is to glean information about the near future. There are, 
of course, degenerate uses for random numbers in testing, but to test, 
build, or, heaven forbid - optimize, using random time series is pointless 
at best. 
> > 
> >wrt coin tosses: the coin is a Markov process of 
length 0 - no memory. If the market is the same, we may as well head for 
Vegas and get free drinks as we go broke. (OK - the market MAY be a fair 
game, but I doubt it.) 
> > 
> >Just some thoughts... 
> > 
> >Richard 
> > ----- Original Message ----- 
> > From: Al Venosa 
> > To: amibroker@xxxxxxxxxxxxxxx 
> > Sent: Tuesday, June 18, 2002 9:57 AM 
> > Subject: Re: [amibroker] Re: MetaStock and AmiBroker 
> > 
> > 
> > Well, Richard, to be honest, I haven't yet done any 
back (er, I mean forward) testing of the random time bars created by 
Scrambler. However, I have looked at the results of randomly scrambling 
the bars and watching them form on the screen, and you can see definite 
trends develop that can last several months, much like in real life. So, 
at least theoretically, it seems to me that, if you are testing a 
short-term or intermediate-term trend-following system, it ought to be 
able to pick up on those short, seemingly non-random trends (even though 
they were developed from a random number generator). Remember, if youflip 
an honest coin 1000 times, there is a low but finite probability thatit 
will come up heads 10 or 12 or 15 times in a row. 
> > 
> > AV 
> > 
> > 
> > >From: "Richard Alford" 
> > 
> > >Reply-To: amibroker@xxxxxxxxxxxxxxx 
> > >To: 
> > >Subject: Re: [amibroker] Re: MetaStock and 
AmiBroker 
> > >Date: Tue, 18 Jun 2002 09:48:35 -0500 
> > > 
> > >I understood that the underlying premise of 
technical analysis was that there IS information in the price-volume 
behavior. If you test against random data the results had better result in 
random behavior - perhaps mean behavior is more accurate. 
> > > 
> > >Perhaps I am delusional, however, I would 
appreciate insight into the value of random time bars? 
> > > 
> > >Cordially, 
> > >Richard 
> > > ----- Original Message ----- 
> > > From: Al Venosa 
> > > To: amibroker@xxxxxxxxxxxxxxx 
> > > Sent: Tuesday, June 18, 2002 9:23 AM 
> > > Subject: Re: [amibroker] Re: MetaStock and 
AmiBroker 
> > > 
> > > 
> > > Sorry, Dimitris, I thought you knew about it. Go 
to post 19331 and download it, following Leo Timmerman's instructions. 
What he has done is created a VBscript tool based on Tuchar Chande's price 
bar scrambler that scrambles all the bars of a given ticker from the 
previous x bars (bars and ticker set by the user) and then re-orders those 
bars randomly into the future, thus creating a ticker that you can use in 
forward-testing. If you set the no. of bars to, say, 130, you can generate 
6 months worth of new data for each ticker you do this on. Very, very 
cool. It differs from William's Simulator in that it actually generates 
new data rather than repeating existing data from the last x bars. Try it, 
you'll like it. 
> > > 
> > > Al Venosa 
> > > 
> > > >From: "dtsokakis" 
> > > 
> > > >Reply-To: amibroker@xxxxxxxxxxxxxxx 
> > > >To: amibroker@xxxxxxxxxxxxxxx 
> > > >Subject: [amibroker] Re: MetaStock and 
AmiBroker 
> > > >Date: Tue, 18 Jun 2002 13:49:30 -0000 
> > > > 
> > > >What is Leo's scrambler ?? 
> > > >DT 
> > > >--- In amibroker@xxxx, "Al Venosa" wrote: 
> > > > > 
> > > > 
> > > 
> > > 
> > 
>------------------------------------------------------------------------------ 

> > > Chat with friends online, try MSN Messenger: 
Click Here 
> > > 
> > > Your use of Yahoo! Groups is subject to the 
Yahoo! Terms of Service. 
> > > 
> > 
> > 
> 
>------------------------------------------------------------------------------ 

> > Get your FREE download of MSN Explorer at 
http://explorer.msn.com. 
> > 
> > Your use of Yahoo! Groups is subject to the Yahoo! 
Terms of Service. 
> > 
> 
> 
>-------------------------------------------------------------------------- 

> Join the world's largest e-mail service with MSN Hotmail. 
Click Here 
> 
> 
> Your use of Yahoo! Groups is subject to the Yahoo! Terms 
of Service. 
> 
> 
> 
> Your use of Yahoo! Groups is subject to the Yahoo! Terms 
of Service. 
> 
> 
> Yahoo! Groups Sponsor 
> ADVERTISEMENT 
> 
> 
> 
> 
> Your use of Yahoo! Groups is subject to the Yahoo! Terms 
of Service. 
> 


Send and receive Hotmail on your mobile device: <A 
href="">Click HereYouruse of 
Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of Service. 
Your use of Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of Service. 
Your use of Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of Service. 
Your use of Yahoo! Groups is subject to the <A 
href="">Yahoo! Terms of Service.