|
PureBytes Links
Trading Reference Links
|
Great stuff Howard. Much clearer and more practical than any systems development book I've ever read.
Keep brining it.
Kind Regards,
GaryHoward Bandy <howardbandy@xxxxxxxxx> wrote:
Hi Jitu –
Yes, a number.
An optimization is a search for a ˇČbestˇÉ model. Given one afl trading system, say a simple moving average crossover system, the ˇČbestˇÉ model is the combination of moving average lengths that gives the buy and sell signals that gives ˇČbestˇÉ results. Best to you may be quite different than best to some other trader. You might be trading an account that cannot stand deep drawdowns, so you might select a model with a smaller total return than the other person. You might look at K-Ratio, while some other person looks at RAR. Whatever you look at, it is called the ˇČobjective functionˇÉ and it is used to rank the goodness of a group of possible models. If the model development process is to be automatic, you must be comfortable accepting the model that appears at the top of a single column. If there are several criteria that
are important to you, you can develop your own composite objective function. It might be 40% times RAR plus 60% times K-Ratio. Whatever it is, it must reduce to a single, scalar value.
I have already asked Tomasz to expose the metrics reported after an optimization, such as Ulcer Performance Index, for our use within afl code. And also to allow us to add additional fields to the optimization report, so that we can create and report our own custom objective functions. Maybe in a future release of AmiBroker.
My point in the post is that everyone should decide on his or her own measure of goodness before spending a lot of time running the optimizer. We all need to recognize a good model when one appears. If a truly automatic model development process is developed, the person will not see the alternative models and will not make any judgment among them. Let me say that another way. If I look at the list of alternative models sorted according to my objective function and decide, for some reason, that I prefer one other that the one at the top of the list, then I have not included something important in the definition of my objective function. I need to revise my objective function, and rerun earlier test runs and model selections.
I recommend producing many sets of trade results and equity curves, computing statistics for them, and plotting their equity curves. Quantify, based on your judgment and trading requirements, which models are best; and see what metrics they have. Determine your own, personal objective function and paste it on the wall. These do not have to come from real trading systems – you can just make them up in Excel, import the signals into AmiBroker, and let AmiBroker compute its reports.
Now, whenever a series of tests is run, you know exactly which is ˇČbestˇÉ.
Note – it is perfectly valid to have different objective functions for different purposes. For example, I might be modeling the behavior of a sector, say oil services, with the intent of trading individual stocks based on what I learn. In this case, I want to identify periods of rising prices with careful attention to turning points, but without much interest in overall profit. On the other hand, I might be modeling individual high beta tech stocks, in which case my model includes several stop loss techniques and I care most about avoiding drawdowns.
It might be interesting to hear what AmiBroker users feel is important to them, and why, and what the resulting objective function is.
Thanks,
Howard
-----Original Message-----From: jtelang [mailto:jtelang@xxxxxxxxx] Sent: Thursday, October 16, 2003 12:31 PMTo: amibroker@xxxxxxxxxxxxxxxSubject: [amibroker] Re: Optimization -- again
Very informative, especially the last paragraph, which is what I feel most people, especially new traders such as myself, end up doing unknowingly.Could you please elaborate further on following? Do you mean a number here or something else?<FONT face="Courier New"
size=2>> First, before any modeling begins. Using judgment of management and> comparison of trading profiles of many trading runs (real, simulated, or> imagined), pick an objective function by which the "goodness" of a trading<SPAN
>> system will be measured. This is important, it is a personal or corporate> judgment, and it should not be subject to optimization. Thanks.Jitu--- In amibroker@xxxxxxxxxxxxxxx, "Howard Bandy" <howardbandy@xxxx> wrote:> Greetings --> > In my opinion, anything we do in development of trading systems involves
a> search for a pattern than precedes a profitable trading opportunity. Any> time we examine the results of alternative systems, we are involved in> searching; and when we select the most promising of those <FONT
face="Courier New" size=2>alternatives, we> are optimizing. Only a system based on truly random entries and exits would> not be the result some optimization. So the question of "should we> optimize?" is moot -- we have no choice but to optimize. <FONT
face="Courier New" size=2>Consequently, we> should be aware of our optimization techniques.> > Chuck referred to an optimization technique recommendation I made to the> company we both worked for in Denver a few years ago. This is a short> description of it.> > The company is a Commodity Trading Advisor which traded futures, not> individual
stocks, but the procedures are equally valid for both.> > When I joined the company, they were using very long data series when> developing their models. They used a technique sometimes called folding or<FONT face="Courier
New" size=2>> jackknifing, where the data was divided into several periods -- say ten.> The modeling process made ten passes. During each pass, one period was held> back to be used as out-of-sample data, the other nine were used to <SPAN
>select> the best parameter values. After all ten passes, the results were gathered> together and the parameter values that scored best overall were chosen.<FONT
face="Courier New">> There are several problems with this method. One is the difficulty with the> "ramp up" period at the start of each segment, another is that it is not> valid to use older data for out-of-sample testing than was used for<FONT face="Courier
New">> in-sample development, and another is that the data series were too long.> Chuck and I and others had many interesting discussions about how long the> in-sample data should be. > <FONT face="Courier New"
size=2>> My background is strong in both the theory and the practice of modeling and> simulation, and includes a great deal of experience with analysis of> financial time series. I proposed the following method, which I continue to<SPAN
>> believe is valid.> > First, before any modeling begins. Using judgment of management and> comparison of trading profiles of many trading runs (real, simulated, or> imagined), pick an objective function by which the "goodness" of a trading> system will be measured. This is important, it is a personal or corporate> judgment, and it should not be subject to optimization. > > Divide each data series into a sequence of in-sample and out-of-sample> periods. The length of the out-of-sample period is the "reoptimization"> period. Say there are
about ten years of historical data available> (1/1/1993 through 1/1/2003. Set the in-sample period to two years and the> out-of-sample period to one year. Run the following sequence: Search /> optimize using 1993 and 1994;
pick the "best" model for 1993-1994; forward> test this model for 1995 and save the results; step forward one> reoptimization period and repeat until all the full in-sample periods have> been used. The final optimization will have been
2001 and 2002, with no> out-of-sample data to test. Ignore all in-sample results!! Examine the> concatenated out-of-sample equity curve. If it is acceptable, you have some<SPAN
>> confidence that the parameters select by the final optimization (2001 and> 2002) will be profitable for 2003. No guarantees -- only some confidence.> <FONT
face="Courier New">> How did I pick two years for in-sample and one year for out-of-sample? That> was just an example. The method is to set up an automated search where the> length of the in-sample period and the length of the out-of-sample <FONT
face="Courier New">period --> the reoptimization period -- are variables, and then search through that> space. > > Trading systems work because they identify inefficiencies in <SPAN
>markets. Every> profitable trade reduces the inefficiency until, finally, the trading system> cannot overcome the frictional forces of commission and slippage. This is> the same phenomenon that physicists talk about as entropy.> > My feeling -- and it may be different than Chuck's -- is that the market is> not only non-stationary, but that the probability that it will <FONT
face="Courier New">return to a> previous state is near zero. > > Being non-stationary means that market conditions change with respect to our> trading systems. If I am modeling a physical process, such as a <FONT
face="Courier New" size=2>chemical> reaction, I can count on a predictable modelable output for a given set of> inputs. If I am modeling a financial time series, the output following a> given set of inputs changes over time. If a market were stationary with> respect to an RSI oscillator system, I could always buy a rise of the RSI> through the 20 percent line, to use a very simplistic example. > > I feel that the introduction of microcomputers, trading system development> software, inexpensive individual brokerage accounts, and discussion groups> such as this one have
permanently changed the realm of trading. One,> everyone who is interested can afford to buy a computer, run AmiBroker, and> design and test trading systems. Two, if someone develops a profitable<FONT face="Courier New"
size=2>> system and trades it, the profits it takes reduce the potential profits> available to anyone else who trades it. Consequently, the characteristics> of the market change in a way that moves the market away from that <SPAN
>model> until that trading system is no longer profitable enough to overcome> commission and slippage. Three, a new person beginning to study trading> system development typically tests a lot of old systems. If one is <SPAN
>found to> be profitable and they start trading it, the market moves back to being> efficient. Consequently, trading systems that used to work, but no longer<FONT
face="Courier New">> work, are very unlikely to ever work again.> > So, I feel that the in-sample period should be short so that the market> conditions do not change much over that period. That is, I am looking for a<FONT
face="Courier New" size=2>> data series that is stationary relative to my model. The stationary> relationship must extend beyond the in-sample period far enough that the> model will be profitable when used for trading in the out-of-sample data.<FONT face="Courier New"
size=2>> The length of the extension determines the reoptimization period. It could> be years, months, or even one day. Note that the holding period of a> typical trade is very much related to the length of both the in-<SPAN
>sample and> out-of-sample periods. The typical trade should be much shorter than the> in-sample period and somewhat shorter than the out-of-sample period.> <FONT
face="Courier New">> The important point in all this is that the only results being analyzed are> the concatenated out-of-sample trades.> > As with all model development, every time I look at the out-of-sample<FONT face="Courier
New" size=2>> results in any way, I reduce the probability that future trading results> will be profitable. That means that I should not perform thousands of tests> of model parameters, in-sample periods, and out-of-sample periods, <SPAN
>on the> same data series and then pick the best model base on my examination of> thousands of out-of-sample results. In effect, I will have just converted<FONT
face="Courier New">> all those out-of-sample results into in-sample data for another step in the> development. That is legitimate, just be aware of what is happening.> > Thanks for listening,<FONT face="Courier New"
size=2>> Howard
Send BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to suggest@xxxxxxxxxxxxx-----------------------------------------Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
Send BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to suggest@xxxxxxxxxxxxx-----------------------------------------Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|