|
PureBytes Links
Trading Reference Links
|
Chuck,
Related to your comments to Fred:
The software has (if I remember correctly) over 100 objective functions by which out-of-sample results can be measured.
Could you elaborate a bit on what you mean by objective functions. I think of MAR, CAR, K-Ratio, UPI, etc. when I read your comments above, and wanted to make sure I'm on track.
Unfortunately, a humanoid has to decide which objective function to use and which results look "best". This is where (IMO) the problem arises. While the fund is still making nice profits for its clients, it could be doing a lot better with the right person driving the research software.
"Best" in terms of looking for the smoothest curve in sensitivity analysis? Clearly you are alluding to the importance of experience in making this decision. However, you've also alluded in past posts to the fact that less experience (i.e. myself) can get a good solution by looking for the smooth curves, and not getting overly complex.
I believe that parameters are chosen based on too short of learning period. More importantly, too few trades are used when determing parameters going forward.
What do you think is a reasonable number of trades for a good sample size.
Thanks in advance,
Gary
Chuck Rademacher <chuck_rademacher@xxxxxxxxxx> wrote:
Fred,
I couldn't help but reply to your question below.
IMO, the software that Howard and I used when we worked together was "capable" of doing the automated task of optimizing and selecting parameters to trade going forward. It was (and is) being used by a fairly successful futures fund in Denver. This particular fund has been ranked in the top-ten based on several different criteria over its 15 years of operation. I still work for the company, but Howard has moved on to do other things.
Before addressing the issue you raised, I have to tell you a bit about this software. It was written by a fella' by the name of Jim Yonan, with a lot of input from Howard and others working for this particular company. To me, the most interesting thing about it was the way it could distribute processing over an entire network of PC's. We have about 20 PC's in the office and some in the homes of staff members. I have 13 PC's in my home here in New Zealand. The software is capable of sending out small batches of optimization runs to any number of PC's and integrating the results as each batch finishes. It's truly amazing to watch!
The software has (if I remember correctly) over 100 objective functions by which out-of-sample results can be measured. Howard actually created most of these functions and they are excellent. Unfortunately, a humanoid has to decide which objective function to use and which results look "best". This is where (IMO) the problem arises. While the fund is still making nice profits for its clients, it could be doing a lot better with the right person driving the research software. I believe that parameters are chosen based on too short of learning period. More importantly, too few trades are used when determing parameters going forward.
So, I believe the answer to your question is that such software exists. It has been sold to other fund managers, but the price tag is over $1 million so it isn't going to help the average punter. But the concept can be fairly easily re-created by a couple of good programmers in a month or two.
BTW, this particular futures fund is 100% mechancal. The trading software monitors every tick in 47 different futures markets and automatically generates orders. Orders are either printed out for a dealer to call into the floor or, in some cases, go straight to the floor electronically.
The manager of the fund is about to launch a U.S. based stock hedge fund using similar concepts.
-----Original Message-----From: Fred [mailto:fctonetti@xxxxxxxxx]Sent: Friday, October 17, 2003 11:01 AMTo: amibroker@xxxxxxxxxxxxxxxSubject: [amibroker] Re: Optimization -- againWalk forward testing and optimization is a great concept in theory and although I've seen lots of ideas for how to set it up and be used over the years, unfortunately I've yet to see anyone demonstrate that it actually works over a variety of market conditions. There are I'm sure lots of reasons for this which I won't delve into here but the question remains, has anyone actually seen this put into practice where the result has been a viable system to use in mechanical trading ? If so can you please point at something that could be looked at objectively that doesn't reside in a black box ?--- In amibroker@xxxxxxxxxxxxxxx,
"Howard Bandy" <howardbandy@xxxx> wrote:> Greetings --> > In my opinion, anything we do in development of trading systems involves a> search for a pattern than precedes a profitable trading opportunity. Any> time we examine the results of alternative systems, we are involved in> searching; and when we select the most promising of those alternatives, we> are optimizing. Only a system based on truly random entries and exits would> not be the result some optimization. So the question of "should we> optimize?" is moot -- we have no choice but to optimize. Consequently, we> should be aware of our optimization techniques.> > Chuck referred to an optimization technique recommendation I made to the> company we both worked for in Denver a few years ago. This is a short> description of it.> > The company is a
Commodity Trading Advisor which traded futures, not> individual stocks, but the procedures are equally valid for both.> > When I joined the company, they were using very long data series when> developing their models. They used a technique sometimes called folding or> jackknifing, where the data was divided into several periods -- say ten.> The modeling process made ten passes. During each pass, one period was held> back to be used as out-of-sample data, the other nine were used to select> the best parameter values. After all ten passes, the results were gathered> together and the parameter values that scored best overall were chosen.> There are several problems with this method. One is the difficulty with the> "ramp up" period at the start of each segment, another is that it is not> valid to use older data for out-of-sample testing than was
used for> in-sample development, and another is that the data series were too long.> Chuck and I and others had many interesting discussions about how long the> in-sample data should be. > > My background is strong in both the theory and the practice of modeling and> simulation, and includes a great deal of experience with analysis of> financial time series. I proposed the following method, which I continue to> believe is valid.> > First, before any modeling begins. Using judgment of management and> comparison of trading profiles of many trading runs (real, simulated, or> imagined), pick an objective function by which the "goodness" of a trading> system will be measured. This is important, it is a personal or corporate> judgment, and it should not be subject to optimization. > > Divide each data series into a
sequence of in-sample and out-of-sample> periods. The length of the out-of-sample period is the "reoptimization"> period. Say there are about ten years of historical data available> (1/1/1993 through 1/1/2003. Set the in-sample period to two years and the> out-of-sample period to one year. Run the following sequence: Search /> optimize using 1993 and 1994; pick the "best" model for 1993-1994; forward> test this model for 1995 and save the results; step forward one> reoptimization period and repeat until all the full in-sample periods have> been used. The final optimization will have been 2001 and 2002, with no> out-of-sample data to test. Ignore all in-sample results!! Examine the> concatenated out-of-sample equity curve. If it is acceptable, you have some> confidence that the parameters select by the final
optimization (2001 and> 2002) will be profitable for 2003. No guarantees -- only some confidence.> > How did I pick two years for in-sample and one year for out-of-sample? That> was just an example. The method is to set up an automated search where the> length of the in-sample period and the length of the out-of-sample period --> the reoptimization period -- are variables, and then search through that> space. > > Trading systems work because they identify inefficiencies in markets. Every> profitable trade reduces the inefficiency until, finally, the trading system> cannot overcome the frictional forces of commission and slippage. This is> the same phenomenon that physicists talk about as entropy.> > My feeling -- and it may be different than Chuck's -- is that the market is> not only non-stationary, but
that the probability that it will return to a> previous state is near zero. > > Being non-stationary means that market conditions change with respect to our> trading systems. If I am modeling a physical process, such as a chemical> reaction, I can count on a predictable modelable output for a given set of> inputs. If I am modeling a financial time series, the output following a> given set of inputs changes over time. If a market were stationary with> respect to an RSI oscillator system, I could always buy a rise of the RSI> through the 20 percent line, to use a very simplistic example. > > I feel that the introduction of microcomputers, trading system development> software, inexpensive individual brokerage accounts, and discussion groups> such as this one have permanently changed the realm of trading. One,> everyone
who is interested can afford to buy a computer, run AmiBroker, and> design and test trading systems. Two, if someone develops a profitable> system and trades it, the profits it takes reduce the potential profits> available to anyone else who trades it. Consequently, the characteristics> of the market change in a way that moves the market away from that model> until that trading system is no longer profitable enough to overcome> commission and slippage. Three, a new person beginning to study trading> system development typically tests a lot of old systems. If one is found to> be profitable and they start trading it, the market moves back to being> efficient. Consequently, trading systems that used to work, but no longer> work, are very unlikely to ever work again.> > So, I feel that the in-sample period should be short so that the
market> conditions do not change much over that period. That is, I am looking for a> data series that is stationary relative to my model. The stationary> relationship must extend beyond the in-sample period far enough that the> model will be profitable when used for trading in the out-of-sample data.> The length of the extension determines the reoptimization period. It could> be years, months, or even one day. Note that the holding period of a> typical trade is very much related to the length of both the in-sample and> out-of-sample periods. The typical trade should be much shorter than the> in-sample period and somewhat shorter than the out-of-sample period.> > The important point in all this is that the only results being analyzed are> the concatenated out-of-sample trades.> > As with all model development, every time I
look at the out-of-sample> results in any way, I reduce the probability that future trading results> will be profitable. That means that I should not perform thousands of tests> of model parameters, in-sample periods, and out-of-sample periods, on the> same data series and then pick the best model base on my examination of> thousands of out-of-sample results. In effect, I will have just converted> all those out-of-sample results into in-sample data for another step in the> development. That is legitimate, just be aware of what is happening.> > Thanks for listening,> HowardSend BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to suggest@xxxxxxxxxxxxx-----------------------------------------Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: <A
href="">http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. Send BUG REPORTS to bugs@xxxxxxxxxxxxxSend SUGGESTIONS to suggest@xxxxxxxxxxxxx-----------------------------------------Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx (Web page: http://groups.yahoo.com/group/amiquote/messages/)--------------------------------------------Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
Do you Yahoo!?
The New Yahoo! Shopping - with improved product search
Yahoo! Groups Sponsor
ADVERTISEMENT
Send BUG REPORTS to bugs@xxxxxxxxxxxxx
Send SUGGESTIONS to suggest@xxxxxxxxxxxxx
-----------------------------------------
Post AmiQuote-related messages ONLY to: amiquote@xxxxxxxxxxxxxxx
(Web page: http://groups.yahoo.com/group/amiquote/messages/)
--------------------------------------------
Check group FAQ at: http://groups.yahoo.com/group/amibroker/files/groupfaq.html
Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
|