|
PureBytes Links
Trading Reference Links
|
Trying to answer that question is very difficult. There are many settings in the MS tester that can impact results.
You also did not account for survivorship bias.
The system tester in MS is very basic and very crude. There are better testers. TradeSim, Trading Blox, Trade Station, Apama, and Market System Analyzer are just a few that either do general back testing or special types of back tests. It all depends on what you can afford and how much time you want to put into it.
In addition, if you want to remove survivorship bias, then you'll need historically accurate data like Compustat, which is very expensive.
None of the testers when given exactly the same data, over exactly the same time frame, using the same trading algorithm yields results that are the same. Every tester approaches testing using a different set of computational algorithms. Some disclose what they are doing and others don't. I generally have to work through the data by hand to figure it out and reconcile any problems I find. Very time consuming!
Once a set of rational backtest results are generated, they are simply probabilities, so there is still the issue of the confidence interval and understanding how that impacts the range of trading results. Almost all backtest programs present backtest results as a set of parameters with fixed results. In reality, those parameters lie in a confidence interval which contains a range of values for that parameter which can be expected 90%, 95% or 99% of the time. The more precise the confidence interval the wider the range of parameter values. The future never exactly duplicates the past, thus the confidence interval.
Sometimes the confidence interval spread can be analyzed using Monte Carlo Simulation, but again the range of interpretations of what constitutes an accurate MCS test is so broad, and so poorly executed by most backtesting software, I wouldn't bet my money on the results.
In addition, live trading results never match backtest results. Even when systems are completely mechanical, the live results are different. The problem is: are they different enough to cause someone to lose money?
Backtest results are only good for RELATIVE comparisons only, not for precise projections of what someone is going to make using XYZ system.
I have used several testers, including TA, QA and FA testers, costing a couple of hundred bucks to well over $10K. Yippee!
I have also gotten test results from other professional traders that I trusted who had done a lot of testing but with different testers and a different source of data. Thomas Stridsman is a professional systems developer and tester. His books and articles have a lot of test results in them.
Then of course you could read Evidence Based Technical Analysis by David Aronson, if you really want to get an idea of what works.
If you read every backtest book written to date, you won't find many parameters that are consistent between the test descriptions, or that answer questions about how much data to use, how to address curve fitting, how much data is needed for walk forward tests, how many out of sample tests are adequate. or how survivorship bias is addressed, to name just a few things that make testing complex.
The more you test, the less likely you are to put much faith in the test results. As I said, test results are good for relative comparisons only.
Several people have written all kinds of analysis of the MS tester. Some of the problem descriptions are in the files section of this group.
Have fun!
Super
--- In equismetastock@xxxxxxxxxxxxxxx, "witness52" <witness52@xxx> wrote:
>
> I did a long term system test on the Nasdaq 100 index from 1-5-1990 to 7-2-2009.
>
> According to the system tester the "buy and hold" results were 492% or 25% annaulized.
>
> In January 1990 the .NDX was at 201. On 7-2-09 it was at 1446.
>
> 1446/201 = 619 percent return.
>
> That is 10.33 percent per year compounded annually.
>
> Is anyone else seeing this kind of bad output from the system tester?
>
> I also notice that the tester does not execute when it should.
>
> Any thoughts? Can this system test be relied upon?
>
------------------------------------
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/equismetastock/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/equismetastock/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:equismetastock-digest@xxxxxxxxxxxxxxx
mailto:equismetastock-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
equismetastock-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|