|
PureBytes Links
Trading Reference Links
|
Hello All,
I have just completed a few days of testing several modeling/backtesting programs. I thought, perhaps, the other members of the list might find the results useful; as I am new here, hopefully this can serve as my first productive contribution.
I do a lot of testing & modeling on (a) daily bars (looking to execute intraday trades) and (b) tick data (for short-term trades). In the case of the first, I need to create signals on a daily series but execute orders against a 1M intraday series. So, in order to test an idea over five years of data, I have to do it over 5Y of 1M data. This gets into performance issues when you start wanting to run a test on several issues and want to run multiple revisions of the test. As for testing on tick data, I am sure you are familiar (at least conceptually) with the performance issues there.
I took a look at NinjaTrader, TradeStation, OpenQuant & AmiBroker. I created an ASCII file of 5Y of EURUSD, USDJPY, GBPUSD, AUDUSD & USDCAD 1M data. In each program, I ran a simple EMA crossover test (10/50). It was an obnoxious test, resulting in +300,000 trades, but it was easy to implement and was a good stress test. What I wanted to do was see if I could get: a) reliable (e.g. reproducible) results from a single-security test and b) a test of the five FX pairs as a portfolio (again, in a reliable/reproducible manner).
1. Ninja: I use Ninja daily to scalp with and execute some short-term system code. I had dim hopes for the backtesting since I am familiar with the program - Ninja really is an execution platform first and an analytical platform second. The results were more or less what I expected: I could get it to test one issue with reproducible results at an okay speed (about 3:00M) but it would start to go into fits when I ran it against all five at once. The results of the five issue test would vary from instance to instance - it would usually show the results for the first 3 issues correctly, but on the last 2 it would suffer some kind of memory issue and give me numbers that were totally off. In one pass, it even managed to corrupt itself and I had to reload all the data.
2. TradeStation: I have a lot of time invested in TradeStation and I was already familar with it's problems - mainly, that over a large test set, TS will return different test results. I have talked with TS support and posted on the message board about this, but I never got anyone interested in what I found to be a critical issue. The results of this test were as expected: I could not get two results to match. Any time I would refresh data from the TS data servers and run the test again, I would get a different result. Sometimes it was as much as test 1 being -$190K and test 2 being +$74K. I do not understand how anyone can use this tool for successful modeling if they are testing over a large dataset; just making up a number would have been as useful. I even exported and imported the data to ensure that it wasn't an issue with the TS data servers. Same inconsistency. I couldn't test all five pairs together since TS does not do portfolio backtesting. As
for time of a single test, it is hard to tell with TS as to what, exactly, it is waiting for/trying to do at any given moment, but I would say it would usually take about 2/4M per test (although, I have no idea what it was doing in that 2/4M since the results it returned seemed random at times).
3. OpenQuant: This platform looked interesting. I set it up, imported the data and ran a test. After 20M I noticed that the first EURUSD 1M test was at 10%. I closed it down and uninstalled it. The performance was just not going to work for me.
4. AmiBroker: You can likely guess the results as I am here as a new member. AB was able to test EURUSD in about 30 seconds and was able to do the portfolio of all 5 pairs in about 2M. No matter how many times I ran the test, the numbers it would return were the same. This fact is pretty crucial to my work. I understand that data will have errors in it, but I at least need my modeling software to consistently return results (which AB did). I even created new databases and populated them with the same data - all test results returned were the same.
As you all know, AB was, by far, the cheapest option out of all of the above.
I hope some members find these results useful/interesting and I think that these results really are a credit to Tomasz.
I would be
interested to hear from anyone who has tested AB against any other
off-the-shelf tools that I did not look at.
- Tim
------------------------------------
**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.
TO GET TECHNICAL SUPPORT send an e-mail directly to
SUPPORT {at} amibroker.com
TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)
For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/amibroker/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/amibroker/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:amibroker-digest@xxxxxxxxxxxxxxx
mailto:amibroker-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
amibroker-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|