Re: [amibroker] AA Optimization: Stable Plateaus vs Unstable Peaks - and Light Tables!, AmiBroker Email List Archive

As far as the rest of what you describe which I termed ... "Sensitivity Directed Optimization" ... and I described as ... "Perform a sensitivity analysis of parameter values and utilize parameter sensitivity as a means of influencing and directing intelligent optimization thereby increasing system robustness" ... I suggest you read pages 36-41 of the IO doc and see the directives that precede this ... This will give you some insight into the complexity of the problem even without dealing with poteentially diverse solutions from multiple optimization runs.

----- Original Message -----
From: Keith McCombs
Date: Sunday, April 19, 2009 3:20 am
Subject: Re: [amibroker] AA Optimization: Stable Plateaus vs Unstable Peaks - and Light Tables!
To: amibroker@xxxxxxxxxxxxxxx

> Ozzy --
> I'm not sure I understand what you mean by "I then flatten the
> 3D
> optimization graphs to 2D for each month, and print out the six
> charts
> on thin sheets of paper. I stack the papers one on top of the
> other and
> place them on a makeshift light table. This essentially gives me
> superimposed Venn diagrams and it is easy to circle the best
> common area
> for the six months."
>
> But let me make a stab at it and you let me know how close I am.
> Say for example, you optimize over two parameters as in:
> small = Optimize("small", 4, 1, 10, 1);
> big = Optimize("big", 40, 10, 100, 10);
> And you optimize for CAR/MDD.
>
> Then, for each month, you produce a single chart with 10 lines,
> CAR/MDD
> vs 'small' for each of the 'big' values. You then overlay the 6
> months
> on one another. You would have the same effect, without
> overlaying, if
> you plotted sixty lines for each of the 'big' values. You might
> plot
> all of the lines that use the same 'big' parameter the same
> color,
> independent of what month they are from.
>
> You may be able to automate much this. And then use your
> 'eyeball' for
> selecting the most promising areas?
>
> ------------------------------
>
> If you want to do less 'eyeballing', you might want to invent
> some
> metrics that might be based on the average value and standard
> error of
> the CAR/MDD for each combination of parameters over the 6
> months.
> And/or you could also 'punish' each pair of parameters based on
> the
> value of their nearest neighbors. For example: each CAR/MD for
> a pair
> of small and big would be represented as CARMD[small, big].
> CARMD[4,40]
> could have a new 'punished' value PCARMD[4,40] which would be
> reduced or
> pulled down by its nearest neighbors in both small and big
> directions,
> but NOT increased or pulled up by them. If you took this
> approach, you
> would be wise to compare your results with your 'eyeball'
> evaluations so
> as to increase, or reduce, confidence in either or both these
> attempts
> at evaluation.
>
> BTW, Fred Tonetti's IO has some provision for introducing
> parameter
> 'sensitivity' into the optimization system (though I don't
> understand
> the nitty-gritty of it).
>
> BTW2, I really appreciate your posting. New thoughts --
> definitely
> worth thinking.
> -- Keith
>
>
>
>
>
> ozzyapeman wrote:
> >
> >
> > I am wondering if anyone here has already created a CBT
> procedure that
> > solves this problem. And if not, wondering if anyone actually
> would
> > volunteer to create one. It's a bit beyond my scope at the
> present time.
> >
> > When one does an In-Sample / Out-Of-Sample walk forward test
> in AB's
> > AA, the AA engine will use the "optimum" variables from the IS
> period
> > to use on the OOS period. But more often than not, these
> values are
> > not really stable.
> >
> > In a two-variable 3D optimization graph, for example, the
> 'best'
> > values are usually on some sharp peak somewhere. If you use
> these
> > values next month (i.e. for the OOS period), they can easily
> fall of
> > that peak and result in a big loss. One would have been better
> off
> > choosing values that were on a more stable plateau, even if
> they
> > represented a significantly lower CAR/MDD or Net Profit or
> whatever
> > value you are optimizing against.
> >
> > *19th Century Light-Table Method*
> >
> > What I have currently found useful is to optimize my system
> over six
> > individual months. I then flatten the 3D optimization graphs
> to 2D for
> > each month, and print out the six charts on thin sheets of
> paper. I
> > stack the papers one on top of the other and place them on a
> makeshift
> > light table. This essentially gives me superimposed Venn
> diagrams and
> > it is easy to circle the best common area for the six months.
> I then
> > choose the center of that common area, and get my X and Y
> coordinates
> > which corresponds to values of Profit and Stop (e.g. 30/100).
> When I
> > test these values on the OOS 7th month, I always end up with a
> > significant profit. I've done a lot of laborious runs like
> this, and
> > it always works.
> >
> > Now of course my mechanical method of printing out graphs and
> placing
> > them over a light is very primitive (perhaps shamefully so).
> In order
> > to truly validate this, I essentially need to program this
> method so I
> > can use it for computerized statistical validation over my
> historical
> > data.
> >
> > I essentially wa nt to be able to automatically:
> >
> > 1. Optimize my system on six individual months (not a single
> > six-month period)
> > 2. Find the the stable plateaus for Profit/Loss for each month,
> > that correspond to a minimum acceptable optimization level
> > 3. Compare the six stable plateaus to find the "Common Plateau"
> > 4. Find the center of the six-month Common Plateau, to
> obtain best
> > Profit/Loss values
> > 5. Walk-Forward to the OOS month (i.e. the 7th month) and
> backtest> with the Profit/Loss values found in step 4
> > 6. Repeat steps 1-5 for as many steps in my Walk-Forward
> analysis.>
> >
> > Of course the above is easier said than done. Or is it? Maybe
> there is
> > some simple way to fandangle this, that I have not thought of.
> >
> > Any suggestions on how to approach this? Or better yet, anyone
> feeling
> > up to the challenge of actually slapping it together as a
> CBT/Jscript
> > piece of coding brilliance?
> >
> > In its simplest form, it really only need to do steps 1-4
> > above,perhaps callable by a jscript. It can then easily be
> used with
> > AB's WF capability.
> >
>