Composites: problem found - appears when running very large scans

["Tomasz Janeczko" <amibroker@xxxx>]

Hello,

There are three things that cause problems:

1. (The most important)
The bug in AmiBroker that exihibits only when working with large databases
and doing large scans.
Strictly speaking it appears only when number of symbol scanned is larger
than "in-memory cache" set in Tools->Preferences->Data.
Usually "in-memory cache" is set to few thousands so the problem does
not appear until your scan exceeds a few thousands of symbols.

What happens is that composite symbol may be flushed from "in-memory" cache
and then when AddToComposite tries to delete all data it "thinks" that
there are no quotes in composite (because it is flushed and cache is not filled again),
so in fact composite in that case is not reset to zero.

The problem will be fixed in a new maintenance release 4.20.5
that will appear today. 

2. (important if industry names /sector names are too long)

When building composite symbols from Sector/Market/Industry
name the string is truncated after 15th character because this is the maximum length
of ticker symbol used in AmiBroker.

Quotes Plus database has a few such sectors/industries that differ
only with characters past 15th.

For example there is
Aerospace-Defense
and
Aerospace-Defence Eqp+
industry in QP2 database.

When the AddToComposite is run first 15 characters are used
to create a symbol. In both cases it results in:
~Aerospace-Defe

This causes that data for symbols belonging to different categories
affect single composite that fools AddToComposite routine.

Similar effect was created by sector names starting with exactly the
same string as industry name. (In QP2 database industry names
have sector name prefixed so it happens quite often)

So, if your database uses long sector industry names you may consider
making them unique by the following code:

symI="~" + StrLeft( IndustryID(1), 11 ) + WriteVal( IndustryID(0), 1.0 ) ;
AddToComposite(O,symI,"O");
AddToComposite(H,symI,"H");
AddToComposite(L,symI,"L");
AddToComposite(C,symI,"C");
AddToComposite(V,symI,"V");

symM="~M_" + MarketID(1);
AddToComposite(O,symM,"O");
AddToComposite(H,symM,"H");
AddToComposite(L,symM,"L");
AddToComposite(C,symM,"C");
AddToComposite(V,symM,"V");

symS="~S_" + SectorID(1);
AddToComposite(O,symS,"O");
AddToComposite(H,symS,"H");
AddToComposite(L,symS,"L");
AddToComposite(C,symS,"C");
AddToComposite(V,symS,"V");


3. (this is not that important, but good to mention)

Using non-default setting for flags.
AddToComposite(O,symM,"O",flags=1);

Flags = 1 mean atcFlagResetValues = 1 - reset values at the beginning of scan (recommended)

But... an important flag (2) is missing:

atcFlagCompositeGroup = 2 - put composite ticker into group 253 and EXCLUDE all other tickers from group 253 (avoids adding
composite to composite)

Without this flag composite tickers are NOT excluded from adding to themselves which is wrong.

By default both atcFlagResetValues and atcFlagCompositeGroup are switched on
and you should NOT change the defaults unless you really know what you are doing.

So you should use:
AddToComposite(O,symM,"O"); // use DEFAULTS !!!

Best regards,
Tomasz Janeczko
amibroker.com