|
PureBytes Links
Trading Reference Links
|
I don't know because I don't know what you mean by "basic data scraping
techniques". Basically, I parse out a list in one of two ways. One, I just
copy the entire web page and paste it into Excel. Saving that as a .csv
file and reopening strips out all the formatting and pictures. If the
formatting of the list I am after is consistent, I can usually just delete
what I don't want and leave only the list. Or Two, I download and save the
pages HTML code and find coding consistencies to tell me where the data is
that I want and save that to file. I use this second method for the nightly
automated update of constituent lists. It took a while to program, but once
done, I can update over 140 index/ETF constituent lists in just a few
minutes without any input from me.
Usually, the .csv format provided lists on web sites are useless you are
doing things manually. Every site seems to have a "better way" and formats
are totally different. Hard as it is to believe, its easier just to use the
web page HTML.
Best,
L. P. Carhartt
MasterDATA
The Only Source for Index & ETF Composite/Breadth Reports, Charts & Data
<http://www.masterdata.com/> www.MasterDATA.com
<mailto:lc@xxxxxxxxxxxxxx> lc@xxxxxxxxxxxxxx
Important for Subscribers: Please be sure to set up your email program to
allow email from the following email addresses:
lc@xxxxxxxxxxxxxx
support@xxxxxxxxxxxxxx
subscriptions@xxxxxxxxxxxxxx
-----Original Message-----
From: equismetastock@xxxxxxxxxxxxxxx [mailto:equismetastock@xxxxxxxxxxxxxxx]
On Behalf Of rsb_44
Sent: Wednesday, January 30, 2008 9:50 AM
To: equismetastock@xxxxxxxxxxxxxxx
Subject: [EquisMetaStock Group] Re: Inserting Data in the Downloader Much
simpler version!!
PERFECT!
Thanks. I think I can apply this elsewhere. When you convert a web
page (table) to .csv are you using the basic data scraping techniques
from the Source page?
--- In equismetastock@ <mailto:equismetastock%40yahoogroups.com>
yahoogroups.com, "L. Carhartt" <lc@xxx> wrote:
>
> Sorry for the delay. I was making (and then eating) breakfast.
>
> I had the web page data you supplied basically converted to.csv
then started
> thinking that I knew there had to be a better way. Converting a
web page to
> .csv is not particularly difficult, but can be time consuming when
done
> manually. Most of the (index & ETF) constituent lists we gather
nightly are
> parsed out of web pages automatically so I am more than familiar
with the
> process.
>
> Anyway, try this link instead for an already put together .csv file:
>
> http://research. <http://research.stlouisfed.org/fred2/series/HOUST/>
stlouisfed.org/fred2/series/HOUST/
>
> Under the black headline, click on "Download Data". Scroll down
the new
> page and select "Text, comma separated" and then click on "Download
Data".
> Not only is this site THE original source for similar historic
data, but it
> is also free. The file provided is easily displayed in MetaStock
using the
> MetaStockCSV Plug-in or, if you are a glutton for punishment,
converted to
> MetaStock price format files and used that way.
>
>
> Best,
>
> L. P. Carhartt
>
>
> -----Original Message-----
> From: equismetastock@ <mailto:equismetastock%40yahoogroups.com>
yahoogroups.com
[mailto:equismetastock@ <mailto:equismetastock%40yahoogroups.com>
yahoogroups.com]
> On Behalf Of rsb_44
> Sent: Wednesday, January 30, 2008 7:45 AM
> To: equismetastock@ <mailto:equismetastock%40yahoogroups.com>
yahoogroups.com
> Subject: [EquisMetaStock Group] Re: Inserting Data in the
Downloader Much
> simpler version!!
>
>
>
> The page given
>
> http://www.economag <http://www.economag <http://www.economagic.com/em->
ic.com/em-
cgi/data.exe/feddal/hsm>
> ic.com/em-cgi/data.exe/feddal/hsm
>
> has the data in a table which is inconsistantly formatted.
> How does one get from the table to a .csv file?
>
> BTW Making the Lite version available for free is a real service to
> the Excel/Metastock community. Thank you.
>
> --- In equismetastock@ <mailto:equismetastock%40yahoogroups.com>
> yahoogroups.com, "L. Carhartt" <lc@> wrote:
> >
> > By the way, I should add, the plug-in will only plot lines and
> histograms.
> > For price charts (bar, candlestick charts, etc.) you do need to
use
> > MetaStock format price files.
> >
> > Best,
> >
> > L. P. Carhartt
> >
> >
> > -----Original Message-----
> > From: equismetastock@ <mailto:equismetastock%40yahoogroups.com>
> yahoogroups.com
> [mailto:equismetastock@ <mailto:equismetastock%40yahoogroups.com>
> yahoogroups.com]
> > On Behalf Of rsb_44
> > Sent: Wednesday, January 30, 2008 6:42 AM
> > To: equismetastock@ <mailto:equismetastock%40yahoogroups.com>
> yahoogroups.com
> > Subject: [EquisMetaStock Group] Re: Inserting Data in the
> Downloader Much
> > simpler version!!
> >
> >
> >
> > In Excel Pute the word DATE in A1
> > in A2 and below put the correct date in mm/dd/yyyy format
> > In B1 Put the word CLOSE
> > in B2 and below put the value for the starts. Beware of the
hidden
> > characters in the table.
> > Save the file as an .xls file
> > Convert the file with the Downloader. (It's help file says how)
> > Follow the rest of the instructions from the prior message.
> >
> > Sorry for the earlier message. It is actually a very simple way
> with
> > consistantly formatted data.
> >
> > --- In equismetastock@ <mailto:equismetastock%40yahoogroups.com>
> > yahoogroups.com, "rsb_44" <rsb_44@> wrote:
> > >
> > > This isn't a simple task as the data isn't consistantly
> formatted.
> > In
> > > fact copying it by hand doesn't work as some of the data is
> hidden.
> > >
> > > If you know Excel, the broad outline is:
> > > 1. Copy the data into Excel (The usual click , shift click copy
> > paste
> > > 2. Parse the data in Excel
> > > 3. Assemble a date
> > > 4. Convert the date to the format the Downloader accepts
> > > 5. Carry the actual starts to it's own column.
> > > 6. Lable the date column DATE and the Data Column CLOSE
> > > 7. Convert the data using the Downloader (You'll have to supply
a
> > > ticker ID)
> > > 8. Create an indicator MyClose (It's simply C, but used as an
> > > indicator.)
> > > 9. Open the ticker
> > > 10.Make the plot the same color as the background.
> > > 11. Apply the myclose indicator use the histogram style.
> > >
> > > You wind up with a chart showing housing starts a series of
> columns
> > > much like a volumn chart
> > >
> > > Is that what you want?
> > >
> > >
> > >
> > > --- In equismetastock@ <mailto:equismetastock%40yahoogroups.com>
> > yahoogroups.com, "Alex Spiroglou"
> > > <a.spiroglou@> wrote:
> > > >
> > > > I would appreciate your help on this please...
> > > >
> > > > I would like to chart "Housing Starts" in Metastock.
> > > > I found the data through Economagic
> > > > http://www.economag
> > <http://www.economag
> <http://www.economag
<http://www.economagic.com/em-cgi/data.exe/feddal/hsm>
ic.com/em-cgi/data.exe/feddal/hsm>
> ic.com/em-cgi/data.exe/feddal/hsm>
> > ic.com/em-cgi/data.exe/feddal/hsm
> > > >
> > > > How could I insert the data in the Downloader ?
> > > >
> > > >
> > > > thank you in advance
> > > > Alex Spiroglou
> > > >
> > > >
> > > > P.S. I would like to chart data from 2000 and on,
> > > > so I would not even mind to do it manually
> > > >
> > > >
> > > > [Non-text portions of this message have been removed]
> > > >
> > >
> >
> >
> >
> >
> >
> >
> >
> > [Non-text portions of this message have been removed]
> >
>
>
>
>
>
>
>
> [Non-text portions of this message have been removed]
>
[Non-text portions of this message have been removed]
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/equismetastock/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/equismetastock/join
(Yahoo! ID required)
<*> To change settings via email:
mailto:equismetastock-digest@xxxxxxxxxxxxxxx
mailto:equismetastock-fullfeatured@xxxxxxxxxxxxxxx
<*> To unsubscribe from this group, send an email to:
equismetastock-unsubscribe@xxxxxxxxxxxxxxx
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/
|