|
PureBytes Links
Trading Reference Links
|
On Sat, 2004-05-15 at 01:13, Sven Napolean Montessori wrote:
> With a little perl, I was easily downloading 10 in parallel, thus 701
> seconds for your example. Checking the bandwidth later, I could have
> easily have been downloading 100 in parallel or 71 seconds.
Again FYI and sorry if I am bringing something up already covered:
The following URl will bring back, in CSV format, up to some 200 symbols
at a time. You'll get unpredictable results if you try more. I grab
"chunks" of 190 per http request and shove them all into file locally
until all are back and then process them further.
http://quote.yahoo.com/d/quotes.csv?s=IBM%2BQCOM&e=.csv&f=snxohgl1vd1c1a2jkt8dyr1qj1b4p6erp9j4s6s7ij2
"IBM","INTL BUS MACHINE","NYSE",86.65,87.35,86.07,86.41,5197800,"5/14/2004",-0.78,5083727,78.73,100.43,108.00,0.72,0.83,"Jun 10","May 6","145.6B",16.723,5.21,4.48,19.46,1.61,"15.191B","91.316B",2.31,"cnsprmiIed", 1,685,370,000
"QCOM","QUALCOMM INC","NasdaqNM",64.60,65.19,63.55,63.65,8169416,"5/14/2004",-0.82,9160772,29.58,69.38,69.94,0.40,0.62,"Jun 25","May 26","51.569B",10.578,6.09,1.38,46.72,12.28,"1.785B","4.253B",2.51,"cnsprmiIed", 810,193,000
This is the function that builds the query string:
def make_current_url (symbols):
""" symbols : list of strings (symbol_yahoo)
symbols_to_retrieve is one of:
a single "IBM", a string list "IBM,YHOO", or a list ["IBM,"YHOO"]
See YAHOO_API.doc for details on format
"""
base_url = 'http://quote.yahoo.com/d/quotes.csv'
symbols = ','.join(symbols)
query = {'f':'snxohgl1vd1d3c1a2jkt8dyr1qj1b4p6erp9j4s6s7ij2','s':symbols,'e':'.csv'}
url = base_url + '?' + urllib.urlencode(query)
return url
Here's the function that gets the data. symbollist is a list data type
containing up to 195 or so symbols ['IBM','QCOM','XIU.TO'] etc.
def get_current_chunk(symbollist):
"""actually grabs the data
"""
url = make_current_url(symbollist)
print url
buffer = []
reader = csv.reader(urllib.urlopen(url))
for r in reader:
print r
if str(r[6]) == '0.0' or r[6] == 0.00: # can't be real or is disused...
continue
buffer.append(r)
return buffer
Deciphering what the parameter values return takes some time and
sleuthing, Yahoo does not document them. Some of the perl and python
projects that went before me were downloading some of the values; I
added to this list by manually trying as many parameter combinations as
possible.
The following function snippet assigns values from the returned data to
variables. It won't do anything for you but serves as part documentation
of what the Yahoo query parameters return. One of these days I'll finish
documenting them, in the meantime:
def store_current(symboldata):
"""Do any conversions required and shove this stuff in the database
"""
from quoteserver.objects import Symbol
#~ http://quote.yahoo.com/d/quotes.csv?s=IBM%2CQCOM&e=.csv&f=snxohgl1vd1d3c1a2jkt8dyr1qj1b4p6erp9j4s6s7ij2
#~ raw back from yahoo
#~ "IBM","INTL BUS MACHINE","NYSE",98.45,99.97,98.41,99.61,4057500,"2/10/2004",+0.66,5544363,73.17,100.43,106.67,0.64,0.65,"Mar 10","Feb 6","171.4B",16.309,6.07,4.34,22.80,1.91,"10.089B","89.131B",3.24,"cnsprmiIed", 1,720,420,000
#~ "QCOM","QUALCOMM INC","NasdaqNM",57.05,58.25,57.01,58.09,5257929,"2/10/2004",+0.93,8603863,29.58,60.73,57.66,0.28,0.49,"Mar 26","Feb 25","46.769B",9.88,5.79,1.14,50.14,11.18,"1.383B","4.116B",3.03,"cnsprmiIed", 805,112,000
#~ as supplied to this function
#~ ['IBM', 'INTL BUS MACHINE', 'NYSE', '86.65', '87.35', '86.07', '86.41', '5197800', '5/14/2004', 'May 14', '-0.78', '5083727', '78.73', '100.43', '108.00', '0.72', '0.83', 'Jun 10', 'May 6', '145.6B', '16.723', '5.21', '4.48', '19.46', '1.61', '15.191B', '91.316B', '2.31', 'cnsprmiIed', ' 1', '685', '370', '000']
#~ ['QCOM', 'QUALCOMM INC', 'NasdaqNM', '64.60', '65.19', '63.55', '63.65', '8169416', '5/14/2004', 'May 14', '-0.82', '9160772', '29.58', '69.38', '69.94', '0.40', '0.62', 'Jun 25', 'May 26', '51.569B', '10.578', '6.09', '1.38', '46.72', '12.28', '1.785B', '4.253B', '2.51', 'cnsprmiIed', ' 810', '193', '000']
count = 0
for d in symboldata:
#~ try:
count += 1
symbol = str(d[0]).upper()
try:
s = Symbol.select(Symbol.q.symbol_yahoo == symbol )[0]
except:
print "not found", symbol
assert(s.symbol_yahoo is not None)
s.set(
# set name seperately - see below
# trade data
last_open = make_float(d[3]) # 98.45,
, last_high = make_float(d[4]) # 99.97,
, last_low = make_float(d[5]) # 98.41,
, last_close = make_float(d[6]) # 99.61,
, last_volume = make_number(d[7]) # 4057500,
, last_datetime = make_datetime(d[8],d[9]) # "2/10/2004","4:00pm"
# net change not done d[10]
# history
, volume_avg_90 = make_number(d[11]) # 5544363,
, year_range_low = make_float(d[12]) # 73.17,
, year_range_high = make_float(d[13])# 100.43
, year_target = make_float(d[14]) # 106.67,
# dividend
, dividend = make_float(d[15]) # 0.64,
, dividend_yield = make_float(d[16]) # 0.65,
, dividend_date = make_datetime(d[17]) # "Mar 10", TODO convert to DateTime
, dividend_ex_date = make_datetime(d[18]) # "Feb 6",old show up "30-Mar-99" style)
# financial
, market_cap = make_number(d[19]) # 171.4B, j1 market_cap (note, not string, not number, trailing
, book_value_per_share = make_float(d[20]) # 16.309, b4 book_value_per_share
, price_book = make_float(d[21]) # 6.07, p6 price_book
, price_earnings = make_float(d[22]) # 4.34, e price_earnings
, price_earnings_ratio = make_float(d[23]) # 22.80, r price_earnings_ratio
, price_sales = make_float(d[24]) # 1.92, p9 price_sales
, ebitda = make_number(d[25]) # 10.089B, j4 ebitda
, revenue = make_number(d[26]) # 89.131B, s6 revenue
, short_ratio = make_float(d[27]) # 3.24, s7 short_ratio
, yahoo_info = str(d[28]) # "cnsprmiIed" i yahoo_info (indication of what links are available)
, share_float = None # 1,720,420,000 j2 float
)
if s.description is None or s.description == '':
s.description = d[1]
Cheers
Mike
--
Michael Watkins
TrendVue.com
mw@xxxxxxxxxxxx
|