Re: Need design advice. What's my best approach for storing this data?


"Mudcat" <mnations@xxxxxxxxx> wrote in message
news:1142615283.733346.74260@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Hi,

I am trying to build a tool that analyzes stock data. Therefore I am
going to download and store quite a vast amount of it. Just for a
general number - assuming there are about 7000 listed stocks on the two
major markets plus some extras, 255 tradying days a year for 20 years,
that is about 36 million entries.


On a different tack, to avoid thinking about any db issues, consider
subscribing
to TC2000 (tc2000.com)... they already have all that data,
in a database which takes about 900Mb when fully installed.
They also have an API which allows you full access to the database
(including from Python via COM). The API is pretty robust and allows
you do pre-filtering (e.g. give me last 20 years of all stocks over $50
with ave daily vol > 100k) at the db level meaning you can focus on using
Python for analysis. The database is also updated daily.

If you don't need daily updates, then subscribe (first 30 days free) and
cancel, and you've got a snapshot db of all the data you need.

They also used to send out an evaluation CD which had all
the history data barring the last 3 months or so which is certainly
good enough for analysis and testing. Not sure if they still do that.

HTH.


.

Relevant Pages

  • Re: How do I do this without database?
    ... > 1100 stocks on the list and I want to save them in a single file ... > array then rewrite the whole array to the file. ... If you're more interested in speed, use a RDBMS - such as MySQL, ... DBD::ODBC module to access the database. ...
    (comp.lang.perl.misc)
  • Re: Very large database
    ... how do you define very large database? ... Or is it on the network? ... I asked it to return all stocks from a list of ... I have a basic understanding of normalization and it ...
    (microsoft.public.access.queries)
  • Re: sad, not free OODBMS that really works to use
    ... I was trying to make later changing of database easier. ... Design would be better for ... You can also insert Dictionaries, and that's when things start to get ... "Stocks" indexed by strings, and then the values of the dictionary are ...
    (comp.lang.smalltalk)
  • Re: Very large database
    ... how do you define very large database? ... Or is it on the network? ... - Shorten the name of the backend MDB ... I asked it to return all stocks from a list of stocks ...
    (microsoft.public.access.queries)
  • Re: Database Timestamp conversion error
    ... I am populating a mySQL database with data from the MS Access ... and I can insert the data successfully into mySQL with Python. ... be a malformed dbiDate object when I insert certain records. ... I didn't start using Python on databases till after DB API ...
    (comp.lang.python)