Re: PySQLLite Speed

From: Gerhard Haering (gh_at_ghaering.de)
Date: 12/03/04 

Date: Fri, 3 Dec 2004 13:36:25 +0100
To: python-list@python.org

On Fri, Dec 03, 2004 at 06:06:11AM -0500, Kent Johnson wrote:
> If your data is (or can be) created by an iterator, you can use this recipe
> to group the data into batches of whatever size you choose and write the
> individual batches to the db.
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/303279

If your data is (or can be) created by an iterator, then you might find it
interesting that *pysqlite2*'s .executemany() not only works on lists, but also
on iterators.

Example:

import pysqlite2.dbapi2 as sqlite
...
# A generator function (which returns an iterator)
def gen():
    for i in xrange(5):
        yield (5, 'foo')

cu.executemany("insert into foo(x, y) values (?, ?)", gen())

So, in pysqlite2, .executemany() and iterators provide best
performance. .executemany() reuses the compiled SQL statement (so the
engine only needs to parse it once), and the iterator, if used
smartly, reduces the amount of memory used because you don't need to
construct large lists any more.

I hope I don't create too much confusion here ;-)

-- Gerhard

Relevant Pages

  • Re: mutable list iterators - a proposal
    ... > Replacing items, which is by far the most common need, works fine. ... >> through which the iterator has already passed, ... The fact that lists don't ... def SetBottom: ...
    (comp.lang.python)
  • Re: Fast expanding macros /Was: How to declare recursive factorial function?
    ... that works purely by expansion, in which cases is it possible to make it ... and a purely expandable macro comparing two items ... Let the iterator modify these arguments during iteration... ... lists you always need to decide upon the treatment of "edge- ...
    (comp.text.tex)
  • Re: Iterator Class?
    ... >> there is a significant performace penalty when using getfor long Lists. ... >> available albeit at a slight performance penalty compared to Iterator. ... Farsight Systems Corporation ...
    (comp.lang.java.programmer)
  • Re: Iterator Class?
    ... >> there is a significant performace penalty when using getfor long Lists. ... >> available albeit at a slight performance penalty compared to Iterator. ... Farsight Systems Corporation ...
    (comp.lang.java.help)
  • Event listener list implementation
    ... I am looking for java.util.List implementation for event listeners ... of synchronization using copy-on-write mode. ... dominant ops are reads (such as event listener lists) then it is ... Related to, when firing events using an iterator, iterator ...
    (comp.lang.java.developer)