Re: dynamic allocation file buffer

On 15 Set, 22:09, "Aaron \"Castironpi\" Brady" <castiro...@xxxxxxxxx>
wrote:
On Sep 15, 4:34 am, Francesc <fal...@xxxxxxxxx> wrote:



On 12 Set, 14:39, "Aaron \"Castironpi\" Brady" <castiro...@xxxxxxxxx>
wrote:

A consideration of other storage formats such as HDF5 might
be appropriate:

http://hdf.ncsa.uiuc.edu/HDF5/whatishdf5.html

There are, of course, HDF5 tools available for Python.

PyTablescame up within the past few weeks on the list.

"When the file is created, the metadata in the object tree is updated
in memory while the actual data is saved to disk. When you close the
file the object tree is no longer available. However, when you reopen
this file the object tree will be reconstructed in memory from the
metadata on disk...."

This is different from what I had in mind, but the extremity depends
on how slow the 'reconstructed in memory' step is. (Fromhttp://www.pytables.org/docs/manual/ch01.html#id2506782). The
counterexample would be needing random access into multiple data
files, which don't all fit in memory at once, but the maturity of the
package might outweigh that. Reconstruction will form a bottleneck
anyway.

Hmm, this was a part of a documentation that needed to be updated.
Now, the object tree is reconstructed in a lazy way (i.e. on-demand),
in order to avoid the bottleneck that you mentioned. I have corrected
the docs in:

http://www.pytables.org/trac/changeset/3714/trunk

Thanks for (indirectly ;-) bringing this to my attention,

Francesc

Depending on how lazy the reconstruction is, would it be possible to
modify separate tables from separate processes concurrently?

No, modification of different tables in the same file simultaneously
is not supported yet. This is a limitation of the HDF5 library
itself. The HDF Group said that they have plans to address this, but
this is probably a long-term task.

Francesc
.

Relevant Pages

  • Re: dynamic allocation file buffer
    ... in memory while the actual data is saved to disk. ... file the object tree is no longer available. ... Now, the object tree is reconstructed in a lazy way, ... Depending on how lazy the reconstruction is, ...
    (comp.lang.python)
  • Re: dynamic allocation file buffer
    ... in memory while the actual data is saved to disk. ... file the object tree is no longer available. ... Now, the object tree is reconstructed in a lazy way, ... Depending on how lazy the reconstruction is, ...
    (comp.lang.python)
  • Re: dynamic allocation file buffer
    ... the metadata in the object tree is updated ... in memory while the actual data is saved to disk. ... Reconstruction will form a bottleneck ...
    (comp.lang.python)
  • Re: OT: Baseballs Doc Ellis LSD assisted no-hitter
    ... You are playing the expert again, ... expressing your own opinions of what you have read about memory and its ... the reconstruction takes place AFTER the event that causes one ... but my limited contact with those who have personal experience ...
    (rec.gambling.poker)
  • Re: The Nature of "Now"
    ... the future is conjecture made up by extrapolation from the memory. ... Arnold Neumaier ... So can be our reconstruction of the future. ...
    (sci.physics.research)