Re: Building Very Large Records
From: Dave Brueck (dave_at_pythonapocrypha.com)
Date: 09/10/04
- Next message: Istvan Albert: "ANN: LOWESS fitting (python wrapper)"
- Previous message: Jeff Hagelberg: "Solved: Dynamic Linking Problems with Intel Compiler"
- Maybe in reply to: Greg Lindstrom: "Building Very Large Records"
- Next in thread: Alex Martelli: "Re: Building Very Large Records"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Fri, 10 Sep 2004 12:52:08 -0600 To: python-list@python.org
Greg Lindstrom wrote:
> Hello-
>
> I am working on a routine to pull information from an Oracle database and
> format it into fixed-length records. My problem is that the record layout
> is quite long (over 500 bytes) and contains dozens of fields. How would
> *you* go about building such a beast? I know that using the += operator is
> a bad idea, as it creates a new copy of the string each time. How about the
> struct.pack() method? A very big '%-12.12s%-50.50s.......' statement?
> Other ideas? It does not have to be blazingly fast, but it doesn't have to
> suck, either. I don't know enough about Oracle SQL yet, could I format the
> data in the SQL call?
Here's what I do, maybe somebody can suggest some improvements upon it:
Derive your record from ctypes' Structure class:
class XHeader(Structure):
_pack_ = 1 # No alignment padding
_fields_ = [('ucid', c_ulonglong),
('majorVer', c_ubyte),
('minorVer', c_ubyte),
('extent', c_ulonglong),
('future', c_byte),
('titleLen', c_ubyte)]
Then to serialize it to a string:
def ObjAsString(obj):
'Takes a ctypes object (works on structures too) and returns it as a string'
# There is probably a better way, but this seems to work ok
sz = sizeof(obj)
RawType = c_char * sz
ptr = RawType.from_address(addressof(obj))
return ptr.raw[:sz]
To go the other direction:
def StringAsObject(s, ObjType):
'Takes a string and returns it as a ctypes object'
z = c_buffer(s)
x = ObjType.from_address(addressof(z))
x.__internal = z #ugly hack - save a ref to the c_buffer!
return x
Basic usage is:
x = XHeader()
x.majorVer = 5
...etc...
StuffIntoDB(ObjAsString(x))
newX = StringAsObject(ReadFromDB(), XHeader)
I haven't used it with tons of object instances, so you may need to see if you
end up with memory leaks.
-Dave
- Next message: Istvan Albert: "ANN: LOWESS fitting (python wrapper)"
- Previous message: Jeff Hagelberg: "Solved: Dynamic Linking Problems with Intel Compiler"
- Maybe in reply to: Greg Lindstrom: "Building Very Large Records"
- Next in thread: Alex Martelli: "Re: Building Very Large Records"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
