pickle, cPickle, & HIGHEST_PROTOCOL

From: A.B., Khalid (abkhd_at_earth.co.jp)
Date: 01/30/05 

Date: 30 Jan 2005 14:27:30 -0800

I wonder if someone can explain what is wrong here. I am pickling a
list of dictionaries (see code attached) and unpickling it back using
the HIGHEST_PROTOCOL of pickle and cPickle. I am getting an error
message and trace backs if the list exceeds eight items. Whether I use
pickle or cPickle does not matter, i.e., the eight number causes a
problem in both modules, although the trace backs are of course
dissimilar.

This pickling and unpickling of the list of dictionaries worked when I
stopped using the HIGHEST_PROTOCOL in both modules (pickle and
cPickle), which got Python to use the ASCII format (I suppose) as I can
read the pickled data.

This behavior was observed in Python 2.3.4 (final), and 2.4 (final) on
Win98.

Any comments?

Regards,
Khalid

# Sample program tester.py begin
!
! import pickle as pkl
! import os
! #-----------------------------
! def test_pickle():
! fn = 'rkeys.txt'
! f = file(fn, 'r')
! lines = f.readlines()
! f.close()
! _test_list = []
! for line in lines:
! sline = line.split(',')
! #print sline
! key, value = sline[0], sline[1].strip()
! _test_dict = {}
! _test_dict[key] = value
! _test_list.append(_test_dict)
!
! # Let's see the contents of our object
! print _test_list
!
! # Then pickle it
! f = file('pkl_' + fn, 'w')
! pkl.dump(_test_list, f, pkl.HIGHEST_PROTOCOL)
! f.close()
!
! # Empty it
! _test_list = []
! print _test_list
!
! # Unpickling object here:
! f = file('pkl_' + fn, 'r')
! _test_list = pkl.load(f)
! f.close()
!
! # See contents after loading
! print _test_list
!#-----------------------------
!if __name__ == '__main__':
! test_pickle()
!
!# Sample program end

# Contents of file rkeys.txt (without the triple quotes):
"""
'1','v1'
'2','v2'
'3','v3'
'4','v4'
'5','v5'
'6','v6'
'7','v7'
'8','v8'
'9','v9'
"""

# Output (without the triple quotes)
# Using "import pickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
! File "tester.py", line 41, in ?
! test_pickle()
! File "tester.py", line 34, in test_pickle
! _test_list = pkl.load(f)
! File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 1390, in load
! return Unpickler(file).load()
! File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 872, in load
! dispatch[key](self)
! File "D:\PY23\PYTHON\DIST\SRC\lib\pickle.py", line 1189, in
load_binput
! i = ord(self.read(1))
!TypeError: ord() expected a character, but string of length 0 found
"""

# Output (without the triple quotes)
# Using "import cPickle as pkl":
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
!Traceback (most recent call last):
! File "tester.py", line 41, in ?
! test_pickle()
! File "tester.py", line 34, in test_pickle
! _test_list = pkl.load(f)
!EOFError
"""

# Output (without the triple quotes)
# Using "import cPickle as pkl", or "import pickle as pkl"
# but _not_ using the HIGHEST_PROTOCOL:
"""
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
[]
[{"'1'": "'v1'"}, {"'2'": "'v2'"}, {"'3'": "'v3'"}, {"'4'": "'v4'"},
{"'5'": "'v5'"}, {"'6'": "'v6'"}, {"'7'": "'v7'"}, {"'8'": "'v8'"},
{"'9'": "'v9'"}]
"""

Relevant Pages

  • Re: Shelve operations are very slow and create huge files
    ... seconds with shelve. ... With cPickle, ... With gzip & cPickle, it also needed 11 seconds (the file was now around ... Using pickle instead of cPickle, ...
    (comp.lang.python)
  • Re: Load a list subset with pickle?
    ... load the first a few elements in the list. ... just pickle each item from the list in order to the same file and only ... In: import cPickle ... "I have come to believe that the whole world is an enigma, a harmless enigma ...
    (comp.lang.python)
  • Strange KeyError using cPickle
    ... I'm experiencing strange errors both with pickle and cPickle in the ... from string import ascii_uppercase ...
    (comp.lang.python)
  • Re: cPickle.dumps differs from Pickle.dumps; looks like a bug.
    ... I've found the following strange behavior of cPickle. ... from cPickle import dumps as cdumps ... that by default pickle and cPickle will create a longer ... ASCII representation, for a binary representation use a higher pickle ...
    (comp.lang.python)
  • RE: Mixing protocols in pickles?
    ... The unpickling implementation has no idea which protocol is in use. ... pickletools.py library module, BTW, and a symbolic pickle disassembler ... A protocol 1 string pickle just contains the raw string bytes; ...
    (comp.lang.python)