generic tokenizer

From: Angus Mackay (yeah_at_right.com)
Date: 08/31/04 

Date: Tue, 31 Aug 2004 18:30:26 GMT

I remember python having a generic tokenizer in the library. all I want
is to set a list of token seperators and then read tokens out of a
stream, the token seperators should be returned as themselves.

is there anything like this?

cheers, Angus.