fast access to large amounts of data

From: Eric (emedlin_at_myrealbox.com)
Date: 03/31/04 

Date: 31 Mar 2004 10:07:57 -0800

I have a tiff viewer program that loads mult. 1bit tiffs (upto 9) at
once. The program allows you to zoom in and see the screen dots and
move around. These tiff are hugh resolution wise ie 30k x 30k. Since
they are compress in CCITT Group 4 the files size is not always that
large a few meg. These are prepress files though and somtimes they
don't compress much so the files size can be 100 megs. I want to be
able to access the data as fast as I can. Using libtiff to read and
decompress the file takes about 1 sec. +/- 0.5 per file, so that is 9
secs to get the data for every file. Is there anyway to reduce this
time. Keeping the uncompress file in memory will not work well.
Uncompress one file is over 100meg. That would be almost 1gig my
program would use for just image data. Since I can read data off the
HD faster than I can use libtiff to load and decompress the file. I
though about writing uncompressed data to the HD and using memory
mapped io to read it when I need it. Would this be any faster than
just putting it all in memory and letting the OS's virtual memory
manager handle it. I am open to other suggestions. Note: This is
being wrote on a powerbook running OS X, but will be ran on G5's.

Relevant Pages

  • Re: Scanned image format
    ... As I'm only scanning text I have saved as a tiff ... >Don't keep adjusting and saving your JPG (once or twice ... >| I thought acrobat was the standard for viewing scanned ... >|>| another way I can reduce the memory? ...
    (microsoft.public.frontpage.client)
  • Re: key to speed - use byte array instead of writing to locked image bytes
    ... |>of the major graphic formats. ... |>it's memory footprint. ... |>regardless of the size on disc. ... |>A 750 meg TIFF might be ok as long as it was a simple format but TIFF is ...
    (microsoft.public.dotnet.framework.drawing)
  • Re: Out of memory when loading an image file
    ... I have 512MB memory, but I set virtual memory to 5GB. ... While Michael made a good point about the misleading errors, if your TIFF is actually 2GB it's not necessarily surprising that you would get an "out of memory" error trying to load it. ... If I recall correctly, even under 64-bit Windows, .NET is still 32-bit though, and can't deal with objects larger than 2GB. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Picture and Fax viewer with tiled TIFFs
    ... tiled; the TIFF spec allows for two main methods of writing; scanline (which ... grab huge blocks of continuous memory. ...
    (microsoft.public.windowsxp.photos)