Re: Best database for implementing a cache

On Mar 31, 6:11 am, Arne Vajhøj <a...@xxxxxxxxxx> wrote:
vj wrote:
Why not just write then to the disk directly? Most File Systems can be
treated as a form a database, and it seems like thats kind of what you
want anyway.

Yes i am writing the image files directly to the filesystem as it is.
However i need to limit the size of the cache so that it may not grow
arbitarily large. that would require me to maintain access count
metric for evicting existing stored images. Since filesystem itself
would not let me assiciate this info with the image I tought of a
utilizing a database that would store the access count as well as all
the available images in the cache.

What about keeping files on disk, having meta data in memory and
simply rebuilding meta data at startup (list all files on disk
and start with zero counters) ?

Arne

Certainly thats a good idea. How aboute mainting a hashtable and a
priority queue. The hashtable will store all the meta data associated
with a file and the priority queue will implement file aviction
policy. as soon as a a file is hit i will increase the hit count. On a
cache miss i will fetch the image , create an entry in the hash table
and push an entry in the queue. if the queue size if creater than a
certain treshhold i will pop the element from its head (with the
lowest access count) delete its corrosponding file / hashtable entry.

This seems to be a good idea but i have a small doubt. Does propirity
queue dynamicaly orders elements. I mean since the priority queue is
implemented as a heap in java hence element ordring only takes place
when elements are added. This will create problems as because if i
update access count in hashtable then they wont be reflected in the
structure of the queue. Hence when i pop element from its head for
eviction it might not be the one with minimum access count.

What do you think, Any other data structure that we can use for fast
lookup of elements

.

Relevant Pages

  • Re[2]: How does disk caching work?
    ... >> While you are correct that when cache is emtry kenrel will dip into the inactive queue. ... Pages on the cache queue still have the association. ... > You may want to study the kernel sources some more, ...
    (freebsd-performance)
  • Re: [RFC] [PATCH 0/3] ioat: DMA engine support
    ... > the queue of copies long (to amortize the cost of ... Don't forget that there are benefits of not polluting the cache with the ... userland data on the return-to-userspace code path. ... send the line "unsubscribe linux-kernel" in ...
    (Linux-Kernel)
  • Re: Lock-free not useful? (was Re: Reconciling Garbage Collection with Deterministic Finaliz
    ... the amount of cache traffic. ... activity than lock-free queue operations do without slowing things down. ... The other threads wanting to use the resource wait on the lock. ...
    (comp.programming.threads)
  • Re: Why Semaphore Hardware-Dependent?
    ... list then some more cache bounces might be avoidable. ... a batch of readers at the front of the queue. ... Hardly a problem to make that slower. ... Currently interruptible rwsems are not available, but that may change, and ...
    (Linux-Kernel)
  • Save Binary data.
    ... I have a program that downloads 'gigabytes' of Axis NetCam photos per day. ... Right now, I set up the process to put the images into a queue, and every 30 ...
    (comp.lang.python)