Re: CL Scaling for High Traffic Web Sites

(message (Hello 'bob)
(you :wrote :on '(29 Apr 2007 12:48:00 -0700))
(

b> I guess I could use sticky session on the load balancer. However,
b> there's still the problem with caching database queries in memory once
b> they grow too big for in-memory data structures and have to be stored
b> on disk constantly. Caching them in local hash tables would result in
b> a lot of duplications. This is a fairly common problem now a days.

lot of duplications where? in memory of different machines, that is actually
what caching is meant to do -- have data closer. if duplications on same
machine, that should be eliminated..

b> address space. DragonFly BSD has some features that makes it easy to
b> do this over a slow and loosely-connected ethernet (they aim to do
b> this efficiently over the internet in the future!).

i think HT over network will introduce at least order-of-magnitude overhead
comparing to direct hash tables.
but it might still acceptable if it introduces benefits comparing to RDBMS
performance.
i didn't evaluate memcached performance, but i saw that RDBMS performance
was killed with network communications -- sending each packet introduces
some overhead, and applications was sending megatons of those packets.. so
time was spent not on web server or database, but on OS system time..
certainly, network HT might have more optimal protocol than RDBMS over
network, or you can fine-tune requests better..

but you are going to check automated shared memory. i don't see how it can
work fast :)
if you'll do testing, please post here some benchmarking results.

)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"I am everything you want and I am everything you need")


.

Relevant Pages

  • Re: CL Scaling for High Traffic Web Sites
    ... b> there's still the problem with caching database queries in memory once ... in memory of different machines, ... i think HT over network will introduce at least order-of-magnitude overhead ...
    (comp.lang.lisp)
  • Re: Maximum Swap Partition Size
    ... So they go straight from disk to the network. ... > Haven't you ever learned about file caching? ... in 4GB of main memory without any need for either swapping or dumping. ...
    (alt.os.linux)
  • [RFC][PATCH 0/9] Network receive deadlock prevention for NBD
    ... Convergence of network and storage paths" ... "Net vm deadlock fix " ... Maybe it is another memory deadlock, ... dipping into the memalloc reserve if it must. ...
    (Linux-Kernel)
  • Re: [PATCH] [Request for inclusion] Filesystem in Userspace
    ... > to allocate network buffers down to the PF_MEMALLOC boundary. ... > the network stack is only allowed to allocate memory down to the ... A lot of people think that mempool solves this problem, ... normal allocation, but that fails, so they start eating into the mempool ...
    (Linux-Kernel)
  • Re: [RFC 0/3] Recursive reclaim (on __PF_MEMALLOC)
    ... then you should not kill the other things that want memory. ... The users of memory are various subsystems. ... the network subsystem because its build on kmalloc! ... The reserves are spread over all kernel mapped zones, ...
    (Linux-Kernel)