Re: Synchronization routine

For some funny reason I see your text as a attachment in outlook express...

"Ertugrul Soeylemez" <never@xxxxxxxxxxxxxx> wrote in message
news:20051031043743.0328ed08@xxxxxxxxxxxxxxx
>"Joris Dobbelsteen" <joris.dobbelsteen@xxxxxxxx> (Sun, 30 Oct 2005 21:30:48
>+0100):
>> Basically I have two lists, one on my firewall and one in a database. The
>> list on the firewall must become the same as the list in the database.
>> The
>> current sizes are over 1 million entries and I expect to see 2 million
>> rows
>> in the future.
>
>Taking into account that you have an old machine, you might be very low
>on memory, so this isn't practical. For about one million entries, the
>memory consumtion is the same as the size of the individual entries, in
>MB! If every entry is 128 bytes in size, then a million entries will
>consume roughly 128 MB. And the data size is well under-estimated. You
>should consider this.

I did consider this.

The average entry size is 24 bytes. So this makes 24 MB for 1 M entries.
With pointers and such it will grow probably several MBs too, but not that
far. The data from the database was pulled for a sequential scan only, not
to comsume too much memory.

The previous (probably flawed) implementation used over 128 MB for only 40k
entries. The dataset was very similair, so I would have expected only
several MB, not growing fast too. Probably something here went wrong.

- Joris


.

Relevant Pages

  • Re: DI-604 "no space to add this entry"
    ... Ian F KF4AON said in ... I expected a finite number of firewall entries, ... There isn't much memory in the DI-604. ...
    (comp.security.firewalls)
  • Synchronization routine
    ... I'm looking into a writitng a efficient routine and am looking for different ... Basically I have two lists, one on my firewall and one in a database. ... The entries in A are largely ordered so ...
    (comp.programming)
  • Re: Sorted and reversed on huge dict ?
    ... Depends on how much memory your machine has. ... If the lists are really large, your next step after the above is ... probably to use an external sort, but 150000 entries is not that many, ... dicts in memory at all will probably also be a strain. ...
    (comp.lang.python)
  • Re: Max connections on a FW-1 cluster
    ... connections, etc., as Greg advised. ... They want to install a new firewall to protect the ... that at 40k entries the firewalls wouldn't accept any more connections, ... in trouble again - she's got 50k entries in the memory of each system. ...
    (comp.security.firewalls)
  • Re: Max connections on a FW-1 cluster
    ... connections, etc., as Greg advised. ... They want to install a new firewall to protect the ... that at 40k entries the firewalls wouldn't accept any more connections, ... in trouble again - she's got 50k entries in the memory of each system. ...
    (comp.security.firewalls)