Re: large and sparse matrices

"mediratta@xxxxxxxxx" wrote:

Hi,

I want to allocate memory for a large matrix, whose size will be
around 2.5 million x 17000. Three fourth of its rows will have all
zeroes, but it is not known which will be those rows. If I try to
allocate memory for this huge array, then I get a segmentation fault
saying:

Program received signal SIGSEGV, Segmentation fault.
0xb7dd5226 in mallopt () from /lib/tls/i686/cmov/libc.so.6

I have not given any compiler options. I think that the error is
because I am allocating too big a size ?
[...]

Rather than attempting to allocate one huge array, allocate an
array of pointers to each row. (Are there 2.5 million rows, or
17,000 rows?) Initialize this array with NULL pointers. Then
allocate each row only as they're used.

However, even then, how much memory is needed for one fourth of
the rows? You are talking about 42.5 billion entries, so one
fourth of them occupied is still over 10 billion entries. Given
that your system (i386 Linux) has only 32 bits of address, that
means 4 billion addresses. Obviously, you cannot address 10
billion entries in 32 bits.

It sounds like you'll need to do some bookkeeping on your end to
keep rows on disk (in separate files -- again for the 32-bit
limit on your system) and swap them in and out as needed.

And you better hope that you have enough disk space for all of
this data as well.

--
+-------------------------+--------------------+-----------------------+
| Kenneth J. Brody | www.hvcomputer.com | #include |
| kenbrody/at\spamcop.net | www.fptech.com | <std_disclaimer.h> |
+-------------------------+--------------------+-----------------------+
Don't e-mail me at: <mailto:ThisIsASpamTrap@xxxxxxxxx>


.

Relevant Pages

  • Re: Linked List & Dynamic Memory Allocation
    ... Although it seems kind of pointless given that these items are already in an array. ... typdef struct node ... I am trying to dynamically allocate memory if user chooses to enter ...
    (microsoft.public.vc.mfc)
  • Re: About c++ pointer
    ... >>because the object(string array) is created in stack. ... >>is fine since you dynamically allocate memory in the heap. ...
    (comp.lang.cpp)
  • Re: [PATCH 09/10] Change table chaining layout
    ... so I need to allocate an array and fill ... it in with DBDMA command blocks (and then dma-map it and point the ... SG list may have _fewer_ requests in it in case some of those entries got ...
    (Linux-Kernel)
  • Re: About c++ pointer
    ... > The code 1 will not work, the pointer will be empty when it returns, ... > because the object(string array) is created in stack. ... > is fine since you dynamically allocate memory in the heap. ...
    (comp.lang.cpp)
  • Re: Buffer or Realloc?
    ... better to allocate memory and realloc it for the size of the what is ... between deciding to use a fixed size buffer or allocating memory ... becomes extremely small when I realloc it. ... I would have to allocate memory. ...
    (comp.lang.c)