Re: Creating huge data in very less time.



On Mar 31, 1:15 pm, Steven D'Aprano
<ste...@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
On Mon, 30 Mar 2009 22:44:41 -0700, venutaurus...@xxxxxxxxx wrote:
Hello all,
            I've a requirement where I need to create around 1000
files under a given folder with each file size of around 1GB. The
constraints here are each file should have random data and no two files
should be unique even if I run the same script multiple times.

I don't understand what you mean. "No two files should be unique" means
literally that only *one* file is unique, the others are copies of each
other.

Do you mean that no two files should be the same?

Moreover
the filenames should also be unique every time I run the script. One
possibility is that we can use Unix time format for the file   names
with some extensions.

That's easy. Start a counter at 0, and every time you create a new file,
name the file by that counter, then increase the counter by one.

Can this be done within few minutes of time. Is it
possble only using threads or can be done in any other way. This has to
be done in Windows.

Is it possible? Sure. In a couple of minutes? I doubt it. 1000 files of
1GB each means you are writing 1TB of data to a HDD. The fastest HDDs can
reach about 125 MB per second under ideal circumstances, so that will
take at least 8 seconds per 1GB file or 8000 seconds in total. If you try
to write them all in parallel, you'll probably just make the HDD waste
time seeking backwards and forwards from one place to another.

--
Steven

That time is reasonable. The randomness should be in such a way that
MD5 checksum of no two files should be the same.The main reason for
having such a huge data is for doing stress testing of our product.
.



Relevant Pages

  • Re: Creating huge data in very less time.
    ... constraints here are each file should have random data and no two files ... should be unique even if I run the same script multiple times. ... 1GB each means you are writing 1TB of data to a HDD. ...
    (comp.lang.python)
  • Re: SATA HDD in external USB2 case no longer recognised after 1st reboot
    ... The reason why this is important to me is that, ... the less essential files to my ext HDD. ... Thanks Anna, and whoever else might want to contribute. ... responders to search out all the posts in this thread and force them to ...
    (microsoft.public.windowsxp.hardware)
  • Re: ? Partitioning
    ... So that in the event your working HDD fails because of some ... electronic/mechanical reason or because of some corruption of your OS so ... simply create a single partition on their HDD that contains the drive's ... If you want any further recommendations from me re the above so indicate and ...
    (microsoft.public.windowsxp.general)
  • Re: Creating huge data in very less time.
    ... should be unique even if I run the same script multiple times. ... 1GB each means you are writing 1TB of data to a HDD. ... MD5 checksum of no two files should be the same.The main reason for ... What nobody has suggested yet is that you can *simulate* the files by making a large set of custom file-like object and feed that to your application. ...
    (comp.lang.python)
  • Re: Hardware change and XP on C:/
    ... Just be sure to perform a proper shutdown before the cloning, ... You've indicated the reason you purchased the new HDD as the same capacity ... reasonably certain or have even verified that your present HDD is defective? ... You've concluded that there's no other hardware/software problem affecting ...
    (microsoft.public.windowsxp.hardware)