Re: Help me, a friend of mine wrote a program in C# and I wrote the same program in C..His is faster than mine on the same machine...How Come ?

Stephen Sprunk wrote:
"Tor Rustad" <tor_rustad@xxxxxxxxxxx> wrote in message news:8uydnTeonY9bVv_a4p2dnAA@xxxxxxxxxxxxxx
Stephen Sprunk wrote:
"Tor Rustad" <tor_rustad@xxxxxxxxxxx> wrote in message news:o8mdnaZ89OVSVvza4p2dnAA@xxxxxxxxxxxxxx
Standard C IO no buffering
Buffering turned off
Written 1221 pages of ca. size 65536 (80019456 bytes)
CPU time 0.26
DiskIO 300.55 Mb/s
...
300.55 Mb/s is too good, the theoretical peek should
have been 150 Mb/s, I don't know why Linux is this
blitzing fast! Does it have anything to do with dual
core CPU??? *strange*

The speed you're measuring is how fast the fwrite() call returns, not the disk write performance. Most likely, fwrite() returns after putting the data into a buffer but before the hardware actually acknowledges it's been written to disk. Modern OSes do all sorts of things like that to improve performance.

Nope, the clock was started *before* fopen(), and stopped *after* fclose().

fclose() just closes the FILE* (and related fd); it does _not_ guarantee that the data is actually physically on the disk. There may be some OS-specific function that will give you the indication you're wrongly assuming you're getting, but it's not on by default.

If Linux is using lazy commit, i.e. allowing a file to be e.g. closed, before flushing system IO buffers, that would be highly interesting/bad, particularly for those working with critical data.

It's completely normal. That's why one should always shut down machines cleanly instead of pulling the plug -- and why data (and filesystems) tend to get corrupted when the power goes out. Even the disks lie to the OS about when data is written; as soon as the data is in the drive's cache, it tells the OS it's done writing so that the OS can reuse the buffer(s). Only top-of-the-line controllers with battery-backed caches are immune from these sorts of problems (and even then, you have to boot the machine again and let it finish writing before removing the disk from the system).

Some years ago, I was told that lazy commit, was the reason we didn't run *any* production systems on Linux, now it's allowed in *some* cases to use this OS. If an OS don't flush system buffers when a file is closed, or is doing a fflush() in an async manner, then it is impossible to code a bullet-proof C program with file I/O, in that environment.

The C standard is silent here, data has only to be transfered to the host environment, depending on QoI, this data may written to persistent storage, before fclose() return, or program exit().

Just consider this, a DB program start a transaction with "begin work", then perform lots of updates, before "commit work"... if you can't trust that the data has been saved on disk, then you risk that the transaction can be lost, in case of e.g. power outage or HW failures (e.g. disk controller failure).

If a disk controller lie, the flaw is limited to that unit. If an OS lie, that is an important thing to know about, and I don't think I would like to use such an OS, for work related production systems.


Added another test case, this time using low-level POSIX IO
functions and sync'ing the file *before* close'ing it, the result was sky
high IO, 434 Mb/s! So, the IO subsystem file writes is more than 10
times faster on Linux, than under Windows XP, tests was done with
identical HW and C source.

*amazing*

It's not so amazing when you realize you're measuring the speed of the OS's I/O system, not the hardware's.

Well, in the benchmark I ran, the result couldn't be explained by disk cache alone, since this is somewhere in the range of 8 Mb - 32 Mb, while the measured IO peek, was >150 Mb/s too high.

Also, since I did an explicit call to fsync() in the last test-case, the data should have been committed to disk, before the stop timer was set. A clear sign that something was wrong, was that fsync()'ing, boosted the performance by another 100 Mb/s!

I think the simple answer here, is that the clock() implementation on Linux, measured processor time used in *user* space only, ignoring the (significant) time spent in system calls. :)

--
Tor <bwzcab@xxxxxxxxx | tr i-za-h a-z>
.

Relevant Pages

  • [PATCH 2/2] ext3: add data=guarded mode
    ... a workqueue where the real work of updating the on disk i_size is done. ... end_io handler for buffers that are marked as guarded. ... When we start tracking guarded buffers on a given inode, ... and it also takes a reference on the buffer head. ...
    (Linux-Kernel)
  • Re: [RFC] fsblock
    ... The other reasons are that supporting larger logical block sizes than PAGE_CACHE_SIZE becomes a pain if it is not done this way when the write targets a hole as that requires all pages in the hole to be locked simultaneously which would mean dropping the page lock to acquire the others that are of lower page index and to then re-take the page lock which is horrible - much better to lock all at once from the outset and the other reason is that in NTFS there is such a thing as the initialized size of an attribute which basically states "anything past this byte offset must be returned as 0 on read, i.e. it does not have to be read from disk at all, and on write beyond the initialized_size you have to zero on disk everything between the old initialized size and the start of the write before you begin writing and certainly before you update the initalized_size otherwise a concurrent read would see random old data from the disk. ... commit_write the copied pages by dirtying their buffers ...
    (Linux-Kernel)
  • kernel 2.6.13 buffer strangeness
    ... It it's a Feature, then I'll find some way to work around it, but it seems to me that it's a Bug :-) ... effectively no more RAM to allocate for buffers. ... It's as if the kernel isn't bothering to write out any of my data until it runs out of RAM. ... Now, I'm pretty sure that the data is streaming out to the disk, cause I can see the disk activity light solidly lit on ...
    (Linux-Kernel)
  • Re: New benchmarks (was Re: SortMerge avoids thrashing (was: Baileys "two pass" FFT algorithm questi
    ... fast cache, it sounds like you can do the following: ... second disk drive, so each disk drive is running sequentially, no ... While Mike is juggling his tiny buffers, ... sys 0m0.020s ...
    (comp.programming)
  • Re: 10g RAC: max performance & min cost with miSCSI?
    ... :-) Due to reliability and minimun downtime requirements we are planning to build a 2 node Linux Oracle 10g RAC. ... To minimize the cost we are planning to run the nodes 1 CPU each since Oracle lisences are per CPU. ... The bottleneck seems almost always be on the disk system so starting the design on disk system seems the right way to go. ... Since we have 2 CPUs on RAC the disk system should be able to deliver 400 MB/s, let's say 500 MB/s to be safe. ...
    (comp.databases.oracle.server)