Re: where can I find this...

From: Davids (nowhere_at_nowthere.com)
Date: 12/20/03 

Date: Fri, 19 Dec 2003 16:31:10 -0700

I understand about what you're saying. I have lots of experience programming in
various shells and c/c++ under both *nix and DOS.

The programs I'm running are very plain apps written in vanilla 'c' code
designed to run as simple command-line apps in numerous environments.
Everything is printed via printf function calls. Nothing's going to stderr.

(If you're really curious, you can download them, along with the examples I'm
trying to run, from this URL: http://abacus.gene.ucl.ac.uk/software/paml.html .)

When I run these (baseml.exe and codeml.exe) from redcon, I get the same output
as if I run them directly from a command prompt. Well, except for some sporadic
newlines added to the data obtained from redcon's pipes.

The problem is that these apps generate some info just as soon as you run them.
They usually take a while to run, so they'll spit out some stuff then appear to
hang for a while. Redcon's mechanism of grabbing the stdout pipe doesn't
display anything in some cases until the program is completely finished
running. Again, the output is pretty much the same (ie., sans newlines); it's
the timing that's the problem.

These are convergence-driven apps. While they're digesting their data, they
spit out a line whenever they can narrow their search space a little bit. After
a while, the delta becomes small enough and they quit. This works fine from the
command line.

We want our Delphi app to spawn the desired app with a custom parameter file,
then get the update messages from stdout and update a graph or something to
indicate progress of the convergence. If the app is running for 10-15 minutes
before anything gets displayed, it tends to bother people.

In fact, the command-line version of the app does NOT run that long without
outputting SOMETHING. It's just that running it under redcon, I got exactly
three screen updates over the 25 minutes it took to run one example. That's
with the initial output and 60 lines periodically indicating changes in
convergence data. Clearly, something is buffering the output from the apps.

Here's an example of some output captured through redcon that illustrates the
newline problem (see lines prefixed by 6, 20, 33, 47, and 48):
---------------------------
  1 h-m-p 0.0000 0.0000 10903.6339 28454.425884 3 0.0000 42 | 0/32
  2 h-m-p 0.0000 0.0000 12666.7480 26577.191787 m 0.0000 77 | 0/32
  3 h-m-p 0.0001 0.0003 2598.8205 26103.259621 2 0.0002 116 | 0/32
  4 h-m-p 0.0002 0.0008 887.4341 25864.474965 3 0.0007 156 | 0/32
  5 h-m-p 0.0002 0.0009 1132.6708 25762.945884 1 0.0004 192 | 0/32
  6 h-m-
p 0.0003 0.0014 614.3020 25648.024875 2 0.0010 230 | 0/32
  7 h-m-p 0.0002 0.0011 566.3375 25579.719307 1 0.0009 268 | 0/32
  8 h-m-p 0.0004 0.0021 419.0274 25549.437737 1 0.0008 304 | 0/32
  9 h-m-p 0.0008 0.0038 432.6577 25492.238832 1 0.0017 340 | 0/32
 10 h-m-p 0.0007 0.0035 324.7013 25463.807693 1 0.0014 376 | 0/32
 11 h-m-p 0.0008 0.0042 164.9031 25442.421159 1 0.0029 413 | 0/32
 12 h-m-p 0.0010 0.0051 237.4340 25404.138205 1 0.0040 451 | 0/32
 13 h-m-p 0.0005 0.0024 784.9129 25328.070911 m 0.0024 486 | 0/32
 14 h-m-p 0.0009 0.0043 555.3111 25284.214894 1 0.0021 522 | 0/32
 15 h-m-p 0.0007 0.0036 241.3858 25275.356281 1 0.0012 558 | 0/32
 16 h-m-p 0.0017 0.0085 83.6509 25271.924678 0 0.0018 593 | 0/32
 17 h-m-p 0.0034 0.0169 22.9935 25270.825450 1 0.0027 629 | 0/32
 18 h-m-p 0.0023 0.0113 21.3352 25268.596803 1 0.0045 665 | 0/32
 19 h-m-p 0.0014 0.0072 38.2847 25258.344127 m 0.0072 700 | 0/32
 20 h-m-p 0.0000 0.0000
  58.7142
h-m-p: 0.00000000e+000 0.00000000e+000 5.87141666e+001 25258.344127
.. | 0/32
 21 h-m-p 0.0000 0.0003 2418.2076 25252.794852 2 0.0000 770 | 0/32
 22 h-m-p 0.0000 0.0002 2373.8905 25162.682457 1 0.0001 808 | 0/32
 23 h-m-p 0.0002 0.0012 418.1333 25125.609624 1 0.0004 844 | 0/32
 24 h-m-p 0.0002 0.0011 255.0021 25107.856223 2 0.0007 883 | 0/32
 25 h-m-p 0.0009 0.0043 115.7372 25097.203013 1 0.0020 919 | 0/32
 26 h-m-p 0.0008 0.0038 128.0400 25091.106952 1 0.0015 955 | 0/32
 27 h-m-p 0.0017 0.0085 80.0126 25089.035687 1 0.0012 991 | 0/32
 28 h-m-p 0.0019 0.0100 51.6956 25086.829471 1 0.0029 1028 | 0/32
 29 h-m-p 0.0029 0.0176 52.4268 25084.699890 1 0.0037 1065 | 0/32
 30 h-m-p 0.0019 0.0093 66.6314 25083.332376 1 0.0022 1102 | 0/32
 31 h-m-p 0.0031 0.0155 36.0921 25082.609418 0 0.0027 1137 | 0/32
 32 h-m-p 0.0043 0.0216 17.6468 25082.371535 1 0.0026 1173 | 0/32
 33 h-m-p 0.0033 0.0762 14.1152 25082.156946 0 0.
0040 1208 | 0/32
 34 h-m-p 0.0028 0.0530 20.0558 25081.906954 0 0.0037 1243 | 0/32
 35 h-m-p 0.0036 0.0324 20.6331 25081.762253 1 0.0023 1279 | 0/32
 36 h-m-p 0.0041 0.1117 11.5193 25081.679683 0 0.0027 1314 | 0/32
 37 h-m-p 0.0025 0.0501 12.3034 25081.602577 0 0.0024 1349 | 0/32
 38 h-m-p 0.0036 0.0931 8.2168 25081.478189 0 0.0057 1384 | 0/32
 39 h-m-p 0.0043 0.0710 10.9368 25081.342430 0 0.0044 1419 | 0/32
 40 h-m-p 0.0108 0.0776 4.4338 25081.203853 0 0.0075 1454 | 0/32
 41 h-m-p 0.0025 0.0686 13.3666 25080.818149 1 0.0059 1490 | 0/32
 42 h-m-p 0.0046 0.0664 16.8905 25079.661289 1 0.0115 1526 | 0/32
 43 h-m-p 0.0031 0.0155 28.2720 25078.714027 1 0.0050 1563 | 0/32
 44 h-m-p 0.0078 0.0635 18.1928 25077.390497 1 0.0107 1600 | 0/32
 45 h-m-p 0.0060 0.0298 21.4836 25076.870488 1 0.0040 1636 | 0/32
 46 h-m-p 0.0074 0.0369 4.8048 25076.771396 0 0.0063 1671 | 0/32
 47 h-m-p 0.0069 0.1327 4.3903 25076.574653 1 0.0201 1708 | 0/3
2
 48 h-m-p 0.0052 0.1269 16.9744 25076.181194 1 0.0
117 1744 | 0/32
 49 h-m-p 0.0289 0.1447 3.0125 25076.168347 0 0.0039 1779 | 0/32
 50 h-m-p 0.0228 1.2297 0.5093 25076.167539 0 0.0048 1814 | 0/32

---------------------------

Given 60 lines of output like this, one would expect to see something come
throught the stdout pipe every 30 seconds or so, on average. Not three "chunks"
of output over a 25 minute period. (Indeed, these apps *DO* put out data to
stdout quite frequently while they're running!)

Any thoughts?

Thx
-David

"Ben Hochstrasser [FF]" wrote:

> Davids wrote:
>
> > Supposedly, in WinNT (and 2K by analogy), the data is not supposed to
> > be buffered by Windows. However, there is some mechanism clearly
> > buffering the stdout data and even chopping it up into some other
> > chunks; ie., every so many lines, a newline is inserted into a line at
> > some arbitrary place in the middle of the line. This makes it rather
> > difficult to parse the output and do anything useful with it.
>
> First of all - there are two output channels: STDOUT and STDERR which
> usually both are redirected to the console. So...how do STDOUT and STDERR
> look like? ("yourprogram.exe > output.txt 2>error.txt")
> Then, many console applications don't strictly use only STDOUT and the
> like, they play with cursor positions and the like (remember old Pascal's
> CRT unit?), eg to display a progress message etc. one could write
>
> while i < 100 do
> begin
> write(#13, i, '% done...');
> do_something_useful;
> inc(i);
> end;
>
> The output of this looks fine on the screen but horrible on the caught
> output.
> Then, my RedCon unit is constantly polling stdin, stdout and stderr to
> see if there's any work to do. Of course, some loops take longer than
> others, and a bunch of characters accumulate in the output buffers - you
> get 'chunky' output.
> I don't quite understand the interspersed newline characters - at least
> in theory the caught stdout and stderr should look the same as if they
> had been redirected to files on the command line. Can you post an example
> (both RedCon results and cmdline output?)
>
> Follow-up set to borland.public.delphi.nativeapi
>
> --
> Ben
>
> Mach es wie die Eieruhr - zähl die weichen Eier nur!

Relevant Pages

  • RE: SQL Server slows down when after the apps runs for several hours.
    ... SQL SELECT Optimization Levels and Performance ... >that the longer he use the apps in a day, the longer my apps would take ... I found that the 'command' he referred to is ... General SQL Server question; Anybody got a clue what is the cause? ...
    (microsoft.public.fox.programmer.exchange)
  • VB newbie - exposing VB subs to VBA?
    ... differences of the apps on the two machines, and also that the programs are ... How do I expose VB subs via COM? ... For instance in Excel I ... in on the command line easily enough. ...
    (microsoft.public.vb.com)
  • Cross process communication (sort of)
    ... I have an application which uses several command line utility apps ... One option to accomplish this is to make the manager app an ActiveX ...
    (microsoft.public.vb.general.discussion)
  • Re: Console Applications
    ... Or you open command line first and then execute ... your executable from command line instead of double-clicking it. ... because I want to build Windows apps and not console apps right now? ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Redirecting Output
    ... You modify the apps so that it's a console program, and then write to stdout. ...
    (microsoft.public.vb.general.discussion)
Loading