Re: serving a file to a client --- background fcopy read fileevent event puts socket channel blocking nonblocking
- From: Fredderic <my-name-here@xxxxxxxxxx>
- Date: Wed, 2 Apr 2008 01:51:33 +1000
On Sun, 30 Mar 2008 02:10:05 -0700 (PDT),
Alexandre Ferrieux <alexandre.ferrieux@xxxxxxxxx> wrote:
On Mar 30, 5:00 am, Fredderic <my-name-h...@xxxxxxxxxx> wrote:
My choice, would be to split [fcopy] in half. An engine thatHmmm... I think you've missed one tiny bit about [fcopy -size]'s
invokes a callback when there's both data waiting and somewhere to
put it (output buffers contain less than a block of data, for
example), and a mechanism for transferring up to a certain amount
of data from one channel to another with minimal memory copying:
fcopy infile outfile -onready command
fcopy infile outfile -single ?-size size?
The -onready option would be compatible with -command, though
probably not -size (although having -size pre-load the input buffer
when used in conjunction with -onready might offer some benefit, or
at least make [chan pending] more useful). In essence, the default
-onready command would simply invoke one execution of [fcopy]'s
existing data transfer code, which is exactly what the -single
option does. I haven't looked at the source to [fcopy], but I
wouldn't imagine it to be a particularly complex change.
semantics :-)
It does exactly what you call "-once mode".
No, it doesn't. Let me explain, with an example (VERY quick-and-dirty):
# test1.tcl:
set fid [socket localhost 9988]
puts $fid 12345678912345
flush $fid
after 10000
puts $fid 12345678912345
flush $fid
after 10000
close $fid
# test2.tcl
set fid [socket -server accept 9988]
proc accept {fid addr port} {
puts "Accepting $fid from $addr at $port"
fcopy $fid stderr -size 20 -command [list lappend ::forever]
puts "Waiting ..."
}
vwait forever
And the output...
(waiting for connection)
Accepting sock6 from 127.0.0.1 at 57498
Waiting ...
12345678912345
(waiting for more input)
12345
(finishes as expected)
That's only a crude example, and there may well be some interplay with
blocking and buffering settings that I'm not aware of. But based on
that, my reading of the doc's, and a moderate look over the source
code, I still stand by my original message.
That "waiting for more input" is the problem that prevents accurate
rate and progress reporting. And as mentioned in my original post,
you can't even observe the progress of the [fcopy] from outside (ie.
some kind of periodic event).
Thus, today's fcopy offers in a single "primitive" (in the sense that
it cannot be emulated at full efficiency in pure Tcl,
NOTHING that is written in decent quality C (or just about any other
optimised compiled language) can be "emulated at full efficiency in pure
Tcl". You know better than to use marketing phrases like that.
Anyhow... Shame on you.
- progress notification with configurable granularity
Yes, granularity is configurable. But it means trading off that
efficiency you mentioned for accuracy. That is what I'm getting at.
- bandwidth throttling with a simple [after] in the callback
Yes, I know the way it works. It's quite good enough for basic
bandwidth throttling. Again [fcopy] is fine as long as it's being used
as a background type operation.
The thing I'm getting at, is if you can tell [fcopy] to move whatever's
waiting (optionally limited by the -size value) and then return, you get
all the efficiency, plus a measure of control simply unavailable.
Think of progress reporting. By default, [fcopy] seems to move data in
4096 byte chunks. When data is really rocking, that's a good thing;
trying to move a couple MB in 20-byte chunks is going to have a bit of
an impact. And setting the -size limit higher causes great long stalls
when the traffic isn't moving so fast. Progress reporting on a telnet
session in 4096 byte chunks isn't going to a whole lot better.
The division of functionality I mentioned should have negligible impact
on efficiency, have absolutely no impact on existing usage, but make
using [fcopy] under unpredictable circumstances simple.
- all of this with the guarantee of never accumulating data in
memory if the input is faster than the output (a mistake that is easy
to make when doing it by hand)
Hence the reason I mentioned that issue several times in my original
post.
The above doesn't strike me as "seriously bodgy". I see it rather as a
well-designed compromise. (Notice that this doesn't weaken the
importance of fixing its current implementation bugs, which we are
actively analyzing right now. The point is that the API is good,
regardless of the quality of the underlying code at any given time).
You'll fit in just fine... There's that lack of imagination again. ;)
Although I can't post over there from the address I signed up with, I
have been watching the progress with some degree of interest. The API
is good, as you say, within the limits of how it's presented. My
assertion is that the code is almost all there to do a whole lot more.
From what I can tell, most of what would be required to realise my
idea is to re-organse the CopyData() (and perhaps one of two of its
associated functions) to factor out the translate-and-transfer, and
minimise buffering outside of TCL's existing IO system. Handling of
the extra functions, and making sure you don't get excessive -onready
invocations, is all that would be needed to round it out.
Ans as I also pointed out, the functionality present in [fcopy],
without the actual copying of data, is itself rather useful. The
-onready functionality is where most of the code in my response was
aimed. Just look at the code that makes up [fcopy]. You've got a
chunk of argument shaping and validation, a chunk of asynchronous flow
control, and then this tiny little bit of task-specific code
responsible for the actual data transfer. Most people think of [fcopy]
as a function for moving data from A to B. So why, then, if that's all
it is, is that function such a small part of the code involved?
Now if there's one practical situation that it doesn't handle, or does
awkwardly, please describe it. This may result in another attempt to
push the envelope, and we both enjoy that :-)
I believe I already did. Accurate timely progress reporting. [fcopy]
simply doesn't support it in a generically applicable fashion. Plus,
if you can do it in response to data being moved, it allows you to do
your calculations and what-not so that next time through the main loop
the data can be shipped out to its destination, rather than being
broken up into little chunks, fiddled across bit by bit, causing
excessive updates and what-not, re-accumulated, and then at some stage
sent off.
But besides what it doesn't do, is what it does. I've had to emulate
the asynchonous functionality already present in [fcopy] in several
projects (three that comes to mind), and have come across others that
have done it themselves.
Ultimately, I think [fcopy]s asynchronous functionality should be moved
into [fileevent] or something similar, and if it's really that
significant, its data transfer functionality highlights a place of
interest in optimising the existing IO handling; for example, perhaps
[read] with non-blocking mode could simply take the input channel's data
buffer (maybe after seeing if it can soak any more data out of the
system buffers) and return the buffer itself as a bytearray until
something decides to convert it into a "string", or hook it onto another
channel's output buffer, thus allowing a standard TCL script [read] and
[puts] to achieve the same data transfer efficiency as [fcopy]. (No
idea if that would actually work, through. ;) )
fileevent $in through $out puts $out -fromchan $in
Now, if that was all that was needed to achieve functionality as good
as the basic [fcopy] command, that would be impressive.
Fredderic
.
- Follow-Ups:
- Prev by Date: Re: Tcl Web Service with Authorization kills Tcl
- Next by Date: HTTP 1.1
- Previous by thread: XOTcl abstraction class
- Next by thread: Re: serving a file to a client --- background fcopy read fileevent event puts socket channel blocking nonblocking
- Index(es):
Relevant Pages
|
