Re: serving a file to a client --- background fcopy read fileevent event puts socket channel blocking nonblocking

On Sat, 29 Mar 2008 05:03:50 -0700 (PDT),
Alexandre Ferrieux <alexandre.ferrieux@xxxxxxxxx> wrote:

On Mar 29, 2:08 am, vit...@xxxxxxxxx wrote:
How do you write a file to socket copy procedure that notifies me
about it's progress every so many bytes and does not block while
doing so.
Easy: use [fcopy -size ... -command ...]
The callback will of course have to repost the fcopy for the next
chunk until eof.

I've only ever used [fcopy] once, and now I remember why. Doesn't that
kind of usage strike anyone as being a seriously bodgy way to get
progress feedback, speed limiting, or, well, to do just about anything
other than pure automagical data movement...?

I tried using [ftell] on the inbound channel, from a periodic timer
that runs until the [fcopy -command] gets invoked, and it just reports
-1 all the time. That would at least make progress reporting possible.

I suppose, if we could tell how much data is waiting at the inbound
channel, we can cap it at some arbitrary value and use that as the -size
argument. But I don't know how to do that, and you'd still need your
own [fileevent] listener to run [fcopy].

Likewise for transfer rate limiting, there's no way to tell [fcopy] to
transfer one buffer full and finish so we can update our progress,
insert a pause, or whatever else. Perhaps a -once option would be
useful, but even then we still have to implement our own [fileevent]
handler.

And as soon as you start to dip into [fileevent] handling, you get the
problems I mentioned in my previous post. If the handler you're
listening to is on a channel that's significantly faster than the other
one, it'll spin like a mad man, so you need a means to listen to
whichever channel is slower. And just how big do you make that
transfer size? If your input stream is coming in sporadically, it'll
sit there waiting because the amount of data that came through in the
last burst was a single byte shorter than the size you specified. In
the meantime you could have posted the current progress, and/or done
your speed limiting calculations and associated adjustments.

I think personally, that being able to copy a single block (or as much
of one as is available), would make [fcopy] actually useful for
non-trivial functions. It's probably still not entirely optimal with
all the setup I suspect it's doing, but it's probably the best bet for
making it truly useful anytime soon.


Of course, it is possible to reimplement [fcopy] in pure Tcl, and it
even is a good exercise for whoever wants to learn about the event
loop. But this will never have the performance of [fcopy], which is
documented as "leverag[ing] the buffering in the Tcl I/O system to
avoid extra copies and to avoid buffering too much data in main memory
when copying large files to slow destinations like network sockets".

Alternatively, moving data from one channel to another is a fairly
common activity from what I've seen, both with and without processing
inbetwee. So I wonder if would be worth providing a means to create a
[fileevent] that only triggers when BOTH the input and output channels
are ready. (One-shot [fileevent]s would probably help there, we could
then post something lean and mean on the wiki to do the job, without
having to fiddle with continually setting and unsetting the [fileevent]
handlers.)

My choice, would be to split [fcopy] in half. An engine that invokes a
callback when there's both data waiting and somewhere to put it (output
buffers contain less than a block of data, for example), and a
mechanism for transferring up to a certain amount of data from one
channel to another with minimal memory copying:

fcopy infile outfile -onready command
fcopy infile outfile -single ?-size size?

The basic existing [fcopy] command could then be "expanded" as:

fcopy $in $out -onready [list fcopy $in $out -single]

The -onready option would be compatible with -command, though probably
not -size (although having -size pre-load the input buffer when used in
conjunction with -onready might offer some benefit, or at least
make [chan pending] more useful). In essence, the default -onready
command would simply invoke one execution of [fcopy]'s existing data
transfer code, which is exactly what the -single option does. I
haven't looked at the source to [fcopy], but I wouldn't imagine it to
be a particularly complex change.

Most importantly, rate limiting, progress reporting, and whatever else,
would grow quite readily out of that basic split in functionality.
Although for rate limiting you probably will want some means to "pause"
and/or cancel the handler (return value?), or a -continue option (or
something similar) that would normally be invoked internally by -single
(perhaps even just "-single -size 0" would be acceptible, albeit ugly).


Comments, anyone?


Fredderic

.

Relevant Pages

  • Re: serving a file to a client --- background fcopy read fileevent event puts socket cha
    ... I've only ever used [fcopy] once, ... progress feedback, speed limiting, or, well, to do just about anything ... I tried using on the inbound channel, ... Most importantly, rate limiting, progress reporting, and whatever else, ...
    (comp.lang.tcl)
  • Re: background fcopy and tracing a variable
    ... updated via another channel, while fcopy is in progress. ... Unfortunately, the command is not invoked, until fcopy has finished -- then ...
    (comp.lang.tcl)
  • Re: background fcopy and tracing a variable
    ... updated via another channel, while fcopy is in progress. ... Unfortunately, the command is not invoked, until fcopy has finished -- then ...
    (comp.lang.tcl)
  • background fcopy and tracing a variable
    ... I initiate a background fcopy and go into vwait. ... updated via another channel, while fcopy is in progress. ... Unfortunately, the command is not invoked, until fcopy has finished -- then ...
    (comp.lang.tcl)
  • Re: Clashed recordings / Sky+ 40GB
    ... In article, Richard Sobey writes ... can't remember the exact time or channel). ... Try something that you can't while two recordings are in progress, such as looking at the synopsis from the /i/ button during the EPG or even shifting the EPG beyond two days ahead. ...
    (uk.tech.tv.sky)