Re: How do I scp extremely large files

On Sat, 4 Feb 2012 18:03:22 -0800 (PST), Mike <mikesmith813@xxxxxxxxx>
wrote, quoted or indirectly quoted someone who said :

I need to copy extremely large files (30-100G) from a remove server to
a machine where my code will be running. I have code in place now
that uses an sftp connection to scan the file directory watching for
files to show up for me to copy. My question is how do I read in
chunks of these large files and write them out in chunks? I cannot
hold the bytes in memory obviously. Any help or pseudo code is
greatly appreaciated!

If your roll your own system, you might want to consider bundling the
small files and zipping as the Replicator does. You would have to
run a script on the server to unpack them. The Replicator only uploads
files that have changed. See
http://mindprod.com/webstart/replicator.html

If the files have only minor changes, you might consider hosting a
version control system on the server. The server can then check out
files that have changed. You get the advantage of atomic updates,
something you will not get with an FTP system. You also can check in
files from many sources directly to the server.

--
Roedy Green Canadian Mind Products
http://mindprod.com
One of the most useful comments you can put in a program is
"If you change this, remember to change ?XXX? too".

.

Relevant Pages

  • Re: Microkernel X11?
    ... Sometimes (i.e. with the Via driver) X actually freezes to where it ... Yes but I meant the server being implemented as such, ... When the drawing thread ... Can't drawing processes lock chunks ...
    (comp.os.minix)
  • Re: file transfer with sockets
    ... confirmation from client for each chunk before sending the next? ... to server, server interprets command, executes it, then simply returns ... files (hence the reading in chunks). ... sending the bytes over the stream. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Microkernel X11?
    ... Sometimes (i.e. with the Via driver) X actually freezes to where it ... Yes but I meant the server being implemented as such, ... When the drawing thread ... Can't drawing processes lock chunks ...
    (comp.os.minix)
  • Re: Large/unreliable file uploading over HTTP
    ... into their server to get large files, and with them being a print ... I am using a PHP5 Script which the $USER download and execute on her/his ... get the md5sum of the original file ... split the original file into chunks of e.g. 1 MByte ...
    (php.general)
  • Re: Client Server shared folder/file upload (security)
    ... I would split up the files into 64K chunks and send them in byte arrays to a web service and insert them into a staging tables in the database. ... my server code would "commit" the data in the staging tables to the production tables. ... client, so they can't see database connection and other setting. ...
    (microsoft.public.dotnet.distributed_apps)