Re: program to copy files - problems - unix ksh to java
From: Thomas Weidenfeller (nobody_at_ericsson.invalid)
Date: 02/11/04
- Next message: Michael Pitoniak: "Re: JDK 1.5 Tiger Layout Manager Problems anyone?"
- Previous message: Barry White: "Re: How to seperate data in a string?"
- In reply to: kaeli: "program to copy files - problems - unix ksh to java"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Wed, 11 Feb 2004 13:19:48 +0100
kaeli wrote:
> I've got a shell script (ksh) that does some file copying and deleting.
> I'm running into some problems with it that I'm wondering if I can
> solve. Since I plan on re-writing it with java (1.4), I figure I might
> as well do it right this time.
There are several ways to fix this (to a "good enough" level), without
using Java. One example:
> Here's the drill:
> Cron runs this code every 5 minutes.
> Program looks on one machine,
Check for the particular file. If found, rename the file to something
temporary - all in one operation. E.g. (Bourne-Shell syntax):
if [ mv "$file" "$file.$$" ] ; then
# Found file, renamed it.
# can start copying
# A second invocation will not find
# this file any more, and leave it alone.
> uses ssh to copy a file to another
fi
> machine, changes the filename, the owner and permissions (chmod), and
> then deletes the file from the source machine.
Delete the renamed file instead.
> Sounds simple enough...
It is. You might want to add a sanity check which e.g. runs once a day
and checks if there are old renamed files lying around and either
collect them, or delete them.
Other solutions include setting empty files as markers to indicate if a
file is already copied. But this can result in a race condition if you
start the script multiple times at the same time:
if [ ! -f "$file.mark" ] ; then
# race condition can happen here
# place a mark
touch "$file.mark"
# now copy
# after copy, delete
rm "$file" "$file.mark"
fi
Instead of setting the marker on the remote machine, you could also set
the marker on the local machine, but you would have to add the remote
host name in order to distinguish the markers.
Another solution would be to separate the script into two scripts. One
doing the copying, and another one checking if there is already a
copying script running for a particular remote machine. Have fun with ps
or pgrep.
> Problems:
> Large files (we're talking gigabytes) take more than 5 minutes to copy.
> Something is causing the file to be deleted before it has finished
> copying.
There is something else wrong. Try to find this "something" first. Most
likely it is the application writing the file, or there is something
wrong in your script. Unix is robust when it comes to the deletion of
files which are currently in use. A deletion during a copy should not
affect the copy.
> I was going to solve this with the usual .running type fix, but we
> really need the program to actually run every 5 minutes (more than one
> instance will be needed). If an instance is already copying the file,
> the file should just be ignored. The file should not be deleted if the
> copy hasn't finished.
You do check the exit code of the copy command, don't you?
> Does anyone know of any system stuff I should be looking at for java?
There is absolutely no need for Java. In fact, you will find that you
gain nothing by using Java, but that you will e.g. get problems in
setting the file owner and mode. You would have to invoke the Unix
commands from Java via exec(), or the system calls via JNI.
> Specifically, Unix Solaris interface so I can tell if a file is in use?
Java has no public platform interface, not even on Sun. You would have
to use exec() or JNI.
> Also, how can I make sure that the copy was finished before deleting the
> source?
Check the return code of the copy command.
May I suggest a good book for learning Unix scripting and a lot of other
Unix command-line tricks? "Unix Power Tools" by Peek, O'Reilly, and
Loukides.
> I expected the script to wait for the copy to finish before
> deleting, but it appears that it is not doing that. Should I use threads
> for this?
You already have concurrency problems, and you want to use threads to
move your concurrency problems to another level? I would not do this.
/Thomas
- Next message: Michael Pitoniak: "Re: JDK 1.5 Tiger Layout Manager Problems anyone?"
- Previous message: Barry White: "Re: How to seperate data in a string?"
- In reply to: kaeli: "program to copy files - problems - unix ksh to java"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
