Re: How to remove a single line from a flat file (Still very off-topic.)

Eric wrote:
But the fact that if you want to (say) edit the syslog file you have to turn off syslog (so it's not writing to the file while you edit it), that tells me you're missing something in the kernel.

If you need to edit it, it's not a log file.

Editting includes rotating the log file, removing old entries, etc.

So you want to allow infinite log files?

Sure, why not? NTFS does, as well as anything else where you can delete from the front of the file without messing up the back of the file. Look up, for example, how the USN journal is implemented. Each entry has a unique serial number, which is the lseek() offset in the journal file where the entry was written. Entries get deleted off the front, but that doesn't change where the later entries are - it just reclaims the space for entries that everyone has presumedly seen.

Anything that writes a log
should have a way of splitting "old" from "current"

And that's my point. Every program that writes a log file under UNIX needs a way to split the old from the current. If *every* program has to implement a mechanism for doing something with a file, wouldn't it make sense to put that in the file system?

As it is, it's fairly difficult to split a log file at a precise boundary, such as exactly at midnight, unless it's built into the program that happens to be writing the log file, and you know in advance that's what you want to do. (BTDT.)

Contrast with a file system that actually stores records you can delete. Simply open the log file, copy the records you want to "rotate", and delete them off the front of the log file as you go. You can even delete "normal" records and keep the "abnormal" records in the log file, for example. I've never seen, for example, anyone write a "SQL Table Rotate" program. You just select into another table, and delete from the original, and you rotate exactly the records you want to rotate.

Missing? If two processes need to change the same thing you need a
lock (type unspecified).

Yes. And that lock (or contention resolution) should be in the file system, rather than as a user-level work-around for a lack of functionality. I shouldn't have to lock the entire file to delete one message out of it. I shouldn't need 2G of disk space to delete data from a file holding 1G of data. Try working with a 40-gig tar file some time, trying to keep it synced to a directory tree somewhere, just for example. It takes way longer than syncing two file systems, because you have to frob 80 gig of data for every operation. As data gets bigger, this will get more and more painful over time, to the point where folks with big data are already reimplementing the file system in various ways to use better semantics.

If I could delete individual records from a file, sendmail could just append messages as new records, and my MUA could delete anything already there, and neither of us would have to explicitly lock the file. Of course, there would be locking of some sort inside the code implementing the file system, but that's where it belongs, so everyone can use it consistently.

Note that you can say "well, implement that sort of thing yourself", but then all your tools like grep and CRM114 and all that have to know your privileged file format.

If you think "work-around" you will do the wrong thing. Unix/Linux is
a low-level platform.

That's what I'm saying, yes. The things I mention are work-arounds for the fact that Linux is a low-level platform. You may not view them that way, but I've used enough FSes to view them that way.

I program mostly in Tcl, too, because C is a low-level platform. Anything I want to do in Tcl, I can obviously do in C also. That doesn't mean that C is as good as Tcl.

--
Darren New / San Diego, CA, USA (PST)
His kernel fu is strong.
He studied at the Shao Linux Temple.
.

Relevant Pages

  • RE: file i/o in multithreaded apps
    ... I'm seeing gaps in the log file using the code I posted - ... the output to a file and that doesn't seem to be missing lines. ... what to lock. ... Get notification to my posts through email? ...
    (microsoft.public.dotnet.framework)
  • Re: Minor HEADSUP - "New order" for newsyslog
    ... waits 10 seconds (after all signals are sent) ... are done in a safer order, and any single daemon is sent only ... one signal (instead of one signal for each log file that needed ... to rotate a large number of apache log files, ...
    (freebsd-current)
  • Re: Queue object with strange behaviour??
    ... my log file integrity analyzer said there ... private void LoggingWithThreads() ... lock (LoggingQueue) ... Thanks again Marc. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: logrotate Problem
    ... I have a Problem to create the log file. ... It says logrotate is controled be a cron job. ... # rotate log files weekly ... # RPM packages drop log rotation information into this directory ...
    (RedHat)
  • Re: Renaming an open file
    ... Christian wrote: ... > On linux, my program writes some stuff in a log file, which is rotated using ... > is renamed on the same file system? ... Rename doesn't do anything to the file. ...
    (comp.unix.programmer)