Re: Data transfer problem - ideas/solutions wanted (please)



NC wrote:

E.T. Grey wrote:

I have a (LARGE) set of historical data that I want to keep
on a central server, as several separate files.


How large exactly?

At last count, there are about 65,000 distinct files (and increasing)



I want a client process to be able to request the data in a
specific file by specifying the file name, start date/time and
end date/time.


The start/end date/time bit actually is a rather fat hint that you
should consider using a database... Searching through large files will
eat up enormous amounts of disk and processor time.


Not necessarily true. Each file has the equivalent of approx 1M rows (yes - thats 1 million) - yet the binary files (which use compression algos) are approx 10k-15K in size. If you multiply the number of rows (on avg) by the number of files - you can quickly see why using a db as a repository would be a poor design choice.


New data will be appended to these files each day, by a
(PHP) script.


Yet another reason to consider a database...


See above

What is the best (i.e. most efficient and fast way) to transfer data
from the server to clients ?.


Assuming you are using HTTP, compressed (gzip) CSV will probably be the
fastest.


This involves converting the read data to a string first, before (possibly) zipping it and sending it. This incurrs overhead (that I would like to avoid) on both server and client.

How can I insure that that the (binary?) data sent from the Unix server
can be correctly interpreted at the client side?


Why should the data be binary? Compressed CSV is likely to be at least
as compact as binary data, plus CSV will be human-readable, which
should help during debugging.



See above

How can I prevent clients from directly accessing the files
(to prevent malicious or accidental corruption of the data files.?


Import them into a database and lock the originals in a safe place.

Cheers, NC


.

Relevant Pages

  • Re: Converting the time from one timezone to another
    ... Carry out some calculation at the server using a date/time pasesed ... from the client that represents the a point in time at the client ... For those who haven't caught up yet, UTC ...
    (microsoft.public.dotnet.framework)
  • Re: How to pass GETDATE() as SqlParameter
    ... would like to ensure that date/time is accurate by using the date/time of the ... CURRENT_TIMESTAMP), runs at the server. ... Best way is to create a stored procedure that you call from the client. ... IF @TheDate IS NULL ...
    (microsoft.public.sqlserver.clients)
  • Re: Converting the time from one timezone to another
    ... add/subtract from UTC that might be something like this: ... Carry out some calculation at the server using a date/time pasesed ... from the client that represents the a point in time at the client ...
    (microsoft.public.dotnet.framework)
  • Re: Reading file from Client instead of Server
    ... How are you converting the CSV to DataTable? ... Webform that allows the user to find their CSV file, and I read that CSV file ... I didn't want to manage these files on the Server when I was done with them. ... You would need to set special security on each client ...
    (microsoft.public.dotnet.framework.aspnet)
  • Re: DateTime fundamentally broken
    ... No, it is returning the current date/time OF the server, converted to UTC, ... DateTime.UtcNow method, the date/time was in terms of UTC. ... > Is it not returning the UTC time at the client? ...
    (microsoft.public.dotnet.framework.webservices)