Re: Hello: not so much php, more like HTML..?

The Natural Philosopher wrote:
Gary L. Burnore wrote:
On Sat, 01 Sep 2007 21:26:46 +0200, "Rik Wasmus"
<luiheidsgoeroe@xxxxxxxxxxx> wrote:

On Sat, 01 Sep 2007 20:39:06 +0200, The Natural Philosopher <a@xxx> wrote:

Rik Wasmus wrote:
On Sat, 01 Sep 2007 13:13:33 +0200, Rik Wasmus <luiheidsgoeroe@xxxxxxxxxxx> wrote:

On Sat, 01 Sep 2007 11:42:59 +0200, The Natural Philosopher <a@xxx> wrote:
If you really want a chunked upload check out the user comments at <http://nl3.php.net/manual/en/function.readfile.php>

Are you dyslexic, I want to DOWNLOAD. Upload I have done already.

bah, 't was late last night... I meant to say download, allthough it depends wether you think usercentric or usercentric which is which :P
Hmmmz, haven't made a full recovery yet I see, "usercentric or servercentric"...

its Saturday, Get a bottle of decent spirits and relax. ;-)

Anyway I have enough onfo op spec out that part of e jobn. Muy necxt problem is wheher
Hehe, taking you own advice? :P

its more efficient to have 5000 files all called 00001, 00002...05000 in one directory, or whether to split them up over several..and whether to keep their names and extensions intact, or just be lazy, knowing the data base knows what they were called.
Hmm, depends on the file system, not really an expert there. Jerry would tell you to just serve them up straight from the database, and forget about the filesystem, I'm not so sure :). You can do a little testing offcourse, see what works best.

The more files you store in one directory, the harder the OS has to
work to list through them.
Ah, but the more subdirectories you have in a file system, the more work...

;-)

I.e. I think that the case where each file is in its own subdirectory is of similar order to no subdirs at all.

I suspect the answer is that for n files, us the pth root of n as the number of subdirs, where p is the depth of the subdirs...but a lot depends on caching algorithms in the directories. AND the way the OS searches them.

I don't know what a directory entry is these days..256 bytes? well 10,000 of them is only 2.56Mytes or so. Should be well within cache range. let's say it has to do maybe 1000 machine cycles for every byte...thats 500K bytes a second searched at 500Mhz..4 seconds for a linear search to find the last one. Mm. That's significant.

whereas a two level hierearchy? a 100 dirs in the first, and 100 files in each? 80 milliseconds. Give or take...starting to get into disk latency times anyway..

well I have no idea how the OS searches the directory, but it seems to me that a hundred dirs of a hundred files each has to be the way to go.

Shouldn't be that hard either: just use an autoincrement on each tag record, and every time it gets to modulo one hundred create a new directory.




further info: Post some research EXT3 filesystems can have 'dir_index' set which adds a hashed Btree into the equation. Mine appear to have this..i.e a lot of what splitting the big directory into smaller ones would do, is already done by this mechanism.

I suspect this means that retrieval time using the database itself, and a ext3 hashed file structure, are pretty similar.

Ergo the answers would seem to be

- On a hashed treed fileystem there is little advantage to going to subdirs provided you KNOW THE FILE YOU WANT.
- Access to a database LONGBLOB is probably no faster, and not much slower.
- however the real issue is the file storage time into the database itself. And how PHP might cope with super large 'strings'

Here is an interesting fragment..complete with at least one error..they don't delete the temporary file..

http://www.php-mysql-tutorial.com/php-mysql-upload.php

I shudder to think how long adding slashes to a 60Mbyte binary image might take, or indeed how much virtual memory holding it in a 60Mbyte php string might use..

Anyone ever handled BLOB objects that large, into a Mysql database?

I guess it's test time...it WOULD be kinda nice to store it all in the database.

Lets see..whats an A0 image..48x24 inches - at 24 bit 300dpi? Hmm Uncompressed, about 60Mbyte..

I guess I could set a limit, and force them to compress down to 10Mbyte or so..

PHP appears to set no intrinsic limit on string sizes, so reading the whole thing into a string is at least feasible in theory. What worries me is the addslashes bit. all of which is then removed by the mysql API..yuk. so instead of a byte for byte copy, there has to be a 'read file into ram, create new slashed string and deallocate memory from first, give to sql, create de-slashed BLOB in RAM, write to ISAM file, deallocate memory,do garbage..etc etc..

What would be better is to write to a BLOB from a file directly.

LOAD DATA might work..not quite sure how..insert a record, and load data into just the blob?

The MySQL server also has restrictions on the maxiumum data size it can accept over the socket ..hmm. Mine's latest and is 16Mbyte..so thats OK..configurable..upwards..

Oh..I think this may work and avoid all this addslashing nonsense

$query= ("UPDATE file_table SET file_contents=LOAD_FILE('".$tmpfilname."') where id=$id");

Any comments?




.

Relevant Pages

  • Re: python an sqlite objects
    ... Just pickle the object and save the resulting string. ... Be sure to save it as BLOB, ... Suppose you have serialized your object as Python bytestring. ... put arbitrary serialized strings into the database. ...
    (comp.lang.python)
  • Re: PrintWriter as blob with mysql
    ... PrintWriter pw = new PrintWriter(new ... then i "insert" it in a blob field in a mysql table ... You don't really show us your code, so some parts of our responses must be speculative, however the field contents after you "insert" your PrintWriter show that it contains a string, not a Writer at all. ... It could be that your way of determining the contents is actually returning the default toStringof the PrintWriter or Blob object of interest, not the actual contents of the database field. ...
    (comp.lang.java.help)
  • Re: A little Rolodex [revised]
    ... including alpha sort and searching for any embedded string, ... In this application, a database is a directory, ... you may optionally provide any alternate UCASE program, ... NN -> first store NN as key length ...
    (comp.sys.hp48)
  • Re: return multiple rows from sql statement
    ... strings from input values is almost certainly a safe path to SQL ... Also, being a MySQL function, it knows what MySQL needs or uses. ... All characters that are entered in the fields make their way into the database unaltered. ... The insert of what surprisinlgly was NOT a syntax error, but a string called "mysql_insert_id" into an integer field resulted in the value zero being put in. ...
    (comp.lang.php)
  • Re: Invalid variant type conversion
    ... because within regular programming, there would be no agreement upon what ... applications built by standard Delphi controls, ... NULL is the same as "" in a database. ... >> The empty string tells it all. ...
    (comp.lang.pascal.delphi.misc)