Re: [FAQ] How do I retrieve a page from a web site? (Was: PHP behaving like a user using a browser)



Q: How do I retrieve a page from a web site?
A: Pass a URL to file() or file_get_contents(). The former returns the
contents as an array of lines. The latter returns the same as string.

Example:

$html = file_get_contents('http://www.example.com/');

Q: How do I retrieve a page from a web site that does browser
detection?
A: Use ini_set() to change the configuration option "user_agent." This
sets the User-Agent header sent by PHP.

Example:

ini_set('user_agent', 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1)');
$html = file_get_contents('http://www.example.com/');

Q: How do I retrieve a page from a web site that requires a cookie?
A: Use stream_context_create() to create a HTTP context with Cookie as
one of the headers. Then, if you are coding in PHP 5, pass the context
to file() or file_get_contents() as the third parameter. In PHP 4
either function accepts a context, so you need to open the URL with
fopen() and retrieve the data a chunk at a time with fread().

Example:

$opts = array(
'http'=>array(
'method'=> 'GET',
'header'=>
"Accept-language: en\r\n" .
"Cookie: foo=bar\r\n"
)
);

$context = stream_context_create($opts);
$f = fopen($url, "rb", false, $context);
while($data = fread($f, 1024)) {
echo $data;

}

stream_context_create() is available in PHP 4.3.0 and above. If you are
using an older version, you would need the cURL functions or use
fsockopen() to open the connection and send the cookie header with
fputs().

Example 1:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER,
array('Cookie: foo=bar'));
curl_exec($ch);
curl_close($ch);

Example 2:

$fp = fsockopen($host, $port);
fputs($fp, "GET / HTTP/1.0\r\n");
fputs($fp, "Host: $host\r\n");
fputs($fp, "Cookie: foo=bar\r\n\r\n");

while ($data = fgets($fp, 1024)) {
echo $data;
}

Refer:
http://www.php.net/curl
http://curl.haxx.se/libcurl/php/examples/?ex=cookiejar.php

Q. My PHP application retrieves a page from a web site that uses
cookies. How do I get the cookie?
A. Use fopen() to open a connection to the server, then call
stream_get_meta_data() to obtain information about the connection. The
function returns an associative array. The 'wrapper_data' element holds
an array containing the HTTP response headers. Loop through it and
parse the string that begins with "Set-Cookie."

Refer:
http://www.phpclasses.org/httpclient
http://pear.php.net/package/HTTP_Client

++++++
@revision 2 Janwillem Borleffs added other examples
@revision 3 URLs changed to www.example.com. Added reference links. Few
double quotes quickly converted to single quotes
@todo Cleanup. Trim

.



Relevant Pages

  • Re: SCP backup script not recursing
    ... > I retrieve this list of files into an array using this ... Unsure why you are using php? ...
    (comp.unix.shell)
  • Re: SCP backup script not recursing
    ... >> Im trying to write a scriptthat backs up a directory. ... >> I retrieve this list of files into an array using this ... The company I work for wants everything PHP. ...
    (comp.unix.shell)
  • Re: Simple PHP question
    ... I want to be able to build a static array with the values hardcoded into the array, and then be able to randomly select an item from the array and retrieve the image, name and description. ... I am new to PHP, but have been programming C/C++ for over 10 years. ... std::string PathNameconst; ...
    (comp.lang.php)
  • [PHP 5.2.8] Retrieve cookies from remote host and pass them back to client.
    ... I have a PHP 5.2.8 web application that needs to retrieve a resource ... the remote host also sends a cookie back ... it's grabbing the cookie from the remote URL that is ...
    (comp.lang.php)
  • filter_input(INPUT_COOKIE) problems
    ... I've started using the excellent new filter functionality in PHP 5.2.1. ... In my humble opinion they are a great addition to PHP. ... I do though have a problem with filter_input(INPUT_COOKIE) when the cookie value is in an array. ...
    (php.general)