Traversing bytes of a 500 MB file hosted remotely on a server in PHP
I have a large file hosted on a remote server (500 MB). This file contains identifiable chunks of bytes embedded using AMF3. These chunks are identifiable by a pre-defined string prefix that I've set ahead of time. In this case, I am appending the string 'prefix' into the file at various points during creation.
I want to traverse the entire length of the file using PHP, find the exact position of these string prefixes without copying the file locally on the server, but I am having problems. Right now, I am using a simple file_get_contents with an HTTP URL pointing to the file on the remote server like so:
$file = file_get_contents('http://remote.server.com/file.xxx')
while($offset = strpos($file, 'prefix', $offset + 1)){
//find prefix string value here using regex
// store the position of the value somewhere
}
Unfortunately, it doesn't work too well on really large sized files and I get a 500 Internal Server Error. Is there a better way to traverse through the bytes of an entire file hosted remotely without first copying the file locally?
这非常强烈
$file = fopen('http://remote.server.com/file.xxx');
$contents = '';
while (!feof($file )) {
$contents .= fread($file , 8192);
$found = strpos($contents , 'prefix') ;
if ($found > 0)
{
//do your thing
$contents = substr($contents,$found,8192) ;
}
}
You could use fopen() and then read the file in blocks, rather than reading the whole thing, searching for the prefix strings as you go. But this is likely to get tricky if the prefix happens to straddle a block boundary.