PoC: Real Bandwidth Limiting with PHP

by apoc · februar 21 2009 · leave a comment

vBrowseFile uses a slightly modificated Version of HTTP_Download. HTTP_Download is a Pear-Module to send local Files to Browser:

Provides an interface to easily send hidden files or any arbitrary data to HTTP clients. HTTP_Download can gain its data from variables, files or stream resources.

It features:
- Basic caching capabilities
- Basic throttling mechanism
- On-the-fly gzip-compression
- Ranges (partial downloads and resuming)
- Delivery of on-the-fly generated archives through Archive_Tar and Archive_Zip
- Sending of PgSQL LOBs without the need to read all data in prior to sending

I added a ThrottleCallback functionality to HTTP_Download, a function that is executed after the BufferSize is transferred and the ThrottleDelay Time is waited. Modifications in HTTP/Download.php:

    var $throttleCallback = null;
    [...]
    function setThrottleCallback($callback)
    {
        $this->throttleCallback = $callback;
    }
    [...]
    function callThrottleCallback()
    {
        // execute callback
        if($this->throttleCallback != null)
        {
            $callback = $this->throttleCallback;
            $callback($this);
        }
    }
    [...]
    function sendChunk($chunk, $cType = null, $bound = null)
    {
    [...]
        // right after sleep is ex
        $this->callThrottleCallback();
    [...]
    }

This Callback could be used to track the download progress and to create a real bandwidth limitation. HTTP_Download supports a basic throttling mechanism to limit the bandwidth, but this only works for one connection, so e.g. you could limit 2 downloads to each 100 KiB/s but not all 2 downloads to each 50 KiB/s to limit the overall bandwidth to 100 KiB/s, instead it would be 200 KiB/s.
For this real bandwidth limiting you could use Apache modules like cband(seems not to work with php) or mod-bw, or you could use tc. Siyb from Geekosphere wrote an article about mod-bw: Limit Bandwidth per vHost in Apache2 (on Debian/Lenny)

Another possible way to limit the bandwidth is to implement it with HTTP_Download. The Theory is, to track all running downloads and change the BufferSize to := download limit * delay / active connections every seconds with the callback function from above.

More detailed:

We generate a random connection id(i call it rcid) for the tracking file.
ThrottleDelay is set to 1 seconds.
BufferSize is set to 100 KiB * 1 Seconds * 1024 to setup a maximum speed of 100KiB/s.
The Callback would now be executed every 1 Seconds:
- create or overwrite our own rcid file with time() inside.
- search for other rcid files and check there time-stamp. tracking files older then 5 seconds would be deleted.
- count all rcid tracking files. e.g. 2 means there 2 open download connections.
- change the BufferSize. e.g. for 2 open connections: (100 KiB * 1 Seconds * 1024) / 2

This would limit the overall bandwidth of all active connections to a maximum of 100 KiB/s with a delay of ~ <5 Seconds. Without any Apache Modules or other standalone traffic sharping tools. For small installations this should work with no problem but for bigger installations a more efficient traffic shaping tool would be more effective.

Example Implementation:

You need the patched Download.php File and the pear module must be installed. (Download)

<?php
/* This is an experimental HTTP_Download bandwidth limiting script
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 *
 * There many problems with this method:
 * - no gzip is possible, because of its headers 
 * (maybe with longer delay)
 * - to many connection problems must solved. 
 * (a max conn, and remove of trackfiles before start send)
 * - heavy really heavy server load because mutch hdd 
 * reading/writing, and get/put_contents/scandir not 
 * the fastest thing to do every second. also this is
 * not really optimized in any way.
 * - i guess many many more... ;)
 *
 * @license GPLv2
 * @author apoc <apoc@sixserv.org>
**/
 
require("Download.php");
 
$file = "path/to/some/big/file/for/testing";
 
$track_folder = "tracks/";
 
$bandwidth = 100; // in KiB/s
$throttle_delay = 1; // in seconds
$buffer_size = $bandwidth * 1024 * $throttle_delay;
$track_expire = 5; // time in seconds before a trackfile expires
 
$dl = &new HTTP_Download();
$dl->setFile($file);
$dl->setGzip(false);
$dl->setBufferSize($buffer_size);
$dl->setThrottleDelay($throttle_delay);
 
if(!is_dir($track_folder))
    mkdir($track_folder);
 
if(!is_dir($track_folder) || !is_writeable($track_folder))
    die("Could not write to track folder.");
 
// generate the rcid
$chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
do // okay lets make sure the file doesnt exists..
{
    $rcid = "";
    for($i = 0; $i < 10; $i++)
    {
        $rcid .= $chars[rand(0, strlen($chars)-1)];
    }
}
while(file_exists($track_folder."/".$rcid));
 
$dl->setThrottleCallback("tracking");
function tracking(&$dl)
{
    global $track_folder, $rcid, $bandwidth, $throttle_delay, $track_expire;
 
    // active tracked connections:
    $active_conns = 1;
 
    // search for other trackfiles
    foreach(scandir($track_folder) as $other_rcid)
    {
        if($other_rcid != "." && $other_rcid != ".." && $other_rcid != $rcid)
        {
            // load trackfile
            $last_seen = trim(file_get_contents($track_folder."/".$other_rcid));
 
            // older then <n> seconds? -> delete trackfile
            if(time() - $last_seen >= $track_expire)
                unlink($track_folder."/".$other_rcid);
            else // this looks like an active connection:
                $active_conns++;
        }
    }
 
    // write/overwrite own trackerdata
    file_put_contents($track_folder."/".$rcid, time());
 
    // now adjust buffer size
    $dl->setBufferSize( ($bandwidth * 1024 * $throttle_delay) / $active_conns );
}
 
$dl->setContentDisposition(HTTP_DOWNLOAD_ATTACHMENT, basename($file));
$dl->guessContentType();
$dl->send();
 
unlink($track_folder."/".$rcid);
 
?>

I know this is some sort of dirty hack to do, its just an experiment! Also this is not just for bandwidth limiting, for vBrowseFile is planned(all of this is an optional feature btw.) to monitor the downloads and create a traffic counter for each user.

Tagged: Coding, Networking, PHP, vbrowsefile

sixserv blog