| HTTP::Proxy - A pure Perl HTTP proxy |
HTTP::Proxy - A pure Perl HTTP proxy
use HTTP::Proxy;
# initialisation
my $proxy = HTTP::Proxy->new( port => 3128 );
# alternate initialisation
my $proxy = HTTP::Proxy->new;
$proxy->port( 3128 ); # the classical accessors are here!
# this is a MainLoop-like method
$proxy->start;
This module implements a HTTP proxy, using a HTTP::Daemon to accept client connections, and a LWP::UserAgent to ask for the requested pages.
The most interesting feature of this proxy object is its ability to filter the HTTP requests and responses through user-defined filters.
Once the proxy is created, with the new() method, it is possible
to alter its behaviour by adding so-called ``filters''. This is
done by the push_filter() method. Once the filter is ready to
run, it can be launched, with the start() method. This method
does not normally return until the proxy is killed or otherwise
stopped.
An important thing to note is that the proxy is (except when running
the NoFork engine) a forking proxy: it doesn't support passing
information between child processes, and you can count on reliable
information passing only during a single HTTP connection (request +
response).
You can alter the way the default HTTP::Proxy works by plugging callbacks (filter objects, actually) at different stages of the request/response handling.
When a request is received by the HTTP::Proxy object, it is filtered through
a standard filter that transform this request accordingly to RFC 2616
(by adding the Via: header, and a few other transformations). This is
the default, bare minimum behaviour.
The response is also filtered in the same manner. There is a total of four
filter chains: request-headers, request-body, reponse-headers and
response-body.
You can add your own filters to the default ones with the
push_filter() method. The method pushes a filter on the appropriate
filter stack.
$proxy->push_filter( response => $filter );
The headers/body category is determined by the base class of the filter.
There are two base classes for filters, which are
HTTP::Proxy::HeaderFilter and HTTP::Proxy::BodyFilter (the names
are self-explanatory). See the documentation of those two classes
to find out how to write your own header or body filters.
The named parameter is used to determine the request/response part.
It is possible to push the same filter on the request and response stacks, as in the following example:
$proxy->push_filter( request => $filter, response => $filter );
If several filters match the message, they will be applied in the order they were pushed on their filter stack.
Named parameters can be used to create the match routine. They are:
method - the request method
scheme - the URI scheme
host - the URI authority (host:port)
path - the URI path
query - the URI query string
mime - the MIME type (for a response-body filter)
The filters are applied only when all the the parameters match the request or the response. All these named parameters have default values, which are:
method => 'OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT'
scheme => 'http'
host => ''
path => ''
query => ''
mime => 'text/*'
The mime parameter is a glob-like string, with a required /
character and a * as a joker. Thus, */* matches all responses,
and "" those with no Content-Type: header. To match any
reponse (with or without a Content-Type: header), use undef.
The mime parameter is only meaningful with the response-body
filter stack. It is ignored if passed to any other filter stack.
The method and scheme parameters are strings consisting of
comma-separated values. The host and path parameters are regular
expressions.
A match routine is compiled by the proxy and used to check if a particular request or response must be filtered through a particular filter.
It is also possible to push several filters on the same stack with the same match subroutine:
# convert italics to bold
$proxy->push_filter(
mime => 'text/html',
response => HTTP::Proxy::BodyFilter::tags->new(),
response => HTTP::Proxy::BodyFilter::simple->new(
sub { ${ $_[1] } =~ s!(</?)i>!$1b>!ig }
)
);
For more details regarding the creation of new filters, check the
HTTP::Proxy::HeaderFilter and HTTP::Proxy::BodyFilter documentation.
Here's an example of subclassing a base filter class:
# fixes a common typo ;-)
# but chances are that this will modify a correct URL
{
package FilterPerl;
use base qw( HTTP::Proxy::BodyFilter );
sub filter {
my ( $self, $dataref, $message, $protocol, $buffer ) = @_;
$$dataref =~ s/PERL/Perl/g;
}
}
$proxy->push_filter( response => FilterPerl->new() );
Other examples can be found in the documentation for
HTTP::Proxy::HeaderFilter, HTTP::Proxy::BodyFilter,
HTTP::Proxy::HeaderFilter::simple, HTTP::Proxy::BodyFilter::simple.
# a simple anonymiser
# see eg/anonymiser.pl for the complete code
$proxy->push_filter(
mime => undef,
request => HTTP::Proxy::HeaderFilter::simple->new(
sub { $_[0]->remove_header(qw( User-Agent From Referer Cookie )) },
),
response => HTTP::Proxy::HeaderFilter::simple->new(
sub { $_[0]->remove_header(qw( Set-Cookie )); },
)
);
IMPORTANT: If you use your own LWP::UserAgent, you must install it
before your calls to push_filter(), otherwise
the match method will make wrong assumptions about the schemes your
agent supports.
NOTE: It is likely that possibility of changing the agent or the daemon may disappear in future versions.
new()new() method creates a new HTTP::Proxy object. All attributes can
be passed as parameters to replace the default.
Parameters that are not HTTP::Proxy attributes will be ignored and
passed to the chosen HTTP::Proxy::Engine object.
init()init() initialise the proxy without starting it. It is usually not
needed.
This method is called by start() if needed.
push_filter()push_filter() method is used to add filters to the proxy.
It is fully described in section FILTERS.
The HTTP::Proxy has several accessors and mutators.
Called with arguments, the accessor returns the current value. Called with a single argument, it sets the current value and returns the previous one, in case you want to keep it.
If you call a read-only accessor with a parameter, this parameter will be ignored.
The defined accessors are (in alphabetical order):
Client-Aborted, Client-Bad-Header-Line, Client-Date,
Client-Junk, Client-Peer, Client-Request-Num,
Client-Response-Num, Client-SSL-Cert-Issuer,
Client-SSL-Cert-Subject, Client-SSL-Cipher, Client-SSL-Warning,
Client-Transfer-Encoding, Client-Warning).
They are removed by the filter HTTP::Proxy::HeaderFilter::standard from the request and response objects received by the proxy.
If a filter (such as a SSL certificate verification filter) need to access them, it must do it through this accessor.
Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization,
TE, Trailers, Transfer-Encoding, Upgrade).
They are removed by the filter HTTP::Proxy::HeaderFilter::standard from the request and response objects received by the proxy.
If a filter (such as a proxy authorisation filter) need to access them, it must do it through this accessor.
This means that by default, the proxy answers only to clients on the
local machine. You can pass a specific interface address or ""/undef
for any interface.
This default prevents your proxy to be used as an anonymous proxy by script kiddies.
HTTP::Proxy. Methods are grouped by type. Known method groups are:
HTTP, WebDAV and DeltaV.
Called with an empty list, this method will return all known methods.
This method is case-insensitive, and will carp() if an unknown
group name is passed.
Here are the various elements that can be added to the mask (their values are powers of 2, starting from 0 and listed here in ascending order):
NONE - Log only errors
PROXY - Proxy information
STATUS - Requested URL, reponse status and total number
of connections processed
PROCESS - Subprocesses information (fork, wait, etc.)
SOCKET - Information about low-level sockets
HEADERS - Full request and response headers are sent along
FILTERS - Filter information
DATA - Data received by the filters
CONNECT - Data transmitted by the CONNECT method
ENGINE - Engine information
ALL - Log all of the above
If you only want status and process information, you can use:
$proxy->logmask( STATUS | PROCESS );
Note that all the logging constants are not exported by default, but
by the :log tag. They can also be exported one by one.
This method is currently delegated to the HTTP::Proxy::Engine object.
maxchild is deprecated and will disappear.
maxconn is deprecated.
Note: max_connections will be deprecated soon, for two reasons: 1)
it is more of an HTTP::Proxy::Engine attribute, 2) not all engines will
support it.
MaxRequestsPerChild in Apache)
maxserve is deprecated.
HTTP::Daemon port (default: 8080).
undef until the proxy actually receives the beginning
of a response from the origin server.
If one of the request filters sets this attribute, it ``short-circuits'' the request/response scheme, and the proxy will return this response (which is NOT filtered through the response filter stacks) instead of the expected origin server response. This is useful for caching (though Squid does it much better) and proxy authentication, for example.
The stash() method can be used to set the whole hash (with a HASH reference).
To access individual keys simply do:
$proxy->stash( 'bloop' );
To set it, type:
$proxy->stash( bloop => 'owww' );
It's also possibly to get a reference to the stash:
my $s = $filter->proxy->stash();
$s->{bang} = 'bam';
# $proxy->stash( 'bang' ) will now return 'bam'
Warning: since the proxy forks for each TCP connection, the data is only shared between filters in the same child process.
$hostname (HTTP::Proxy/$VERSION))
X-Forwarded-For: header.
(default: true)
start()MainLoop: you hand over control to the
HTTP::Proxy object you created and configured.
If maxconn is not zero, start() will return after accepting
at most that many connections. It will return the total number of
connexions.
serve_connections()
$message at the end of logfh, if $level matches logmask.
The log() method also prints a timestamp.
The output looks like:
[Thu Dec 5 12:30:12 2002] ($$) $prefix: $message
where $$ is the current processus id.
If $message is a multiline string, several log lines will be output,
each line starting with $prefix.
This method is only used internaly.
It is essential to allow HTTP::Proxy users to create ``pseudo-schemes'' that LWP doesn't know about, but that one of the proxy filters can handle directly. New schemes are added as follows:
$proxy->init(); # required to get an agent
$proxy->agent->protocols_allowed(
[ @{ $proxy->agent->protocols_allowed }, 'myhttp' ] );
new_connection()HTTP::Proxy::Engine objects.
HTTP::Proxy has several Apache-like attributes that control the
way the HTTP and TCP connections are handled.
The following attributes control the TCP connection. They are passed to
the underlying HTTP::Proxy::Engine, which may (or may not) use them
to change its behaviour.
Those attributes control the HTTP connection:
No symbols are exported by default. The :log tag exports all the
logging constants.
This module does not work under Windows, but I can't see why, and do not have a development platform under that system. Patches and explanations very welcome.
I guess it is because fork() is not well supported.
$proxy->maxchild(0);
Several people have tried to help, but we haven't found a way to make it work correctly yet.
As from version 0.16, the default engine is HTTP::Proxy::Engine::NoFork.
Let me know if it works better.
the HTTP::Proxy::Engine manpage, the HTTP::Proxy::BodyFilter manpage, the HTTP::Proxy::HeaderFilter manpage, the examples in eg/.
Philippe ``BooK'' Bruhat, <book@cpan.org>.
The module has its own web page at http://http-proxy.mongueurs.net/ complete with older versions and repository snapshot.
There are also two mailing-lists: http-proxy@mongueurs.net for general
discussion about HTTP::Proxy and http-proxy-cvs@mongueurs.net for
CVS commits emails.
Many people helped me during the development of this module, either on mailing-lists, IRC or over a beer in a pub...
So, in no particular order, thanks to the libwww-perl team for such a
terrific suite of modules, perl-qa (tips for testing), the French Perl
Mongueurs (for code tricks, beers and encouragements) and my growing
user base... ;-)
I'd like to particularly thank Dan Grigsby, who's been using
HTTP::Proxy since 2003 (before the filter classes even existed). He is
apparently making a living from a product based on HTTP::Proxy. Thanks
a lot for your confidence in my work!
Copyright 2002-2008, Philippe Bruhat.
This module is free software; you can redistribute it or modify it under the same terms as Perl itself.
| HTTP::Proxy - A pure Perl HTTP proxy |