org.apache.manifoldcf.crawler.connectors.webcrawler

Interface IThrottledConnection

All Known Implementing Classes:

ThrottledFetcher.ThrottledConnection

public interface IThrottledConnection

This interface represents an established connection to a URL.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	_rcsid
static int	FETCH_BAD_URI
static int	FETCH_CIRCULAR_REDIRECT
static int	FETCH_INTERRUPTED
static int	FETCH_IO_ERROR
static int	FETCH_NOT_TRIED
static int	FETCH_SEQUENCE_ERROR
static int	FETCH_UNKNOWN_ERROR

Method Summary

Modifier and Type	Method and Description
void	beginFetch(java.lang.String fetchType) Begin the fetch process.
void	close() Close the connection.
void	destroy() Destroy the connection.
void	doneFetch(org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities) Done with the fetch.
void	executeFetch(java.lang.String urlPath, java.lang.String userAgent, java.lang.String from, boolean redirectOK, java.lang.String host, FormData formData, LoginCookies loginCookies) Execute the fetch and get the return code.
LoginCookies	getLastFetchCookies() Get the last fetch cookies.
java.lang.String	getLimitedResponseBody(int maxSize, java.lang.String encoding) Get limited response as a string.
java.io.InputStream	getResponseBodyStream() Get the response input stream.
int	getResponseCode() Get the http response code.
java.lang.String	getResponseHeader(java.lang.String headerName) Get a specified response header, if it exists.
java.util.Map<java.lang.String,java.util.List<java.lang.String>>	getResponseHeaders() Get response headers
boolean	hasExpired(long currentTime) Check whether the connection has expired.
void	noteInterrupted(java.lang.Throwable e) Note that the connection fetch was interrupted by something.
void	setAbortChecker(AbortChecker abortCheck) Set the abort checker.

Field Detail

_rcsid

static final java.lang.String _rcsid

See Also:

Constant Field Values

FETCH_NOT_TRIED

static final int FETCH_NOT_TRIED

See Also:

Constant Field Values

FETCH_CIRCULAR_REDIRECT

static final int FETCH_CIRCULAR_REDIRECT

See Also:

Constant Field Values

FETCH_BAD_URI

static final int FETCH_BAD_URI

See Also:

Constant Field Values

FETCH_SEQUENCE_ERROR

static final int FETCH_SEQUENCE_ERROR

See Also:

Constant Field Values

FETCH_IO_ERROR

static final int FETCH_IO_ERROR

See Also:

Constant Field Values

FETCH_INTERRUPTED

static final int FETCH_INTERRUPTED

See Also:

Constant Field Values

FETCH_UNKNOWN_ERROR

static final int FETCH_UNKNOWN_ERROR

See Also:

Constant Field Values

Method Detail

setAbortChecker

void setAbortChecker(AbortChecker abortCheck)

Set the abort checker. This must be done before the connection is actually used.

hasExpired

boolean hasExpired(long currentTime)

Check whether the connection has expired.

Parameters:

currentTime - is the current time to use to judge if a connection has expired.

Returns:

true if the connection has expired, and should be closed.

beginFetch

void beginFetch(java.lang.String fetchType)
         throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Begin the fetch process.

Parameters:

fetchType - is a short descriptive string describing the kind of fetch being requested. This is used solely for logging purposes.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

executeFetch

void executeFetch(java.lang.String urlPath,
                  java.lang.String userAgent,
                  java.lang.String from,
                  boolean redirectOK,
                  java.lang.String host,
                  FormData formData,
                  LoginCookies loginCookies)
           throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                  org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Execute the fetch and get the return code. This method uses the standard logging mechanism to keep track of the fetch attempt. It also signals the following conditions: ServiceInterruption (if a dynamic error occurs), or ManifoldCFException if a fatal error occurs, or nothing if a standard protocol error occurs. Note that, for proxies etc, the idea is for this fetch request to handle whatever redirections are needed to support proxies.

Parameters:

urlPath - is the path part of the url, e.g. "/robots.txt"

userAgent - is the value of the userAgent header to use.

from - is the value of the from header to use.

redirectOK - should be set to true if you want redirects to be automatically followed.

host - is the value to use as the "Host" header, or null to use the default.

formData - describes additional form arguments and how to fetch the page.

loginCookies - describes the cookies that should be in effect for this page fetch.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseCode

int getResponseCode()
             throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                    org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get the http response code.

Returns:

the response code. This is either an HTTP response code, or one of the codes above.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getLastFetchCookies

LoginCookies getLastFetchCookies()
                          throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                 org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get the last fetch cookies.

Returns:

the cookies now in effect from the last fetch.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseHeaders

java.util.Map<java.lang.String,java.util.List<java.lang.String>> getResponseHeaders()
                                                                             throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                                                                    org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get response headers

Returns:

a map keyed by header name containing a list of values.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseHeader

java.lang.String getResponseHeader(java.lang.String headerName)
                            throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                   org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get a specified response header, if it exists.

Parameters:

headerName - is the name of the header.

Returns:

the header value, or null if it doesn't exist.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getResponseBodyStream

java.io.InputStream getResponseBodyStream()
                                   throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                          org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get the response input stream. It is the responsibility of the caller to close this stream when done.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

getLimitedResponseBody

java.lang.String getLimitedResponseBody(int maxSize,
                                        java.lang.String encoding)
                                 throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

Get limited response as a string.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

org.apache.manifoldcf.agents.interfaces.ServiceInterruption

noteInterrupted

void noteInterrupted(java.lang.Throwable e)

Note that the connection fetch was interrupted by something.

doneFetch

void doneFetch(org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities)
        throws org.apache.manifoldcf.core.interfaces.ManifoldCFException

Done with the fetch. Call this when the fetch has been completed. A log entry will be generated describing what was done.

Throws:

org.apache.manifoldcf.core.interfaces.ManifoldCFException

close

void close()

Close the connection. Call this to return the connection to its pool.

destroy

void destroy()

Destroy the connection. Call this to close the connection.