Package org.apache.manifoldcf.crawler.system

Class CrawlerAgent

java.lang.Object

org.apache.manifoldcf.crawler.system.CrawlerAgent

All Implemented Interfaces:

IAgent

public class CrawlerAgent
extends java.lang.Object
implements IAgent

This is the main agent class for the crawler.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	_rcsid
protected AssessmentThread	assessmentThread
protected DocumentCleanupStufferThread	cleanupStufferThread
protected DocumentCleanupThread[]	cleanupThreads
protected DocumentDeleteStufferThread	deleteStufferThread
protected DocumentDeleteThread[]	deleteThreads
protected DocCleanupResetManager	docCleanupResetManager	Cleanup thread pool reset manager
protected DocDeleteResetManager	docDeleteResetManager	Delete thread pool reset manager
protected ExpireStufferThread	expireStufferThread
protected ExpireThread[]	expireThreads
protected FinisherThread	finisherThread
protected HistoryCleanupThread	historyCleanupThread
protected IdleCleanupThread	idleCleanupThread
protected JobDeleteThread	jobDeleteThread
protected JobResetThread	jobResetThread
protected JobStartThread	jobStartThread
protected float	lowWaterFactor
protected JobNotificationThread	notificationThread
protected int	numCleanupThreads
protected int	numDeleteThreads
protected int	numExpireThreads
protected int	numWorkerThreads
protected java.lang.String	processID	Process identifier for this agent
protected SeedingThread	seedingThread
protected SetPriorityThread	setPriorityThread
protected StartDeleteThread	startDeleteThread
protected StartupThread	startupThread
protected float	stuffAmtFactor
protected StufferThread	stufferThread
protected WorkerResetManager	workerResetManager	Worker thread pool reset manager
protected WorkerThread[]	workerThreads

Constructor Summary

Constructors

Constructor	Description
CrawlerAgent()	Constructor.

Method Summary

Modifier and Type	Method	Description
void	cleanUp(IThreadContext threadContext)	Tear down agent environment.
void	cleanUpAgentData(IThreadContext threadContext, java.lang.String currentProcessID, java.lang.String cleanupProcessID)	Cleanup after agents process.
void	cleanUpAllAgentData(IThreadContext threadContext, java.lang.String currentProcessID)	Cleanup after ALL agents processes.
void	clusterInit(IThreadContext threadContext)	Called ONLY when no other active services of this kind are running.
void	deinstall(IThreadContext threadContext)	Uninstall agent.
void	initialize(IThreadContext threadContext)	Initialize agent environment.
void	install(IThreadContext threadContext)	Install agent.
boolean	isOutputConnectionInUse(IThreadContext threadContext, java.lang.String connName)	Request permission from agent to delete an output connection.
boolean	isTransformationConnectionInUse(IThreadContext threadContext, java.lang.String connName)	Request permission from agent to delete a transformation connection.
void	noteOutputConnectionChange(IThreadContext threadContext, java.lang.String connectionName)	Note a change in configuration for an output connection.
void	noteOutputConnectorDeregistration(IThreadContext threadContext, java.lang.String[] connectionNames)	Note the deregistration of a set of output connections.
void	noteOutputConnectorRegistration(IThreadContext threadContext, java.lang.String[] connectionNames)	Note the registration of a set of output connections.
void	noteTransformationConnectionChange(IThreadContext threadContext, java.lang.String connectionName)	Note a change in configuration for a transformation connection.
void	noteTransformationConnectorDeregistration(IThreadContext threadContext, java.lang.String[] connectionNames)	Note the deregistration of a set of transformation connections.
void	noteTransformationConnectorRegistration(IThreadContext threadContext, java.lang.String[] connectionNames)	Note the registration of a set of transformation connections.
void	startAgent(IThreadContext threadContext, java.lang.String processID)	Start the agent.
void	startSystem(IThreadContext threadContext)	Start everything.
void	stopAgent(IThreadContext threadContext)	Stop the agent.
void	stopSystem(IThreadContext threadContext)	Stop the system.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_rcsid

public static final java.lang.String _rcsid

See Also:

Constant Field Values

jobStartThread

protected JobStartThread jobStartThread

stufferThread

protected StufferThread stufferThread

finisherThread

protected FinisherThread finisherThread

notificationThread

protected JobNotificationThread notificationThread

startupThread

protected StartupThread startupThread

startDeleteThread

protected StartDeleteThread startDeleteThread

jobDeleteThread

protected JobDeleteThread jobDeleteThread

workerThreads

protected WorkerThread[] workerThreads

expireStufferThread

protected ExpireStufferThread expireStufferThread

expireThreads

protected ExpireThread[] expireThreads

deleteStufferThread

protected DocumentDeleteStufferThread deleteStufferThread

deleteThreads

protected DocumentDeleteThread[] deleteThreads

cleanupStufferThread

protected DocumentCleanupStufferThread cleanupStufferThread

cleanupThreads

protected DocumentCleanupThread[] cleanupThreads

jobResetThread

protected JobResetThread jobResetThread

seedingThread

protected SeedingThread seedingThread

idleCleanupThread

protected IdleCleanupThread idleCleanupThread

setPriorityThread

protected SetPriorityThread setPriorityThread

historyCleanupThread

protected HistoryCleanupThread historyCleanupThread

assessmentThread

protected AssessmentThread assessmentThread

workerResetManager

protected WorkerResetManager workerResetManager

Worker thread pool reset manager

docDeleteResetManager

protected DocDeleteResetManager docDeleteResetManager

Delete thread pool reset manager

docCleanupResetManager

protected DocCleanupResetManager docCleanupResetManager

Cleanup thread pool reset manager

numWorkerThreads

protected int numWorkerThreads

numDeleteThreads

protected int numDeleteThreads

numCleanupThreads

protected int numCleanupThreads

numExpireThreads

protected int numExpireThreads

lowWaterFactor

protected float lowWaterFactor

stuffAmtFactor

protected float stuffAmtFactor

processID

protected java.lang.String processID

Process identifier for this agent

Constructor Detail

CrawlerAgent

public CrawlerAgent()
             throws ManifoldCFException

Constructor.

Throws:

ManifoldCFException

Method Detail

initialize

public void initialize(IThreadContext threadContext)
                throws ManifoldCFException

Initialize agent environment. This is called before any of the other operations are called, and is meant to insure that the environment is properly initialized.

Specified by:

initialize in interface IAgent

Throws:

ManifoldCFException

cleanUp

public void cleanUp(IThreadContext threadContext)
             throws ManifoldCFException

Tear down agent environment. This is called after all the other operations are completed, and is meant to allow environment resources to be freed.

Specified by:

cleanUp in interface IAgent

Throws:

ManifoldCFException

install

public void install(IThreadContext threadContext)
             throws ManifoldCFException

Install agent. This usually installs the agent's database tables etc.

Specified by:

install in interface IAgent

Throws:

ManifoldCFException

deinstall

public void deinstall(IThreadContext threadContext)
               throws ManifoldCFException

Uninstall agent. This must clean up everything the agent is responsible for.

Specified by:

deinstall in interface IAgent

Throws:

ManifoldCFException

clusterInit

public void clusterInit(IThreadContext threadContext)
                 throws ManifoldCFException

Called ONLY when no other active services of this kind are running. Meant to be used after the cluster has been down for an indeterminate period of time.

Specified by:

clusterInit in interface IAgent

Throws:

ManifoldCFException

cleanUpAllAgentData

public void cleanUpAllAgentData(IThreadContext threadContext,
                                java.lang.String currentProcessID)
                         throws ManifoldCFException

Cleanup after ALL agents processes. Call this method to clean up dangling persistent state when a cluster is just starting to come up. This method CANNOT be called when there are any active agents processes at all.

Specified by:

cleanUpAllAgentData in interface IAgent

Parameters:

currentProcessID - is the current process ID.

Throws:

ManifoldCFException

cleanUpAgentData

public void cleanUpAgentData(IThreadContext threadContext,
                             java.lang.String currentProcessID,
                             java.lang.String cleanupProcessID)
                      throws ManifoldCFException

Cleanup after agents process. Call this method to clean up dangling persistent state after agent has been stopped. This method CANNOT be called when the agent is active, but it can be called at any time and by any process in order to guarantee that a terminated agent does not block other agents from completing their tasks.

Specified by:

cleanUpAgentData in interface IAgent

Parameters:

currentProcessID - is the current process ID.

cleanupProcessID - is the process ID of the agent to clean up after.

Throws:

ManifoldCFException

startAgent

public void startAgent(IThreadContext threadContext,
                       java.lang.String processID)
                throws ManifoldCFException

Start the agent. This method should spin up the agent threads, and then return.

Specified by:

startAgent in interface IAgent

processID - is the process ID to start up an agent for.

Throws:

ManifoldCFException

stopAgent

public void stopAgent(IThreadContext threadContext)
               throws ManifoldCFException

Stop the agent. This should shut down the agent threads etc.

Specified by:

stopAgent in interface IAgent

Throws:

ManifoldCFException

isOutputConnectionInUse

public boolean isOutputConnectionInUse(IThreadContext threadContext,
                                       java.lang.String connName)
                                throws ManifoldCFException

Request permission from agent to delete an output connection.

Specified by:

isOutputConnectionInUse in interface IAgent

Parameters:

connName - is the name of the output connection.

Returns:

true if the connection is in use, false otherwise.

Throws:

ManifoldCFException

noteOutputConnectorDeregistration

public void noteOutputConnectorDeregistration(IThreadContext threadContext,
                                              java.lang.String[] connectionNames)
                                       throws ManifoldCFException

Note the deregistration of a set of output connections.

Specified by:

noteOutputConnectorDeregistration in interface IAgent

Parameters:

connectionNames - are the names of the connections being deregistered.

Throws:

ManifoldCFException

noteOutputConnectorRegistration

public void noteOutputConnectorRegistration(IThreadContext threadContext,
                                            java.lang.String[] connectionNames)
                                     throws ManifoldCFException

Note the registration of a set of output connections.

Specified by:

noteOutputConnectorRegistration in interface IAgent

Parameters:

connectionNames - are the names of the connections being registered.

Throws:

ManifoldCFException

noteOutputConnectionChange

public void noteOutputConnectionChange(IThreadContext threadContext,
                                       java.lang.String connectionName)
                                throws ManifoldCFException

Note a change in configuration for an output connection.

Specified by:

noteOutputConnectionChange in interface IAgent

Parameters:

connectionName - is the name of the connections being changed.

Throws:

ManifoldCFException

isTransformationConnectionInUse

public boolean isTransformationConnectionInUse(IThreadContext threadContext,
                                               java.lang.String connName)
                                        throws ManifoldCFException

Request permission from agent to delete a transformation connection.

Specified by:

isTransformationConnectionInUse in interface IAgent

Parameters:

connName - is the name of the transformation connection.

Returns:

true if the connection is in use, false otherwise.

Throws:

ManifoldCFException

noteTransformationConnectorDeregistration

public void noteTransformationConnectorDeregistration(IThreadContext threadContext,
                                                      java.lang.String[] connectionNames)
                                               throws ManifoldCFException

Note the deregistration of a set of transformation connections.

Specified by:

noteTransformationConnectorDeregistration in interface IAgent

Parameters:

connectionNames - are the names of the connections being deregistered.

Throws:

ManifoldCFException

noteTransformationConnectorRegistration

public void noteTransformationConnectorRegistration(IThreadContext threadContext,
                                                    java.lang.String[] connectionNames)
                                             throws ManifoldCFException

Note the registration of a set of transformation connections.

Specified by:

noteTransformationConnectorRegistration in interface IAgent

Parameters:

connectionNames - are the names of the connections being registered.

Throws:

ManifoldCFException

noteTransformationConnectionChange

public void noteTransformationConnectionChange(IThreadContext threadContext,
                                               java.lang.String connectionName)
                                        throws ManifoldCFException

Note a change in configuration for a transformation connection.

Specified by:

noteTransformationConnectionChange in interface IAgent

Parameters:

connectionName - is the name of the connection being changed.

Throws:

ManifoldCFException

startSystem

public void startSystem(IThreadContext threadContext)
                 throws ManifoldCFException

Start everything.

Throws:

ManifoldCFException

stopSystem

public void stopSystem(IThreadContext threadContext)
                throws ManifoldCFException

Stop the system.

Throws:

ManifoldCFException