Package org.apache.manifoldcf.crawler.jobs

Class JobManager.ThrottleLimit

  • All Implemented Interfaces:
    ILimitChecker
    Enclosing class:
    JobManager
    protected static class JobManager.ThrottleLimit
extends java.lang.Object
implements ILimitChecker
    This class provides the throttling limits for the job queueing query.

    • Field Detail

      • n

        protected final int n

      • jobConnection

        protected java.util.Map<java.lang.Long,​java.lang.String> jobConnection

      • activeConnections

        protected java.util.Map<java.lang.String,​IRepositoryConnector> activeConnections

      • setSizes

        protected java.util.Map<java.lang.String,​java.lang.Integer> setSizes

      • maxSetSize

        protected int maxSetSize

      • documentsProcessed

        protected int documentsProcessed

      • blockingDocumentArray

        protected final java.util.List<DocumentDescription> blockingDocumentArray

    • Constructor Detail

      • ThrottleLimit

        public ThrottleLimit​(int n)
        Constructor. This class is built up piecemeal, so the constructor does nothing.
        Parameters:
        n - is the maximum number of full job descriptions we want at this time.

    • Method Detail

      • tallyBlockingDocuments

        public void tallyBlockingDocuments​(BlockingDocuments blockingDocuments)
        Transfer blocking documents discovered to BlockingDocuments object

      • addJob

        public void addJob​(java.lang.Long jobID,
                   java.lang.String connectionName)
        Add a job/connection name map entry.
        Parameters:
        jobID - is the job id.
        connectionName - is the connection name.

      • addConnectionName

        public void addConnectionName​(java.lang.String connectionName,
                              IRepositoryConnector connectorInstance)
                       throws ManifoldCFException
        Add an active connection. This is the pool of active connections that will be used for the lifetime of this operation.
        Parameters:
        connectionName - is the connection name.
        Throws:
        ManifoldCFException

      • addLimit

        public void addLimit​(java.lang.String connectionName,
                     java.lang.String regexp,
                     int upperLimit)
        Add a document limit for a specified connection. This is the limit across all matching bins; if any individual matching bin exceeds that limit, then documents that belong to that bin will be excluded.
        Parameters:
        connectionName - is the connection name.
        regexp - is the regular expression, which we will match against various bins.
        upperLimit - is the maximum count associated with the specified job.

      • setConnectionLimit

        public void setConnectionLimit​(java.lang.String connectionName,
                               int maxDocuments)
        Set a connection-based total document limit.

      • doesCompareWork

        public boolean doesCompareWork()
        See if this class can be legitimately compared against another of the same type.
        Specified by:
        doesCompareWork in interface ILimitChecker
        Returns:
        true if comparisons will ever return "true".

      • duplicate

        public ILimitChecker duplicate()
        Create a duplicate of this class instance. All current state should be preserved. NOTE: Since doesCompareWork() returns false, queries using this limit checker cannot be cached, and therefore duplicate() is never called from the query executor. But it can be called from other places.
        Specified by:
        duplicate in interface ILimitChecker
        Returns:
        the duplicate.

      • hashCode

        public int hashCode()
        Find the hashcode for this class. This will only ever be used if doesCompareWork() returns true.
        Specified by:
        hashCode in interface ILimitChecker
        Overrides:
        hashCode in class java.lang.Object
        Returns:
        the hashcode.

      • equals

        public boolean equals​(java.lang.Object object)
        Compare two objects and see if equal. This will only ever be used if doesCompareWork() returns true.
        Specified by:
        equals in interface ILimitChecker
        Overrides:
        equals in class java.lang.Object
        Parameters:
        object - is the object to compare against.
        Returns:
        true if equal.

      • getRemainingDocuments

        public int getRemainingDocuments()
        Get the remaining documents we should query for.
        Returns:
        the maximal remaining count.