Package org.apache.manifoldcf.crawler.connectors.webcrawler

Class ThrottleDescription

  • All Implemented Interfaces:
    org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec
    public class ThrottleDescription
extends java.lang.Object
implements org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec
    This class describes complex throttling criteria pulled from a configuration. The data contained is organized by regular expression performed on a bin. What we store for each regular expression is a Pattern, for efficiency. This structure deals with bandwidth limits, maximum connection limits, and maximum fetch rate limits. Average fetch rate limits are handled in the infrastructure. Generally it is a good thing to limit the number of regexps that need to be evaluated against any given bin value as much as possible. For that reason I've organized this structure accordingly.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      protected static class  ThrottleDescription.ThrottleItem
      Class representing an individual throttle item.

    • Constructor Summary

      Constructors 
      Constructor Description
      ThrottleDescription​(org.apache.manifoldcf.core.interfaces.ConfigParams configData)
      Constructor.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      int getMaxOpenConnections​(java.lang.String binName)
      Given a bin name, find the max open connections to use for that bin.
      double getMinimumMillisecondsPerByte​(java.lang.String binName)
      Look up minimum milliseconds per byte for a bin.
      long getMinimumMillisecondsPerFetch​(java.lang.String binName)
      Look up minimum milliseconds for a fetch for a bin.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • patternHash

        protected java.util.Map<java.lang.String,​ThrottleDescription.ThrottleItem> patternHash
        This is the hash that contains everything. It's keyed by the regexp string itself. Values are ThrottleItem's.

    • Constructor Detail

      • ThrottleDescription

        public ThrottleDescription​(org.apache.manifoldcf.core.interfaces.ConfigParams configData)
                    throws org.apache.manifoldcf.core.interfaces.ManifoldCFException
        Constructor. Build the description from the ConfigParams.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException

    • Method Detail

      • getMaxOpenConnections

        public int getMaxOpenConnections​(java.lang.String binName)
        Given a bin name, find the max open connections to use for that bin.
        Specified by:
        getMaxOpenConnections in interface org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec
        Returns:
        Integer.MAX_VALUE if no limit found.

      • getMinimumMillisecondsPerByte

        public double getMinimumMillisecondsPerByte​(java.lang.String binName)
        Look up minimum milliseconds per byte for a bin.
        Specified by:
        getMinimumMillisecondsPerByte in interface org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec
        Returns:
        0.0 if no limit found.

      • getMinimumMillisecondsPerFetch

        public long getMinimumMillisecondsPerFetch​(java.lang.String binName)
        Look up minimum milliseconds for a fetch for a bin.
        Specified by:
        getMinimumMillisecondsPerFetch in interface org.apache.manifoldcf.connectorcommon.interfaces.IThrottleSpec
        Returns:
        0 if no limit found.