Package org.apache.manifoldcf.crawler.jobs

Class JobQueue

  • public class JobQueue
extends BaseTable
    This is the job queue manager class. It is responsible for managing the jobqueue database table.

    jobqueue
    FieldTypeDescription        
    idBIGINTPrimary Key
    jobidBIGINTReference:jobs.id
    dochashVARCHAR(40)
    docidLONGTEXT
    checktimeBIGINT
    failtimeBIGINT
    failcountBIGINT
    statusCHAR(1)
    isseedCHAR(1)
    docpriorityFLOAT
    checkactionCHAR(1)
    processidVARCHAR(16)
    seedingprocessidVARCHAR(16)
    needpriorityCHAR(1)
    needpriorityprocessidVARCHAR(16)


    • Field Detail

      • STATUS_PENDINGPURGATORY

        public static final int STATUS_PENDINGPURGATORY
        See Also:
        Constant Field Values

      • STATUS_ACTIVENEEDRESCAN

        public static final int STATUS_ACTIVENEEDRESCAN
        See Also:
        Constant Field Values

      • STATUS_ACTIVENEEDRESCANPURGATORY

        public static final int STATUS_ACTIVENEEDRESCANPURGATORY
        See Also:
        Constant Field Values

      • STATUS_ELIGIBLEFORDELETE

        public static final int STATUS_ELIGIBLEFORDELETE
        See Also:
        Constant Field Values

      • NEEDPRIORITY_INPROGRESS

        public static final int NEEDPRIORITY_INPROGRESS
        See Also:
        Constant Field Values

      • docPriorityField

        public static final java.lang.String docPriorityField
        See Also:
        Constant Field Values

      • checkActionField

        public static final java.lang.String checkActionField
        See Also:
        Constant Field Values

      • seedingProcessIDField

        public static final java.lang.String seedingProcessIDField
        See Also:
        Constant Field Values

      • needPriorityField

        public static final java.lang.String needPriorityField
        See Also:
        Constant Field Values

      • needPriorityProcessIDField

        public static final java.lang.String needPriorityProcessIDField
        See Also:
        Constant Field Values

      • nullDocPriority

        public static final java.lang.Double nullDocPriority

      • statusMap

        protected static final java.util.Map<java.lang.String,​java.lang.Integer> statusMap

      • seedstatusMap

        protected static final java.util.Map<java.lang.String,​java.lang.Integer> seedstatusMap

      • actionMap

        protected static final java.util.Map<java.lang.String,​java.lang.Integer> actionMap

      • needPriorityMap

        protected static final java.util.Map<java.lang.String,​java.lang.Integer> needPriorityMap

      • prereqEventManager

        protected PrereqEventManager prereqEventManager
        Prerequisite event manager

      • threadContext

        protected IThreadContext threadContext
        Thread context

      • getNextDocumentsIndexHint

        protected java.lang.String getNextDocumentsIndexHint
        Cached getNextDocuments order-by index hint

    • Method Detail

      • restart

        public void restart​(java.lang.String processID)
             throws ManifoldCFException
        Restart. This method should be called at initial startup time. It resets the status of all documents to something reasonable, so the jobs can be restarted and work properly to completion.
        Parameters:
        processID - is the processID to clean up after.
        Throws:
        ManifoldCFException

      • reactivateHopcountRemovedRecords

        public void reactivateHopcountRemovedRecords​(java.lang.Long jobID)
                                      throws ManifoldCFException
        Flip all records for a job that have status HOPCOUNTREMOVED back to PENDING. NOTE: We need to actually schedule these!!! so the following can't really work.
        Throws:
        ManifoldCFException

      • clearFailTimes

        public void clearFailTimes​(java.lang.Long jobID)
                    throws ManifoldCFException
        Clear the failtimes for all documents associated with a job. This method is called when the system detects that a significant delaying event has occurred, and therefore the "failure clock" needs to be reset.
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • resetDocumentWorkerStatus

        public void resetDocumentWorkerStatus​(java.lang.String processID)
                               throws ManifoldCFException
        Reset as part of restoring document worker threads. This will get called if something went wrong that could have screwed up the status of a worker thread. The threads all die/end, and this method resets any active documents back to the right state (waiting for stuffing).
        Parameters:
        processID - is the current processID.
        Throws:
        ManifoldCFException

      • resetDocCleanupWorkerStatus

        public void resetDocCleanupWorkerStatus​(java.lang.String processID)
                                 throws ManifoldCFException
        Reset doc cleaning worker status.
        Throws:
        ManifoldCFException

      • prepareDeleteScan

        public void prepareDeleteScan​(java.lang.Long jobID)
                       throws ManifoldCFException
        Prepare for a job delete pass. This will not be called unless the job is in an INACTIVE state. Does the following: (1) Delete PENDING entries (2) Maps PENDINGPURGATORY, PURGATORY, and COMPLETED entries to ELIGIBLEFORDELETE
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • prepareFullScan

        public void prepareFullScan​(java.lang.Long jobID)
                     throws ManifoldCFException
        Prepare for a "full scan" job. This will not be called unless the job is in the "INACTIVE" state. This does the following: (1) get rid of all PENDING entries. (2) map PENDINGPURGATORY entries to PURGATORY. (4) map COMPLETED entries to PURGATORY.
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • resetPendingDocumentSchedules

        public void resetPendingDocumentSchedules​(java.lang.Long jobID)
                                   throws ManifoldCFException
        Reset schedule for all PENDINGPURGATORY entries.
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • queueAllExisting

        public void queueAllExisting​(java.lang.Long jobID)
                      throws ManifoldCFException
        For ADD_CHANGE_DELETE jobs where the specifications have been changed, we must reconsider every existing document. So reconsider them all.
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • preparePartialScan

        public void preparePartialScan​(java.lang.Long jobID)
                        throws ManifoldCFException
        Prepare for a "partial" job. This is called ONLY when the job is inactive. This method maps all COMPLETE entries to UNCHANGED. The purpose is to allow discovery to find the documents that need to be processed. If they were marked as COMPLETE that would stop them from being queued.
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • prepareIncrementalScan

        public void prepareIncrementalScan​(java.lang.Long jobID)
                            throws ManifoldCFException
        Prepare for an "incremental" job. This is called ONLY when the job is inactive; that is, there should be no ACTIVE or ACTIVEPURGATORY entries at all. The preparation for starting an incremental job is to requeue all documents that are currently in the system that are marked "COMPLETE". These get marked as "PENDINGPURGATORY", since the idea is to queue them in such a way that we know they were ingested before.
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • deleteIngestedDocumentIdentifiers

        public void deleteIngestedDocumentIdentifiers​(DocumentDescription[] identifiers)
                                       throws ManifoldCFException
        Delete ingested document identifiers (as part of deleting the owning job). The number of identifiers specified is guaranteed to be less than the maxInClauseCount for the database.
        Parameters:
        identifiers - is the set of document identifiers.
        Throws:
        ManifoldCFException

      • deleteAllJobRecords

        public void deleteAllJobRecords​(java.lang.Long jobID)
                         throws ManifoldCFException
        For a job deletion: Delete all records for a job.
        Parameters:
        jobID - is the job identifier.
        Throws:
        ManifoldCFException

      • markNeedPriorityInProgress

        public void markNeedPriorityInProgress​(java.lang.Long rowID,
                                       java.lang.String processID)
                                throws ManifoldCFException
        Prepare to calculate a document priority for a given row.
        Throws:
        ManifoldCFException

      • noDocPriorities

        public void noDocPriorities​(java.lang.Long jobID)
                     throws ManifoldCFException
        Clear all document priorities for a job that is going to sleep
        Throws:
        ManifoldCFException

      • clearAllDocPriorities

        public void clearAllDocPriorities()
                           throws ManifoldCFException
        Clear all document priorities globally for all documents that have priorities set, and signal that we need new priorities for all.
        Throws:
        ManifoldCFException

      • updateCompletedRecord

        public void updateCompletedRecord​(java.lang.Long recID,
                                  int currentStatus)
                           throws ManifoldCFException
        Set the "completed" status for a record.
        Throws:
        ManifoldCFException

      • updateOrHopcountRemoveRecord

        public boolean updateOrHopcountRemoveRecord​(java.lang.Long recID,
                                            int currentStatus)
                                     throws ManifoldCFException
        Either mark a record as hopcountremoved, or set status to "rescan", depending on the record's state.
        Throws:
        ManifoldCFException

      • updateActiveRecord

        public void updateActiveRecord​(java.lang.Long id,
                               int currentStatus,
                               java.lang.String processID)
                        throws ManifoldCFException
        Set the status to active on a record, leaving alone priority or check time.
        Parameters:
        id - is the job queue id.
        currentStatus - is the current status
        Throws:
        ManifoldCFException

      • setRequeuedStatus

        public void setRequeuedStatus​(java.lang.Long id,
                              java.lang.Long checkTime,
                              int action,
                              long failTime,
                              int failCount)
                       throws ManifoldCFException
        Set the status on a record, including check time and priority.
        Parameters:
        id - is the job queue id.
        checkTime - is the check time.
        Throws:
        ManifoldCFException

      • setDeletingStatus

        public void setDeletingStatus​(java.lang.Long id,
                              java.lang.String processID)
                       throws ManifoldCFException
        Set the status of a document to "being deleted".
        Throws:
        ManifoldCFException

      • setUndeletingStatus

        public void setUndeletingStatus​(java.lang.Long id,
                                long checkTime)
                         throws ManifoldCFException
        Set the status of a document to be "no longer deleting"
        Throws:
        ManifoldCFException

      • setCleaningStatus

        public void setCleaningStatus​(java.lang.Long id,
                              java.lang.String processID)
                       throws ManifoldCFException
        Set the status of a document to "being cleaned".
        Throws:
        ManifoldCFException

      • setUncleaningStatus

        public void setUncleaningStatus​(java.lang.Long id,
                                long checkTime)
                         throws ManifoldCFException
        Set the status of a document to be "no longer cleaning"
        Throws:
        ManifoldCFException

      • deleteRecordMultiple

        public void deleteRecordMultiple​(java.lang.Long[] ids)
                          throws ManifoldCFException
        Remove multiple records entirely.
        Parameters:
        ids - is the set of job queue id's
        Throws:
        ManifoldCFException

      • maxClauseDoDeletes

        protected int maxClauseDoDeletes()
        Calculate the number of deletes we can do at once.

      • updateExistingRecordInitial

        public void updateExistingRecordInitial​(java.lang.Long recordID,
                                        int currentStatus,
                                        java.lang.Long checkTimeValue,
                                        long desiredExecuteTime,
                                        IPriorityCalculator desiredPriority,
                                        java.lang.String[] prereqEvents,
                                        java.lang.String processID)
                                 throws ManifoldCFException
        Update an existing record (as the result of an initial add). The record is presumed to exist and have been locked, via "FOR UPDATE".
        Throws:
        ManifoldCFException

      • insertNewRecordInitial

        public void insertNewRecordInitial​(java.lang.Long jobID,
                                   java.lang.String docHash,
                                   java.lang.String docID,
                                   IPriorityCalculator desiredDocPriority,
                                   long desiredExecuteTime,
                                   java.lang.String[] prereqEvents,
                                   java.lang.String processID)
                            throws ManifoldCFException
        Insert a new record into the jobqueue table (as part of adding an initial reference).
        Parameters:
        jobID - is the job identifier.
        docHash - is the hash of the local document identifier.
        docID - is the local document identifier.
        Throws:
        ManifoldCFException

      • addRemainingDocumentsInitial

        public void addRemainingDocumentsInitial​(java.lang.Long jobID,
                                         java.lang.String[] docIDHashes,
                                         java.lang.String processID)
                                  throws ManifoldCFException
        Note the remaining documents that do NOT need to be queued. These are noted so that the doneDocumentsInitial() method does not clean up seeds from previous runs wrongly.
        Throws:
        ManifoldCFException

      • maxClauseProcessRemainingDocuments

        protected int maxClauseProcessRemainingDocuments​(java.lang.Long jobID)
        Calculate max

      • processRemainingDocuments

        protected void processRemainingDocuments​(java.util.Map idMap,
                                         java.lang.Long jobID,
                                         java.util.ArrayList list,
                                         java.util.Map inSet)
                                  throws ManifoldCFException
        Process the specified set of documents.
        Throws:
        ManifoldCFException

      • maxClauseUpdateRemainingDocuments

        protected int maxClauseUpdateRemainingDocuments()
        Get the maximum count

      • updateRemainingDocuments

        protected void updateRemainingDocuments​(java.util.ArrayList list,
                                        java.lang.String processID)
                                 throws ManifoldCFException
        Update the specified set of documents to be "NEWSEED"
        Throws:
        ManifoldCFException

      • doneDocumentsInitial

        public void doneDocumentsInitial​(java.lang.Long jobID,
                                 boolean isPartial)
                          throws ManifoldCFException
        Complete the initial set of documents. This method converts the seeding statuses for the job to their steady-state values. SEEDSTATUS_SEED becomes SEEDSTATUS_NOTSEED, and SEEDSTATUS_NEWSEED becomes SEEDSTATUS_SEED. If the seeding was partial, then all previous seeds are preserved as such.
        Parameters:
        jobID - is the job identifier.
        isPartial - is true of the passed list of seeds is not complete.
        Throws:
        ManifoldCFException

      • getAllSeeds

        public java.lang.String[] getAllSeeds​(java.lang.Long jobID)
                               throws ManifoldCFException
        Get all the current seeds. Returns the seed document identifiers for a job.
        Parameters:
        jobID - is the job identifier.
        Returns:
        the document identifier hashes that are currently considered to be seeds.
        Throws:
        ManifoldCFException

      • updateExistingRecord

        public void updateExistingRecord​(java.lang.Long recordID,
                                 int currentStatus,
                                 java.lang.Long checkTimeValue,
                                 long desiredExecuteTime,
                                 boolean otherChangesSeen,
                                 IPriorityCalculator desiredPriority,
                                 java.lang.String[] prereqEvents)
                          throws ManifoldCFException
        Update an existing record (as the result of a reference add). The record is presumed to exist and have been locked, via "FOR UPDATE".
        Throws:
        ManifoldCFException

      • insertNewRecord

        public void insertNewRecord​(java.lang.Long jobID,
                            java.lang.String docIDHash,
                            java.lang.String docID,
                            IPriorityCalculator desiredDocPriority,
                            long desiredExecuteTime,
                            java.lang.String[] prereqEvents)
                     throws ManifoldCFException
        Insert a new record into the jobqueue table (as part of adding a child reference).
        Throws:
        ManifoldCFException

      • stringToStatus

        public static int stringToStatus​(java.lang.String value)
                          throws ManifoldCFException
        Convert status field value to integer.
        Parameters:
        value - is the string.
        Returns:
        the integer.
        Throws:
        ManifoldCFException

      • statusToString

        public static java.lang.String statusToString​(int status)
                                       throws ManifoldCFException
        Convert status to string.
        Parameters:
        status - is the status value.
        Returns:
        the database string.
        Throws:
        ManifoldCFException

      • getHashCode

        public static java.lang.String getHashCode​(java.lang.String documentIdentifier)
                                    throws ManifoldCFException
        Get a hash value from a document id string. This will convert the string into something that can fit in 20 characters. (Someday this will be an MD5 hash, but for now just use java hashing.)
        Parameters:
        documentIdentifier - is the input document id string.
        Returns:
        the hash code.
        Throws:
        ManifoldCFException