org.apache.manifoldcf.crawler.jobs

Class JobQueue

java.lang.Object

org.apache.manifoldcf.core.database.BaseTable

org.apache.manifoldcf.crawler.jobs.JobQueue

public class JobQueue
extends BaseTable

This is the job queue manager class. It is responsible for managing the jobqueue database table.

jobqueue

Field	Type	Description
id	BIGINT	Primary Key
jobid	BIGINT	Reference:jobs.id
dochash	VARCHAR(40)
docid	LONGTEXT
checktime	BIGINT
failtime	BIGINT
failcount	BIGINT
status	CHAR(1)
isseed	CHAR(1)
docpriority	FLOAT
checkaction	CHAR(1)
processid	VARCHAR(16)
seedingprocessid	VARCHAR(16)
needpriority	CHAR(1)
needpriorityprocessid	VARCHAR(16)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	JobQueue.DuplicateFinder

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	_rcsid
static int	ACTION_REMOVE
static int	ACTION_RESCAN
protected static java.util.Map<java.lang.String,java.lang.Integer>	actionMap
static java.lang.String	checkActionField
static java.lang.String	checkTimeField
static java.lang.String	docHashField
static java.lang.String	docIDField
static java.lang.String	docPriorityField
static java.lang.String	failCountField
static java.lang.String	failTimeField
protected java.lang.String	getNextDocumentsIndexHint Cached getNextDocuments order-by index hint
static java.lang.String	idField
static java.lang.String	isSeedField
static java.lang.String	jobIDField
static int	NEEDPRIORITY_FALSE
static int	NEEDPRIORITY_INPROGRESS
static int	NEEDPRIORITY_TRUE
static java.lang.String	needPriorityField
protected static java.util.Map<java.lang.String,java.lang.Integer>	needPriorityMap
static java.lang.String	needPriorityProcessIDField
static double	noDocPriorityValue
static java.lang.Double	nullDocPriority
protected PrereqEventManager	prereqEventManager Prerequisite event manager
static java.lang.String	processIDField
static java.lang.String	seedingProcessIDField
static int	SEEDSTATUS_NEWSEED
static int	SEEDSTATUS_NOTSEED
static int	SEEDSTATUS_SEED
protected static java.util.Map<java.lang.String,java.lang.Integer>	seedstatusMap
static int	STATUS_ACTIVE
static int	STATUS_ACTIVENEEDRESCAN
static int	STATUS_ACTIVENEEDRESCANPURGATORY
static int	STATUS_ACTIVEPURGATORY
static int	STATUS_BEINGCLEANED
static int	STATUS_BEINGDELETED
static int	STATUS_COMPLETE
static int	STATUS_ELIGIBLEFORDELETE
static int	STATUS_HOPCOUNTREMOVED
static int	STATUS_PENDING
static int	STATUS_PENDINGPURGATORY
static int	STATUS_PURGATORY
static int	STATUS_UNCHANGED
static java.lang.String	statusField
protected static java.util.Map<java.lang.String,java.lang.Integer>	statusMap
protected IThreadContext	threadContext Thread context

Fields inherited from class org.apache.manifoldcf.core.database.BaseTable

dbInterface, tableName

Constructor Summary

Constructors

Constructor and Description
JobQueue(IThreadContext tc, IDBInterface database) Constructor.

Method Summary

Modifier and Type	Method and Description
static java.lang.String	actionToString(int action) Convert integer to action string
void	addRemainingDocumentsInitial(java.lang.Long jobID, java.lang.String[] docIDHashes, java.lang.String processID) Note the remaining documents that do NOT need to be queued.
boolean	checkJobBusy(java.lang.Long jobID) Check if there are any outstanding active documents for a job
void	clearAllDocPriorities() Clear all document priorities globally for all documents that have priorities set, and signal that we need new priorities for all.
void	clearFailTimes(java.lang.Long jobID) Clear the failtimes for all documents associated with a job.
void	deinstall() Uninstall.
void	deleteAllJobRecords(java.lang.Long jobID) For a job deletion: Delete all records for a job.
void	deleteIngestedDocumentIdentifiers(DocumentDescription[] identifiers) Delete ingested document identifiers (as part of deleting the owning job).
void	deleteRecord(java.lang.Long id) Remove a record entirely.
void	deleteRecordMultiple(java.lang.Long[] ids) Remove multiple records entirely.
protected void	doDeletes(java.util.ArrayList list) Do a batch of deletes.
void	doneDocumentsInitial(java.lang.Long jobID, boolean isPartial) Complete the initial set of documents.
java.lang.String[]	getAllSeeds(java.lang.Long jobID) Get all the current seeds.
java.lang.String	getGetNextDocumentsIndexHint() Get the 'getNextDocuments' index hint.
static java.lang.String	getHashCode(java.lang.String documentIdentifier) Get a hash value from a document id string.
void	insertNewRecord(java.lang.Long jobID, java.lang.String docIDHash, java.lang.String docID, IPriorityCalculator desiredDocPriority, long desiredExecuteTime, java.lang.String[] prereqEvents) Insert a new record into the jobqueue table (as part of adding a child reference).
void	insertNewRecordInitial(java.lang.Long jobID, java.lang.String docHash, java.lang.String docID, IPriorityCalculator desiredDocPriority, long desiredExecuteTime, java.lang.String[] prereqEvents, java.lang.String processID) Insert a new record into the jobqueue table (as part of adding an initial reference).
void	install(java.lang.String jobsTable, java.lang.String jobsColumn) Install or upgrade.
void	markNeedPriorityInProgress(java.lang.Long rowID, java.lang.String processID) Prepare to calculate a document priority for a given row.
protected int	maxClauseDoDeletes() Calculate the number of deletes we can do at once.
protected int	maxClauseProcessRemainingDocuments(java.lang.Long jobID) Calculate max
protected int	maxClauseUpdateRemainingDocuments() Get the maximum count
static java.lang.String	needPriorityToString(int value) Convert boolean to need priority value.
void	noDocPriorities(java.lang.Long jobID) Clear all document priorities for a job that is going to sleep
void	prepareDeleteScan(java.lang.Long jobID) Prepare for a job delete pass.
void	prepareFullScan(java.lang.Long jobID) Prepare for a "full scan" job.
void	prepareIncrementalScan(java.lang.Long jobID) Prepare for an "incremental" job.
void	preparePartialScan(java.lang.Long jobID) Prepare for a "partial" job.
void	prioritizeQueuedDocuments(java.lang.Long jobID) Mark queued documents for prioritization
protected void	processRemainingDocuments(java.util.Map idMap, java.lang.Long jobID, java.util.ArrayList list, java.util.Map inSet) Process the specified set of documents.
void	queueAllExisting(java.lang.Long jobID) For ADD_CHANGE_DELETE jobs where the specifications have been changed, we must reconsider every existing document.
void	reactivateHopcountRemovedRecords(java.lang.Long jobID) Flip all records for a job that have status HOPCOUNTREMOVED back to PENDING.
void	resetDocCleanupWorkerStatus(java.lang.String processID) Reset doc cleaning worker status.
void	resetDocDeleteWorkerStatus(java.lang.String processID) Reset doc delete worker status.
void	resetDocumentWorkerStatus(java.lang.String processID) Reset as part of restoring document worker threads.
void	resetPendingDocumentSchedules(java.lang.Long jobID) Reset schedule for all PENDINGPURGATORY entries.
void	restart() Cleanup after all processIDs.
void	restart(java.lang.String processID) Restart.
void	restartCluster() Restart for entire cluster.
static java.lang.String	seedstatusToString(int status) Convert seedstatus value to a string.
void	setCleaningStatus(java.lang.Long id, java.lang.String processID) Set the status of a document to "being cleaned".
void	setDeletingStatus(java.lang.Long id, java.lang.String processID) Set the status of a document to "being deleted".
void	setRequeuedStatus(java.lang.Long id, java.lang.Long checkTime, int action, long failTime, int failCount) Set the status on a record, including check time and priority.
void	setUncleaningStatus(java.lang.Long id, long checkTime) Set the status of a document to be "no longer cleaning"
void	setUndeletingStatus(java.lang.Long id, long checkTime) Set the status of a document to be "no longer deleting"
static java.lang.String	statusToString(int status) Convert status to string.
static int	stringToAction(java.lang.String value) Convert action field value to integer.
static int	stringToNeedPriority(java.lang.String value) Convert need priority value to boolean.
static int	stringToSeedstatus(java.lang.String x) Convert seedstatus field value to a boolean.
static int	stringToStatus(java.lang.String value) Convert status field value to integer.
void	unconditionallyAnalyzeTables() Analyze job tables due to major event
void	updateActiveRecord(java.lang.Long id, int currentStatus, java.lang.String processID) Set the status to active on a record, leaving alone priority or check time.
void	updateCompletedRecord(java.lang.Long recID, int currentStatus) Set the "completed" status for a record.
void	updateExistingRecord(java.lang.Long recordID, int currentStatus, java.lang.Long checkTimeValue, long desiredExecuteTime, boolean otherChangesSeen, IPriorityCalculator desiredPriority, java.lang.String[] prereqEvents) Update an existing record (as the result of a reference add).
void	updateExistingRecordInitial(java.lang.Long recordID, int currentStatus, java.lang.Long checkTimeValue, long desiredExecuteTime, IPriorityCalculator desiredPriority, java.lang.String[] prereqEvents, java.lang.String processID) Update an existing record (as the result of an initial add).
boolean	updateOrHopcountRemoveRecord(java.lang.Long recID, int currentStatus) Either mark a record as hopcountremoved, or set status to "rescan", depending on the record's state.
protected void	updateRemainingDocuments(java.util.ArrayList list, java.lang.String processID) Update the specified set of documents to be "NEWSEED"
void	writeDocPriority(java.lang.Long rowID, IPriorityCalculator priority) Write out a document priority

Methods inherited from class org.apache.manifoldcf.core.database.BaseTable

addTableIndex, analyzeTable, beginTransaction, buildConjunctionClause, constructCountClause, constructDistinctOnClause, constructDoubleCastClause, constructOffsetLimitClause, constructRegexpClause, constructSubstringClause, endTransaction, findConjunctionClauseMax, getDatabaseCacheKey, getDBInterface, getMaxInClause, getMaxOrClause, getSleepAmt, getTableIndexes, getTableName, getTableSchema, getTransactionID, getWindowedReportMaxRows, makeTableKey, noteModifications, performAddIndex, performAlter, performCommit, performCreate, performDelete, performDrop, performInsert, performModification, performQuery, performQuery, performRemoveIndex, performUpdate, prepareRowForSave, readRow, reindexTable, signalRollback, sleepFor

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_rcsid

public static final java.lang.String _rcsid

See Also:

Constant Field Values

SEEDSTATUS_NOTSEED

public static final int SEEDSTATUS_NOTSEED

See Also:

Constant Field Values

SEEDSTATUS_SEED

public static final int SEEDSTATUS_SEED

See Also:

Constant Field Values

SEEDSTATUS_NEWSEED

public static final int SEEDSTATUS_NEWSEED

See Also:

Constant Field Values

STATUS_PENDING

public static final int STATUS_PENDING

See Also:

Constant Field Values

STATUS_ACTIVE

public static final int STATUS_ACTIVE

See Also:

Constant Field Values

STATUS_COMPLETE

public static final int STATUS_COMPLETE

See Also:

Constant Field Values

STATUS_UNCHANGED

public static final int STATUS_UNCHANGED

See Also:

Constant Field Values

STATUS_PENDINGPURGATORY

public static final int STATUS_PENDINGPURGATORY

See Also:

Constant Field Values

STATUS_ACTIVEPURGATORY

public static final int STATUS_ACTIVEPURGATORY

See Also:

Constant Field Values

STATUS_PURGATORY

public static final int STATUS_PURGATORY

See Also:

Constant Field Values

STATUS_BEINGDELETED

public static final int STATUS_BEINGDELETED

See Also:

Constant Field Values

STATUS_ACTIVENEEDRESCAN

public static final int STATUS_ACTIVENEEDRESCAN

See Also:

Constant Field Values

STATUS_ACTIVENEEDRESCANPURGATORY

public static final int STATUS_ACTIVENEEDRESCANPURGATORY

See Also:

Constant Field Values

STATUS_BEINGCLEANED

public static final int STATUS_BEINGCLEANED

See Also:

Constant Field Values

STATUS_ELIGIBLEFORDELETE

public static final int STATUS_ELIGIBLEFORDELETE

See Also:

Constant Field Values

STATUS_HOPCOUNTREMOVED

public static final int STATUS_HOPCOUNTREMOVED

See Also:

Constant Field Values

ACTION_RESCAN

public static final int ACTION_RESCAN

See Also:

Constant Field Values

ACTION_REMOVE

public static final int ACTION_REMOVE

See Also:

Constant Field Values

NEEDPRIORITY_FALSE

public static final int NEEDPRIORITY_FALSE

See Also:

Constant Field Values

NEEDPRIORITY_INPROGRESS

public static final int NEEDPRIORITY_INPROGRESS

See Also:

Constant Field Values

NEEDPRIORITY_TRUE

public static final int NEEDPRIORITY_TRUE

See Also:

Constant Field Values

idField

public static final java.lang.String idField

See Also:

Constant Field Values

jobIDField

public static final java.lang.String jobIDField

See Also:

Constant Field Values

docHashField

public static final java.lang.String docHashField

See Also:

Constant Field Values

docIDField

public static final java.lang.String docIDField

See Also:

Constant Field Values

checkTimeField

public static final java.lang.String checkTimeField

See Also:

Constant Field Values

statusField

public static final java.lang.String statusField

See Also:

Constant Field Values

failTimeField

public static final java.lang.String failTimeField

See Also:

Constant Field Values

failCountField

public static final java.lang.String failCountField

See Also:

Constant Field Values

isSeedField

public static final java.lang.String isSeedField

See Also:

Constant Field Values

docPriorityField

public static final java.lang.String docPriorityField

See Also:

Constant Field Values

checkActionField

public static final java.lang.String checkActionField

See Also:

Constant Field Values

processIDField

public static final java.lang.String processIDField

See Also:

Constant Field Values

seedingProcessIDField

public static final java.lang.String seedingProcessIDField

See Also:

Constant Field Values

needPriorityField

public static final java.lang.String needPriorityField

See Also:

Constant Field Values

needPriorityProcessIDField

public static final java.lang.String needPriorityProcessIDField

See Also:

Constant Field Values

noDocPriorityValue

public static final double noDocPriorityValue

See Also:

Constant Field Values

nullDocPriority

public static final java.lang.Double nullDocPriority

statusMap

protected static final java.util.Map<java.lang.String,java.lang.Integer> statusMap

seedstatusMap

protected static final java.util.Map<java.lang.String,java.lang.Integer> seedstatusMap

actionMap

protected static final java.util.Map<java.lang.String,java.lang.Integer> actionMap

needPriorityMap

protected static final java.util.Map<java.lang.String,java.lang.Integer> needPriorityMap

prereqEventManager

protected PrereqEventManager prereqEventManager

Prerequisite event manager

threadContext

protected IThreadContext threadContext

Thread context

getNextDocumentsIndexHint

protected java.lang.String getNextDocumentsIndexHint

Cached getNextDocuments order-by index hint

Constructor Detail

JobQueue

public JobQueue(IThreadContext tc,
                IDBInterface database)
         throws ManifoldCFException

Constructor.

Parameters:

database - is the database handle.

Throws:

ManifoldCFException

Method Detail

install

public void install(java.lang.String jobsTable,
                    java.lang.String jobsColumn)
             throws ManifoldCFException

Install or upgrade.

Throws:

ManifoldCFException

getGetNextDocumentsIndexHint

public java.lang.String getGetNextDocumentsIndexHint()
                                              throws ManifoldCFException

Get the 'getNextDocuments' index hint.

Throws:

ManifoldCFException

unconditionallyAnalyzeTables

public void unconditionallyAnalyzeTables()
                                  throws ManifoldCFException

Analyze job tables due to major event

Throws:

ManifoldCFException

deinstall

public void deinstall()
               throws ManifoldCFException

Uninstall.

Throws:

ManifoldCFException

restart

public void restart(java.lang.String processID)
             throws ManifoldCFException

Restart. This method should be called at initial startup time. It resets the status of all documents to something reasonable, so the jobs can be restarted and work properly to completion.

Parameters:

processID - is the processID to clean up after.

Throws:

ManifoldCFException

restart

public void restart()
             throws ManifoldCFException

Cleanup after all processIDs.

Throws:

ManifoldCFException

restartCluster

public void restartCluster()
                    throws ManifoldCFException

Restart for entire cluster.

Throws:

ManifoldCFException

reactivateHopcountRemovedRecords

public void reactivateHopcountRemovedRecords(java.lang.Long jobID)
                                      throws ManifoldCFException

Flip all records for a job that have status HOPCOUNTREMOVED back to PENDING. NOTE: We need to actually schedule these!!! so the following can't really work.

Throws:

ManifoldCFException

clearFailTimes

public void clearFailTimes(java.lang.Long jobID)
                    throws ManifoldCFException

Clear the failtimes for all documents associated with a job. This method is called when the system detects that a significant delaying event has occurred, and therefore the "failure clock" needs to be reset.

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

resetDocumentWorkerStatus

public void resetDocumentWorkerStatus(java.lang.String processID)
                               throws ManifoldCFException

Reset as part of restoring document worker threads. This will get called if something went wrong that could have screwed up the status of a worker thread. The threads all die/end, and this method resets any active documents back to the right state (waiting for stuffing).

Parameters:

processID - is the current processID.

Throws:

ManifoldCFException

resetDocDeleteWorkerStatus

public void resetDocDeleteWorkerStatus(java.lang.String processID)
                                throws ManifoldCFException

Reset doc delete worker status.

Throws:

ManifoldCFException

resetDocCleanupWorkerStatus

public void resetDocCleanupWorkerStatus(java.lang.String processID)
                                 throws ManifoldCFException

Reset doc cleaning worker status.

Throws:

ManifoldCFException

prepareDeleteScan

public void prepareDeleteScan(java.lang.Long jobID)
                       throws ManifoldCFException

Prepare for a job delete pass. This will not be called unless the job is in an INACTIVE state. Does the following: (1) Delete PENDING entries (2) Maps PENDINGPURGATORY, PURGATORY, and COMPLETED entries to ELIGIBLEFORDELETE

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

prepareFullScan

public void prepareFullScan(java.lang.Long jobID)
                     throws ManifoldCFException

Prepare for a "full scan" job. This will not be called unless the job is in the "INACTIVE" state. This does the following: (1) get rid of all PENDING entries. (2) map PENDINGPURGATORY entries to PURGATORY. (4) map COMPLETED entries to PURGATORY.

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

resetPendingDocumentSchedules

public void resetPendingDocumentSchedules(java.lang.Long jobID)
                                   throws ManifoldCFException

Reset schedule for all PENDINGPURGATORY entries.

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

queueAllExisting

public void queueAllExisting(java.lang.Long jobID)
                      throws ManifoldCFException

For ADD_CHANGE_DELETE jobs where the specifications have been changed, we must reconsider every existing document. So reconsider them all.

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

preparePartialScan

public void preparePartialScan(java.lang.Long jobID)
                        throws ManifoldCFException

Prepare for a "partial" job. This is called ONLY when the job is inactive. This method maps all COMPLETE entries to UNCHANGED. The purpose is to allow discovery to find the documents that need to be processed. If they were marked as COMPLETE that would stop them from being queued.

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

prepareIncrementalScan

public void prepareIncrementalScan(java.lang.Long jobID)
                            throws ManifoldCFException

Prepare for an "incremental" job. This is called ONLY when the job is inactive; that is, there should be no ACTIVE or ACTIVEPURGATORY entries at all. The preparation for starting an incremental job is to requeue all documents that are currently in the system that are marked "COMPLETE". These get marked as "PENDINGPURGATORY", since the idea is to queue them in such a way that we know they were ingested before.

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

deleteIngestedDocumentIdentifiers

public void deleteIngestedDocumentIdentifiers(DocumentDescription[] identifiers)
                                       throws ManifoldCFException

Delete ingested document identifiers (as part of deleting the owning job). The number of identifiers specified is guaranteed to be less than the maxInClauseCount for the database.

Parameters:

identifiers - is the set of document identifiers.

Throws:

ManifoldCFException

checkJobBusy

public boolean checkJobBusy(java.lang.Long jobID)
                     throws ManifoldCFException

Check if there are any outstanding active documents for a job

Throws:

ManifoldCFException

deleteAllJobRecords

public void deleteAllJobRecords(java.lang.Long jobID)
                         throws ManifoldCFException

For a job deletion: Delete all records for a job.

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

markNeedPriorityInProgress

public void markNeedPriorityInProgress(java.lang.Long rowID,
                                       java.lang.String processID)
                                throws ManifoldCFException

Prepare to calculate a document priority for a given row.

Throws:

ManifoldCFException

writeDocPriority

public void writeDocPriority(java.lang.Long rowID,
                             IPriorityCalculator priority)
                      throws ManifoldCFException

Write out a document priority

Throws:

ManifoldCFException

prioritizeQueuedDocuments

public void prioritizeQueuedDocuments(java.lang.Long jobID)
                               throws ManifoldCFException

Mark queued documents for prioritization

Throws:

ManifoldCFException

noDocPriorities

public void noDocPriorities(java.lang.Long jobID)
                     throws ManifoldCFException

Clear all document priorities for a job that is going to sleep

Throws:

ManifoldCFException

clearAllDocPriorities

public void clearAllDocPriorities()
                           throws ManifoldCFException

Clear all document priorities globally for all documents that have priorities set, and signal that we need new priorities for all.

Throws:

ManifoldCFException

updateCompletedRecord

public void updateCompletedRecord(java.lang.Long recID,
                                  int currentStatus)
                           throws ManifoldCFException

Set the "completed" status for a record.

Throws:

ManifoldCFException

updateOrHopcountRemoveRecord

public boolean updateOrHopcountRemoveRecord(java.lang.Long recID,
                                            int currentStatus)
                                     throws ManifoldCFException

Either mark a record as hopcountremoved, or set status to "rescan", depending on the record's state.

Throws:

ManifoldCFException

updateActiveRecord

public void updateActiveRecord(java.lang.Long id,
                               int currentStatus,
                               java.lang.String processID)
                        throws ManifoldCFException

Set the status to active on a record, leaving alone priority or check time.

Parameters:

id - is the job queue id.

currentStatus - is the current status

Throws:

ManifoldCFException

setRequeuedStatus

public void setRequeuedStatus(java.lang.Long id,
                              java.lang.Long checkTime,
                              int action,
                              long failTime,
                              int failCount)
                       throws ManifoldCFException

Set the status on a record, including check time and priority.

Parameters:

id - is the job queue id.

checkTime - is the check time.

Throws:

ManifoldCFException

setDeletingStatus

public void setDeletingStatus(java.lang.Long id,
                              java.lang.String processID)
                       throws ManifoldCFException

Set the status of a document to "being deleted".

Throws:

ManifoldCFException

setUndeletingStatus

public void setUndeletingStatus(java.lang.Long id,
                                long checkTime)
                         throws ManifoldCFException

Set the status of a document to be "no longer deleting"

Throws:

ManifoldCFException

setCleaningStatus

public void setCleaningStatus(java.lang.Long id,
                              java.lang.String processID)
                       throws ManifoldCFException

Set the status of a document to "being cleaned".

Throws:

ManifoldCFException

setUncleaningStatus

public void setUncleaningStatus(java.lang.Long id,
                                long checkTime)
                         throws ManifoldCFException

Set the status of a document to be "no longer cleaning"

Throws:

ManifoldCFException

deleteRecordMultiple

public void deleteRecordMultiple(java.lang.Long[] ids)
                          throws ManifoldCFException

Remove multiple records entirely.

Parameters:

ids - is the set of job queue id's

Throws:

ManifoldCFException

maxClauseDoDeletes

protected int maxClauseDoDeletes()

Calculate the number of deletes we can do at once.

doDeletes

protected void doDeletes(java.util.ArrayList list)
                  throws ManifoldCFException

Do a batch of deletes.

Throws:

ManifoldCFException

deleteRecord

public void deleteRecord(java.lang.Long id)
                  throws ManifoldCFException

Remove a record entirely.

Parameters:

id - is the job queue id.

Throws:

ManifoldCFException

updateExistingRecordInitial

public void updateExistingRecordInitial(java.lang.Long recordID,
                                        int currentStatus,
                                        java.lang.Long checkTimeValue,
                                        long desiredExecuteTime,
                                        IPriorityCalculator desiredPriority,
                                        java.lang.String[] prereqEvents,
                                        java.lang.String processID)
                                 throws ManifoldCFException

Update an existing record (as the result of an initial add). The record is presumed to exist and have been locked, via "FOR UPDATE".

Throws:

ManifoldCFException

insertNewRecordInitial

public void insertNewRecordInitial(java.lang.Long jobID,
                                   java.lang.String docHash,
                                   java.lang.String docID,
                                   IPriorityCalculator desiredDocPriority,
                                   long desiredExecuteTime,
                                   java.lang.String[] prereqEvents,
                                   java.lang.String processID)
                            throws ManifoldCFException

Insert a new record into the jobqueue table (as part of adding an initial reference).

Parameters:

jobID - is the job identifier.

docHash - is the hash of the local document identifier.

docID - is the local document identifier.

Throws:

ManifoldCFException

addRemainingDocumentsInitial

public void addRemainingDocumentsInitial(java.lang.Long jobID,
                                         java.lang.String[] docIDHashes,
                                         java.lang.String processID)
                                  throws ManifoldCFException

Note the remaining documents that do NOT need to be queued. These are noted so that the doneDocumentsInitial() method does not clean up seeds from previous runs wrongly.

Throws:

ManifoldCFException

maxClauseProcessRemainingDocuments

protected int maxClauseProcessRemainingDocuments(java.lang.Long jobID)

Calculate max

processRemainingDocuments

protected void processRemainingDocuments(java.util.Map idMap,
                                         java.lang.Long jobID,
                                         java.util.ArrayList list,
                                         java.util.Map inSet)
                                  throws ManifoldCFException

Process the specified set of documents.

Throws:

ManifoldCFException

maxClauseUpdateRemainingDocuments

protected int maxClauseUpdateRemainingDocuments()

Get the maximum count

updateRemainingDocuments

protected void updateRemainingDocuments(java.util.ArrayList list,
                                        java.lang.String processID)
                                 throws ManifoldCFException

Update the specified set of documents to be "NEWSEED"

Throws:

ManifoldCFException

doneDocumentsInitial

public void doneDocumentsInitial(java.lang.Long jobID,
                                 boolean isPartial)
                          throws ManifoldCFException

Complete the initial set of documents. This method converts the seeding statuses for the job to their steady-state values. SEEDSTATUS_SEED becomes SEEDSTATUS_NOTSEED, and SEEDSTATUS_NEWSEED becomes SEEDSTATUS_SEED. If the seeding was partial, then all previous seeds are preserved as such.

Parameters:

jobID - is the job identifier.

isPartial - is true of the passed list of seeds is not complete.

Throws:

ManifoldCFException

getAllSeeds

public java.lang.String[] getAllSeeds(java.lang.Long jobID)
                               throws ManifoldCFException

Get all the current seeds. Returns the seed document identifiers for a job.

Parameters:

jobID - is the job identifier.

Returns:

the document identifier hashes that are currently considered to be seeds.

Throws:

ManifoldCFException

updateExistingRecord

public void updateExistingRecord(java.lang.Long recordID,
                                 int currentStatus,
                                 java.lang.Long checkTimeValue,
                                 long desiredExecuteTime,
                                 boolean otherChangesSeen,
                                 IPriorityCalculator desiredPriority,
                                 java.lang.String[] prereqEvents)
                          throws ManifoldCFException

Update an existing record (as the result of a reference add). The record is presumed to exist and have been locked, via "FOR UPDATE".

Throws:

ManifoldCFException

insertNewRecord

public void insertNewRecord(java.lang.Long jobID,
                            java.lang.String docIDHash,
                            java.lang.String docID,
                            IPriorityCalculator desiredDocPriority,
                            long desiredExecuteTime,
                            java.lang.String[] prereqEvents)
                     throws ManifoldCFException

Insert a new record into the jobqueue table (as part of adding a child reference).

Throws:

ManifoldCFException

seedstatusToString

public static java.lang.String seedstatusToString(int status)
                                           throws ManifoldCFException

Convert seedstatus value to a string.

Throws:

ManifoldCFException

stringToSeedstatus

public static int stringToSeedstatus(java.lang.String x)
                              throws ManifoldCFException

Convert seedstatus field value to a boolean.

Throws:

ManifoldCFException

stringToAction

public static int stringToAction(java.lang.String value)
                          throws ManifoldCFException

Convert action field value to integer.

Throws:

ManifoldCFException

actionToString

public static java.lang.String actionToString(int action)
                                       throws ManifoldCFException

Convert integer to action string

Throws:

ManifoldCFException

stringToNeedPriority

public static int stringToNeedPriority(java.lang.String value)
                                throws ManifoldCFException

Convert need priority value to boolean.

Throws:

ManifoldCFException

needPriorityToString

public static java.lang.String needPriorityToString(int value)
                                             throws ManifoldCFException

Convert boolean to need priority value.

Throws:

ManifoldCFException

stringToStatus

public static int stringToStatus(java.lang.String value)
                          throws ManifoldCFException

Convert status field value to integer.

Parameters:

value - is the string.

Returns:

the integer.

Throws:

ManifoldCFException

statusToString

public static java.lang.String statusToString(int status)
                                       throws ManifoldCFException

Convert status to string.

Parameters:

status - is the status value.

Returns:

the database string.

Throws:

ManifoldCFException

getHashCode

public static java.lang.String getHashCode(java.lang.String documentIdentifier)
                                    throws ManifoldCFException

Get a hash value from a document id string. This will convert the string into something that can fit in 20 characters. (Someday this will be an MD5 hash, but for now just use java hashing.)

Parameters:

documentIdentifier - is the input document id string.

Returns:

the hash code.

Throws:

ManifoldCFException