org.apache.manifoldcf.crawler.jobs

Class JobManager

java.lang.Object

org.apache.manifoldcf.crawler.jobs.JobManager

All Implemented Interfaces:

IJobManager

public class JobManager
extends java.lang.Object
implements IJobManager

This is the main job manager. It provides methods that support both job definition, and the threads that execute the jobs.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	JobManager.JobqueueRecord Class for tracking existing jobqueue row data
protected static class	JobManager.MutableInteger Mutable integer class.
protected static class	JobManager.QueueHashItem This class contains information per job on how many queue items have so far been accumulated.
protected static class	JobManager.ThrottleJobItem This class represents the information stored PER JOB in the throttling structure.
protected static class	JobManager.ThrottleLimit This class provides the throttling limits for the job queueing query.
protected static class	JobManager.ThrottleLimitSpec This is a class which describes an individual throttle limit, in fetches.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	_rcsid
protected Carrydown	carryDown
protected static java.lang.String	cleanStufferLock
protected IRepositoryConnectionManager	connectionMgr
protected IConnectorManager	connectorMgr
protected IDBInterface	database
protected static java.lang.String	deleteStufferLock
protected EventManager	eventManager
protected static java.lang.String	expireStufferLock
protected HopCount	hopCount
protected static java.lang.String	hopLock
protected JobQueue	jobQueue
protected static java.lang.String	jobResumeLock
protected Jobs	jobs
protected static java.lang.String	jobStopLock
protected ILockManager	lockManager
protected INotificationConnectionManager	notificationMgr
protected IOutputConnectorManager	outputConnectorMgr
protected IOutputConnectionManager	outputMgr
protected static java.util.Random	random
protected IRepositoryConnectorPool	repositoryConnectorPool
protected static java.lang.String	reprioritizationLock
protected static java.lang.String	stufferLock
protected IThreadContext	threadContext
protected ITransformationConnectorManager	transformationConnectorMgr
protected ITransformationConnectionManager	transformationMgr

Fields inherited from interface org.apache.manifoldcf.crawler.interfaces.IJobManager

ACTION_REMOVE, ACTION_RESCAN, DOCSTATE_NEVERPROCESSED, DOCSTATE_OUTOFSCOPE, DOCSTATE_PREVIOUSLYPROCESSED, DOCSTATUS_DELETING, DOCSTATUS_EXPIRING, DOCSTATUS_HOPCOUNTEXCEEDED, DOCSTATUS_INACTIVE, DOCSTATUS_PROCESSING, DOCSTATUS_READYFOREXPIRATION, DOCSTATUS_READYFORPROCESSING, DOCSTATUS_WAITINGFOREVER, DOCSTATUS_WAITINGFOREXPIRATION, DOCSTATUS_WAITINGFORPROCESSING, STOP_ERRORABORT, STOP_MANUALABORT, STOP_MANUALPAUSE, STOP_RESTART, STOP_SCHEDULEPAUSE

Constructor Summary

Constructors

Constructor and Description
JobManager(IThreadContext threadContext, IDBInterface database) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	addBucketExtract(java.lang.StringBuilder sb, java.util.ArrayList list, java.lang.String columnPrefix, java.lang.String columnName, BucketDescription bucketDesc) Turn a bucket description into a return column.
protected boolean	addCriteria(java.lang.StringBuilder sb, java.util.ArrayList list, java.lang.String fieldPrefix, java.lang.String connectionName, StatusFilterCriteria criteria, boolean whereEmitted) Add criteria clauses to query.
void	addDocument(java.lang.String processID, java.lang.Long jobID, java.lang.String[] legalLinkTypes, java.lang.String docIDHash, java.lang.String docID, java.lang.String parentIdentifierHash, java.lang.String relationshipType, int hopcountMethod, java.lang.String[] dataNames, java.lang.Object[][] dataValues, IPriorityCalculator priority, java.lang.String[] prereqEventNames) Add a document to the queue.
void	addDocuments(java.lang.String processID, java.lang.Long jobID, java.lang.String[] legalLinkTypes, java.lang.String[] docIDHashes, java.lang.String[] docIDs, java.lang.String parentIdentifierHash, java.lang.String relationshipType, int hopcountMethod, java.lang.String[][] dataNames, java.lang.Object[][][] dataValues, IPriorityCalculator[] documentPriorities, java.lang.String[][] prereqEventNames) Add documents to the queue in bulk.
void	addDocumentsInitial(java.lang.String processID, java.lang.Long jobID, java.lang.String[] legalLinkTypes, java.lang.String[] docIDHashes, java.lang.String[] docIDs, boolean overrideSchedule, int hopcountMethod, IPriorityCalculator[] documentPriorities, java.lang.String[][] prereqEventNames) Add an initial set of documents to the queue.
protected void	addLimits(java.lang.StringBuilder sb, int startRow, int maxRowCount) Add limit and offset.
protected void	addOrdering(java.lang.StringBuilder sb, java.lang.String[] completeFieldList, SortOrder sort) Add ordering.
void	addRemainingDocumentsInitial(java.lang.String processID, java.lang.Long jobID, java.lang.String[] legalLinkTypes, java.lang.String[] docIDHashes, int hopcountMethod) Add an initial set of remaining documents to the queue.
protected void	addWhereClause(java.lang.StringBuilder sb, java.util.ArrayList list, java.lang.String whereClause, java.util.ArrayList whereParams, boolean wherePresent)
void	assessMarkedJobs() Assess jobs marked to be in need of assessment for connector status changes.
boolean	beginEventSequence(java.lang.String processID, java.lang.String eventName) Begin an event sequence.
protected void	buildCountsUsingGroupBy(java.lang.String whereClause, java.util.ArrayList whereParams, java.util.Map<java.lang.Long,java.lang.Long> set2Hash, java.util.Map<java.lang.Long,java.lang.Long> set3Hash, java.util.Map<java.lang.Long,java.lang.Long> set4Hash, java.util.Map<java.lang.Long,java.lang.Boolean> set2Exact, java.util.Map<java.lang.Long,java.lang.Boolean> set3Exact, java.util.Map<java.lang.Long,java.lang.Boolean> set4Exact)
protected void	buildCountsUsingIndividualQueries(java.lang.String whereClause, java.util.ArrayList whereParams, int maxCount, java.util.Map<java.lang.Long,java.lang.Long> set2Hash, java.util.Map<java.lang.Long,java.lang.Long> set3Hash, java.util.Map<java.lang.Long,java.lang.Long> set4Hash, java.util.Map<java.lang.Long,java.lang.Boolean> set2Exact, java.util.Map<java.lang.Long,java.lang.Boolean> set3Exact, java.util.Map<java.lang.Long,java.lang.Boolean> set4Exact)
protected static ClauseDescription	buildOutstandingClause()
protected static ClauseDescription	buildProcessedClause()
protected static java.util.HashMap	buildReorderMap(java.lang.String[] originalIDHashes, java.lang.String[] reorderedIDHashes) Build a reorder map, describing how to convert an original index into a reordered index.
protected DocumentDescription[]	calculateAffectedDeleteCarrydownChildren(java.lang.Long jobID, java.lang.String[] docIDHashes) Helper method: Find the document descriptions that will be affected due to carrydown row deletions.
protected DocumentDescription[]	calculateAffectedRestoreCarrydownChildren(java.lang.Long jobID, java.lang.String[] parentIDHashes) Helper method: Calculate the unique set of affected carrydown children resulting from a "restoreRecords" operation.
void	carrydownChangeDocument(DocumentDescription documentDescription, IPriorityCalculator docPriority) Requeue a document because of carrydown changes.
void	carrydownChangeDocumentMultiple(DocumentDescription[] documentDescriptions, IPriorityCalculator[] docPriorities) Requeue a document set because of carrydown changes.
boolean	checkIfNotificationReference(java.lang.String connectionName) See if there's a reference to a notification connection name.
boolean	checkIfOutputReference(java.lang.String connectionName) See if there's a reference to an output connection name.
boolean	checkIfReference(java.lang.String connectionName) See if there's a reference to a connection name.
boolean	checkIfTransformationReference(java.lang.String connectionName) See if there's a reference to a transformation connection name.
boolean	checkJobActive(java.lang.Long jobID) Verify that a specific job is indeed still active.
boolean	checkJobBusy(java.lang.Long jobID) Verify if a job is still processing documents, or no longer has any outstanding active documents
protected static java.lang.Long	checkTimeMatch(long startTime, long currentTimestamp, EnumeratedValues daysOfWeek, EnumeratedValues daysOfMonth, EnumeratedValues months, EnumeratedValues years, EnumeratedValues hours, EnumeratedValues minutes, java.lang.String timezone, java.lang.Long duration) Check if the specified job parameters have a 'hit' within the specified interval.
void	cleanupProcessData() Reset the job queue for all process IDs.
void	cleanupProcessData(java.lang.String processID) Reset the job queue for an individual process ID.
void	clearAllDocumentPriorities() Clear all document priorities, in preparation for reprioritization of all previously-prioritized documents.
void	clearJobSeedingState(java.lang.Long jobID) Clear job seeding state.
void	completeEventSequence(java.lang.String eventName) Complete an event sequence.
IJobDescription	createJob() Create a new job.
void	deinstall() Uninstall.
void	deleteIngestedDocumentIdentifiers(DocumentDescription[] identifiers) Delete ingested document identifiers (as part of deleting the owning job).
void	deleteJob(java.lang.Long id) Delete a job.
void	deleteJobsReadyForDelete() Delete jobs in need of being deleted (which are marked "ready for delete").
protected DocumentDescription[]	doDeleteMultiple(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) Delete documents with no repercussions.
void	doneDocumentsInitial(java.lang.Long jobID, java.lang.String[] legalLinkTypes, boolean isPartial, int hopcountMethod) Signal that a seeding pass has been done.
protected static java.lang.String[]	eliminateDuplicates(java.lang.String[] docIDHashes) Eliminate duplicates, and sort
protected boolean	emitClauseStart(java.lang.StringBuilder sb, boolean whereEmitted) Emit a WHERE or an AND, depending...
boolean	errorAbort(java.lang.Long jobID, java.lang.String errorText) Abort a running job due to a fatal error condition.
void	exportConfiguration(java.io.OutputStream os) Export configuration
protected void	fetchAndProcessDocuments(java.util.List<DocumentDescription> answers, java.lang.Long currentTimeValue, java.lang.Long currentPriorityValue, JobManager.ThrottleLimit vList, IRepositoryConnection[] connections, java.lang.String processID) Fetch and process documents matching the passed-in criteria
boolean[]	findHopCounts(java.lang.Long jobID, java.lang.String[] legalLinkTypes, java.lang.String[] docIDHashes, java.lang.String linkType, int limit, int hopcountMethod) Get the specified hop counts, with the limit as described.
IJobDescription[]	findJobsForConnection(java.lang.String connectionName) Get the job IDs associated with a given connection name.
DocumentDescription[]	finishDocuments(java.lang.Long jobID, java.lang.String[] legalLinkTypes, java.lang.String[] parentIdentifierHashes, int hopcountMethod) Complete adding child documents to the queue, for a set of documents.
void	finishJobResumes(long timestamp, java.util.List<IJobDescription> modifiedJobs) Complete the sequence that resumes jobs, either from a pause or from a scheduling window wait.
void	finishJobs() Put all eligible jobs in the "shutting down" state.
void	finishJobStops(long timestamp, java.util.List<IJobDescription> modifiedJobs, java.util.List<java.lang.Integer> stopNotificationTypes) Complete the sequence that stops jobs, either for abort, pause, or because of a scheduling window.
IResultSet	genDocumentStatus(java.lang.String connectionName, StatusFilterCriteria filterCriteria, SortOrder sortOrder, int startRow, int rowCount) Run a 'document status' report.
IResultSet	genQueueStatus(java.lang.String connectionName, StatusFilterCriteria filterCriteria, SortOrder sortOrder, BucketDescription idBucketDescription, int startRow, int rowCount) Run a 'queue status' report.
IJobDescription[]	getAllJobs() Load a sorted list of job descriptions.
java.lang.String[]	getAllSeeds(java.lang.Long jobID) Get all the current seeds.
JobStatus[]	getAllStatus() Get a list of all jobs, and their status information.
JobStatus[]	getAllStatus(boolean includeCounts) Get a list of all jobs, and their status information.
JobStatus[]	getAllStatus(boolean includeCounts, int maxCount) Get a list of all jobs, and their status information.
DocumentSetAndFlags	getExpiredDocuments(java.lang.String processID, int n, long currentTime) Get up to the next n documents to be expired.
JobStatus[]	getFinishedJobs() Get a list of completed jobs, and their statistics.
JobStatus[]	getFinishedJobs(boolean includeCounts) Get a list of completed jobs, and their statistics.
JobStatus[]	getFinishedJobs(boolean includeCounts, int maxCount) Get a list of completed jobs, and their statistics.
protected java.lang.String	getHopLockName(java.lang.Long jobID) Get the hoplock for a given job ID
JobNotifyRecord[]	getJobsReadyForDelete(java.lang.String processID) Find the list of jobs that need to have their connectors notified of job deletion.
JobDeleteRecord[]	getJobsReadyForDeleteCleanup(java.lang.String processID) Get the list of jobs that are ready for delete cleanup.
JobNotifyRecord[]	getJobsReadyForInactivity(java.lang.String processID) Find the list of jobs that need to have their connectors notified of job completion.
JobSeedingRecord[]	getJobsReadyForSeeding(java.lang.String processID, long currentTime) Get the list of jobs that are ready for seeding.
JobStartRecord[]	getJobsReadyForStartup(java.lang.String processID) Get the list of jobs that are ready for startup.
DocumentSetAndFlags	getNextCleanableDocuments(java.lang.String processID, int maxCount, long currentTime) Get list of cleanable document descriptions.
DocumentDescription[]	getNextDeletableDocuments(java.lang.String processID, int maxCount, long currentTime) Get list of deletable document descriptions.
DocumentDescription[]	getNextDocuments(java.lang.String processID, int n, long currentTime, long interval, BlockingDocuments blockingDocuments, PerformanceStatistics statistics, DepthStatistics scanRecord) /** Get up to the next n document(s) to be fetched and processed.
DocumentDescription[]	getNextNotYetProcessedReprioritizationDocuments(java.lang.String processID, int n) Get a list of not-yet-processed documents to reprioritize.
protected long	getRandomAmount() Sleep a random amount of time after a transaction abort.
JobStatus[]	getRunningJobs() Get a list of running jobs.
JobStatus[]	getRunningJobs(boolean includeCounts) Get a list of running jobs.
JobStatus[]	getRunningJobs(boolean includeCounts, int maxCount) Get a list of running jobs.
JobStatus	getStatus(java.lang.Long jobID) Get the status of a job.
JobStatus	getStatus(java.lang.Long jobID, boolean includeCounts) Get the status of a job.
JobStatus	getStatus(java.lang.Long jobID, boolean includeCounts, int maxCount) Get the status of a job.
protected java.lang.String[]	getUnindexableDocumentIdentifiers(DocumentDescription[] documentIdentifiers, java.lang.String connectionName) Get a list of document identifiers that should actually be deleted from the index, from a list that might contain identifiers that are shared with other jobs, which are targeted to the same output connection.
void	importConfiguration(java.io.InputStream is) Import configuration
void	inactivateJob(java.lang.Long jobID) Inactivate a job, from the notification state.
void	install() Install.
IJobDescription	load(java.lang.Long id) Load a job for editing.
IJobDescription	load(java.lang.Long id, boolean readOnly) Load a job.
protected static java.lang.String	makeCompositeID(java.lang.String docIDHash, java.lang.String connectionName) Create a composite document hash key.
protected JobStatus[]	makeJobStatus(java.lang.String whereClause, java.util.ArrayList whereParams, boolean includeCounts, int maxCount) Make a job status array from a query result.
void	manualAbort(java.lang.Long jobID) Manually abort a running job.
void	manualAbortRestart(java.lang.Long jobID) Manually restart a running job.
void	manualAbortRestart(java.lang.Long jobID, boolean requestMinimum) Manually restart a running job.
void	manualStart(java.lang.Long jobID) Manually start a job.
void	manualStart(java.lang.Long jobID, boolean requestMinimum) Manually start a job.
protected static java.lang.Integer	mapToNotificationType(int jobStatus, boolean noErrorText)
DocumentDescription[]	markDocumentCleanedUp(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) Delete from queue as a result of cleaning up an unreachable document.
DocumentDescription[]	markDocumentCleanedUpMultiple(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) Delete from queue as a result of cleaning up an unreachable document.
void	markDocumentCompleted(DocumentDescription documentDescription) Note completion of document processing by a job thread of a document.
void	markDocumentCompletedMultiple(DocumentDescription[] documentDescriptions) Note completion of document processing by a job thread of a document.
DocumentDescription[]	markDocumentDeleted(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) Delete from queue as a result of processing of an active document.
DocumentDescription[]	markDocumentDeletedMultiple(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) Delete from queue as a result of processing of an active document.
DocumentDescription[]	markDocumentExpired(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) Delete from queue as a result of expiration of an active document.
DocumentDescription[]	markDocumentExpiredMultiple(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) Delete from queue as a result of expiration of an active document.
DocumentDescription[]	markDocumentHopcountRemoval(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription documentDescription, int hopcountMethod) Mark hopcount removal from queue as a result of processing of an active document.
DocumentDescription[]	markDocumentHopcountRemovalMultiple(java.lang.Long jobID, java.lang.String[] legalLinkTypes, DocumentDescription[] documentDescriptions, int hopcountMethod) Mark hopcount removal from queue as a result of processing of an active document.
protected int	maxClauseProcessDeleteHashSet() Get maximum count.
protected void	noDocPriorities(java.lang.Long jobID)
void	noteConnectionChange(java.lang.String connectionName) Note a change in connection configuration.
protected void	noteConnectionDeregistration(java.util.List<java.lang.String> list) Note deregistration for a batch of connection names.
protected void	noteConnectionRegistration(java.util.List<java.lang.String> list) Note registration for a batch of connection names.
void	noteConnectorDeregistration(java.lang.String[] connectionNames) Note the deregistration of a connector used by the specified connections.
void	noteConnectorRegistration(java.lang.String[] connectionNames) Note the registration of a connector used by the specified connections.
void	noteJobDeleteStarted(java.lang.Long jobID, long startTime) Note job delete started.
void	noteJobSeeded(java.lang.Long jobID, java.lang.String seedingVersion) Note job seeded.
void	noteJobStarted(java.lang.Long jobID, long startTime, java.lang.String seedingVersion) Note job started.
void	noteNotificationConnectionChange(java.lang.String connectionName) Note a change in notification connection configuration.
protected void	noteNotificationConnectionDeregistration(java.util.List<java.lang.String> list) Note deregistration for a batch of notification connection names.
protected void	noteNotificationConnectionRegistration(java.util.List<java.lang.String> list) Note registration for a batch of connection names.
void	noteNotificationConnectorDeregistration(java.lang.String[] connectionNames) Note the deregistration of a notification connector used by the specified connections.
void	noteNotificationConnectorRegistration(java.lang.String[] connectionNames) Note the registration of a notification connector used by the specified connections.
protected void	noteNotificationJobDeregistration(java.util.List<java.lang.Long> jobIDs)
protected void	noteNotificationJobRegistration(java.util.List<java.lang.Long> jobIDs)
void	noteOutputConnectionChange(java.lang.String connectionName) Note a change in output connection configuration.
protected void	noteOutputConnectionDeregistration(java.util.List<java.lang.String> list) Note deregistration for a batch of output connection names.
protected void	noteOutputConnectionRegistration(java.util.List<java.lang.String> list) Note registration for a batch of output connection names.
void	noteOutputConnectorDeregistration(java.lang.String[] connectionNames) Note the deregistration of an output connector used by the specified connections.
void	noteOutputConnectorRegistration(java.lang.String[] connectionNames) Note the registration of an output connector used by the specified connections.
protected void	noteOutputJobDeregistration(java.util.List<java.lang.Long> jobIDs)
protected void	noteOutputJobRegistration(java.util.List<java.lang.Long> jobIDs)
void	noteTransformationConnectionChange(java.lang.String connectionName) Note a change in transformation connection configuration.
protected void	noteTransformationConnectionDeregistration(java.util.List<java.lang.String> list) Note deregistration for a batch of transformation connection names.
protected void	noteTransformationConnectionRegistration(java.util.List<java.lang.String> list) Note registration for a batch of transformation connection names.
void	noteTransformationConnectorDeregistration(java.lang.String[] connectionNames) Note the deregistration of a transformation connector used by the specified connections.
void	noteTransformationConnectorRegistration(java.lang.String[] connectionNames) Note the registration of a transformation connector used by the specified connections.
protected void	noteTransformationJobDeregistration(java.util.List<java.lang.Long> jobIDs)
protected void	noteTransformationJobRegistration(java.util.List<java.lang.Long> jobIDs)
void	pauseJob(java.lang.Long jobID) Pause a job.
void	prepareDeleteScan(java.lang.Long jobID) Prepare for a delete scan.
void	prepareForClusterStart() Prepare to start the entire cluster.
protected void	prepareFullScan(java.lang.Long jobID, java.lang.String[] legalLinkTypes, int hopcountMethod) Prepare for a full scan.
void	prepareJobScan(java.lang.Long jobID, java.lang.String[] legalLinkTypes, int hopcountMethod, int connectorModel, boolean continuousJob, boolean fromBeginningOfTime, boolean requestMinimum) Prepare a job to be run.
protected void	processDeleteHashSet(java.lang.Long jobID, java.util.HashMap resultHash, java.util.ArrayList list) Helper method: look up rows affected by a deleteRecords operation.
protected void	processParentHashSet(java.lang.Long jobID, java.util.HashMap resultHash, java.util.ArrayList list) Helper method: look up rows affected by a restoreRecords operation.
protected void	queueAllExisting(java.lang.Long jobID, java.lang.String[] legalLinkTypes) Queue all existing.
protected EnumeratedValues	readEnumeratedValues(java.io.InputStream is)
void	removeJob(java.lang.Long jobID) Remove a job, from the notification state.
void	requeueDocument(DocumentDescription documentDescription, java.lang.Long executeTime, int action) Requeue a document for further processing in the future.
void	requeueDocumentMultiple(DocumentDescription[] documentDescriptions, java.lang.Long[] executeTimes, int[] actions) Requeue a document for further processing in the future.
void	resetCleaningDocument(DocumentDescription documentDescription, long checkTime) Reset a cleaning document back to its former state.
void	resetCleaningDocumentMultiple(DocumentDescription[] documentDescriptions, long checkTime) Reset a set of cleaning documents for further processing in the future.
void	resetDeleteNotifyJob(java.lang.Long jobID) Reset a job that is delete notifying back to "ready for delete notify" state.
void	resetDeleteStartupWorkerStatus(java.lang.String processID) Reset as part of restoring delete startup threads.
void	resetDeletingDocument(DocumentDescription documentDescription, long checkTime) Reset a deleting document back to its former state.
void	resetDeletingDocumentMultiple(DocumentDescription[] documentDescriptions, long checkTime) Reset a set of deleting documents for further processing in the future.
void	resetDocCleanupWorkerStatus(java.lang.String processID) Reset as part of restoring doc cleanup threads.
void	resetDocDeleteWorkerStatus(java.lang.String processID) Reset as part of restoring doc delete threads.
void	resetDocument(DocumentDescription documentDescription, long executeTime, int action, long failTime, int failCount) Reset an active document back to its former state.
void	resetDocumentMultiple(DocumentDescription[] documentDescriptions, long executeTime, int action, long failTime, int failCount) Reset a set of documents for further processing in the future.
void	resetDocumentWorkerStatus(java.lang.String processID) Reset as part of restoring document worker threads.
void	resetJobs(long currentTime, java.util.List<IJobDescription> resetJobs) Reset eligible jobs either back to the "inactive" state, or make them active again.
void	resetJobSchedule(java.lang.Long jobID) Reset job schedule.
void	resetNotificationWorkerStatus(java.lang.String processID) Reset as part of restoring notification threads.
void	resetNotifyJob(java.lang.Long jobID) Reset a job that is notifying back to "ready for notify" state.
void	resetSeedingWorkerStatus(java.lang.String processID) Reset as part of restoring seeding threads.
void	resetSeedJob(java.lang.Long jobID) Reset a seeding job back to "active" state.
void	resetStartDeleteJob(java.lang.Long jobID) Reset a job starting for delete back to "ready for delete" state.
void	resetStartupJob(java.lang.Long jobID) Reset a starting job back to "ready for startup" state.
void	resetStartupWorkerStatus(java.lang.String processID) Reset as part of restoring startup threads.
void	restartJob(java.lang.Long jobID) Restart a paused job.
java.lang.String[]	retrieveParentData(java.lang.Long jobID, java.lang.String docIDHash, java.lang.String dataName) Retrieve specific parent data for a given document.
CharacterInput[]	retrieveParentDataAsFiles(java.lang.Long jobID, java.lang.String docIDHash, java.lang.String dataName) Retrieve specific parent data for a given document.
void	retryDeleteNotification(JobNotifyRecord jnr, long failTime, int failCount) Retry delete notification.
void	retryNotification(JobNotifyRecord jnr, long failTime, int failCount) Retry notification.
void	retrySeeding(JobSeedingRecord jsr, long failTime, int failCount) Retry seeding.
void	retryStartup(JobStartRecord jsr, long failTime, int failCount) Retry startup.
void	revertDocuments(java.lang.Long jobID, java.lang.String[] legalLinkTypes, java.lang.String[] parentIdentifierHashes) Undo the addition of child documents to the queue, for a set of documents.
void	save(IJobDescription jobDescription) Save a job.
protected void	sleepFor(long amt)
void	startJobs(long currentTime, java.util.List<java.lang.Long> unwaitList) Start all jobs in need of starting.
void	waitJobs(long currentTime, java.util.List<java.lang.Long> waitList) Put active or paused jobs in wait state, if they've exceeded their window.
void	writeDocumentPriorities(DocumentDescription[] documentDescriptions, IPriorityCalculator[] priorities) Save a set of document priorities.
protected static void	writeEnumeratedValues(java.io.OutputStream os, EnumeratedValues ev)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_rcsid

public static final java.lang.String _rcsid

See Also:

Constant Field Values

stufferLock

protected static final java.lang.String stufferLock

See Also:

Constant Field Values

reprioritizationLock

protected static final java.lang.String reprioritizationLock

See Also:

Constant Field Values

deleteStufferLock

protected static final java.lang.String deleteStufferLock

See Also:

Constant Field Values

expireStufferLock

protected static final java.lang.String expireStufferLock

See Also:

Constant Field Values

cleanStufferLock

protected static final java.lang.String cleanStufferLock

See Also:

Constant Field Values

jobStopLock

protected static final java.lang.String jobStopLock

See Also:

Constant Field Values

jobResumeLock

protected static final java.lang.String jobResumeLock

See Also:

Constant Field Values

hopLock

protected static final java.lang.String hopLock

See Also:

Constant Field Values

database

protected final IDBInterface database

outputMgr

protected final IOutputConnectionManager outputMgr

connectionMgr

protected final IRepositoryConnectionManager connectionMgr

notificationMgr

protected final INotificationConnectionManager notificationMgr

transformationMgr

protected final ITransformationConnectionManager transformationMgr

outputConnectorMgr

protected final IOutputConnectorManager outputConnectorMgr

connectorMgr

protected final IConnectorManager connectorMgr

transformationConnectorMgr

protected final ITransformationConnectorManager transformationConnectorMgr

repositoryConnectorPool

protected final IRepositoryConnectorPool repositoryConnectorPool

lockManager

protected final ILockManager lockManager

threadContext

protected final IThreadContext threadContext

jobQueue

protected final JobQueue jobQueue

jobs

protected final Jobs jobs

hopCount

protected final HopCount hopCount

carryDown

protected final Carrydown carryDown

eventManager

protected final EventManager eventManager

random

protected static java.util.Random random

Constructor Detail

JobManager

public JobManager(IThreadContext threadContext,
                  IDBInterface database)
           throws ManifoldCFException

Constructor.

Parameters:

threadContext - is the thread context.

database - is the database.

Throws:

ManifoldCFException

Method Detail

install

public void install()
             throws ManifoldCFException

Install.

Specified by:

install in interface IJobManager

Throws:

ManifoldCFException

deinstall

public void deinstall()
               throws ManifoldCFException

Uninstall.

Specified by:

deinstall in interface IJobManager

Throws:

ManifoldCFException

exportConfiguration

public void exportConfiguration(java.io.OutputStream os)
                         throws java.io.IOException,
                                ManifoldCFException

Export configuration

Specified by:

exportConfiguration in interface IJobManager

Throws:

java.io.IOException

ManifoldCFException

writeEnumeratedValues

protected static void writeEnumeratedValues(java.io.OutputStream os,
                                            EnumeratedValues ev)
                                     throws java.io.IOException

Throws:

java.io.IOException

importConfiguration

public void importConfiguration(java.io.InputStream is)
                         throws java.io.IOException,
                                ManifoldCFException

Import configuration

Specified by:

importConfiguration in interface IJobManager

Throws:

java.io.IOException

ManifoldCFException

readEnumeratedValues

protected EnumeratedValues readEnumeratedValues(java.io.InputStream is)
                                         throws java.io.IOException

Throws:

java.io.IOException

noteConnectorDeregistration

public void noteConnectorDeregistration(java.lang.String[] connectionNames)
                                 throws ManifoldCFException

Note the deregistration of a connector used by the specified connections. This method will be called when the connector is deregistered. Jobs that use these connections must therefore enter appropriate states.

Specified by:

noteConnectorDeregistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteConnectionDeregistration

protected void noteConnectionDeregistration(java.util.List<java.lang.String> list)
                                     throws ManifoldCFException

Note deregistration for a batch of connection names.

Throws:

ManifoldCFException

noteConnectorRegistration

public void noteConnectorRegistration(java.lang.String[] connectionNames)
                               throws ManifoldCFException

Note the registration of a connector used by the specified connections. This method will be called when a connector is registered, on which the specified connections depend.

Specified by:

noteConnectorRegistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteConnectionRegistration

protected void noteConnectionRegistration(java.util.List<java.lang.String> list)
                                   throws ManifoldCFException

Note registration for a batch of connection names.

Throws:

ManifoldCFException

noteNotificationConnectorDeregistration

public void noteNotificationConnectorDeregistration(java.lang.String[] connectionNames)
                                             throws ManifoldCFException

Note the deregistration of a notification connector used by the specified connections. This method will be called when the connector is deregistered. Jobs that use these connections must therefore enter appropriate states.

Specified by:

noteNotificationConnectorDeregistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteNotificationConnectionDeregistration

protected void noteNotificationConnectionDeregistration(java.util.List<java.lang.String> list)
                                                 throws ManifoldCFException

Note deregistration for a batch of notification connection names.

Throws:

ManifoldCFException

noteNotificationJobDeregistration

protected void noteNotificationJobDeregistration(java.util.List<java.lang.Long> jobIDs)
                                          throws ManifoldCFException

Throws:

ManifoldCFException

noteNotificationConnectorRegistration

public void noteNotificationConnectorRegistration(java.lang.String[] connectionNames)
                                           throws ManifoldCFException

Note the registration of a notification connector used by the specified connections. This method will be called when a connector is registered, on which the specified connections depend.

Specified by:

noteNotificationConnectorRegistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteNotificationConnectionRegistration

protected void noteNotificationConnectionRegistration(java.util.List<java.lang.String> list)
                                               throws ManifoldCFException

Note registration for a batch of connection names.

Throws:

ManifoldCFException

noteNotificationJobRegistration

protected void noteNotificationJobRegistration(java.util.List<java.lang.Long> jobIDs)
                                        throws ManifoldCFException

Throws:

ManifoldCFException

noteOutputConnectorDeregistration

public void noteOutputConnectorDeregistration(java.lang.String[] connectionNames)
                                       throws ManifoldCFException

Note the deregistration of an output connector used by the specified connections. This method will be called when the connector is deregistered. Jobs that use these connections must therefore enter appropriate states.

Specified by:

noteOutputConnectorDeregistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteOutputConnectionDeregistration

protected void noteOutputConnectionDeregistration(java.util.List<java.lang.String> list)
                                           throws ManifoldCFException

Note deregistration for a batch of output connection names.

Throws:

ManifoldCFException

noteOutputJobDeregistration

protected void noteOutputJobDeregistration(java.util.List<java.lang.Long> jobIDs)
                                    throws ManifoldCFException

Throws:

ManifoldCFException

noteOutputConnectorRegistration

public void noteOutputConnectorRegistration(java.lang.String[] connectionNames)
                                     throws ManifoldCFException

Note the registration of an output connector used by the specified connections. This method will be called when a connector is registered, on which the specified connections depend.

Specified by:

noteOutputConnectorRegistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteOutputConnectionRegistration

protected void noteOutputConnectionRegistration(java.util.List<java.lang.String> list)
                                         throws ManifoldCFException

Note registration for a batch of output connection names.

Throws:

ManifoldCFException

noteOutputJobRegistration

protected void noteOutputJobRegistration(java.util.List<java.lang.Long> jobIDs)
                                  throws ManifoldCFException

Throws:

ManifoldCFException

noteTransformationConnectorDeregistration

public void noteTransformationConnectorDeregistration(java.lang.String[] connectionNames)
                                               throws ManifoldCFException

Note the deregistration of a transformation connector used by the specified connections. This method will be called when the connector is deregistered. Jobs that use these connections must therefore enter appropriate states.

Specified by:

noteTransformationConnectorDeregistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteTransformationConnectionDeregistration

protected void noteTransformationConnectionDeregistration(java.util.List<java.lang.String> list)
                                                   throws ManifoldCFException

Note deregistration for a batch of transformation connection names.

Throws:

ManifoldCFException

noteTransformationJobDeregistration

protected void noteTransformationJobDeregistration(java.util.List<java.lang.Long> jobIDs)
                                            throws ManifoldCFException

Throws:

ManifoldCFException

noteTransformationConnectorRegistration

public void noteTransformationConnectorRegistration(java.lang.String[] connectionNames)
                                             throws ManifoldCFException

Note the registration of a transformation connector used by the specified connections. This method will be called when a connector is registered, on which the specified connections depend.

Specified by:

noteTransformationConnectorRegistration in interface IJobManager

Parameters:

connectionNames - is the set of connection names.

Throws:

ManifoldCFException

noteTransformationConnectionRegistration

protected void noteTransformationConnectionRegistration(java.util.List<java.lang.String> list)
                                                 throws ManifoldCFException

Note registration for a batch of transformation connection names.

Throws:

ManifoldCFException

noteTransformationJobRegistration

protected void noteTransformationJobRegistration(java.util.List<java.lang.Long> jobIDs)
                                          throws ManifoldCFException

Throws:

ManifoldCFException

noteConnectionChange

public void noteConnectionChange(java.lang.String connectionName)
                          throws ManifoldCFException

Note a change in connection configuration. This method will be called whenever a connection's configuration is modified, or when an external repository change is signalled.

Specified by:

noteConnectionChange in interface IJobManager

Throws:

ManifoldCFException

noteNotificationConnectionChange

public void noteNotificationConnectionChange(java.lang.String connectionName)
                                      throws ManifoldCFException

Note a change in notification connection configuration. This method will be called whenever a notification connection's configuration is modified, or when an external repository change is signalled.

Specified by:

noteNotificationConnectionChange in interface IJobManager

Throws:

ManifoldCFException

noteOutputConnectionChange

public void noteOutputConnectionChange(java.lang.String connectionName)
                                throws ManifoldCFException

Note a change in output connection configuration. This method will be called whenever a connection's configuration is modified, or when an external target config change is signalled.

Specified by:

noteOutputConnectionChange in interface IJobManager

Throws:

ManifoldCFException

noteTransformationConnectionChange

public void noteTransformationConnectionChange(java.lang.String connectionName)
                                        throws ManifoldCFException

Note a change in transformation connection configuration. This method will be called whenever a connection's configuration is modified.

Specified by:

noteTransformationConnectionChange in interface IJobManager

Throws:

ManifoldCFException

assessMarkedJobs

public void assessMarkedJobs()
                      throws ManifoldCFException

Assess jobs marked to be in need of assessment for connector status changes.

Specified by:

assessMarkedJobs in interface IJobManager

Throws:

ManifoldCFException

getAllJobs

public IJobDescription[] getAllJobs()
                             throws ManifoldCFException

Load a sorted list of job descriptions.

Specified by:

getAllJobs in interface IJobManager

Returns:

the list, sorted by description.

Throws:

ManifoldCFException

createJob

public IJobDescription createJob()
                          throws ManifoldCFException

Create a new job.

Specified by:

createJob in interface IJobManager

Returns:

the new job.

Throws:

ManifoldCFException

getHopLockName

protected java.lang.String getHopLockName(java.lang.Long jobID)

Get the hoplock for a given job ID

deleteJob

public void deleteJob(java.lang.Long id)
               throws ManifoldCFException

Delete a job.

Specified by:

deleteJob in interface IJobManager

Parameters:

id - is the job's identifier. This method will purge all the records belonging to the job from the database, as well as remove all documents indexed by the job from the index.

Throws:

ManifoldCFException

load

public IJobDescription load(java.lang.Long id)
                     throws ManifoldCFException

Load a job for editing.

Specified by:

load in interface IJobManager

Parameters:

id - is the job's identifier.

Returns:

null if the job doesn't exist.

Throws:

ManifoldCFException

load

public IJobDescription load(java.lang.Long id,
                            boolean readOnly)
                     throws ManifoldCFException

Load a job.

Specified by:

load in interface IJobManager

Parameters:

id - is the job's identifier.

readOnly - is true if a read-only object is desired.

Returns:

null if the job doesn't exist.

Throws:

ManifoldCFException

save

public void save(IJobDescription jobDescription)
          throws ManifoldCFException

Save a job.

Specified by:

save in interface IJobManager

Parameters:

jobDescription - is the job description.

Throws:

ManifoldCFException

checkIfReference

public boolean checkIfReference(java.lang.String connectionName)
                         throws ManifoldCFException

See if there's a reference to a connection name.

Specified by:

checkIfReference in interface IJobManager

Parameters:

connectionName - is the name of the connection.

Returns:

true if there is a reference, false otherwise.

Throws:

ManifoldCFException

checkIfNotificationReference

public boolean checkIfNotificationReference(java.lang.String connectionName)
                                     throws ManifoldCFException

See if there's a reference to a notification connection name.

Specified by:

checkIfNotificationReference in interface IJobManager

Parameters:

connectionName - is the name of the connection.

Returns:

true if there is a reference, false otherwise.

Throws:

ManifoldCFException

checkIfOutputReference

public boolean checkIfOutputReference(java.lang.String connectionName)
                               throws ManifoldCFException

See if there's a reference to an output connection name.

Specified by:

checkIfOutputReference in interface IJobManager

Parameters:

connectionName - is the name of the connection.

Returns:

true if there is a reference, false otherwise.

Throws:

ManifoldCFException

checkIfTransformationReference

public boolean checkIfTransformationReference(java.lang.String connectionName)
                                       throws ManifoldCFException

See if there's a reference to a transformation connection name.

Specified by:

checkIfTransformationReference in interface IJobManager

Parameters:

connectionName - is the name of the connection.

Returns:

true if there is a reference, false otherwise.

Throws:

ManifoldCFException

findJobsForConnection

public IJobDescription[] findJobsForConnection(java.lang.String connectionName)
                                        throws ManifoldCFException

Get the job IDs associated with a given connection name.

Specified by:

findJobsForConnection in interface IJobManager

Parameters:

connectionName - is the name of the connection.

Returns:

the set of job id's associated with that connection.

Throws:

ManifoldCFException

clearJobSeedingState

public void clearJobSeedingState(java.lang.Long jobID)
                          throws ManifoldCFException

Clear job seeding state.

Specified by:

clearJobSeedingState in interface IJobManager

Parameters:

jobID - is the job ID.

Throws:

ManifoldCFException

cleanupProcessData

public void cleanupProcessData(java.lang.String processID)
                        throws ManifoldCFException

Reset the job queue for an individual process ID. If a node was shut down in the middle of doing something, sufficient information should be around in the database to allow the node's activities to be cleaned up.

Specified by:

cleanupProcessData in interface IJobManager

Parameters:

processID - is the process ID of the node we want to clean up after.

Throws:

ManifoldCFException

cleanupProcessData

public void cleanupProcessData()
                        throws ManifoldCFException

Reset the job queue for all process IDs. If a node was shut down in the middle of doing something, sufficient information should be around in the database to allow the node's activities to be cleaned up.

Specified by:

cleanupProcessData in interface IJobManager

Throws:

ManifoldCFException

prepareForClusterStart

public void prepareForClusterStart()
                            throws ManifoldCFException

Prepare to start the entire cluster. If there are no other nodes alive, then at the time the first node comes up, we need to reset the job queue for ALL processes that had been running before. This method must be called in addition to cleanupProcessData().

Specified by:

prepareForClusterStart in interface IJobManager

Throws:

ManifoldCFException

resetDocumentWorkerStatus

public void resetDocumentWorkerStatus(java.lang.String processID)
                               throws ManifoldCFException

Reset as part of restoring document worker threads.

Specified by:

resetDocumentWorkerStatus in interface IJobManager

Parameters:

processID - is the current process ID.

Throws:

ManifoldCFException

resetSeedingWorkerStatus

public void resetSeedingWorkerStatus(java.lang.String processID)
                              throws ManifoldCFException

Reset as part of restoring seeding threads.

Specified by:

resetSeedingWorkerStatus in interface IJobManager

Parameters:

processID - is the current process ID.

Throws:

ManifoldCFException

resetDocDeleteWorkerStatus

public void resetDocDeleteWorkerStatus(java.lang.String processID)
                                throws ManifoldCFException

Reset as part of restoring doc delete threads.

Specified by:

resetDocDeleteWorkerStatus in interface IJobManager

Parameters:

processID - is the current process ID.

Throws:

ManifoldCFException

resetDocCleanupWorkerStatus

public void resetDocCleanupWorkerStatus(java.lang.String processID)
                                 throws ManifoldCFException

Reset as part of restoring doc cleanup threads.

Specified by:

resetDocCleanupWorkerStatus in interface IJobManager

Parameters:

processID - is the current process ID.

Throws:

ManifoldCFException

resetDeleteStartupWorkerStatus

public void resetDeleteStartupWorkerStatus(java.lang.String processID)
                                    throws ManifoldCFException

Reset as part of restoring delete startup threads.

Specified by:

resetDeleteStartupWorkerStatus in interface IJobManager

Parameters:

processID - is the current process ID.

Throws:

ManifoldCFException

resetNotificationWorkerStatus

public void resetNotificationWorkerStatus(java.lang.String processID)
                                   throws ManifoldCFException

Reset as part of restoring notification threads.

Specified by:

resetNotificationWorkerStatus in interface IJobManager

Parameters:

processID - is the current process ID.

Throws:

ManifoldCFException

resetStartupWorkerStatus

public void resetStartupWorkerStatus(java.lang.String processID)
                              throws ManifoldCFException

Reset as part of restoring startup threads.

Specified by:

resetStartupWorkerStatus in interface IJobManager

Parameters:

processID - is the current process ID.

Throws:

ManifoldCFException

deleteIngestedDocumentIdentifiers

public void deleteIngestedDocumentIdentifiers(DocumentDescription[] identifiers)
                                       throws ManifoldCFException

Delete ingested document identifiers (as part of deleting the owning job). The number of identifiers specified is guaranteed to be less than the maxInClauseCount for the database.

Specified by:

deleteIngestedDocumentIdentifiers in interface IJobManager

Parameters:

identifiers - is the set of document identifiers.

Throws:

ManifoldCFException

getNextCleanableDocuments

public DocumentSetAndFlags getNextCleanableDocuments(java.lang.String processID,
                                                     int maxCount,
                                                     long currentTime)
                                              throws ManifoldCFException

Get list of cleanable document descriptions. This list will take into account multiple jobs that may own the same document. All documents for which a description is returned will be transitioned to the "beingcleaned" state. Documents which are not in transition and are eligible, but are owned by other jobs, will have their jobqueue entries deleted by this method.

Specified by:

getNextCleanableDocuments in interface IJobManager

Parameters:

processID - is the current process ID.

maxCount - is the maximum number of documents to return.

currentTime - is the current time; some fetches do not occur until a specific time.

Returns:

the document descriptions for these documents.

Throws:

ManifoldCFException

makeCompositeID

protected static java.lang.String makeCompositeID(java.lang.String docIDHash,
                                                  java.lang.String connectionName)

Create a composite document hash key. This consists of the document id hash plus the connection name.

getNextDeletableDocuments

public DocumentDescription[] getNextDeletableDocuments(java.lang.String processID,
                                                       int maxCount,
                                                       long currentTime)
                                                throws ManifoldCFException

Get list of deletable document descriptions. This list will take into account multiple jobs that may own the same document. All documents for which a description is returned will be transitioned to the "beingdeleted" state. Documents which are not in transition and are eligible, but are owned by other jobs, will have their jobqueue entries deleted by this method.

Specified by:

getNextDeletableDocuments in interface IJobManager

Parameters:

processID - is the current process ID.

maxCount - is the maximum number of documents to return.

currentTime - is the current time; some fetches do not occur until a specific time.

Returns:

the document descriptions for these documents.

Throws:

ManifoldCFException

getUnindexableDocumentIdentifiers

protected java.lang.String[] getUnindexableDocumentIdentifiers(DocumentDescription[] documentIdentifiers,
                                                               java.lang.String connectionName)
                                                        throws ManifoldCFException

Get a list of document identifiers that should actually be deleted from the index, from a list that might contain identifiers that are shared with other jobs, which are targeted to the same output connection. The input list is guaranteed to be smaller in size than maxInClauseCount for the database.

Parameters:

documentIdentifiers - is the set of document identifiers to consider.

connectionName - is the connection name for ALL the document identifiers.

Returns:

the set of documents which should be removed from the index, where there are no potential conflicts.

Throws:

ManifoldCFException

clearAllDocumentPriorities

public void clearAllDocumentPriorities()
                                throws ManifoldCFException

Clear all document priorities, in preparation for reprioritization of all previously-prioritized documents. This method is called to start the dynamic reprioritization cycle, which follows this method with explicit prioritization of all documents, piece-meal, using getNextNotYetProcessedReprioritizationDocuments(), and writeDocumentPriorities().

Specified by:

clearAllDocumentPriorities in interface IJobManager

Throws:

ManifoldCFException

getNextNotYetProcessedReprioritizationDocuments

public DocumentDescription[] getNextNotYetProcessedReprioritizationDocuments(java.lang.String processID,
                                                                             int n)
                                                                      throws ManifoldCFException

Get a list of not-yet-processed documents to reprioritize. Documents in all jobs will be returned by this method. Up to n document descriptions will be returned.

Specified by:

getNextNotYetProcessedReprioritizationDocuments in interface IJobManager

Parameters:

processID - is the process that requests the reprioritization documents.

n - is the maximum number of document descriptions desired.

Returns:

the document descriptions.

Throws:

ManifoldCFException

writeDocumentPriorities

public void writeDocumentPriorities(DocumentDescription[] documentDescriptions,
                                    IPriorityCalculator[] priorities)
                             throws ManifoldCFException

Save a set of document priorities. In the case where a document was eligible to have its priority set, but it no longer is eligible, then the provided priority will not be written.

Specified by:

writeDocumentPriorities in interface IJobManager

Parameters:

documentDescriptions - are the document descriptions.

priorities - are the desired priorities.

Throws:

ManifoldCFException

getExpiredDocuments

public DocumentSetAndFlags getExpiredDocuments(java.lang.String processID,
                                               int n,
                                               long currentTime)
                                        throws ManifoldCFException

Get up to the next n documents to be expired. This method marks the documents whose descriptions have been returned as "being processed", or active. The same marking is used as is used for documents that have been queued for worker threads. The model is thus identical.

Specified by:

getExpiredDocuments in interface IJobManager

Parameters:

processID - is the current process ID.

n - is the maximum number of records desired.

currentTime - is the current time.

Returns:

the array of document descriptions to expire.

Throws:

ManifoldCFException

getNextDocuments

public DocumentDescription[] getNextDocuments(java.lang.String processID,
                                              int n,
                                              long currentTime,
                                              long interval,
                                              BlockingDocuments blockingDocuments,
                                              PerformanceStatistics statistics,
                                              DepthStatistics scanRecord)
                                       throws ManifoldCFException

/** Get up to the next n document(s) to be fetched and processed. This fetch returns records that contain the document identifier, plus all instructions pertaining to the document's handling (e.g. whether it should be refetched if the version has not changed). This method also marks the documents whose descriptions have be returned as "being processed".

Specified by:

getNextDocuments in interface IJobManager

Parameters:

processID - is the current process ID.

n - is the maximum number of records desired.

currentTime - is the current time; some fetches do not occur until a specific time.

interval - is the number of milliseconds that this set of documents should represent (for throttling).

blockingDocuments - is the place to record documents that were encountered, are eligible for reprioritization, but could not be queued due to throttling considerations.

statistics - are the current performance statistics per connection, which are used to balance the queue stuffing so that individual connections are not overwhelmed.

scanRecord - retains the bins from all documents encountered from the query, even those that were skipped due to being overcommitted.

Returns:

the array of document descriptions to fetch and process.

Throws:

ManifoldCFException

fetchAndProcessDocuments

protected void fetchAndProcessDocuments(java.util.List<DocumentDescription> answers,
                                        java.lang.Long currentTimeValue,
                                        java.lang.Long currentPriorityValue,
                                        JobManager.ThrottleLimit vList,
                                        IRepositoryConnection[] connections,
                                        java.lang.String processID)
                                 throws ManifoldCFException

Fetch and process documents matching the passed-in criteria

Throws:

ManifoldCFException

checkJobActive

public boolean checkJobActive(java.lang.Long jobID)
                       throws ManifoldCFException

Verify that a specific job is indeed still active. This is used to permit abort or pause to be relatively speedy. The query done within MUST be cached in order to not cause undue performance degradation.

Specified by:

checkJobActive in interface IJobManager

Parameters:

jobID - is the job identifier.

Returns:

true if the job is in one of the "active" states.

Throws:

ManifoldCFException

checkJobBusy

public boolean checkJobBusy(java.lang.Long jobID)
                     throws ManifoldCFException

Verify if a job is still processing documents, or no longer has any outstanding active documents

Specified by:

checkJobBusy in interface IJobManager

Throws:

ManifoldCFException

markDocumentCompletedMultiple

public void markDocumentCompletedMultiple(DocumentDescription[] documentDescriptions)
                                   throws ManifoldCFException

Note completion of document processing by a job thread of a document. This method causes the state of the document to be marked as "completed".

Specified by:

markDocumentCompletedMultiple in interface IJobManager

Parameters:

documentDescriptions - are the description objects for the documents that were processed.

Throws:

ManifoldCFException

markDocumentCompleted

public void markDocumentCompleted(DocumentDescription documentDescription)
                           throws ManifoldCFException

Note completion of document processing by a job thread of a document. This method causes the state of the document to be marked as "completed".

Specified by:

markDocumentCompleted in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was processed.

Throws:

ManifoldCFException

markDocumentDeletedMultiple

public DocumentDescription[] markDocumentDeletedMultiple(java.lang.Long jobID,
                                                         java.lang.String[] legalLinkTypes,
                                                         DocumentDescription[] documentDescriptions,
                                                         int hopcountMethod)
                                                  throws ManifoldCFException

Delete from queue as a result of processing of an active document. The document is expected to be in one of the active states: ACTIVE, ACTIVESEEDING, ACTIVENEEDSRESCAN, ACTIVESEEDINGNEEDSRESCAN. The RESCAN variants are interpreted as meaning that the document should not be deleted, but should instead be popped back on the queue for a repeat processing attempt.

Specified by:

markDocumentDeletedMultiple in interface IJobManager

Parameters:

documentDescriptions - are the set of description objects for the documents that were processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

markDocumentDeleted

public DocumentDescription[] markDocumentDeleted(java.lang.Long jobID,
                                                 java.lang.String[] legalLinkTypes,
                                                 DocumentDescription documentDescription,
                                                 int hopcountMethod)
                                          throws ManifoldCFException

Delete from queue as a result of processing of an active document. The document is expected to be in one of the active states: ACTIVE, ACTIVESEEDING, ACTIVENEEDSRESCAN, ACTIVESEEDINGNEEDSRESCAN. The RESCAN variants are interpreted as meaning that the document should not be deleted, but should instead be popped back on the queue for a repeat processing attempt.

Specified by:

markDocumentDeleted in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

markDocumentHopcountRemovalMultiple

public DocumentDescription[] markDocumentHopcountRemovalMultiple(java.lang.Long jobID,
                                                                 java.lang.String[] legalLinkTypes,
                                                                 DocumentDescription[] documentDescriptions,
                                                                 int hopcountMethod)
                                                          throws ManifoldCFException

Mark hopcount removal from queue as a result of processing of an active document. The document is expected to be in one of the active states: ACTIVE, ACTIVESEEDING, ACTIVENEEDSRESCAN, ACTIVESEEDINGNEEDSRESCAN. The RESCAN variants are interpreted as meaning that the document should not be marked as removed, but should instead be popped back on the queue for a repeat processing attempt.

Specified by:

markDocumentHopcountRemovalMultiple in interface IJobManager

Parameters:

documentDescriptions - are the set of description objects for the documents that were processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

markDocumentHopcountRemoval

public DocumentDescription[] markDocumentHopcountRemoval(java.lang.Long jobID,
                                                         java.lang.String[] legalLinkTypes,
                                                         DocumentDescription documentDescription,
                                                         int hopcountMethod)
                                                  throws ManifoldCFException

Mark hopcount removal from queue as a result of processing of an active document. The document is expected to be in one of the active states: ACTIVE, ACTIVESEEDING, ACTIVENEEDSRESCAN, ACTIVESEEDINGNEEDSRESCAN. The RESCAN variants are interpreted as meaning that the document should not be marked as removed, but should instead be popped back on the queue for a repeat processing attempt.

Specified by:

markDocumentHopcountRemoval in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

markDocumentExpiredMultiple

public DocumentDescription[] markDocumentExpiredMultiple(java.lang.Long jobID,
                                                         java.lang.String[] legalLinkTypes,
                                                         DocumentDescription[] documentDescriptions,
                                                         int hopcountMethod)
                                                  throws ManifoldCFException

Delete from queue as a result of expiration of an active document. The document is expected to be in one of the active states: ACTIVE, ACTIVESEEDING, ACTIVENEEDSRESCAN, ACTIVESEEDINGNEEDSRESCAN. Since the document expired, no special activity takes place as a result of the document being in a RESCAN state.

Specified by:

markDocumentExpiredMultiple in interface IJobManager

Parameters:

documentDescriptions - are the set of description objects for the documents that were processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

markDocumentExpired

public DocumentDescription[] markDocumentExpired(java.lang.Long jobID,
                                                 java.lang.String[] legalLinkTypes,
                                                 DocumentDescription documentDescription,
                                                 int hopcountMethod)
                                          throws ManifoldCFException

Delete from queue as a result of expiration of an active document. The document is expected to be in one of the active states: ACTIVE, ACTIVESEEDING, ACTIVENEEDSRESCAN, ACTIVESEEDINGNEEDSRESCAN. Since the document expired, no special activity takes place as a result of the document being in a RESCAN state.

Specified by:

markDocumentExpired in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

markDocumentCleanedUpMultiple

public DocumentDescription[] markDocumentCleanedUpMultiple(java.lang.Long jobID,
                                                           java.lang.String[] legalLinkTypes,
                                                           DocumentDescription[] documentDescriptions,
                                                           int hopcountMethod)
                                                    throws ManifoldCFException

Delete from queue as a result of cleaning up an unreachable document. The document is expected to be in the PURGATORY state. There is never any need to reprocess the document.

Specified by:

markDocumentCleanedUpMultiple in interface IJobManager

Parameters:

documentDescriptions - are the set of description objects for the documents that were processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

markDocumentCleanedUp

public DocumentDescription[] markDocumentCleanedUp(java.lang.Long jobID,
                                                   java.lang.String[] legalLinkTypes,
                                                   DocumentDescription documentDescription,
                                                   int hopcountMethod)
                                            throws ManifoldCFException

Delete from queue as a result of cleaning up an unreachable document. The document is expected to be in the PURGATORY state. There is never any need to reprocess the document.

Specified by:

markDocumentCleanedUp in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

doDeleteMultiple

protected DocumentDescription[] doDeleteMultiple(java.lang.Long jobID,
                                                 java.lang.String[] legalLinkTypes,
                                                 DocumentDescription[] documentDescriptions,
                                                 int hopcountMethod)
                                          throws ManifoldCFException

Delete documents with no repercussions. We don't have to worry about the current state of each document, since the document is definitely going away.

Parameters:

documentDescriptions - are the set of description objects for the documents that were processed.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

calculateAffectedDeleteCarrydownChildren

protected DocumentDescription[] calculateAffectedDeleteCarrydownChildren(java.lang.Long jobID,
                                                                         java.lang.String[] docIDHashes)
                                                                  throws ManifoldCFException

Helper method: Find the document descriptions that will be affected due to carrydown row deletions.

Throws:

ManifoldCFException

maxClauseProcessDeleteHashSet

protected int maxClauseProcessDeleteHashSet()

Get maximum count.

processDeleteHashSet

protected void processDeleteHashSet(java.lang.Long jobID,
                                    java.util.HashMap resultHash,
                                    java.util.ArrayList list)
                             throws ManifoldCFException

Helper method: look up rows affected by a deleteRecords operation.

Throws:

ManifoldCFException

requeueDocumentMultiple

public void requeueDocumentMultiple(DocumentDescription[] documentDescriptions,
                                    java.lang.Long[] executeTimes,
                                    int[] actions)
                             throws ManifoldCFException

Requeue a document for further processing in the future. This method is called after a document is processed, when the job is a "continuous" one. It is essentially equivalent to noting that the document processing is complete, except the document remains on the queue.

Specified by:

requeueDocumentMultiple in interface IJobManager

Parameters:

documentDescriptions - is the set of description objects for the document that was processed.

executeTimes - are the times that the documents should be rescanned. Null indicates "never".

actions - are what should be done when the time arrives. Choices are ACTION_RESCAN or ACTION_REMOVE.

Throws:

ManifoldCFException

requeueDocument

public void requeueDocument(DocumentDescription documentDescription,
                            java.lang.Long executeTime,
                            int action)
                     throws ManifoldCFException

Requeue a document for further processing in the future. This method is called after a document is processed, when the job is a "continuous" one. It is essentially equivalent to noting that the document processing is complete, except the document remains on the queue.

Specified by:

requeueDocument in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was processed.

executeTime - is the time that the document should be rescanned. Null indicates "never".

action - is what should be done when the time arrives. Choices include ACTION_RESCAN or ACTION_REMOVE.

Throws:

ManifoldCFException

resetDocumentMultiple

public void resetDocumentMultiple(DocumentDescription[] documentDescriptions,
                                  long executeTime,
                                  int action,
                                  long failTime,
                                  int failCount)
                           throws ManifoldCFException

Reset a set of documents for further processing in the future. This method is called after some unknown number of the documents were processed, but then a service interruption occurred. Note well: The logic here basically presumes that we cannot know whether the documents were indeed processed or not. If we knew for a fact that none of the documents had been handled, it would be possible to look at the document's current status and decide what the new status ought to be, based on a true rollback scenario. Such cases, however, are rare enough so that special logic is probably not worth it.

Specified by:

resetDocumentMultiple in interface IJobManager

Parameters:

documentDescriptions - is the set of description objects for the document that was processed.

executeTime - is the time that the documents should be rescanned.

failTime - is the time beyond which a service interruption will be considered a hard failure.

failCount - is the number of retries beyond which a service interruption will be considered a hard failure.

Throws:

ManifoldCFException

resetCleaningDocumentMultiple

public void resetCleaningDocumentMultiple(DocumentDescription[] documentDescriptions,
                                          long checkTime)
                                   throws ManifoldCFException

Reset a set of cleaning documents for further processing in the future. This method is called after some unknown number of the documents were cleaned, but then an ingestion service interruption occurred. Note well: The logic here basically presumes that we cannot know whether the documents were indeed cleaned or not. If we knew for a fact that none of the documents had been handled, it would be possible to look at the document's current status and decide what the new status ought to be, based on a true rollback scenario. Such cases, however, are rare enough so that special logic is probably not worth it.

Specified by:

resetCleaningDocumentMultiple in interface IJobManager

Parameters:

documentDescriptions - is the set of description objects for the document that was cleaned.

checkTime - is the minimum time for the next cleaning attempt.

Throws:

ManifoldCFException

resetCleaningDocument

public void resetCleaningDocument(DocumentDescription documentDescription,
                                  long checkTime)
                           throws ManifoldCFException

Reset a cleaning document back to its former state. This gets done when a deleting thread sees a service interruption, etc., from the ingestion system.

Specified by:

resetCleaningDocument in interface IJobManager

Parameters:

documentDescription - is the description of the document that was cleaned.

checkTime - is the minimum time for the next cleaning attempt.

Throws:

ManifoldCFException

resetDeletingDocumentMultiple

public void resetDeletingDocumentMultiple(DocumentDescription[] documentDescriptions,
                                          long checkTime)
                                   throws ManifoldCFException

Reset a set of deleting documents for further processing in the future. This method is called after some unknown number of the documents were deleted, but then an ingestion service interruption occurred. Note well: The logic here basically presumes that we cannot know whether the documents were indeed processed or not. If we knew for a fact that none of the documents had been handled, it would be possible to look at the document's current status and decide what the new status ought to be, based on a true rollback scenario. Such cases, however, are rare enough so that special logic is probably not worth it.

Specified by:

resetDeletingDocumentMultiple in interface IJobManager

Parameters:

documentDescriptions - is the set of description objects for the document that was processed.

checkTime - is the minimum time for the next cleaning attempt.

Throws:

ManifoldCFException

resetDeletingDocument

public void resetDeletingDocument(DocumentDescription documentDescription,
                                  long checkTime)
                           throws ManifoldCFException

Reset a deleting document back to its former state. This gets done when a deleting thread sees a service interruption, etc., from the ingestion system.

Specified by:

resetDeletingDocument in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was cleaned.

checkTime - is the minimum time for the next cleaning attempt.

Throws:

ManifoldCFException

resetDocument

public void resetDocument(DocumentDescription documentDescription,
                          long executeTime,
                          int action,
                          long failTime,
                          int failCount)
                   throws ManifoldCFException

Reset an active document back to its former state. This gets done when there's a service interruption and the document cannot be processed yet. Note well: This method formerly presumed that a perfect rollback was possible, and that there was zero chance of any processing activity occuring before it got called. That assumption appears incorrect, however, so I've opted to now presume that processing has perhaps occurred. Perfect rollback is thus no longer possible.

Specified by:

resetDocument in interface IJobManager

Parameters:

documentDescription - is the description object for the document that was processed.

executeTime - is the time that the document should be rescanned.

failTime - is the time that the document should be considered to have failed, if it has not been successfully read until then.

failCount - is the number of permitted failures before a hard error is signalled.

Throws:

ManifoldCFException

eliminateDuplicates

protected static java.lang.String[] eliminateDuplicates(java.lang.String[] docIDHashes)

Eliminate duplicates, and sort

buildReorderMap

protected static java.util.HashMap buildReorderMap(java.lang.String[] originalIDHashes,
                                                   java.lang.String[] reorderedIDHashes)

Build a reorder map, describing how to convert an original index into a reordered index.

retryStartup

public void retryStartup(JobStartRecord jsr,
                         long failTime,
                         int failCount)
                  throws ManifoldCFException

Retry startup.

Specified by:

retryStartup in interface IJobManager

Parameters:

jsr - is the current job notification record.

failTime - is the new fail time (-1L if none).

failCount - is the new fail retry count (-1 if none).

Throws:

ManifoldCFException

retrySeeding

public void retrySeeding(JobSeedingRecord jsr,
                         long failTime,
                         int failCount)
                  throws ManifoldCFException

Retry seeding.

Specified by:

retrySeeding in interface IJobManager

Parameters:

jsr - is the current job seeding record.

failTime - is the new fail time (-1L if none).

failCount - is the new fail retry count (-1 if none).

Throws:

ManifoldCFException

retryNotification

public void retryNotification(JobNotifyRecord jnr,
                              long failTime,
                              int failCount)
                       throws ManifoldCFException

Retry notification.

Specified by:

retryNotification in interface IJobManager

Parameters:

jnr - is the current job notification record.

failTime - is the new fail time (-1L if none).

failCount - is the new fail retry count (-1 if none).

Throws:

ManifoldCFException

retryDeleteNotification

public void retryDeleteNotification(JobNotifyRecord jnr,
                                    long failTime,
                                    int failCount)
                             throws ManifoldCFException

Retry delete notification.

Specified by:

retryDeleteNotification in interface IJobManager

Parameters:

jnr - is the current job notification record.

failTime - is the new fail time (-1L if none).

failCount - is the new fail retry count (-1 if none).

Throws:

ManifoldCFException

addDocumentsInitial

public void addDocumentsInitial(java.lang.String processID,
                                java.lang.Long jobID,
                                java.lang.String[] legalLinkTypes,
                                java.lang.String[] docIDHashes,
                                java.lang.String[] docIDs,
                                boolean overrideSchedule,
                                int hopcountMethod,
                                IPriorityCalculator[] documentPriorities,
                                java.lang.String[][] prereqEventNames)
                         throws ManifoldCFException

Add an initial set of documents to the queue. This method is called during job startup, when the queue is being loaded. A set of document references is passed to this method, which updates the status of the document in the specified job's queue, according to specific state rules.

Specified by:

addDocumentsInitial in interface IJobManager

Parameters:

processID - is the current process ID.

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

docIDs - are the local document identifiers.

overrideSchedule - is true if any existing document schedule should be overridden.

hopcountMethod - is either accurate, nodelete, or neverdelete.

documentPriorities - are the document priorities corresponding to the document identifiers.

prereqEventNames - are the events that must be completed before each document can be processed.

docIDHashes - are the hashes of the local document identifiers (primary key).

Throws:

ManifoldCFException

addRemainingDocumentsInitial

public void addRemainingDocumentsInitial(java.lang.String processID,
                                         java.lang.Long jobID,
                                         java.lang.String[] legalLinkTypes,
                                         java.lang.String[] docIDHashes,
                                         int hopcountMethod)
                                  throws ManifoldCFException

Add an initial set of remaining documents to the queue. This method is called during job startup, when the queue is being loaded, to list documents that were NOT included by calling addDocumentsInitial(). Documents listed here are simply designed to enable the framework to get rid of old, invalid seeds. They are not queued for processing.

Specified by:

addRemainingDocumentsInitial in interface IJobManager

Parameters:

processID - is the current process ID.

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

docIDHashes - are the local document identifier hashes.

hopcountMethod - is either accurate, nodelete, or neverdelete.

Throws:

ManifoldCFException

doneDocumentsInitial

public void doneDocumentsInitial(java.lang.Long jobID,
                                 java.lang.String[] legalLinkTypes,
                                 boolean isPartial,
                                 int hopcountMethod)
                          throws ManifoldCFException

Signal that a seeding pass has been done. Call this method at the end of a seeding pass. It is used to perform the bookkeeping necessary to maintain the hopcount table.

Specified by:

doneDocumentsInitial in interface IJobManager

Parameters:

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

isPartial - is set if the seeds provided are only a partial list. Some connectors cannot supply a full list of seeds on every seeding iteration; this acknowledges that limitation.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Throws:

ManifoldCFException

findHopCounts

public boolean[] findHopCounts(java.lang.Long jobID,
                               java.lang.String[] legalLinkTypes,
                               java.lang.String[] docIDHashes,
                               java.lang.String linkType,
                               int limit,
                               int hopcountMethod)
                        throws ManifoldCFException

Get the specified hop counts, with the limit as described.

Specified by:

findHopCounts in interface IJobManager

Parameters:

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

docIDHashes - are the hashes for the set of documents to find the hopcount for.

linkType - is the kind of link to find the hopcount for.

limit - is the limit, beyond which a negative distance may be returned.

hopcountMethod - is the method for managing hopcounts that is in effect.

Returns:

a vector of booleans corresponding to the documents requested. A true value is returned if the document is within the specified limit, false otherwise.

Throws:

ManifoldCFException

getAllSeeds

public java.lang.String[] getAllSeeds(java.lang.Long jobID)
                               throws ManifoldCFException

Get all the current seeds. Returns the seed document identifiers for a job.

Specified by:

getAllSeeds in interface IJobManager

Parameters:

jobID - is the job identifier.

Returns:

the document identifiers that are currently considered to be seeds.

Throws:

ManifoldCFException

addDocuments

public void addDocuments(java.lang.String processID,
                         java.lang.Long jobID,
                         java.lang.String[] legalLinkTypes,
                         java.lang.String[] docIDHashes,
                         java.lang.String[] docIDs,
                         java.lang.String parentIdentifierHash,
                         java.lang.String relationshipType,
                         int hopcountMethod,
                         java.lang.String[][] dataNames,
                         java.lang.Object[][][] dataValues,
                         IPriorityCalculator[] documentPriorities,
                         java.lang.String[][] prereqEventNames)
                  throws ManifoldCFException

Add documents to the queue in bulk. This method is called during document processing, when a set of document references are discovered. The document references are passed to this method, which updates the status of the document(s) in the specified job's queue, according to specific state rules.

Specified by:

addDocuments in interface IJobManager

Parameters:

processID - is the process ID.

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

docIDHashes - are the local document identifier hashes.

parentIdentifierHash - is the optional parent identifier hash of this document. Pass null if none. MUST be present in the case of carrydown information.

relationshipType - is the optional link type between this document and its parent. Pass null if there is no relationship with a parent.

hopcountMethod - is the desired method for managing hopcounts.

dataNames - are the names of the data to carry down to the child from this parent.

dataValues - are the values to carry down to the child from this parent, corresponding to dataNames above. If CharacterInput objects are passed in here, it is the caller's responsibility to clean these up.

documentPriorities - are the desired document priorities for the documents.

prereqEventNames - are the events that must be completed before a document can be queued.

docIDs - are the local document identifiers.

Throws:

ManifoldCFException

addDocument

public void addDocument(java.lang.String processID,
                        java.lang.Long jobID,
                        java.lang.String[] legalLinkTypes,
                        java.lang.String docIDHash,
                        java.lang.String docID,
                        java.lang.String parentIdentifierHash,
                        java.lang.String relationshipType,
                        int hopcountMethod,
                        java.lang.String[] dataNames,
                        java.lang.Object[][] dataValues,
                        IPriorityCalculator priority,
                        java.lang.String[] prereqEventNames)
                 throws ManifoldCFException

Add a document to the queue. This method is called during document processing, when a document reference is discovered. The document reference is passed to this method, which updates the status of the document in the specified job's queue, according to specific state rules.

Specified by:

addDocument in interface IJobManager

Parameters:

processID - is the process ID.

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

docIDHash - is the local document identifier hash value.

parentIdentifierHash - is the optional parent identifier hash of this document. Pass null if none. MUST be present in the case of carrydown information.

relationshipType - is the optional link type between this document and its parent. Pass null if there is no relationship with a parent.

hopcountMethod - is the desired method for managing hopcounts.

dataNames - are the names of the data to carry down to the child from this parent.

dataValues - are the values to carry down to the child from this parent, corresponding to dataNames above.

priority - is the desired document priority for the document.

prereqEventNames - are the events that must be completed before the document can be processed.

Throws:

ManifoldCFException

revertDocuments

public void revertDocuments(java.lang.Long jobID,
                            java.lang.String[] legalLinkTypes,
                            java.lang.String[] parentIdentifierHashes)
                     throws ManifoldCFException

Undo the addition of child documents to the queue, for a set of documents. This method is called at the end of document processing, to back out any incomplete additions to the queue, and restore the status quo ante prior to the incomplete additions. Call this method instead of finishDocuments() if the addition of documents was not completed.

Specified by:

revertDocuments in interface IJobManager

Parameters:

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

parentIdentifierHashes - are the hashes of the document identifiers for whom child link extraction just took place.

Throws:

ManifoldCFException

finishDocuments

public DocumentDescription[] finishDocuments(java.lang.Long jobID,
                                             java.lang.String[] legalLinkTypes,
                                             java.lang.String[] parentIdentifierHashes,
                                             int hopcountMethod)
                                      throws ManifoldCFException

Complete adding child documents to the queue, for a set of documents. This method is called at the end of document processing, to help the hopcount tracking engine do its bookkeeping.

Specified by:

finishDocuments in interface IJobManager

Parameters:

jobID - is the job identifier.

legalLinkTypes - is the set of legal link types that this connector generates.

parentIdentifierHashes - are the document identifier hashes for whom child link extraction just took place.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Returns:

the set of documents for which carrydown data was changed by this operation. These documents are likely to be requeued as a result of the change.

Throws:

ManifoldCFException

calculateAffectedRestoreCarrydownChildren

protected DocumentDescription[] calculateAffectedRestoreCarrydownChildren(java.lang.Long jobID,
                                                                          java.lang.String[] parentIDHashes)
                                                                   throws ManifoldCFException

Helper method: Calculate the unique set of affected carrydown children resulting from a "restoreRecords" operation.

Throws:

ManifoldCFException

processParentHashSet

protected void processParentHashSet(java.lang.Long jobID,
                                    java.util.HashMap resultHash,
                                    java.util.ArrayList list)
                             throws ManifoldCFException

Helper method: look up rows affected by a restoreRecords operation.

Throws:

ManifoldCFException

beginEventSequence

public boolean beginEventSequence(java.lang.String processID,
                                  java.lang.String eventName)
                           throws ManifoldCFException

Begin an event sequence.

Specified by:

beginEventSequence in interface IJobManager

Parameters:

processID - is the current process ID.

eventName - is the name of the event.

Returns:

true if the event could be created, or false if it's already there.

Throws:

ManifoldCFException

completeEventSequence

public void completeEventSequence(java.lang.String eventName)
                           throws ManifoldCFException

Complete an event sequence.

Specified by:

completeEventSequence in interface IJobManager

Parameters:

eventName - is the name of the event.

Throws:

ManifoldCFException

carrydownChangeDocumentMultiple

public void carrydownChangeDocumentMultiple(DocumentDescription[] documentDescriptions,
                                            IPriorityCalculator[] docPriorities)
                                     throws ManifoldCFException

Requeue a document set because of carrydown changes. This method is called when carrydown data is modified for a set of documents. The documents must be requeued for immediate reprocessing, even to the extent that if one is *already* being processed, it will need to be done over again.

Specified by:

carrydownChangeDocumentMultiple in interface IJobManager

Parameters:

documentDescriptions - is the set of description objects for the documents that have had their parent carrydown information changed.

docPriorities - are the document priorities to assign to the documents, if needed.

Throws:

ManifoldCFException

carrydownChangeDocument

public void carrydownChangeDocument(DocumentDescription documentDescription,
                                    IPriorityCalculator docPriority)
                             throws ManifoldCFException

Requeue a document because of carrydown changes. This method is called when carrydown data is modified for a document. The document must be requeued for immediate reprocessing, even to the extent that if it is *already* being processed, it will need to be done over again.

Specified by:

carrydownChangeDocument in interface IJobManager

Parameters:

documentDescription - is the description object for the document that has had its parent carrydown information changed.

docPriority - is the document priority to assign to the document, if needed.

Throws:

ManifoldCFException

getRandomAmount

protected long getRandomAmount()

Sleep a random amount of time after a transaction abort.

sleepFor

protected void sleepFor(long amt)
                 throws ManifoldCFException

Throws:

ManifoldCFException

retrieveParentData

public java.lang.String[] retrieveParentData(java.lang.Long jobID,
                                             java.lang.String docIDHash,
                                             java.lang.String dataName)
                                      throws ManifoldCFException

Retrieve specific parent data for a given document.

Specified by:

retrieveParentData in interface IJobManager

Parameters:

jobID - is the job identifier.

docIDHash - is the document identifier hash value.

dataName - is the kind of data to retrieve.

Returns:

the unique data values.

Throws:

ManifoldCFException

retrieveParentDataAsFiles

public CharacterInput[] retrieveParentDataAsFiles(java.lang.Long jobID,
                                                  java.lang.String docIDHash,
                                                  java.lang.String dataName)
                                           throws ManifoldCFException

Retrieve specific parent data for a given document.

Specified by:

retrieveParentDataAsFiles in interface IJobManager

Parameters:

jobID - is the job identifier.

docIDHash - is the document identifier hash value.

dataName - is the kind of data to retrieve.

Returns:

the unique data values.

Throws:

ManifoldCFException

startJobs

public void startJobs(long currentTime,
                      java.util.List<java.lang.Long> unwaitList)
               throws ManifoldCFException

Start all jobs in need of starting. This method marks all the appropriate jobs as "in progress", which is all that should be needed to start them. It's also the case that the start event should be logged in the event log. In order to make it possible for the caller to do this logging, a set of job ID's will be returned containing the jobs that were started.

Specified by:

startJobs in interface IJobManager

Parameters:

currentTime - is the current time in milliseconds since epoch.

unwaitList - is filled in with the set of job ID objects that were resumed.

Throws:

ManifoldCFException

waitJobs

public void waitJobs(long currentTime,
                     java.util.List<java.lang.Long> waitList)
              throws ManifoldCFException

Put active or paused jobs in wait state, if they've exceeded their window.

Specified by:

waitJobs in interface IJobManager

Parameters:

currentTime - is the current time in milliseconds since epoch.

waitList - is filled in with the set of job ID's that were put into a wait state.

Throws:

ManifoldCFException

resetJobSchedule

public void resetJobSchedule(java.lang.Long jobID)
                      throws ManifoldCFException

Reset job schedule. This re-evaluates whether the job should be started now. This method would typically be called after a job's scheduling window has been changed.

Specified by:

resetJobSchedule in interface IJobManager

Parameters:

jobID - is the job identifier.

Throws:

ManifoldCFException

checkTimeMatch

protected static java.lang.Long checkTimeMatch(long startTime,
                                               long currentTimestamp,
                                               EnumeratedValues daysOfWeek,
                                               EnumeratedValues daysOfMonth,
                                               EnumeratedValues months,
                                               EnumeratedValues years,
                                               EnumeratedValues hours,
                                               EnumeratedValues minutes,
                                               java.lang.String timezone,
                                               java.lang.Long duration)

Check if the specified job parameters have a 'hit' within the specified interval.

Parameters:

startTime - is the start time.

currentTimestamp - is the end time.

daysOfWeek - is the enumerated days of the week, or null.

daysOfMonth - is the enumerated days of the month, or null.

months - is the enumerated months, or null.

years - is the enumerated years, or null.

hours - is the enumerated hours, or null.

minutes - is the enumerated minutes, or null.

Returns:

null if there is NO hit within the interval; otherwise the actual time of the hit in milliseconds from epoch is returned.

manualStart

public void manualStart(java.lang.Long jobID)
                 throws ManifoldCFException

Manually start a job. The specified job will be run REGARDLESS of the timed windows, and will not cease until complete. If the job is already running, this operation will assure that the job does not pause when its window ends. The job can be manually paused, or manually aborted.

Specified by:

manualStart in interface IJobManager

Parameters:

jobID - is the ID of the job to start.

Throws:

ManifoldCFException

manualStart

public void manualStart(java.lang.Long jobID,
                        boolean requestMinimum)
                 throws ManifoldCFException

Manually start a job. The specified job will be run REGARDLESS of the timed windows, and will not cease until complete. If the job is already running, this operation will assure that the job does not pause when its window ends. The job can be manually paused, or manually aborted.

Specified by:

manualStart in interface IJobManager

Parameters:

jobID - is the ID of the job to start.

requestMinimum - is true if a minimal job run is requested.

Throws:

ManifoldCFException

noteJobDeleteStarted

public void noteJobDeleteStarted(java.lang.Long jobID,
                                 long startTime)
                          throws ManifoldCFException

Note job delete started.

Specified by:

noteJobDeleteStarted in interface IJobManager

Parameters:

jobID - is the job id.

startTime - is the job delete start time.

Throws:

ManifoldCFException

noteJobStarted

public void noteJobStarted(java.lang.Long jobID,
                           long startTime,
                           java.lang.String seedingVersion)
                    throws ManifoldCFException

Note job started.

Specified by:

noteJobStarted in interface IJobManager

Parameters:

jobID - is the job id.

startTime - is the job start time.

seedingVersion - is the seeding version to record with the job start.

Throws:

ManifoldCFException

noteJobSeeded

public void noteJobSeeded(java.lang.Long jobID,
                          java.lang.String seedingVersion)
                   throws ManifoldCFException

Note job seeded.

Specified by:

noteJobSeeded in interface IJobManager

Parameters:

jobID - is the job id.

seedingVersion - is the job seeding version string to record.

Throws:

ManifoldCFException

prepareDeleteScan

public void prepareDeleteScan(java.lang.Long jobID)
                       throws ManifoldCFException

Prepare for a delete scan.

Specified by:

prepareDeleteScan in interface IJobManager

Parameters:

jobID - is the job id.

Throws:

ManifoldCFException

prepareJobScan

public void prepareJobScan(java.lang.Long jobID,
                           java.lang.String[] legalLinkTypes,
                           int hopcountMethod,
                           int connectorModel,
                           boolean continuousJob,
                           boolean fromBeginningOfTime,
                           boolean requestMinimum)
                    throws ManifoldCFException

Prepare a job to be run. This method is called regardless of the details of the job; what differs is only the flags that are passed in. The code inside will determine the appropriate procedures. (This method replaces prepareFullScan() and prepareIncrementalScan(). )

Specified by:

prepareJobScan in interface IJobManager

Parameters:

jobID - is the job id.

legalLinkTypes - are the link types allowed for the job.

hopcountMethod - describes how to handle deletions for hopcount purposes.

connectorModel - is the model used by the connector for the job.

continuousJob - is true if the job is a continuous one.

fromBeginningOfTime - is true if the job is running starting from time 0.

requestMinimum - is true if the minimal amount of work is requested for the job run.

Throws:

ManifoldCFException

queueAllExisting

protected void queueAllExisting(java.lang.Long jobID,
                                java.lang.String[] legalLinkTypes)
                         throws ManifoldCFException

Queue all existing.

Parameters:

jobID - is the job id.

legalLinkTypes - are the link types allowed for the job.

Throws:

ManifoldCFException

prepareFullScan

protected void prepareFullScan(java.lang.Long jobID,
                               java.lang.String[] legalLinkTypes,
                               int hopcountMethod)
                        throws ManifoldCFException

Prepare for a full scan.

Parameters:

jobID - is the job id.

legalLinkTypes - are the link types allowed for the job.

hopcountMethod - describes how to handle deletions for hopcount purposes.

Throws:

ManifoldCFException

manualAbort

public void manualAbort(java.lang.Long jobID)
                 throws ManifoldCFException

Manually abort a running job. The job will be permanently stopped, and will not run again until automatically started based on schedule, or manually started.

Specified by:

manualAbort in interface IJobManager

Parameters:

jobID - is the job to abort.

Throws:

ManifoldCFException

manualAbortRestart

public void manualAbortRestart(java.lang.Long jobID,
                               boolean requestMinimum)
                        throws ManifoldCFException

Manually restart a running job. The job will be stopped and restarted. Any schedule affinity will be lost, until the job finishes on its own.

Specified by:

manualAbortRestart in interface IJobManager

Parameters:

jobID - is the job to abort.

requestMinimum - is true if a minimal job run is requested.

Throws:

ManifoldCFException

manualAbortRestart

public void manualAbortRestart(java.lang.Long jobID)
                        throws ManifoldCFException

Manually restart a running job. The job will be stopped and restarted. Any schedule affinity will be lost, until the job finishes on its own.

Specified by:

manualAbortRestart in interface IJobManager

Parameters:

jobID - is the job to abort.

Throws:

ManifoldCFException

errorAbort

public boolean errorAbort(java.lang.Long jobID,
                          java.lang.String errorText)
                   throws ManifoldCFException

Abort a running job due to a fatal error condition.

Specified by:

errorAbort in interface IJobManager

Parameters:

jobID - is the job to abort.

errorText - is the error text.

Returns:

true if this is the first logged abort request for this job.

Throws:

ManifoldCFException

pauseJob

public void pauseJob(java.lang.Long jobID)
              throws ManifoldCFException

Pause a job.

Specified by:

pauseJob in interface IJobManager

Parameters:

jobID - is the job identifier to pause.

Throws:

ManifoldCFException

restartJob

public void restartJob(java.lang.Long jobID)
                throws ManifoldCFException

Restart a paused job.

Specified by:

restartJob in interface IJobManager

Parameters:

jobID - is the job identifier to restart.

Throws:

ManifoldCFException

getJobsReadyForSeeding

public JobSeedingRecord[] getJobsReadyForSeeding(java.lang.String processID,
                                                 long currentTime)
                                          throws ManifoldCFException

Get the list of jobs that are ready for seeding.

Specified by:

getJobsReadyForSeeding in interface IJobManager

Parameters:

processID - is the current process ID.

currentTime - is the current time in milliseconds since epoch.

Returns:

jobs that are active and are running in adaptive mode. These will be seeded based on what the connector says should be added to the queue.

Throws:

ManifoldCFException

getJobsReadyForDeleteCleanup

public JobDeleteRecord[] getJobsReadyForDeleteCleanup(java.lang.String processID)
                                               throws ManifoldCFException

Get the list of jobs that are ready for delete cleanup.

Specified by:

getJobsReadyForDeleteCleanup in interface IJobManager

Parameters:

processID - is the current process ID.

Returns:

jobs that were in the "readyfordelete" state.

Throws:

ManifoldCFException

getJobsReadyForStartup

public JobStartRecord[] getJobsReadyForStartup(java.lang.String processID)
                                        throws ManifoldCFException

Get the list of jobs that are ready for startup.

Specified by:

getJobsReadyForStartup in interface IJobManager

Parameters:

processID - is the current process ID.

Returns:

jobs that were in the "readyforstartup" state. These will be marked as being in the "starting up" state.

Throws:

ManifoldCFException

inactivateJob

public void inactivateJob(java.lang.Long jobID)
                   throws ManifoldCFException

Inactivate a job, from the notification state.

Specified by:

inactivateJob in interface IJobManager

Parameters:

jobID - is the ID of the job to inactivate.

Throws:

ManifoldCFException

removeJob

public void removeJob(java.lang.Long jobID)
               throws ManifoldCFException

Remove a job, from the notification state.

Specified by:

removeJob in interface IJobManager

Parameters:

jobID - is the ID of the job to remove.

Throws:

ManifoldCFException

resetStartDeleteJob

public void resetStartDeleteJob(java.lang.Long jobID)
                         throws ManifoldCFException

Reset a job starting for delete back to "ready for delete" state.

Specified by:

resetStartDeleteJob in interface IJobManager

Parameters:

jobID - is the job id.

Throws:

ManifoldCFException

resetNotifyJob

public void resetNotifyJob(java.lang.Long jobID)
                    throws ManifoldCFException

Reset a job that is notifying back to "ready for notify" state.

Specified by:

resetNotifyJob in interface IJobManager

Parameters:

jobID - is the job id.

Throws:

ManifoldCFException

resetDeleteNotifyJob

public void resetDeleteNotifyJob(java.lang.Long jobID)
                          throws ManifoldCFException

Reset a job that is delete notifying back to "ready for delete notify" state.

Specified by:

resetDeleteNotifyJob in interface IJobManager

Parameters:

jobID - is the job id.

Throws:

ManifoldCFException

resetStartupJob

public void resetStartupJob(java.lang.Long jobID)
                     throws ManifoldCFException

Reset a starting job back to "ready for startup" state.

Specified by:

resetStartupJob in interface IJobManager

Parameters:

jobID - is the job id.

Throws:

ManifoldCFException

resetSeedJob

public void resetSeedJob(java.lang.Long jobID)
                  throws ManifoldCFException

Reset a seeding job back to "active" state.

Specified by:

resetSeedJob in interface IJobManager

Parameters:

jobID - is the job id.

Throws:

ManifoldCFException

deleteJobsReadyForDelete

public void deleteJobsReadyForDelete()
                              throws ManifoldCFException

Delete jobs in need of being deleted (which are marked "ready for delete"). This method is meant to be called periodically to perform delete processing on jobs.

Specified by:

deleteJobsReadyForDelete in interface IJobManager

Throws:

ManifoldCFException

finishJobs

public void finishJobs()
                throws ManifoldCFException

Put all eligible jobs in the "shutting down" state.

Specified by:

finishJobs in interface IJobManager

Throws:

ManifoldCFException

getJobsReadyForInactivity

public JobNotifyRecord[] getJobsReadyForInactivity(java.lang.String processID)
                                            throws ManifoldCFException

Find the list of jobs that need to have their connectors notified of job completion.

Specified by:

getJobsReadyForInactivity in interface IJobManager

Parameters:

processID - is the process ID.

Returns:

the ID's of jobs that need their output connectors notified in order to become inactive.

Throws:

ManifoldCFException

getJobsReadyForDelete

public JobNotifyRecord[] getJobsReadyForDelete(java.lang.String processID)
                                        throws ManifoldCFException

Find the list of jobs that need to have their connectors notified of job deletion.

Specified by:

getJobsReadyForDelete in interface IJobManager

Parameters:

processID - is the process ID.

Returns:

the ID's of jobs that need their output connectors notified in order to be removed.

Throws:

ManifoldCFException

finishJobResumes

public void finishJobResumes(long timestamp,
                             java.util.List<IJobDescription> modifiedJobs)
                      throws ManifoldCFException

Complete the sequence that resumes jobs, either from a pause or from a scheduling window wait. The logic will restore the job to an active state (many possibilities depending on connector status), and will record the jobs that have been so modified.

Specified by:

finishJobResumes in interface IJobManager

Parameters:

timestamp - is the current time in milliseconds since epoch.

modifiedJobs - is filled in with the set of IJobDescription objects that were resumed.

Throws:

ManifoldCFException

finishJobStops

public void finishJobStops(long timestamp,
                           java.util.List<IJobDescription> modifiedJobs,
                           java.util.List<java.lang.Integer> stopNotificationTypes)
                    throws ManifoldCFException

Complete the sequence that stops jobs, either for abort, pause, or because of a scheduling window. The logic will move the job to its next state (INACTIVE, PAUSED, ACTIVEWAIT), and will record the jobs that have been so modified.

Specified by:

finishJobStops in interface IJobManager

Parameters:

timestamp - is the current time in milliseconds since epoch.

modifiedJobs - is filled in with the set of IJobDescription objects that were stopped.

stopNotificationTypes - is filled in with the type of stop notification.

Throws:

ManifoldCFException

mapToNotificationType

protected static java.lang.Integer mapToNotificationType(int jobStatus,
                                                         boolean noErrorText)

noDocPriorities

protected void noDocPriorities(java.lang.Long jobID)
                        throws ManifoldCFException

Throws:

ManifoldCFException

resetJobs

public void resetJobs(long currentTime,
                      java.util.List<IJobDescription> resetJobs)
               throws ManifoldCFException

Reset eligible jobs either back to the "inactive" state, or make them active again. The latter will occur if the cleanup phase of the job generated more pending documents. This method is used to pick up all jobs in the shutting down state whose purgatory or being-cleaned records have been all processed.

Specified by:

resetJobs in interface IJobManager

Parameters:

currentTime - is the current time in milliseconds since epoch.

resetJobs - is filled in with the set of IJobDescription objects that were reset.

Throws:

ManifoldCFException

getStatus

public JobStatus getStatus(java.lang.Long jobID)
                    throws ManifoldCFException

Get the status of a job.

Specified by:

getStatus in interface IJobManager

Parameters:

jobID - is the job ID.

Returns:

the status object for the specified job.

Throws:

ManifoldCFException

getAllStatus

public JobStatus[] getAllStatus()
                         throws ManifoldCFException

Get a list of all jobs, and their status information.

Specified by:

getAllStatus in interface IJobManager

Returns:

an ordered array of job status objects.

Throws:

ManifoldCFException

getRunningJobs

public JobStatus[] getRunningJobs()
                           throws ManifoldCFException

Get a list of running jobs. This is for status reporting.

Specified by:

getRunningJobs in interface IJobManager

Returns:

an array of the job status objects.

Throws:

ManifoldCFException

getFinishedJobs

public JobStatus[] getFinishedJobs()
                            throws ManifoldCFException

Get a list of completed jobs, and their statistics.

Specified by:

getFinishedJobs in interface IJobManager

Returns:

an array of the job status objects.

Throws:

ManifoldCFException

getStatus

public JobStatus getStatus(java.lang.Long jobID,
                           boolean includeCounts)
                    throws ManifoldCFException

Get the status of a job.

Specified by:

getStatus in interface IJobManager

Parameters:

jobID - is the job ID.

includeCounts - is true if document counts should be included.

Returns:

the status object for the specified job.

Throws:

ManifoldCFException

getAllStatus

public JobStatus[] getAllStatus(boolean includeCounts)
                         throws ManifoldCFException

Get a list of all jobs, and their status information.

Specified by:

getAllStatus in interface IJobManager

Parameters:

includeCounts - is true if document counts should be included.

Returns:

an ordered array of job status objects.

Throws:

ManifoldCFException

getRunningJobs

public JobStatus[] getRunningJobs(boolean includeCounts)
                           throws ManifoldCFException

Get a list of running jobs. This is for status reporting.

Specified by:

getRunningJobs in interface IJobManager

Parameters:

includeCounts - is true if document counts should be included.

Returns:

an array of the job status objects.

Throws:

ManifoldCFException

getFinishedJobs

public JobStatus[] getFinishedJobs(boolean includeCounts)
                            throws ManifoldCFException

Get a list of completed jobs, and their statistics.

Specified by:

getFinishedJobs in interface IJobManager

Parameters:

includeCounts - is true if document counts should be included.

Returns:

an array of the job status objects.

Throws:

ManifoldCFException

getStatus

public JobStatus getStatus(java.lang.Long jobID,
                           boolean includeCounts,
                           int maxCount)
                    throws ManifoldCFException

Get the status of a job.

Specified by:

getStatus in interface IJobManager

Parameters:

includeCounts - is true if document counts should be included.

jobID - is the job ID.

maxCount - is the maximum number of documents we want to count for each status.

Returns:

the status object for the specified job.

Throws:

ManifoldCFException

getAllStatus

public JobStatus[] getAllStatus(boolean includeCounts,
                                int maxCount)
                         throws ManifoldCFException

Get a list of all jobs, and their status information.

Specified by:

getAllStatus in interface IJobManager

Parameters:

includeCounts - is true if document counts should be included.

maxCount - is the maximum number of documents we want to count for each status.

Returns:

an ordered array of job status objects.

Throws:

ManifoldCFException

getRunningJobs

public JobStatus[] getRunningJobs(boolean includeCounts,
                                  int maxCount)
                           throws ManifoldCFException

Get a list of running jobs. This is for status reporting.

Specified by:

getRunningJobs in interface IJobManager

Parameters:

includeCounts - is true if document counts should be included.

maxCount - is the maximum number of documents we want to count for each status.

Returns:

an array of the job status objects.

Throws:

ManifoldCFException

getFinishedJobs

public JobStatus[] getFinishedJobs(boolean includeCounts,
                                   int maxCount)
                            throws ManifoldCFException

Get a list of completed jobs, and their statistics.

Specified by:

getFinishedJobs in interface IJobManager

Parameters:

includeCounts - is true if document counts should be included.

maxCount - is the maximum number of documents we want to count for each status.

Returns:

an array of the job status objects.

Throws:

ManifoldCFException

makeJobStatus

protected JobStatus[] makeJobStatus(java.lang.String whereClause,
                                    java.util.ArrayList whereParams,
                                    boolean includeCounts,
                                    int maxCount)
                             throws ManifoldCFException

Make a job status array from a query result.

Parameters:

whereClause - is the where clause for the jobs we are interested in.

Returns:

the status array.

Throws:

ManifoldCFException

buildOutstandingClause

protected static ClauseDescription buildOutstandingClause()
                                                   throws ManifoldCFException

Throws:

ManifoldCFException

buildProcessedClause

protected static ClauseDescription buildProcessedClause()
                                                 throws ManifoldCFException

Throws:

ManifoldCFException

buildCountsUsingIndividualQueries

protected void buildCountsUsingIndividualQueries(java.lang.String whereClause,
                                                 java.util.ArrayList whereParams,
                                                 int maxCount,
                                                 java.util.Map<java.lang.Long,java.lang.Long> set2Hash,
                                                 java.util.Map<java.lang.Long,java.lang.Long> set3Hash,
                                                 java.util.Map<java.lang.Long,java.lang.Long> set4Hash,
                                                 java.util.Map<java.lang.Long,java.lang.Boolean> set2Exact,
                                                 java.util.Map<java.lang.Long,java.lang.Boolean> set3Exact,
                                                 java.util.Map<java.lang.Long,java.lang.Boolean> set4Exact)
                                          throws ManifoldCFException

Throws:

ManifoldCFException

buildCountsUsingGroupBy

protected void buildCountsUsingGroupBy(java.lang.String whereClause,
                                       java.util.ArrayList whereParams,
                                       java.util.Map<java.lang.Long,java.lang.Long> set2Hash,
                                       java.util.Map<java.lang.Long,java.lang.Long> set3Hash,
                                       java.util.Map<java.lang.Long,java.lang.Long> set4Hash,
                                       java.util.Map<java.lang.Long,java.lang.Boolean> set2Exact,
                                       java.util.Map<java.lang.Long,java.lang.Boolean> set3Exact,
                                       java.util.Map<java.lang.Long,java.lang.Boolean> set4Exact)
                                throws ManifoldCFException

Throws:

ManifoldCFException

addWhereClause

protected void addWhereClause(java.lang.StringBuilder sb,
                              java.util.ArrayList list,
                              java.lang.String whereClause,
                              java.util.ArrayList whereParams,
                              boolean wherePresent)

genDocumentStatus

public IResultSet genDocumentStatus(java.lang.String connectionName,
                                    StatusFilterCriteria filterCriteria,
                                    SortOrder sortOrder,
                                    int startRow,
                                    int rowCount)
                             throws ManifoldCFException

Run a 'document status' report.

Specified by:

genDocumentStatus in interface IJobManager

Parameters:

connectionName - is the name of the connection.

filterCriteria - are the criteria used to limit the records considered for the report.

sortOrder - is the specified sort order of the final report.

startRow - is the first row to include.

rowCount - is the number of rows to include.

Returns:

the results, with the following columns: identifier, job, state, status, scheduled, action, retrycount, retrylimit. The "scheduled" column and the "retrylimit" column are long values representing a time; all other values will be user-friendly strings.

Throws:

ManifoldCFException

genQueueStatus

public IResultSet genQueueStatus(java.lang.String connectionName,
                                 StatusFilterCriteria filterCriteria,
                                 SortOrder sortOrder,
                                 BucketDescription idBucketDescription,
                                 int startRow,
                                 int rowCount)
                          throws ManifoldCFException

Run a 'queue status' report.

Specified by:

genQueueStatus in interface IJobManager

Parameters:

connectionName - is the name of the connection.

filterCriteria - are the criteria used to limit the records considered for the report.

sortOrder - is the specified sort order of the final report.

idBucketDescription - is the bucket description for generating the identifier class.

startRow - is the first row to include.

rowCount - is the number of rows to include.

Returns:

the results, with the following columns: idbucket, inactive, processing, expiring, deleting, processready, expireready, processwaiting, expirewaiting

Throws:

ManifoldCFException

addBucketExtract

protected void addBucketExtract(java.lang.StringBuilder sb,
                                java.util.ArrayList list,
                                java.lang.String columnPrefix,
                                java.lang.String columnName,
                                BucketDescription bucketDesc)

Turn a bucket description into a return column. This is complicated by the fact that the extraction code is inherently case sensitive. So if case insensitive is desired, that means we whack the whole thing to lower case before doing the match.

addCriteria

protected boolean addCriteria(java.lang.StringBuilder sb,
                              java.util.ArrayList list,
                              java.lang.String fieldPrefix,
                              java.lang.String connectionName,
                              StatusFilterCriteria criteria,
                              boolean whereEmitted)
                       throws ManifoldCFException

Add criteria clauses to query.

Throws:

ManifoldCFException

emitClauseStart

protected boolean emitClauseStart(java.lang.StringBuilder sb,
                                  boolean whereEmitted)

Emit a WHERE or an AND, depending...

addOrdering

protected void addOrdering(java.lang.StringBuilder sb,
                           java.lang.String[] completeFieldList,
                           SortOrder sort)

Add ordering.

addLimits

protected void addLimits(java.lang.StringBuilder sb,
                         int startRow,
                         int maxRowCount)

Add limit and offset.