Package org.apache.manifoldcf.crawler.jobs

Class Carrydown

java.lang.Object

org.apache.manifoldcf.core.database.BaseTable

org.apache.manifoldcf.crawler.jobs.Carrydown

public class Carrydown
extends BaseTable

This class manages the table that keeps track of intrinsic relationships between documents.

carrydown

Field	Type	Description
jobid	BIGINT	Reference:jobs.id
parentidhash	VARCHAR(40)
childidhash	VARCHAR(40)
dataname	VARCHAR(255)
datavaluehash	VARCHAR(40)
datavalue	LONGTEXT
isnew	CHAR(1)
processid	VARCHAR(16)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	Carrydown.DuplicateFinder
protected static class	Carrydown.ResultDuplicateEliminator	Limit checker which removes duplicate rows, based on datavaluehash
protected static class	Carrydown.ValueRecord

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	_rcsid
static java.lang.String	childIDHashField
static java.lang.String	dataNameField
static java.lang.String	dataValueField
static java.lang.String	dataValueHashField
protected static int	ISNEW_BASE	The standard value for the "isnew" field.
protected static int	ISNEW_EXISTING	This value means that the link existed before, and has been found during this scan.
protected static int	ISNEW_NEW	This value means that the link is brand-new; it did not exist before this pass.
protected static java.util.Map	isNewMap
static java.lang.String	jobIDField
static java.lang.String	newField
static java.lang.String	parentIDHashField
static java.lang.String	processIDField

Fields inherited from class org.apache.manifoldcf.core.database.BaseTable

dbInterface, tableName

Constructor Summary

Constructors

Constructor	Description
Carrydown(IDBInterface database)	Constructor.

Method Summary

Modifier and Type	Method	Description
void	analyzeTables()	Analyze job tables that need analysis.
void	deinstall()	Uninstall.
void	deleteOwner(java.lang.Long jobID)	Delete an owning job (and clean up the corresponding carrydown rows).
void	deleteRecords(java.lang.Long jobID, java.lang.String[] documentIDHashes)	Delete all records that mention a particular set of document identifiers.
java.lang.String[]	getDataValues(java.lang.Long jobID, java.lang.String documentIdentifierHash, java.lang.String dataName)	Get unique values given a document identifier, data name, an job identifier
CharacterInput[]	getDataValuesAsFiles(java.lang.Long jobID, java.lang.String documentIdentifierHash, java.lang.String dataName)	Get unique values given a document identifier, data name, an job identifier
void	install(java.lang.String jobsTable, java.lang.String jobsColumn)	Install or upgrade.
protected int	maxClausePerformDeleteRecords(java.lang.Long jobID)
protected void	performDeleteRecords(java.lang.Long jobID, java.util.List<java.lang.String> list)
protected void	performExistsCheck(java.util.Set<Carrydown.ValueRecord> presentSet, java.lang.String query, java.util.ArrayList list)	Do the exists check, in batch.
protected void	performRestoreRecords(java.lang.String query, java.lang.Long jobID, java.util.List<java.lang.String> list)
protected void	performRevertRecords(java.lang.String query, java.lang.Long jobID, java.util.List<java.lang.String> list)
boolean	recordCarrydownData(java.lang.Long jobID, java.lang.String parentDocumentIDHash, java.lang.String childDocumentIDHash, java.lang.String[] documentDataNames, java.lang.String[][] documentDataValueHashes, java.lang.Object[][] documentDataValues, java.lang.String processID)	Add carrydown data for a given parent/child pair.
boolean[]	recordCarrydownDataMultiple(java.lang.Long jobID, java.lang.String parentDocumentIDHash, java.lang.String[] childDocumentIDHashes, java.lang.String[][] dataNames, java.lang.String[][][] dataValueHashes, java.lang.Object[][][] dataValues, java.lang.String processID)	Add carrydown data to the table.
void	restart()	Clean up after all process IDs.
void	restart(java.lang.String processID)	Reset, at startup time.
void	restartCluster()	Reset, at startup time, entire cluster
void	restoreRecords(java.lang.Long jobID, java.lang.String[] parentDocumentIDHashes)	Return all records belonging to the specified parent documents to the base state, and delete the old (eliminated) child records.
void	revertRecords(java.lang.Long jobID, java.lang.String[] parentDocumentIDHashes)	Revert all records belonging to the specified parent documents to their original, pre-modified, state.
static java.lang.String	statusToString(int status)	Convert link status to string
static int	stringToStatus(java.lang.String status)	Convert string to link status.

Methods inherited from class org.apache.manifoldcf.core.database.BaseTable

addTableIndex, analyzeTable, beginTransaction, buildConjunctionClause, constructCountClause, constructDistinctOnClause, constructDoubleCastClause, constructOffsetLimitClause, constructRegexpClause, constructSubstringClause, endTransaction, findConjunctionClauseMax, getDatabaseCacheKey, getDBInterface, getMaxInClause, getMaxOrClause, getSleepAmt, getTableIndexes, getTableName, getTableSchema, getTransactionID, getWindowedReportMaxRows, makeTableKey, noteModifications, performAddIndex, performAlter, performCommit, performCreate, performDelete, performDrop, performInsert, performModification, performQuery, performQuery, performRemoveIndex, performUpdate, prepareRowForSave, readRow, reindexTable, signalRollback, sleepFor

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_rcsid

public static final java.lang.String _rcsid

See Also:

Constant Field Values

jobIDField

public static final java.lang.String jobIDField

See Also:

Constant Field Values

parentIDHashField

public static final java.lang.String parentIDHashField

See Also:

Constant Field Values

childIDHashField

public static final java.lang.String childIDHashField

See Also:

Constant Field Values

dataNameField

public static final java.lang.String dataNameField

See Also:

Constant Field Values

dataValueHashField

public static final java.lang.String dataValueHashField

See Also:

Constant Field Values

dataValueField

public static final java.lang.String dataValueField

See Also:

Constant Field Values

newField

public static final java.lang.String newField

See Also:

Constant Field Values

processIDField

public static final java.lang.String processIDField

See Also:

Constant Field Values

ISNEW_BASE

protected static final int ISNEW_BASE

The standard value for the "isnew" field. Means that the link existed prior to this scan, and no new link was found yet.

See Also:

Constant Field Values

ISNEW_NEW

protected static final int ISNEW_NEW

This value means that the link is brand-new; it did not exist before this pass.

See Also:

Constant Field Values

ISNEW_EXISTING

protected static final int ISNEW_EXISTING

This value means that the link existed before, and has been found during this scan.

See Also:

Constant Field Values

isNewMap

protected static java.util.Map isNewMap

Constructor Detail

Carrydown

public Carrydown(IDBInterface database)
          throws ManifoldCFException

Constructor.

Parameters:

database - is the database handle.

Throws:

ManifoldCFException

Method Detail

install

public void install(java.lang.String jobsTable,
                    java.lang.String jobsColumn)
             throws ManifoldCFException

Install or upgrade.

Throws:

ManifoldCFException

deinstall

public void deinstall()
               throws ManifoldCFException

Uninstall.

Throws:

ManifoldCFException

analyzeTables

public void analyzeTables()
                   throws ManifoldCFException

Analyze job tables that need analysis.

Throws:

ManifoldCFException

deleteOwner

public void deleteOwner(java.lang.Long jobID)
                 throws ManifoldCFException

Delete an owning job (and clean up the corresponding carrydown rows).

Throws:

ManifoldCFException

restart

public void restart(java.lang.String processID)
             throws ManifoldCFException

Reset, at startup time.

Parameters:

processID - is the process ID.

Throws:

ManifoldCFException

restart

public void restart()
             throws ManifoldCFException

Clean up after all process IDs.

Throws:

ManifoldCFException

restartCluster

public void restartCluster()
                    throws ManifoldCFException

Reset, at startup time, entire cluster

Throws:

ManifoldCFException

recordCarrydownData

public boolean recordCarrydownData(java.lang.Long jobID,
                                   java.lang.String parentDocumentIDHash,
                                   java.lang.String childDocumentIDHash,
                                   java.lang.String[] documentDataNames,
                                   java.lang.String[][] documentDataValueHashes,
                                   java.lang.Object[][] documentDataValues,
                                   java.lang.String processID)
                            throws ManifoldCFException

Add carrydown data for a given parent/child pair.

Returns:

true if new carrydown data was recorded; false otherwise.

Throws:

ManifoldCFException

recordCarrydownDataMultiple

public boolean[] recordCarrydownDataMultiple(java.lang.Long jobID,
                                             java.lang.String parentDocumentIDHash,
                                             java.lang.String[] childDocumentIDHashes,
                                             java.lang.String[][] dataNames,
                                             java.lang.String[][][] dataValueHashes,
                                             java.lang.Object[][][] dataValues,
                                             java.lang.String processID)
                                      throws ManifoldCFException

Add carrydown data to the table.

Throws:

ManifoldCFException

performExistsCheck

protected void performExistsCheck(java.util.Set<Carrydown.ValueRecord> presentSet,
                                  java.lang.String query,
                                  java.util.ArrayList list)
                           throws ManifoldCFException

Do the exists check, in batch.

Throws:

ManifoldCFException

revertRecords

public void revertRecords(java.lang.Long jobID,
                          java.lang.String[] parentDocumentIDHashes)
                   throws ManifoldCFException

Revert all records belonging to the specified parent documents to their original, pre-modified, state.

Throws:

ManifoldCFException

performRevertRecords

protected void performRevertRecords(java.lang.String query,
                                    java.lang.Long jobID,
                                    java.util.List<java.lang.String> list)
                             throws ManifoldCFException

Throws:

ManifoldCFException

restoreRecords

public void restoreRecords(java.lang.Long jobID,
                           java.lang.String[] parentDocumentIDHashes)
                    throws ManifoldCFException

Return all records belonging to the specified parent documents to the base state, and delete the old (eliminated) child records.

Throws:

ManifoldCFException

performRestoreRecords

protected void performRestoreRecords(java.lang.String query,
                                     java.lang.Long jobID,
                                     java.util.List<java.lang.String> list)
                              throws ManifoldCFException

Throws:

ManifoldCFException

deleteRecords

public void deleteRecords(java.lang.Long jobID,
                          java.lang.String[] documentIDHashes)
                   throws ManifoldCFException

Delete all records that mention a particular set of document identifiers.

Throws:

ManifoldCFException

maxClausePerformDeleteRecords

protected int maxClausePerformDeleteRecords(java.lang.Long jobID)

performDeleteRecords

protected void performDeleteRecords(java.lang.Long jobID,
                                    java.util.List<java.lang.String> list)
                             throws ManifoldCFException

Throws:

ManifoldCFException

getDataValues

public java.lang.String[] getDataValues(java.lang.Long jobID,
                                        java.lang.String documentIdentifierHash,
                                        java.lang.String dataName)
                                 throws ManifoldCFException

Get unique values given a document identifier, data name, an job identifier

Throws:

ManifoldCFException

getDataValuesAsFiles

public CharacterInput[] getDataValuesAsFiles(java.lang.Long jobID,
                                             java.lang.String documentIdentifierHash,
                                             java.lang.String dataName)
                                      throws ManifoldCFException

Get unique values given a document identifier, data name, an job identifier

Throws:

ManifoldCFException

stringToStatus

public static int stringToStatus(java.lang.String status)

Convert string to link status.

statusToString

public static java.lang.String statusToString(int status)

Convert link status to string