org.archive.modules.writer

Class HBaseWriterProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.writer.WriterPoolProcessor

org.archive.modules.writer.HBaseWriterProcessor

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.io.warc.WARCWriterPoolSettings, org.archive.io.WriterPoolSettings, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class HBaseWriterProcessor
extends org.archive.modules.writer.WriterPoolProcessor
implements org.archive.io.warc.WARCWriterPoolSettings

A Heritrix 3 processor that writes to Hadoop HBase. The following example shows how to configure the crawl job configuration.

 <bean id="hbaseParameterSettings" class="org.archive.io.hbase.HBaseParameters">
 	<!-- These settings are required -->
 	<property name="zkQuorum" value="localhost" />
 	<property name="hbaseTableName" value="crawl" />
 
 	<!-- This should reflect your installation, but 2181 is the default -->
 	<property name="zkPort" value="2181" />
 
 	<!-- All other settings are optional -->
 	<property name="onlyProcessNewRecords" value="false" />
 	<property name="onlyWriteNewRecords" value="false" />
 	<property name="contentColumnFamily" value="newcontent" />
  <property name="defaultMaxFileSizeInBytes" value="26214400" />
  <!-- 25 *  1024 * 1024 = 26214400 bytes -->
 	<!-- Overwrite more options here -->
 </bean>
 
 <bean id="hbaseWriterProcessor" class="org.archive.modules.writer.HBaseWriterProcessor">
 	<property name="hbaseParameters">
 		 <ref bean="hbaseParameterSettings"/> 
 	</property>
 </bean>
 
 <bean id="dispositionProcessors" class="org.archive.modules.DispositionChain">
 	<property name="processors">
 		 <list>
 			<ref bean="hbaseWriterProcessor"/>
 			<!-- other references -->
 		</list>
 	 </property>
 </bean>
 
 

See Also:

{@link org.archive.io.hbase.HBaseParameters} for defining hbaseParameters

Field Summary

Fields

Modifier and Type	Field and Description
static long	serialVersionUID The Constant serialVersionUID.

Fields inherited from class org.archive.modules.writer.WriterPoolProcessor

ANNOTATION_UNWRITTEN, compress, directory, frequentFlushes, maxFileSizeBytes, maxTotalBytesToWrite, maxWaitForIdleMs, poolMaxActive, prefix, serverCache, skipIdenticalDigests, startNewFilesOnCheckpoint, storePaths, template, writeBufferSize

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
HBaseWriterProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
protected org.archive.io.WriterPool	generateWriterPool(AtomicInteger serial)
long	getDefaultMaxFileSize() Gets the default max file size.
List<org.archive.spring.ConfigPath>	getDefaultStorePaths() Gets the default store paths.
HBaseParameters	getHbaseParameters() Gets the hbase parameters.
List<String>	getMetadata()
org.archive.uid.RecordIDGenerator	getRecordIDGenerator()
protected org.archive.modules.ProcessResult	innerProcessResult(org.archive.modules.CrawlURI uri)
void	setHbaseParameters(HBaseParameters options) Sets the hbase parameters.
protected void	setupPool(AtomicInteger serial)
protected boolean	shouldProcess(org.archive.modules.CrawlURI curi)
protected boolean	shouldWrite(org.archive.modules.CrawlURI curi) Whether the given CrawlURI should be written to archive files.
protected org.archive.modules.ProcessResult	write(org.archive.modules.CrawlURI curi, long recordLength, boolean doNotWriteContent) Write to HBase.

Methods inherited from class org.archive.modules.writer.WriterPoolProcessor

calcOutputDirs, checkBytesWritten, copyForwardWriteTagIfDupe, doCheckpoint, fromCheckpointJson, getCompress, getDirectory, getFrequentFlushes, getHostAddress, getMaxFileSizeBytes, getMaxTotalBytesToWrite, getMaxWaitForIdleMs, getMetadataProvider, getPool, getPoolMaxActive, getPrefix, getSerialNo, getServerCache, getSkipIdenticalDigests, getStartNewFilesOnCheckpoint, getStorePaths, getTemplate, getTotalBytesWritten, getWriteBufferSize, innerProcess, innerRejectProcess, setCompress, setDirectory, setFrequentFlushes, setMaxFileSizeBytes, setMaxTotalBytesToWrite, setMaxWaitForIdleMs, setMetadataProvider, setPool, setPoolMaxActive, setPrefix, setServerCache, setSkipIdenticalDigests, setStartNewFilesOnCheckpoint, setStorePaths, setTemplate, setTotalBytesWritten, setWriteBufferSize, start, stop, toCheckpointJson

Methods inherited from class org.archive.modules.Processor

finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.archive.io.WriterPoolSettings

calcOutputDirs, getCompress, getFrequentFlushes, getMaxFileSizeBytes, getPrefix, getTemplate, getWriteBufferSize

Methods inherited from interface org.springframework.context.Lifecycle

isRunning

Methods inherited from interface org.archive.checkpointing.Checkpointable

finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

Field Detail

serialVersionUID

public static final long serialVersionUID

The Constant serialVersionUID.

See Also:

Constant Field Values

Constructor Detail

HBaseWriterProcessor

public HBaseWriterProcessor()

Method Detail

getHbaseParameters

public HBaseParameters getHbaseParameters()

Gets the hbase parameters.

Returns:

the hbase parameters

setHbaseParameters

public void setHbaseParameters(HBaseParameters options)

Sets the hbase parameters.

Parameters:

options - the new hbase parameters

getDefaultMaxFileSize

public long getDefaultMaxFileSize()

Gets the default max file size.

Specified by:

getDefaultMaxFileSize in class org.archive.modules.writer.WriterPoolProcessor

Returns:

the default max file size

setupPool

protected void setupPool(AtomicInteger serial)

Specified by:

setupPool in class org.archive.modules.writer.WriterPoolProcessor

generateWriterPool

protected org.archive.io.WriterPool generateWriterPool(AtomicInteger serial)

innerProcessResult

protected org.archive.modules.ProcessResult innerProcessResult(org.archive.modules.CrawlURI uri)

Specified by:

innerProcessResult in class org.archive.modules.writer.WriterPoolProcessor

shouldProcess

protected boolean shouldProcess(org.archive.modules.CrawlURI curi)

Overrides:

shouldProcess in class org.archive.modules.writer.WriterPoolProcessor

shouldWrite

protected boolean shouldWrite(org.archive.modules.CrawlURI curi)

Whether the given CrawlURI should be written to archive files. Annotates CrawlURI with a reason for any negative answer.

Overrides:

shouldWrite in class org.archive.modules.writer.WriterPoolProcessor

Parameters:

curi - CrawlURI

Returns:

true if URI should be written; false otherwise

write

protected org.archive.modules.ProcessResult write(org.archive.modules.CrawlURI curi,
                                      long recordLength,
                                      boolean doNotWriteContent)
                                           throws IOException

Write to HBase.

Parameters:

curi - the curi

recordLength - the record length

in - the in

Returns:

the process result

Throws:

IOException - Signals that an I/O exception has occurred.

getDefaultStorePaths

public List<org.archive.spring.ConfigPath> getDefaultStorePaths()

Gets the default store paths.

Specified by:

getDefaultStorePaths in class org.archive.modules.writer.WriterPoolProcessor

Returns:

the default store paths

getMetadata

public List<String> getMetadata()

Specified by:

getMetadata in interface org.archive.io.WriterPoolSettings

Specified by:

getMetadata in class org.archive.modules.writer.WriterPoolProcessor

getRecordIDGenerator

public org.archive.uid.RecordIDGenerator getRecordIDGenerator()

Specified by:

getRecordIDGenerator in interface org.archive.io.warc.WARCWriterPoolSettings