org.archive.modules.writer

Class HBaseWriterProcessor

java.lang.Object

org.archive.modules.Processor

org.archive.modules.writer.WriterPoolProcessor

org.archive.modules.writer.HBaseWriterProcessor

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.io.warc.WARCWriterPoolSettings, org.archive.io.WriterPoolSettings, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.Aware, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class HBaseWriterProcessor
extends org.archive.modules.writer.WriterPoolProcessor
implements org.archive.io.warc.WARCWriterPoolSettings

A Heritrix 3 processor that writes to Hadoop HBase. The following example shows how to configure the crawl job configuration.

 <bean id="hbaseParameterSettings" class="org.archive.io.hbase.HBaseParameters">
 	<!-- These settings are required -->
 	<property name="zkQuorum" value="localhost" />
 	<property name="hbaseTableName" value="crawl" />
 
 	<!-- This should reflect your installation, but 2181 is the default -->
 	<property name="zkPort" value="2181" />
 
 	<!-- All other settings are optional -->
 	<property name="onlyProcessNewRecords" value="false" />
 	<property name="onlyWriteNewRecords" value="false" />
 	<property name="contentColumnFamily" value="newcontent" />
  <property name="defaultMaxFileSizeInBytes" value="26214400" />
  <!-- 25 *  1024 * 1024 = 26214400 bytes -->
 	<!-- Overwrite more options here -->
 </bean>
 
 <bean id="hbaseWriterProcessor" class="org.archive.modules.writer.HBaseWriterProcessor">
 	<property name="hbaseParameters">
 		 <ref bean="hbaseParameterSettings"/> 
 	</property>
 </bean>
 
 <bean id="dispositionProcessors" class="org.archive.modules.DispositionChain">
 	<property name="processors">
 		 <list>
 			<ref bean="hbaseWriterProcessor"/>
 			<!-- other references -->
 		</list>
 	 </property>
 </bean>
 
 

See Also:

{@link org.archive.io.hbase.HBaseParameters} for defining hbaseParameters

Field Summary

Fields

Modifier and Type	Field and Description
static long	serialVersionUID The Constant serialVersionUID.

Fields inherited from class org.archive.modules.writer.WriterPoolProcessor

ANNOTATION_UNWRITTEN, compress, directory, frequentFlushes, maxFileSizeBytes, maxTotalBytesToWrite, maxWaitForIdleMs, poolMaxActive, prefix, serverCache, skipIdenticalDigests, startNewFilesOnCheckpoint, storePaths, template, writeBufferSize

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
HBaseWriterProcessor()

Method Summary

Methods

Modifier and Type	Method and Description
protected org.archive.io.WriterPool	generateWriterPool(AtomicInteger serial)
long	getDefaultMaxFileSize() Gets the default max file size.
List<org.archive.spring.ConfigPath>	getDefaultStorePaths() Gets the default store paths.
HBaseParameters	getHbaseParameters() Gets the hbase parameters.
List<String>	getMetadata()
org.archive.uid.RecordIDGenerator	getRecordIDGenerator()
protected org.archive.modules.ProcessResult	innerProcessResult(org.archive.modules.CrawlURI uri)
void	modifyPut(HBaseParameters hBaseParameters, org.archive.modules.CrawlURI curi, String ip, org.apache.hadoop.hbase.client.Put put, org.archive.io.RecordingOutputStream recordingOutputStream, org.archive.io.RecordingInputStream recordingInputStream) * This is a stub method and is here to allow extension/overriding for custom content parsing, data manipulation and to populate new columns.
void	setHbaseParameters(HBaseParameters options) Sets the hbase parameters.
protected void	setupPool(AtomicInteger serial)
protected boolean	shouldProcess(org.archive.modules.CrawlURI curi)
protected boolean	shouldWrite(org.archive.modules.CrawlURI curi) Whether the given CrawlURI should be written to archive files.
protected org.archive.modules.ProcessResult	write(org.archive.modules.CrawlURI curi) Write to HBase.

Methods inherited from class org.archive.modules.writer.WriterPoolProcessor

calcOutputDirs, checkBytesWritten, copyForwardWriteTagIfDupe, doCheckpoint, fromCheckpointJson, getCompress, getDirectory, getFrequentFlushes, getHostAddress, getMaxFileSizeBytes, getMaxTotalBytesToWrite, getMaxWaitForIdleMs, getMetadataProvider, getPool, getPoolMaxActive, getPrefix, getSerialNo, getServerCache, getSkipIdenticalDigests, getStartNewFilesOnCheckpoint, getStorePaths, getTemplate, getTotalBytesWritten, getWriteBufferSize, innerProcess, innerRejectProcess, setCompress, setDirectory, setFrequentFlushes, setMaxFileSizeBytes, setMaxTotalBytesToWrite, setMaxWaitForIdleMs, setMetadataProvider, setPool, setPoolMaxActive, setPrefix, setServerCache, setSkipIdenticalDigests, setStartNewFilesOnCheckpoint, setStorePaths, setTemplate, setTotalBytesWritten, setWriteBufferSize, start, stop, toCheckpointJson

Methods inherited from class org.archive.modules.Processor

finishCheckpoint, flattenVia, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, isRunning, isSuccess, process, report, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.archive.io.WriterPoolSettings

calcOutputDirs, getCompress, getFrequentFlushes, getMaxFileSizeBytes, getPrefix, getTemplate, getWriteBufferSize

Methods inherited from interface org.springframework.context.Lifecycle

isRunning

Methods inherited from interface org.archive.checkpointing.Checkpointable

finishCheckpoint, setRecoveryCheckpoint, startCheckpoint

Field Detail

serialVersionUID

public static final long serialVersionUID

The Constant serialVersionUID.

See Also:

Constant Field Values

Constructor Detail

HBaseWriterProcessor

public HBaseWriterProcessor()

Method Detail

getHbaseParameters

public HBaseParameters getHbaseParameters()

Gets the hbase parameters.

Returns:

the hbase parameters

setHbaseParameters

public void setHbaseParameters(HBaseParameters options)

Sets the hbase parameters.

Parameters:

options - the new hbase parameters

getDefaultMaxFileSize

public long getDefaultMaxFileSize()

Gets the default max file size.

Specified by:

getDefaultMaxFileSize in class org.archive.modules.writer.WriterPoolProcessor

Returns:

the default max file size

setupPool

protected void setupPool(AtomicInteger serial)

Specified by:

setupPool in class org.archive.modules.writer.WriterPoolProcessor

generateWriterPool

protected org.archive.io.WriterPool generateWriterPool(AtomicInteger serial)

innerProcessResult

protected org.archive.modules.ProcessResult innerProcessResult(org.archive.modules.CrawlURI uri)

Specified by:

innerProcessResult in class org.archive.modules.writer.WriterPoolProcessor

shouldProcess

protected boolean shouldProcess(org.archive.modules.CrawlURI curi)

Overrides:

shouldProcess in class org.archive.modules.writer.WriterPoolProcessor

shouldWrite

protected boolean shouldWrite(org.archive.modules.CrawlURI curi)

Whether the given CrawlURI should be written to archive files. Annotates CrawlURI with a reason for any negative answer.

Overrides:

shouldWrite in class org.archive.modules.writer.WriterPoolProcessor

Parameters:

curi - CrawlURI

Returns:

true if URI should be written; false otherwise

write

protected org.archive.modules.ProcessResult write(org.archive.modules.CrawlURI curi)
                                           throws IOException

Write to HBase.

Parameters:

curi - the curi

recordLength - the record length

in - the in

Returns:

the process result

Throws:

IOException - Signals that an I/O exception has occurred.

modifyPut

public void modifyPut(HBaseParameters hBaseParameters,
             org.archive.modules.CrawlURI curi,
             String ip,
             org.apache.hadoop.hbase.client.Put put,
             org.archive.io.RecordingOutputStream recordingOutputStream,
             org.archive.io.RecordingInputStream recordingInputStream)
               throws IOException

* This is a stub method and is here to allow extension/overriding for custom content parsing, data manipulation and to populate new columns. For Example : html parsing, text extraction, analysis and transformation and storing the results in new column families/columns using the batch update object. Or even saving the values in other custom hbase tables or other remote data sources. (a.k.a. anything you want)

Parameters:

hbaseParameters - - the configured hbase parameters for this crawl job

curi - - This requested uri for this content

ip - - the ip the host name in the uri resolves to

put - the stateful put object containing all the row data to be written. This is the 'output' object.

recordingOutputStream - - request to the server (output from us to the server)

recordingInputStream - - the server response (input from the server to us)

Throws:

IOException

getDefaultStorePaths

public List<org.archive.spring.ConfigPath> getDefaultStorePaths()

Gets the default store paths.

Specified by:

getDefaultStorePaths in class org.archive.modules.writer.WriterPoolProcessor

Returns:

the default store paths

getMetadata

public List<String> getMetadata()

Specified by:

getMetadata in interface org.archive.io.WriterPoolSettings

Specified by:

getMetadata in class org.archive.modules.writer.WriterPoolProcessor

getRecordIDGenerator

public org.archive.uid.RecordIDGenerator getRecordIDGenerator()

Specified by:

getRecordIDGenerator in interface org.archive.io.warc.WARCWriterPoolSettings