org.archive.io.hbase

Class HBaseWriter

java.lang.Object

org.archive.io.WriterPoolMember

org.archive.io.hbase.HBaseWriter

All Implemented Interfaces:

org.archive.format.ArchiveFileConstants, Serializer

public class HBaseWriter
extends org.archive.io.WriterPoolMember
implements Serializer

HBaseWriter implementation.

Field Summary

Fields inherited from class org.archive.io.WriterPoolMember

countOut, currentBasename, currentTimestamp, DEFAULT_PREFIX, DEFAULT_TEMPLATE, f, out, rebuf, roundRobinIndex, scratchbuffer, serialNoFormatter, settings, UTF8

Fields inherited from interface org.archive.format.ArchiveFileConstants

ABSOLUTE_OFFSET_KEY, CDX, CDX_FILE, CDX_LINE_BUFFER_SIZE, CRLF, DATE_FIELD_KEY, DEFAULT_DIGEST_METHOD, DOT_COMPRESSED_FILE_EXTENSION, DUMP, GZIP_DUMP, HEADER, INVALID_SUFFIX, LENGTH_FIELD_KEY, MIMETYPE_FIELD_KEY, NOHEAD, OCCUPIED_SUFFIX, READER_IDENTIFIER_FIELD_KEY, RECORD_IDENTIFIER_FIELD_KEY, SINGLE_SPACE, TYPE_FIELD_KEY, URL_FIELD_KEY, VERSION_FIELD_KEY

Constructor Summary

Constructors

Constructor and Description
HBaseWriter(AtomicInteger serialNo, org.archive.io.WriterPoolSettings settings, HBaseParameters parameters) Instantiates a new h base writer.

Method Summary

Methods

Modifier and Type	Method and Description
static String	createRowKeyFromUrl(String url) This is a stub method and is here to allow extension/overriding for custom content parsing, data manipulation and to populate new columns.
static String	createUrlFromRowKey(String rowKey)
static byte[]	getByteArrayFromInputStream(org.archive.io.ReplayInputStream replayInputStream, int streamSize) Read the ReplayInputStream and write it to the given BatchUpdate with the given column.
HBaseParameters	getHbaseParameters() Gets the hbase options.
org.apache.hadoop.hbase.client.HTable	getHTable() Gets the HTable client.
protected void	initializeCrawlTable(org.apache.hadoop.conf.Configuration hbaseConfiguration, String hbaseTableName) Creates the crawl table in HBase.
byte[]	serialize(byte[] bytes) Implement if you want to serialize bytes in a custom manner.
void	write(HBaseWriterProcessor hBaseWriterProcessor, org.archive.modules.CrawlURI curi, String ip, org.archive.io.RecordingOutputStream recordingOutputStream, org.archive.io.RecordingInputStream recordingInputStream, long recordedSize) Write the crawled output to the configured HBase table.

Methods inherited from class org.archive.io.WriterPoolMember

checkSize, checkWriteable, close, copyFrom, createFile, createFile, flush, generateNewBasename, getBaseFilename, getFile, getNextDirectory, getOutputStream, getPosition, isCompressed, isOversize, postWriteRecordTasks, preWriteRecordTasks, write, write, write

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HBaseWriter

public HBaseWriter(AtomicInteger serialNo,
           org.archive.io.WriterPoolSettings settings,
           HBaseParameters parameters)
            throws IOException

Instantiates a new h base writer.

Parameters:

serialNo - the serial no

settings - the settings

parameters - the parameters

Throws:

IOException - Signals that an I/O exception has occurred.

Method Detail

getHbaseParameters

public HBaseParameters getHbaseParameters()

Gets the hbase options.

Returns:

the hbase options

See Also:

HBaseParameters

getHTable

public org.apache.hadoop.hbase.client.HTable getHTable()

Gets the HTable client.

Returns:

the client

initializeCrawlTable

protected void initializeCrawlTable(org.apache.hadoop.conf.Configuration hbaseConfiguration,
                        String hbaseTableName)
                             throws IOException

Creates the crawl table in HBase.

Parameters:

hbaseConfiguration - the c

hbaseTableName - the table

Throws:

IOException - Signals that an I/O exception has occurred.

getByteArrayFromInputStream

public static byte[] getByteArrayFromInputStream(org.archive.io.ReplayInputStream replayInputStream,
                                 int streamSize)
                                          throws IOException

Read the ReplayInputStream and write it to the given BatchUpdate with the given column.

Parameters:

replayInputStream - the ris the cell data as a replay input stream

streamSize - the size

Returns:

the byte array from input stream

Throws:

IOException - Signals that an I/O exception has occurred.

createRowKeyFromUrl

public static String createRowKeyFromUrl(String url)

This is a stub method and is here to allow extension/overriding for custom content parsing, data manipulation and to populate new columns. For Example : html parsing, text extraction, analysis and transformation and storing the results in new column families/columns using the batch update object. Or even saving the values in other custom hbase tables or other remote data sources. (a.k.a. anything you want)

Parameters:

put - the stateful put object containing all the row data to be written.

replayInputStream - the replay input stream containing the raw content gotten by heritrix crawler.

streamSize - the stream size

Throws:

IOException - Signals that an I/O exception has occurred.

createUrlFromRowKey

public static String createUrlFromRowKey(String rowKey)

write

public void write(HBaseWriterProcessor hBaseWriterProcessor,
         org.archive.modules.CrawlURI curi,
         String ip,
         org.archive.io.RecordingOutputStream recordingOutputStream,
         org.archive.io.RecordingInputStream recordingInputStream,
         long recordedSize)
           throws IOException

Write the crawled output to the configured HBase table. Write each row key as the url with reverse domain and optionally process any content.

Parameters:

curi - URI of crawled document

ip - IP of remote machine.

recordingOutputStream - recording input stream that captured the response

recordingInputStream - recording output stream that captured the GET request

Throws:

IOException - Signals that an I/O exception has occurred.

serialize

public byte[] serialize(byte[] bytes)

Description copied from interface: Serializer

Implement if you want to serialize bytes in a custom manner.

Specified by:

serialize in interface Serializer

Parameters:

bytes - the bytes

Returns:

serialized bytes