C G H I M P S T W

C

checkBytesWritten(StateProvider) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

close() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

com.powerset.heritrix.writer - package com.powerset.heritrix.writer

Provides HBase writer for heritrix.

CONTENT_COLUMN - Static variable in class com.powerset.heritrix.writer.HBaseWriter

CONTENT_COLUMN_FAMILY - Static variable in class com.powerset.heritrix.writer.HBaseWriter

CONTENT_MAX_SIZE - Static variable in class com.powerset.heritrix.writer.HBaseWriterProcessor

Maximum allowable content size.

createCrawlTable(HBaseConfiguration, String) - Method in class com.powerset.heritrix.writer.HBaseWriter

CURI_COLUMN_FAMILY - Static variable in class com.powerset.heritrix.writer.HBaseWriter

G

getHostAddress(ProcessorURI) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

Return IP address of given URI suitable for recording (as in a classic ARC 5-field header line).

getMaster() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

getMaxActive() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

getMaxWait() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

getPool() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

getTable() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

getTotalBytesWritten() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

H

HBaseWriter - Class in com.powerset.heritrix.writer

Write to HBase.

HBaseWriter(String, String) - Constructor for class com.powerset.heritrix.writer.HBaseWriter

HBaseWriterPool - Class in com.powerset.heritrix.writer

A pool of HBaseWriters.

HBaseWriterPool(String, String, int, int) - Constructor for class com.powerset.heritrix.writer.HBaseWriterPool

Constructor

HBaseWriterProcessor - Class in com.powerset.heritrix.writer

An heritrix2 processor that writes to Hadoop HBase.

HBaseWriterProcessor() - Constructor for class com.powerset.heritrix.writer.HBaseWriterProcessor

I

initialTasks(StateProvider) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

innerProcess(ProcessorURI) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

innerProcessResult(ProcessorURI) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

M

MASTER - Static variable in class com.powerset.heritrix.writer.HBaseWriterProcessor

Location of hbase master.

P

POOL_MAX_ACTIVE - Static variable in class com.powerset.heritrix.writer.HBaseWriterProcessor

Maximum active files in pool.

POOL_MAX_WAIT - Static variable in class com.powerset.heritrix.writer.HBaseWriterProcessor

Maximum time to wait on pool element (milliseconds).

processContent(BatchUpdate) - Method in class com.powerset.heritrix.writer.HBaseWriter

This is a stub method and is here to allow extension/overriding for custom content parsing, data manipulation and to populate new columns.

S

SERVER_CACHE - Static variable in class com.powerset.heritrix.writer.HBaseWriterProcessor

setPool(WriterPool) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

setTotalBytesWritten(long) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

setupPool() - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

shouldProcess(ProcessorURI) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

shouldWrite(ProcessorURI) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

Whether the given ProcessorURI should be written to archive files.

T

TABLE - Static variable in class com.powerset.heritrix.writer.HBaseWriterProcessor

HBase table to crawl into.

TOTAL_BYTES_TO_WRITE - Static variable in class com.powerset.heritrix.writer.HBaseWriterProcessor

Total file bytes to write to disk.

W

write(ProcessorURI, String, RecordingOutputStream, RecordingInputStream) - Method in class com.powerset.heritrix.writer.HBaseWriter

write(ProcessorURI, long, InputStream, String) - Method in class com.powerset.heritrix.writer.HBaseWriterProcessor

C G H I M P S T W