DiskBackedPartitionStore (Apache Giraph Parent 1.4.0-SNAPSHOT API)

java.lang.Object
- org.apache.giraph.ooc.data.DiskBackedDataStore<ExtendedDataOutput>
- - org.apache.giraph.ooc.data.DiskBackedPartitionStore<I,V,E>

Type Parameters:

I - Vertex id

V - Vertex data

E - Edge data

All Implemented Interfaces:

PartitionStore<I,V,E>
```
public class DiskBackedPartitionStore
extends DiskBackedDataStore<ExtendedDataOutput>
implements PartitionStore<I,V,E>
```
Implementation of a partition-store used for out-of-core mechanism. Partition store is responsible for partition data, as well as data buffers in INPUT_SUPERSTEP ("raw data buffer" -- defined in DiskBackedDataStore -- refers to vertex buffers in INPUT_SUPERSTEP).

Field Summary
- Fields inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore
  hasPartitionDataOnFile, MINIMUM_BUFFER_SIZE_TO_FLUSH, oocEngine

Constructor Summary

Constructors
Constructor and Description
`DiskBackedPartitionStore(PartitionStore<I,V,E> partitionStore, ImmutableClassesGiraphConfiguration<I,V,E> conf, org.apache.hadoop.mapreduce.Mapper.Context context, OutOfCoreEngine oocEngine)` Constructor.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`addEntryToInMemoryPartitionData(int partitionId, ExtendedDataOutput vertices)` Adds a single entry for a given partition to the in-memory data store.
`boolean`	`addPartition(Partition<I,V,E> partition)` Add a new partition to the store.
`void`	`addPartitionVertices(Integer partitionId, ExtendedDataOutput extendedDataOutput)` Add vertices to a given partition from a given DataOutput instance.
`protected int`	`entrySerializedSize(ExtendedDataOutput vertices)` Gets the size of a given entry in bytes.
`Partition<I,V,E>`	`getNextPartition()` Return the next partition in iteration for the current superstep.
`int`	`getNumPartitions()` Return the number of stored partitions.
`long`	`getPartitionEdgeCount(Integer partitionId)` Return the number of edges in a partition.
`Iterable<Integer>`	`getPartitionIds()` Return the ids of all the stored partitions as an Iterable.
`long`	`getPartitionVertexCount(Integer partitionId)` Return the number of vertices in a partition.
`boolean`	`hasPartition(Integer partitionId)` Whether a specific partition is present in the store.
`void`	`initialize()` Called at the beginning of the computation.
`boolean`	`isEmpty()` Whether the partition store is empty.
`protected long`	`loadInMemoryPartitionData(int partitionId, int ioThreadId, DataIndex index)` Loads data of a partition into data store.
`long`	`loadPartitionData(int partitionId)` Loads and assembles all data for a given partition, and put it into the data store.
`long`	`offloadBuffers(int partitionId)` Offloads raw data buffers of a given partition to disk, and returns the number of bytes offloaded from memory to disk.
`protected long`	`offloadInMemoryPartitionData(int partitionId, int ioThreadId, DataIndex index)` Offloads data of a partition in data store to disk.
`long`	`offloadPartitionData(int partitionId)` Offloads partition data of a given partition in the data store to disk, and returns the number of bytes offloaded from memory to disk.
`void`	`putPartition(Partition<I,V,E> partition)` Put a partition back to the store.
`protected ExtendedDataOutput`	`readNextEntry(DataInput in)` Reads the next available raw entry from a given input stream.
`Partition<I,V,E>`	`removePartition(Integer partitionId)` Remove a partition and return it.
`void`	`shutdown()` Called at the end of the computation.
`void`	`startIteration()` Start the iteration cycle to iterate over partitions.
`protected void`	`writeEntry(ExtendedDataOutput vertices, DataOutput out)` Writes a single raw entry to a given output stream.

Methods inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore
addEntry, getCandidateBuffersToOffload, loadPartitionDataProxy, offloadBuffersProxy, offloadPartitionDataProxy

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
 - DiskBackedPartitionStore
```
public DiskBackedPartitionStore(PartitionStore<I,V,E> partitionStore,
 ImmutableClassesGiraphConfiguration<I,V,E> conf,
 org.apache.hadoop.mapreduce.Mapper.Context context,
 OutOfCoreEngine oocEngine)
```
 Constructor.
 
 Parameters:
 
 partitionStore - In-memory partition store for which out-of-code partition store would be a wrapper
 
 conf - Configuration
 
 context - Job context
 
 oocEngine - Out-of-core engine
- Method Detail
 - addPartition
```
public boolean addPartition(Partition<I,V,E> partition)
```
 Description copied from interface: PartitionStore
 
 Add a *new* partition to the store. If the partition is already existed, it does not add the partition and returns false. Note: this method is not thread-safe and should be called by a single thread.
 
 Specified by:
 
 addPartition in interface PartitionStore
 
 Parameters:
 
 partition - Partition to add
 
 Returns:
 
 Whether the addition made any change in the partition store
 - removePartition
```
public Partition<I,V,E> removePartition(Integer partitionId)
```
 Description copied from interface: PartitionStore
 
 Remove a partition and return it. Called from a single thread, *not* from within a scheduling cycle. This method should *not* be called in INPUT_SUPERSTEP.
 
 Specified by:
 
 removePartition in interface PartitionStore
 
 Parameters:
 
 partitionId - Partition id
 
 Returns:
 
 The removed partition
 - hasPartition
```
public boolean hasPartition(Integer partitionId)
```
 Description copied from interface: PartitionStore
 
 Whether a specific partition is present in the store.
 
 Specified by:
 
 hasPartition in interface PartitionStore
 
 Parameters:
 
 partitionId - Partition id
 
 Returns:
 
 True iff the partition is present
 - getPartitionIds
```
public Iterable<Integer> getPartitionIds()
```
 Description copied from interface: PartitionStore
 
 Return the ids of all the stored partitions as an Iterable.
 
 Specified by:
 
 getPartitionIds in interface PartitionStore
 
 Returns:
 
 The partition ids
 - getNumPartitions
```
public int getNumPartitions()
```
 Description copied from interface: PartitionStore
 
 Return the number of stored partitions.
 
 Specified by:
 
 getNumPartitions in interface PartitionStore
 
 Returns:
 
 The number of partitions
 - getPartitionVertexCount
```
public long getPartitionVertexCount(Integer partitionId)
```
 Description copied from interface: PartitionStore
 
 Return the number of vertices in a partition.
 
 Specified by:
 
 getPartitionVertexCount in interface PartitionStore
 
 Parameters:
 
 partitionId - Partition id
 
 Returns:
 
 The number of vertices in the specified partition
 - getPartitionEdgeCount
```
public long getPartitionEdgeCount(Integer partitionId)
```
 Description copied from interface: PartitionStore
 
 Return the number of edges in a partition.
 
 Specified by:
 
 getPartitionEdgeCount in interface PartitionStore
 
 Parameters:
 
 partitionId - Partition id
 
 Returns:
 
 The number of edges in the specified partition
 - isEmpty
```
public boolean isEmpty()
```
 Description copied from interface: PartitionStore
 
 Whether the partition store is empty.
 
 Specified by:
 
 isEmpty in interface PartitionStore
 
 Returns:
 
 True iff there are no partitions in the store
 - startIteration
```
public void startIteration()
```
 Description copied from interface: PartitionStore
 
 Start the iteration cycle to iterate over partitions. Note that each iteration cycle *must* iterate over *all* partitions. Usually an iteration cycle is necessary for 1) moving edges (from edge store) to vertices after edge input splits are loaded in INPUT_SUPERSTEP, 2) computing partitions in each superstep (called once per superstep), 3) saving vertices/edges in the output superstep. 4) any sort of populating a data-structure based on the partitions in this store. After an iteration is started, multiple threads can access the partition store using PartitionStore.getNextPartition() to iterate over the partitions. Each time PartitionStore.getNextPartition() is called an unprocessed partition in the current iteration is returned. After processing of the partition is done, partition should be put back in the store using PartitionStore.putPartition(Partition). Here is an example of the entire workflow: In the main thread: partitionStore.startIteration(); In multiple threads iterating over the partitions: Partition partition = partitionStore.getNextPartition(); ... do stuff with partition ... partitionStore.putPartition(partition); Called from a single thread.
 
 Specified by:
 
 startIteration in interface PartitionStore
 - getNextPartition
```
public Partition<I,V,E> getNextPartition()
```
 Description copied from interface: PartitionStore
 
 Return the next partition in iteration for the current superstep. Note: user has to put back the partition to the store through PartitionStore.putPartition(Partition) after use. Look at comments on PartitionStore.startIteration() for more detail.
 
 Specified by:
 
 getNextPartition in interface PartitionStore
 
 Returns:
 
 The next partition to process
 - putPartition
```
public void putPartition(Partition<I,V,E> partition)
```
 Description copied from interface: PartitionStore
 
 Put a partition back to the store. Use this method to put a partition back after it has been retrieved through PartitionStore.getNextPartition(). Look at comments on PartitionStore.startIteration() for more detail.
 
 Specified by:
 
 putPartition in interface PartitionStore
 
 Parameters:
 
 partition - Partition
 - addPartitionVertices
```
public void addPartitionVertices(Integer partitionId,
 ExtendedDataOutput extendedDataOutput)
```
 Description copied from interface: PartitionStore
 
 Add vertices to a given partition from a given DataOutput instance. This method is called right after receipt of vertex request in INPUT_SUPERSTEP.
 
 Specified by:
 
 addPartitionVertices in interface PartitionStore
 
 Parameters:
 
 partitionId - Partition id
 
 extendedDataOutput - Output containing serialized vertex data
 - shutdown
```
public void shutdown()
```
 Description copied from interface: PartitionStore
 
 Called at the end of the computation. Called from a single thread.
 
 Specified by:
 
 shutdown in interface PartitionStore
 - initialize
```
public void initialize()
```
 Description copied from interface: PartitionStore
 
 Called at the beginning of the computation. Called from a single thread.
 
 Specified by:
 
 initialize in interface PartitionStore
 - loadInMemoryPartitionData
```
protected long loadInMemoryPartitionData(int partitionId,
 int ioThreadId,
 DataIndex index)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Loads data of a partition into data store. Returns number of bytes loaded.
 
 Specified by:
 
 loadInMemoryPartitionData in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 partitionId - id of the partition to load its data
 
 ioThreadId - id of the IO thread performing the load
 
 index - data index chain for the data to load
 
 Returns:
 
 number of bytes loaded from disk to memory
 
 Throws:
 
 IOException
 - readNextEntry
```
protected ExtendedDataOutput readNextEntry(DataInput in)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Reads the next available raw entry from a given input stream.
 
 Specified by:
 
 readNextEntry in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 in - input stream to read the entry from
 
 Returns:
 
 entry read from an input stream
 
 Throws:
 
 IOException
 - addEntryToInMemoryPartitionData
```
protected void addEntryToInMemoryPartitionData(int partitionId,
 ExtendedDataOutput vertices)
```
 Description copied from class: DiskBackedDataStore
 
 Adds a single entry for a given partition to the in-memory data store.
 
 Specified by:
 
 addEntryToInMemoryPartitionData in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 partitionId - id of the partition to add the data to
 
 vertices - input entry to add to the data store
 - loadPartitionData
```
public long loadPartitionData(int partitionId)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Loads and assembles all data for a given partition, and put it into the data store. Returns the number of bytes transferred from disk to memory in the loading process.
 
 Specified by:
 
 loadPartitionData in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 partitionId - id of the partition to load and assemble all data for
 
 Returns:
 
 number of bytes loaded from disk to memory
 
 Throws:
 
 IOException
 - offloadPartitionData
```
public long offloadPartitionData(int partitionId)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Offloads partition data of a given partition in the data store to disk, and returns the number of bytes offloaded from memory to disk.
 
 Specified by:
 
 offloadPartitionData in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 partitionId - id of the partition to offload its data
 
 Returns:
 
 number of bytes offloaded from memory to disk
 
 Throws:
 
 IOException
 - offloadInMemoryPartitionData
```
protected long offloadInMemoryPartitionData(int partitionId,
 int ioThreadId,
 DataIndex index)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Offloads data of a partition in data store to disk. Returns the number of bytes offloaded to disk
 
 Specified by:
 
 offloadInMemoryPartitionData in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 partitionId - id of the partition to offload to disk
 
 ioThreadId - id of the IO thread performing the offload
 
 index - data index chain for the data to offload
 
 Returns:
 
 number of bytes offloaded from memory to disk
 
 Throws:
 
 IOException
 - writeEntry
```
protected void writeEntry(ExtendedDataOutput vertices,
 DataOutput out)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Writes a single raw entry to a given output stream.
 
 Specified by:
 
 writeEntry in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 vertices - entry to write to output
 
 out - output stream to write the entry to
 
 Throws:
 
 IOException
 - offloadBuffers
```
public long offloadBuffers(int partitionId)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Offloads raw data buffers of a given partition to disk, and returns the number of bytes offloaded from memory to disk.
 
 Specified by:
 
 offloadBuffers in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 partitionId - id of the partition to offload its raw data buffers
 
 Returns:
 
 number of bytes offloaded from memory to disk
 
 Throws:
 
 IOException
 - entrySerializedSize
```
protected int entrySerializedSize(ExtendedDataOutput vertices)
```
 Description copied from class: DiskBackedDataStore
 
 Gets the size of a given entry in bytes.
 
 Specified by:
 
 entrySerializedSize in class DiskBackedDataStore<ExtendedDataOutput>
 
 Parameters:
 
 vertices - input entry to find its size
 
 Returns:
 
 size of given input entry in bytes

Class DiskBackedPartitionStore<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Field Summary

Fields inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore

Constructor Summary

Method Summary

Methods inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore

Methods inherited from class java.lang.Object

Constructor Detail

DiskBackedPartitionStore

Method Detail

addPartition

removePartition

hasPartition

getPartitionIds

getNumPartitions

getPartitionVertexCount

getPartitionEdgeCount

isEmpty

startIteration

getNextPartition

putPartition

addPartitionVertices

shutdown

initialize

loadInMemoryPartitionData

readNextEntry

addEntryToInMemoryPartitionData

loadPartitionData

offloadPartitionData

offloadInMemoryPartitionData

writeEntry

offloadBuffers

entrySerializedSize