org.apache.giraph.partition

Interface PartitionStore<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Type Parameters:

I - Vertex id

V - Vertex data

E - Edge data

All Known Implementing Classes:

DiskBackedPartitionStore, SimplePartitionStore

public interface PartitionStore<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Structure that stores partitions for a worker. PartitionStore does not allow random accesses to partitions except upon removal. This structure is thread-safe, unless otherwise specified.

Method Summary

Modifier and Type	Method and Description
boolean	addPartition(Partition<I,V,E> partition) Add a *new* partition to the store.
void	addPartitionVertices(Integer partitionId, ExtendedDataOutput extendedDataOutput) Add vertices to a given partition from a given DataOutput instance.
Partition<I,V,E>	getNextPartition() Return the next partition in iteration for the current superstep.
int	getNumPartitions() Return the number of stored partitions.
long	getPartitionEdgeCount(Integer partitionId) Return the number of edges in a partition.
Iterable<Integer>	getPartitionIds() Return the ids of all the stored partitions as an Iterable.
long	getPartitionVertexCount(Integer partitionId) Return the number of vertices in a partition.
boolean	hasPartition(Integer partitionId) Whether a specific partition is present in the store.
void	initialize() Called at the beginning of the computation.
boolean	isEmpty() Whether the partition store is empty.
void	putPartition(Partition<I,V,E> partition) Put a partition back to the store.
Partition<I,V,E>	removePartition(Integer partitionId) Remove a partition and return it.
void	shutdown() Called at the end of the computation.
void	startIteration() Start the iteration cycle to iterate over partitions.

Method Detail

addPartition

boolean addPartition(Partition<I,V,E> partition)

Add a *new* partition to the store. If the partition is already existed, it does not add the partition and returns false. Note: this method is not thread-safe and should be called by a single thread.

Parameters:

partition - Partition to add

Returns:

Whether the addition made any change in the partition store

removePartition

Partition<I,V,E> removePartition(Integer partitionId)

Remove a partition and return it. Called from a single thread, *not* from within a scheduling cycle. This method should *not* be called in INPUT_SUPERSTEP.

Parameters:

partitionId - Partition id

Returns:

The removed partition

hasPartition

boolean hasPartition(Integer partitionId)

Whether a specific partition is present in the store.

Parameters:

partitionId - Partition id

Returns:

True iff the partition is present

getPartitionIds

Iterable<Integer> getPartitionIds()

Return the ids of all the stored partitions as an Iterable.

Returns:

The partition ids

getNumPartitions

int getNumPartitions()

Return the number of stored partitions.

Returns:

The number of partitions

getPartitionVertexCount

long getPartitionVertexCount(Integer partitionId)

Return the number of vertices in a partition.

Parameters:

partitionId - Partition id

Returns:

The number of vertices in the specified partition

getPartitionEdgeCount

long getPartitionEdgeCount(Integer partitionId)

Return the number of edges in a partition.

Parameters:

partitionId - Partition id

Returns:

The number of edges in the specified partition

isEmpty

boolean isEmpty()

Whether the partition store is empty.

Returns:

True iff there are no partitions in the store

addPartitionVertices

void addPartitionVertices(Integer partitionId,
                          ExtendedDataOutput extendedDataOutput)

Add vertices to a given partition from a given DataOutput instance. This method is called right after receipt of vertex request in INPUT_SUPERSTEP.

Parameters:

partitionId - Partition id

extendedDataOutput - Output containing serialized vertex data

shutdown

void shutdown()

Called at the end of the computation. Called from a single thread.

initialize

void initialize()

Called at the beginning of the computation. Called from a single thread.

startIteration

void startIteration()

Start the iteration cycle to iterate over partitions. Note that each iteration cycle *must* iterate over *all* partitions. Usually an iteration cycle is necessary for 1) moving edges (from edge store) to vertices after edge input splits are loaded in INPUT_SUPERSTEP, 2) computing partitions in each superstep (called once per superstep), 3) saving vertices/edges in the output superstep. 4) any sort of populating a data-structure based on the partitions in this store. After an iteration is started, multiple threads can access the partition store using getNextPartition() to iterate over the partitions. Each time getNextPartition() is called an unprocessed partition in the current iteration is returned. After processing of the partition is done, partition should be put back in the store using putPartition(Partition). Here is an example of the entire workflow: In the main thread: partitionStore.startIteration(); In multiple threads iterating over the partitions: Partition partition = partitionStore.getNextPartition(); ... do stuff with partition ... partitionStore.putPartition(partition); Called from a single thread.

getNextPartition

Partition<I,V,E> getNextPartition()

Return the next partition in iteration for the current superstep. Note: user has to put back the partition to the store through putPartition(Partition) after use. Look at comments on startIteration() for more detail.

Returns:

The next partition to process

putPartition

void putPartition(Partition<I,V,E> partition)

Put a partition back to the store. Use this method to put a partition back after it has been retrieved through getNextPartition(). Look at comments on startIteration() for more detail.

Parameters:

partition - Partition