org.apache.giraph.utils

Class CheckpointingUtils

java.lang.Object

org.apache.giraph.utils.CheckpointingUtils

public class CheckpointingUtils
extends Object

Holds useful functions to get checkpoint paths in hdfs.

Field Summary

Fields

Modifier and Type	Field and Description
static String	CHECKPOINT_DATA_POSTFIX If at the end of a checkpoint file, indicates that we store WorkerContext and aggregator handler data.
static String	CHECKPOINT_FINALIZED_POSTFIX If at the end of a checkpoint file, indicates the stitched checkpoint file prefixes.
static String	CHECKPOINT_METADATA_POSTFIX If at the end of a checkpoint file, indicates metadata
static String	CHECKPOINT_VALID_POSTFIX If at the end of a checkpoint file, indicates metadata and data is valid for the same filenames without .valid
static String	CHECKPOINT_VERTICES_POSTFIX If at the end of a checkpoint file, indicates vertices, edges, messages, etc.

Method Summary

Modifier and Type	Method and Description
static String	getCheckpointBasePath(org.apache.hadoop.conf.Configuration conf, String jobId) Path to the checkpoint's root (including job id)
static org.apache.hadoop.fs.Path	getCheckpointMarkPath(org.apache.hadoop.conf.Configuration conf, String jobId) Path to checkpoint&halt node in hdfs.
static long	getLastCheckpointedSuperstep(org.apache.hadoop.fs.FileSystem fs, String checkpointBasePath) Get the last saved superstep.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CHECKPOINT_METADATA_POSTFIX

public static final String CHECKPOINT_METADATA_POSTFIX

If at the end of a checkpoint file, indicates metadata

See Also:

Constant Field Values

CHECKPOINT_VERTICES_POSTFIX

public static final String CHECKPOINT_VERTICES_POSTFIX

If at the end of a checkpoint file, indicates vertices, edges, messages, etc.

See Also:

Constant Field Values

CHECKPOINT_VALID_POSTFIX

public static final String CHECKPOINT_VALID_POSTFIX

If at the end of a checkpoint file, indicates metadata and data is valid for the same filenames without .valid

See Also:

Constant Field Values

CHECKPOINT_DATA_POSTFIX

public static final String CHECKPOINT_DATA_POSTFIX

If at the end of a checkpoint file, indicates that we store WorkerContext and aggregator handler data.

See Also:

Constant Field Values

CHECKPOINT_FINALIZED_POSTFIX

public static final String CHECKPOINT_FINALIZED_POSTFIX

If at the end of a checkpoint file, indicates the stitched checkpoint file prefixes. A checkpoint is not valid if this file does not exist.

See Also:

Constant Field Values

Method Detail

getCheckpointBasePath

public static String getCheckpointBasePath(org.apache.hadoop.conf.Configuration conf,
                                           String jobId)

Path to the checkpoint's root (including job id)

Parameters:

conf - Immutable configuration of the job

jobId - job ID

Returns:

checkpoint's root

getCheckpointMarkPath

public static org.apache.hadoop.fs.Path getCheckpointMarkPath(org.apache.hadoop.conf.Configuration conf,
                                                              String jobId)

Path to checkpoint&halt node in hdfs. It is set to let client know that master has successfully finished checkpointing and job can be restarted.

Parameters:

conf - Immutable configuration of the job

jobId - job ID

Returns:

path to checkpoint&halt node in hdfs.

getLastCheckpointedSuperstep

public static long getLastCheckpointedSuperstep(org.apache.hadoop.fs.FileSystem fs,
                                                String checkpointBasePath)
                                         throws IOException

Get the last saved superstep.

Parameters:

fs - file system where checkpoint is stored.

checkpointBasePath - path to checkpoints folder

Returns:

Last good superstep number

Throws:

IOException