public class YarnUtils extends Object
Modifier and Type | Method and Description |
---|---|
static void |
addFileToResourceMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources,
org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path target)
Boilerplate to add a file to the local resources..
|
static void |
addFsResourcesToMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> map,
GiraphConfiguration giraphConf,
org.apache.hadoop.yarn.api.records.ApplicationId appId)
Populates the LocalResources list with the HDFS paths listed in
the conf under GiraphConstants.GIRAPH_YARN_LIBJARS, and the
GiraphConfiguration for this job.
|
static void |
addGiraphConfToLocalResourceMap(GiraphConfiguration giraphConf,
org.apache.hadoop.yarn.api.records.ApplicationId appId,
Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResourceMap)
Populate the LocalResources list with the GiraphConf XML file's HDFS path.
|
static void |
addLocalClasspathToEnv(Map<String,String> env,
GiraphConfiguration giraphConf)
Popuate the environment string map to be added to the environment vars
in a remote execution container.
|
static void |
exportGiraphConfiguration(GiraphConfiguration giraphConf,
org.apache.hadoop.yarn.api.records.ApplicationId appId)
Export our populated GiraphConfiguration as an XML file to be used by the
ApplicationMaster's exec container, and register it with LocalResources.
|
static org.apache.hadoop.fs.Path |
getFsCachePath(org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.yarn.api.records.ApplicationId appId)
Get the base HDFS dir we will be storing our LocalResources in.
|
static Set<org.apache.hadoop.fs.Path> |
getLocalFiles(Set<String> fileNames)
Utility function to locate local JAR files and other resources
recursively in the dirs on the local CLASSPATH.
|
public static void addFsResourcesToMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> map, GiraphConfiguration giraphConf, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
map
- the LocalResources list to populate.giraphConf
- the configuration to use to select jars to include.appId
- the ApplicationId, naming the the HDFS base dir for job jars.IOException
public static Set<org.apache.hadoop.fs.Path> getLocalFiles(Set<String> fileNames)
fileNames
are found, we stop and return the results.fileNames
- the file name of the jars, without path information.public static void addFileToResourceMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path target) throws IOException
localResources
- the LocalResources map to populate.fs
- handle to the HDFS file system.target
- the file to send to the remote container.IOException
public static org.apache.hadoop.fs.Path getFsCachePath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.yarn.api.records.ApplicationId appId)
fs
- the file system.appId
- the ApplicationId under which our resources will be stored.public static void addLocalClasspathToEnv(Map<String,String> env, GiraphConfiguration giraphConf)
env
- the map of env var values.giraphConf
- the GiraphConfiguration to pull values from.public static void addGiraphConfToLocalResourceMap(GiraphConfiguration giraphConf, org.apache.hadoop.yarn.api.records.ApplicationId appId, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResourceMap) throws IOException
giraphConf
- the GiraphConfifuration to export for worker tasks.appId
- the ApplicationId for this YARN app.localResourceMap
- the LocalResource map of files to export to tasks.IOException
public static void exportGiraphConfiguration(GiraphConfiguration giraphConf, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
giraphConf
- the current Configuration object to be published.appId
- the ApplicationId to stamp this app's base HDFS resources dir.IOException
Copyright © 2011-2020 The Apache Software Foundation. All Rights Reserved.