Package | Description |
---|---|
org.apache.hadoop.yarn.server.federation.failover |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.
|
org.apache.hadoop.yarn.server.federation.policies |
Federation Policies.
|
org.apache.hadoop.yarn.server.federation.policies.amrmproxy |
AMRMPRoxy policies.
|
org.apache.hadoop.yarn.server.federation.policies.router |
Router policies.
|
org.apache.hadoop.yarn.server.federation.resolver |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.
|
org.apache.hadoop.yarn.server.federation.store.records |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.
|
org.apache.hadoop.yarn.server.federation.store.records.impl.pb |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.
|
org.apache.hadoop.yarn.server.federation.store.utils |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.
|
org.apache.hadoop.yarn.server.federation.utils |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.
|
Modifier and Type | Method and Description |
---|---|
static <T> T |
FederationProxyProviderUtil.createRMProxy(org.apache.hadoop.conf.Configuration configuration,
Class<T> protocol,
SubClusterId subClusterId,
org.apache.hadoop.security.UserGroupInformation user)
Create a proxy for the specified protocol in the context of Federation.
|
static <T> T |
FederationProxyProviderUtil.createRMProxy(org.apache.hadoop.conf.Configuration configuration,
Class<T> protocol,
SubClusterId subClusterId,
org.apache.hadoop.security.UserGroupInformation user,
org.apache.hadoop.security.token.Token<? extends org.apache.hadoop.security.token.TokenIdentifier> token)
Create a proxy for the specified protocol in the context of Federation.
|
Modifier and Type | Method and Description |
---|---|
SubClusterId |
FederationPolicyInitializationContext.getHomeSubcluster()
Returns the current home sub-cluster.
|
SubClusterId |
RouterPolicyFacade.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
This method provides a wrapper of all policy functionalities for routing .
|
Modifier and Type | Method and Description |
---|---|
protected Map<SubClusterId,SubClusterInfo> |
AbstractConfigurableFederationPolicy.getActiveSubclusters()
This methods gets active subclusters map from the
FederationStateStoreFacade and validate it not being null/empty. |
Modifier and Type | Method and Description |
---|---|
static FederationAMRMProxyPolicy |
FederationPolicyUtils.loadAMRMPolicy(String queue,
FederationAMRMProxyPolicy oldPolicy,
org.apache.hadoop.conf.Configuration conf,
FederationStateStoreFacade federationFacade,
SubClusterId homeSubClusterId)
Get AMRMProxy policy from state store, using default queue and
configuration as fallback.
|
void |
FederationPolicyInitializationContext.setHomeSubcluster(SubClusterId homeSubcluster)
Sets in the context the home sub-cluster.
|
Modifier and Type | Method and Description |
---|---|
SubClusterId |
RouterPolicyFacade.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
This method provides a wrapper of all policy functionalities for routing .
|
static void |
FederationPolicyUtils.validateSubClusterAvailability(List<SubClusterId> activeSubClusters,
List<SubClusterId> blackListSubClusters)
Validate if there is any active subcluster that is not blacklisted, it will
throw an exception if there are no usable subclusters.
|
static void |
FederationPolicyUtils.validateSubClusterAvailability(List<SubClusterId> activeSubClusters,
List<SubClusterId> blackListSubClusters)
Validate if there is any active subcluster that is not blacklisted, it will
throw an exception if there are no usable subclusters.
|
Constructor and Description |
---|
FederationPolicyInitializationContext(SubClusterPolicyConfiguration policy,
SubClusterResolver resolver,
FederationStateStoreFacade storeFacade,
SubClusterId home) |
RouterPolicyFacade(org.apache.hadoop.conf.Configuration conf,
FederationStateStoreFacade facade,
SubClusterResolver resolver,
SubClusterId homeSubcluster) |
Modifier and Type | Method and Description |
---|---|
protected SubClusterId |
LocalityMulticastAMRMProxyPolicy.getSubClusterForUnResolvedRequest(LocalityMulticastAMRMProxyPolicy.AllocationBookkeeper bookKeeper,
long allocationId)
For unit test to override.
|
Modifier and Type | Method and Description |
---|---|
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
RejectAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
LocalityMulticastAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
HomeAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
FederationAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters)
Splits the
ResourceRequest s from the client across one or more
sub-clusters based on the policy semantics (e.g., broadcast, load-based). |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
BroadcastAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Modifier and Type | Method and Description |
---|---|
void |
LocalityMulticastAMRMProxyPolicy.notifyOfResponse(SubClusterId subClusterId,
org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse response) |
void |
FederationAMRMProxyPolicy.notifyOfResponse(SubClusterId subClusterId,
org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse response)
This method should be invoked to notify the policy about responses being
received.
|
void |
AbstractAMRMProxyPolicy.notifyOfResponse(SubClusterId subClusterId,
org.apache.hadoop.yarn.api.protocolrecords.AllocateResponse response) |
Modifier and Type | Method and Description |
---|---|
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
RejectAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
LocalityMulticastAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
HomeAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
FederationAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters)
Splits the
ResourceRequest s from the client across one or more
sub-clusters based on the policy semantics (e.g., broadcast, load-based). |
Map<SubClusterId,List<org.apache.hadoop.yarn.api.records.ResourceRequest>> |
BroadcastAMRMProxyPolicy.splitResourceRequests(List<org.apache.hadoop.yarn.api.records.ResourceRequest> resourceRequests,
Set<SubClusterId> timedOutSubClusters) |
Modifier and Type | Method and Description |
---|---|
SubClusterId |
WeightedRandomRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blacklist) |
SubClusterId |
UniformRandomRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
Simply picks a random active subCluster to start the AM (this does NOT
depend on the weights in the policy).
|
SubClusterId |
RejectRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
The policy always reject requests.
|
SubClusterId |
PriorityRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blacklist) |
SubClusterId |
LocalityRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters) |
SubClusterId |
LoadBasedRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blacklist) |
SubClusterId |
HashBasedRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
Simply picks from alphabetically-sorted active subclusters based on the
hash of quey name.
|
SubClusterId |
FederationRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
Determines the sub-cluster that the user application submission should be
routed to.
|
Modifier and Type | Method and Description |
---|---|
SubClusterId |
WeightedRandomRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blacklist) |
SubClusterId |
UniformRandomRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
Simply picks a random active subCluster to start the AM (this does NOT
depend on the weights in the policy).
|
SubClusterId |
RejectRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
The policy always reject requests.
|
SubClusterId |
PriorityRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blacklist) |
SubClusterId |
LocalityRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters) |
SubClusterId |
LoadBasedRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blacklist) |
SubClusterId |
HashBasedRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
Simply picks from alphabetically-sorted active subclusters based on the
hash of quey name.
|
SubClusterId |
FederationRouterPolicy.getHomeSubcluster(org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext appSubmissionContext,
List<SubClusterId> blackListSubClusters)
Determines the sub-cluster that the user application submission should be
routed to.
|
Modifier and Type | Method and Description |
---|---|
SubClusterId |
SubClusterResolver.getSubClusterForNode(String nodename)
Obtain the sub-cluster that a specified node belongs to.
|
SubClusterId |
DefaultSubClusterResolverImpl.getSubClusterForNode(String nodename) |
SubClusterId |
AbstractSubClusterResolver.getSubClusterForNode(String nodename) |
Modifier and Type | Method and Description |
---|---|
Map<String,SubClusterId> |
AbstractSubClusterResolver.getNodeToSubCluster() |
Map<String,Set<SubClusterId>> |
AbstractSubClusterResolver.getRackToSubClusters() |
Set<SubClusterId> |
SubClusterResolver.getSubClustersForRack(String rackname)
Obtain the sub-clusters that have nodes on a specified rack.
|
Set<SubClusterId> |
DefaultSubClusterResolverImpl.getSubClustersForRack(String rackname) |
Set<SubClusterId> |
AbstractSubClusterResolver.getSubClustersForRack(String rackname) |
Modifier and Type | Method and Description |
---|---|
abstract SubClusterId |
ApplicationHomeSubCluster.getHomeSubCluster()
Get the
SubClusterId representing the unique identifier of the home
subcluster in which the ApplicationMaster of the application is running. |
abstract SubClusterId |
AddApplicationHomeSubClusterResponse.getHomeSubCluster()
Get the home sub-cluster that this application has been assigned to.
|
abstract SubClusterId |
SubClusterInfo.getSubClusterId()
Get the
SubClusterId representing the unique identifier of the
subcluster. |
abstract SubClusterId |
SubClusterHeartbeatRequest.getSubClusterId()
Get the
SubClusterId representing the unique identifier of the
subcluster. |
abstract SubClusterId |
SubClusterDeregisterRequest.getSubClusterId()
Get the
SubClusterId representing the unique identifier of the
subcluster. |
abstract SubClusterId |
GetSubClusterInfoRequest.getSubClusterId()
Get the
SubClusterId representing the unique identifier of the
subcluster. |
static SubClusterId |
SubClusterId.newInstance(String subClusterId) |
SubClusterId |
SubClusterIdInfo.toId()
Get the sub-cluster identifier as
SubClusterId . |
Modifier and Type | Method and Description |
---|---|
int |
SubClusterId.compareTo(SubClusterId other) |
static ApplicationHomeSubCluster |
ApplicationHomeSubCluster.newInstance(org.apache.hadoop.yarn.api.records.ApplicationId appId,
SubClusterId homeSubCluster) |
static GetSubClusterInfoRequest |
GetSubClusterInfoRequest.newInstance(SubClusterId subClusterId) |
static AddApplicationHomeSubClusterResponse |
AddApplicationHomeSubClusterResponse.newInstance(SubClusterId homeSubCluster) |
static SubClusterHeartbeatRequest |
SubClusterHeartbeatRequest.newInstance(SubClusterId subClusterId,
long lastHeartBeat,
SubClusterState state,
String capability) |
static SubClusterInfo |
SubClusterInfo.newInstance(SubClusterId subClusterId,
String amRMServiceAddress,
String clientRMServiceAddress,
String rmAdminServiceAddress,
String rmWebServiceAddress,
long lastHeartBeat,
SubClusterState state,
long lastStartTime,
String capability) |
static SubClusterInfo |
SubClusterInfo.newInstance(SubClusterId subClusterId,
String amRMServiceAddress,
String clientRMServiceAddress,
String rmAdminServiceAddress,
String rmWebServiceAddress,
SubClusterState state,
long lastStartTime,
String capability) |
static SubClusterDeregisterRequest |
SubClusterDeregisterRequest.newInstance(SubClusterId subClusterId,
SubClusterState subClusterState) |
static SubClusterHeartbeatRequest |
SubClusterHeartbeatRequest.newInstance(SubClusterId subClusterId,
SubClusterState state,
String capability) |
abstract void |
ApplicationHomeSubCluster.setHomeSubCluster(SubClusterId homeSubCluster)
Set the
SubClusterId representing the unique identifier of the home
subcluster in which the ApplicationMaster of the application is running. |
abstract void |
AddApplicationHomeSubClusterResponse.setHomeSubCluster(SubClusterId homeSubCluster)
Set the home sub-cluster that this application has been assigned to.
|
abstract void |
SubClusterInfo.setSubClusterId(SubClusterId subClusterId)
Set the
SubClusterId representing the unique identifier of the
subCluster. |
abstract void |
SubClusterHeartbeatRequest.setSubClusterId(SubClusterId subClusterId)
Set the
SubClusterId representing the unique identifier of the
subCluster. |
abstract void |
SubClusterDeregisterRequest.setSubClusterId(SubClusterId subClusterId)
Set the
SubClusterId representing the unique identifier of the
subcluster. |
abstract void |
GetSubClusterInfoRequest.setSubClusterId(SubClusterId subClusterId)
Set the
SubClusterId representing the unique identifier of the
subcluster. |
Constructor and Description |
---|
SubClusterIdInfo(SubClusterId subClusterId) |
Modifier and Type | Class and Description |
---|---|
class |
SubClusterIdPBImpl
Protocol buffer based implementation of
SubClusterId . |
Modifier and Type | Method and Description |
---|---|
SubClusterId |
ApplicationHomeSubClusterPBImpl.getHomeSubCluster() |
SubClusterId |
AddApplicationHomeSubClusterResponsePBImpl.getHomeSubCluster() |
SubClusterId |
SubClusterInfoPBImpl.getSubClusterId() |
SubClusterId |
SubClusterHeartbeatRequestPBImpl.getSubClusterId() |
SubClusterId |
SubClusterDeregisterRequestPBImpl.getSubClusterId() |
SubClusterId |
GetSubClusterInfoRequestPBImpl.getSubClusterId() |
Modifier and Type | Method and Description |
---|---|
void |
ApplicationHomeSubClusterPBImpl.setHomeSubCluster(SubClusterId homeSubCluster) |
void |
AddApplicationHomeSubClusterResponsePBImpl.setHomeSubCluster(SubClusterId homeSubCluster) |
void |
SubClusterInfoPBImpl.setSubClusterId(SubClusterId subClusterId) |
void |
SubClusterHeartbeatRequestPBImpl.setSubClusterId(SubClusterId subClusterId) |
void |
SubClusterDeregisterRequestPBImpl.setSubClusterId(SubClusterId subClusterId) |
void |
GetSubClusterInfoRequestPBImpl.setSubClusterId(SubClusterId subClusterId) |
Modifier and Type | Method and Description |
---|---|
protected static void |
FederationMembershipStateStoreInputValidator.checkSubClusterId(SubClusterId subClusterId)
Validate if the SubCluster Id is present or not.
|
Modifier and Type | Method and Description |
---|---|
SubClusterId |
FederationStateStoreFacade.addApplicationHomeSubCluster(ApplicationHomeSubCluster appHomeSubCluster)
Adds the home
SubClusterId for the specified ApplicationId . |
SubClusterId |
FederationStateStoreFacade.getApplicationHomeSubCluster(org.apache.hadoop.yarn.api.records.ApplicationId appId)
Returns the home
SubClusterId for the specified
ApplicationId . |
Modifier and Type | Method and Description |
---|---|
Map<SubClusterId,SubClusterInfo> |
FederationStateStoreFacade.getSubClusters(boolean filterInactiveSubClusters)
Returns the
SubClusterInfo of all active sub cluster(s). |
Modifier and Type | Method and Description |
---|---|
SubClusterInfo |
FederationStateStoreFacade.getSubCluster(SubClusterId subClusterId)
Returns the
SubClusterInfo for the specified SubClusterId . |
SubClusterInfo |
FederationStateStoreFacade.getSubCluster(SubClusterId subClusterId,
boolean flushCache)
Updates the cache with the central
FederationStateStore and returns
the SubClusterInfo for the specified SubClusterId . |
Copyright © 2008–2022 Apache Software Foundation. All rights reserved.