# One failed embedded ServiceDef prevents services from being created…

Open blekinge opened this issue 9 years ago • 1 comments

… in Ranger

EmbeddedServiceDefsUtil have this method (I removed some log statements to make the code more clear)

public class EmbeddedServiceDefsUtil {
    ...
	private RangerServiceDef getOrCreateServiceDef(ServiceStore store, String serviceDefName) {
		....
		RangerServiceDef ret = null;
		boolean createServiceDef = (CollectionUtils.isEmpty(supportedServiceDefs) || supportedServiceDefs.contains(serviceDefName));
		try {
			ret = store.getServiceDefByName(serviceDefName);
			if(ret == null && createEmbeddedServiceDefs && createServiceDef) {
				ret = loadEmbeddedServiceDef(serviceDefName);
				if (ret.getId() != null) {
					store.setPopulateExistingBaseFields(true);
					ret = store.createServiceDef(ret);
					store.setPopulateExistingBaseFields(false);
				} else {
					ret = store.createServiceDef(ret);
				}
				LOG.info("created embedded service-def " + serviceDefName);
			}
		} catch(Exception excp) {
			LOG.fatal("EmbeddedServiceDefsUtil.getOrCreateServiceDef(): failed to load/create serviceType " + serviceDefName, excp);
		}
		...
		return ret;
	}
}

When creating a service def, the flag PopulateExistingBaseFields must be set to true, but it should be set to false afterwards.

If the statement ret = store.createServiceDef(ret); throws an exception, the flag is never reset. But since the catch statement do not rethrow the exception, null will be returned and Ranger will start normally.

Now, the flag PopulateExistingBaseFields are used by a lot of methods in ServiceDBStore

public RangerServiceDef createServiceDef(RangerServiceDef serviceDef) throws Exception
public RangerService createService(RangerService service) throws Exception
public RangerService updateService(RangerService service) throws Exception
public RangerPolicy createPolicy(RangerPolicy policy) throws Exception {

For createService and updateService, it will cause the method to select between svcServiceWithAssignedId and svcService. svcServiceWithAssignedId require that the incoming service have an ID. Services created via the web gui and via the ambari integration do NOT have a pre-selected ID. They cannot be created, and thus you cannot actually use Ranger.

Now, how can ret = store.createServiceDef(ret); fail? I do not know, but it happened on my system with this exception log entry

2016-12-06 09:34:25,850 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:136) - ==> EmbeddedServiceDefsUtil.getOrCreateServiceDef(kms)

2016-12-06 09:34:25,850 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.biz.ServiceDBStore (ServiceDBStore.java:934) - ==> ServiceDefDBStore.getServiceDefByName(kms)

2016-12-06 09:34:25,855 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.biz.ServiceDBStore (ServiceDBStore.java:946) - == ServiceDefDBStore.getServiceDefByName(kms): null

2016-12-06 09:34:25,855 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:168) - ==> EmbeddedServiceDefsUtil.loadEmbeddedServiceDef(kms)

2016-12-06 09:34:25,867 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:182) - ==> EmbeddedServiceDefsUtil.loadEmbeddedServiceDef(kms)

2016-12-06 09:34:25,867 [kact-man-001.kact.sblokalnet-startStop-1] INFO  org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:146) - creating embedded service-def kms

2016-12-06 09:34:25,868 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.biz.ServiceDBStore (ServiceDBStore.java:242) - ==> ServiceDefDBStore.createServiceDef(RangerServiceDef={id={7} guid={null} isEnabled={true} createdBy={null} updatedBy={null} createTime={null} updateTime={null} version={null} name={kms} implClass={org.apache.ranger.services.kms.RangerServiceKMS} label={KMS} description={KMS} rbKeyLabel={null} rbKeyDescription={null} c
onfigs={RangerServiceConfigDef={itemId={provider} name={provider} type={string} subType={null} mandatory={true} defaultValue={null} validationRegEx={null} validationMessage={null} uiHint={null} label={KMS URL} description={null} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }RangerServiceConfigDef={itemId={username} name={username} type={string} subType={null} mandatory={true} defaultValue={null} validationRegEx={null} validationM
essage={null} uiHint={null} label={Username} description={null} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }RangerServiceConfigDef={itemId={password} name={password} type={password} subType={null} mandatory={true} defaultValue={null} validationRegEx={null} validationMessage={null} uiHint={null} label={Password} description={null} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }} resources={RangerResourc
eDef={itemId={1} name={keyname} type={string} level={10} parent={} mandatory={true} lookupSupported={true} recursiveSupported={false} excludesSupported={false} matcher={org.apache.ranger.plugin.resourcematcher.RangerDefaultResourceMatcher} matcherOptions={{wildCard=true, ignoreCase=false}} validationRegEx={} validationMessage={} uiHint={} label={Key Name} description={Key Name} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }} acce
ssTypes={RangerAccessTypeDef={itemId={1} name={create} label={Create} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={2} name={delete} label={Delete} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={3} name={rollover} label={Rollover} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={4} name={setkeymaterial} label={Set Key Material} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={5} name={get
} label={Get} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={6} name={getkeys} label={Get Keys} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={7} name={getmetadata} label={Get Metadata} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={8} name={generateeek} label={Generate EEK} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={9} name={decrypteek} label={Decrypt EEK} rbKeyLabel={null} implie
dGrants={} }} policyConditions={} contextEnrichers={} enums={} })

2016-12-06 09:34:25,870 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.common.db.BaseDao (BaseDao.java:239) - Ignoring BaseDao.setIdentityInsert(). This should be executed if DB flavor is sqlserver.

2016-12-06 09:34:25,872 [kact-man-001.kact.sblokalnet-startStop-1] FATAL org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:156) - EmbeddedServiceDefsUtil.getOrCreateServiceDef(): failed to load/create serviceType kms
java.lang.NullPointerException
        at org.apache.ranger.service.RangerServiceDefServiceBase.mapViewToEntityBean(RangerServiceDefServiceBase.java:158)
        at org.apache.ranger.service.RangerServiceDefWithAssignedIdService.mapViewToEntityBean(RangerServiceDefWithAssignedIdService.java:31)
        at org.apache.ranger.service.RangerServiceDefWithAssignedIdService.mapViewToEntityBean(RangerServiceDefWithAssignedIdService.java:25)
        at org.apache.ranger.service.RangerBaseModelService.populateEntityBeanForCreate(RangerBaseModelService.java:180)
        at org.apache.ranger.service.RangerBaseModelService.preCreate(RangerBaseModelService.java:217)
        at org.apache.ranger.service.RangerBaseModelService.create(RangerBaseModelService.java:225)
        at org.apache.ranger.biz.ServiceDBStore.createServiceDef(ServiceDBStore.java:268)
        at org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil.getOrCreateServiceDef(EmbeddedServiceDefsUtil.java:149)
        at org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil.init(EmbeddedServiceDefsUtil.java:96)
        at org.apache.ranger.biz.ServiceDBStore$1.doInTransaction(ServiceDBStore.java:223)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at org.apache.ranger.biz.ServiceDBStore.initStore(ServiceDBStore.java:220)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:346)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:299)
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:132)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:394)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1448)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:872)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:814)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:731)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:485)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:92)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:872)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:814)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:731)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:485)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:92)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:872)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:814)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:731)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:485)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:92)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:872)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:814)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:731)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:485)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:92)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:605)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:383)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:283)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4992)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5490)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

2016-12-06 09:34:25,875 [kact-man-001.kact.sblokalnet-startStop-1] DEBUG org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:160) - <== EmbeddedServiceDefsUtil.getOrCreateServiceDef(kms): RangerServiceDef={id={7} guid={null} isEnabled={true} createdBy={null} updatedBy={null} createTime={null} updateTime={null} version={1} name={kms} implClass={org.apache.ranger.services.kms.RangerServiceKMS} label={KMS} description={KMS} rbKeyLabel={null} rbKeyDescription={null} configs={RangerServiceConfigDef={itemId={provider} name={provider} type={string} subType={null} mandatory={true} defaultValue={null} validationRegEx={null} validationMessage={null} uiHint={null} label={KMS URL} description={null} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }RangerServiceConfigDef={itemId={username} name={username} type={string} subType={null} mandatory={true} defaultValue={null} validationRegEx={null} validationMessage={null} uiHint={null} label={Username} description={null} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }RangerServiceConfigDef={itemId={password} name={password} type={password} subType={null} mandatory={true} defaultValue={null} validationRegEx={null} validationMessage={null} uiHint={null} label={Password} description={null} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }} resources={RangerResourceDef={itemId={1} name={keyname} type={string} level={10} parent={} mandatory={true} lookupSupported={true} recursiveSupported={false} excludesSupported={false} matcher={org.apache.ranger.plugin.resourcematcher.RangerDefaultResourceMatcher} matcherOptions={{wildCard=true, ignoreCase=false}} validationRegEx={} validationMessage={} uiHint={} label={Key Name} description={Key Name} rbKeyLabel={null} rbKeyDescription={null} rbKeyValidationMessage={null} }} accessTypes={RangerAccessTypeDef={itemId={1} name={create} label={Create} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={2} name={delete} label={Delete} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={3} name={rollover} label={Rollover} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={4} name={setkeymaterial} label={Set Key Material} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={5} name={get} label={Get} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={6} name={getkeys} label={Get Keys} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={7} name={getmetadata} label={Get Metadata} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={8} name={generateeek} label={Generate EEK} rbKeyLabel={null} impliedGrants={} }RangerAccessTypeDef={itemId={9} name={decrypteek} label={Decrypt EEK} rbKeyLabel={null} impliedGrants={} }} policyConditions={} contextEnrichers={} enums={} }

Dec 06 '16 10:12 blekinge

JIRA issue https://issues.apache.org/jira/browse/RANGER-1243

Dec 06 '16 10:12 blekinge