Need support to create webdriver connection from a NTLM proxy environment

[kulin24]

QAF Version - Latest

Note: only the latest version is supported

Steps To Reproduce

1. In an environment which has NTLM proxy and if our selenium server is outside this environment. Then our webdriver is not able to connect to the server. Many banks have NTLM configured in their networks and this restricts the use of QAF to connect with selenium servers.

We have a customized version of UIDriverFactory to create a driver with this proxy configurations.

`private QAFExtendedWebDriver getDriver(String url, WebDriverCommandLogger reporter) {

		try
		{
			ConfigurationManager.getBundle().getString("proxyHost");
		}
		catch(Exception ex)
		{
			return standardConnect(url, reporter);	
		}
		
		
		if(!ConfigurationManager.getBundle().getString("proxyHost").equals(""))
		{
			return proxyConnect(url, reporter);
		}
		else
		{				
			return standardConnect(url, reporter);				
		}
	}
	
	private QAFExtendedWebDriver standardConnect(String url, WebDriverCommandLogger reporter)
	{
		logger.info("Direct Driver Connect");
		DesiredCapabilities desiredCapabilities = getDesiredCapabilities();
		Collection<QAFWebDriverCommandListener> listners = getDriverListeners();
		beforeInitialize(desiredCapabilities, listners);
		try {
			if (StringUtil
					.isNotBlank(ApplicationProperties.WEBDRIVER_REMOTE_SESSION
							.getStringVal())
					|| desiredCapabilities.asMap().containsKey(
							ApplicationProperties.WEBDRIVER_REMOTE_SESSION.key)) {

				Constructor<?> constructor = Class
						.forName(
								"com.qmetry.qaf.automation.ui.webdriver.LiveIsExtendedWebDriver")
						.getDeclaredConstructor(URL.class, Capabilities.class,
								WebDriverCommandLogger.class);
				return (QAFExtendedWebDriver) constructor.newInstance(new URL(url),
						desiredCapabilities, reporter);
			}
			return new QAFExtendedWebDriver(new URL(url), desiredCapabilities,
					reporter);
		} catch (Throwable e) {
			onInitializationFailure(desiredCapabilities, e, listners);

			throw new AutomationError(
					"Unable to Create Driver Instance " + e.getMessage(), e);
		}
	}
	
	private QAFExtendedWebDriver proxyConnect(String url, WebDriverCommandLogger reporter)
	{
		logger.info("Proxy Driver Connect");
		DesiredCapabilities desiredCapabilities = getDesiredCapabilities();
		Collection<QAFWebDriverCommandListener> listners = getDriverListeners();
		
		String proxyHost = ConfigurationManager.getBundle().getString("proxyHost");
		int proxyPort = Integer.parseInt(ConfigurationManager.getBundle().getString("proxyPort"));
		String proxyUserDomain = ConfigurationManager.getBundle().getString("proxyDomain");
		String proxyUser = ConfigurationManager.getBundle().getString("proxyUser");
		String proxyPassword = ConfigurationManager.getBundle().getString("proxyPassword");
		
		URL urls;
		try {
			urls = new URL(ConfigurationManager.getBundle().getString("remote.server"));
		} catch (MalformedURLException e) {
			throw new RuntimeException(e.getMessage(), e);
		}
		HttpClientBuilder builder = HttpClientBuilder.create();
		HttpHost proxy = new HttpHost(proxyHost, proxyPort);
		CredentialsProvider credsProvider = new BasicCredentialsProvider();
		credsProvider.setCredentials(new AuthScope(proxyHost, proxyPort),
				new NTCredentials(proxyUser, proxyPassword, getWorkstation(), proxyUserDomain));
		if (urls.getUserInfo() != null && !urls.getUserInfo().isEmpty()) {
			credsProvider.setCredentials(
					new AuthScope(urls.getHost(), (urls.getPort() > 0 ? urls.getPort() : urls.getDefaultPort())),
					new UsernamePasswordCredentials(urls.getUserInfo()));
		}
		builder.setProxy(proxy);
		builder.setDefaultCredentialsProvider(credsProvider);
		Factory factory = new MyHttpClientFactory(builder);
		HttpCommandExecutor executor = new HttpCommandExecutor(new HashMap<String, CommandInfo>(), urls, factory);

		beforeInitialize(desiredCapabilities, listners);
		try {
			return new QAFExtendedWebDriver(executor, urls, desiredCapabilities, reporter);

		} catch (Throwable e) {

			throw new AutomationError("Unable to Create Driver Instance " + e.getMessage(), e);
		}
	}

	private String getWorkstation() {
		Map<String, String> env = System.getenv();
		if (env.containsKey("COMPUTERNAME")) {
			// Windows
			return env.get("COMPUTERNAME");
		} else if (env.containsKey("HOSTNAME")) {
			// Unix/Linux/MacOS
			return env.get("HOSTNAME");
		} else {
			// From DNS
			try {
				return InetAddress.getLocalHost().getHostName();
			} catch (UnknownHostException ex) {
				return "Unknown";
			}
		}
	}
}`

Is it possible to use HTTPCommand executor to create a driver with proxy configurations as shown below?

I have attached the UIDriverFactory file where this code is implemented.

UiDriverFactory.java.zip

[cjayswal]

Use WinHttpProxySelector or similar implementation instead.

For example in before suite place below code:

import java.net.ProxySelector; 
....
ProxySelector.setDefault(new WinHttpProxySelector()); 

[jeremyperfecto]

I'm not entirely sure I understand how this would be used. Do you have an example of how you used it in the past? This particular project looks like it sets up a proxy on your machine that inherits your system level proxy auth and details but maybe I'm wrong. How do we pass the credentials and such in via a CI build using this since that system wouldn't have authentication?

With the implementation I applied to QAF class we just set the proxy details in application properties and its taken care of but its inclusive unfortunately to NTLM however it may work with other auth schemes but i haven't been in an environment yet to try it.

[cjayswal]

WinHttpProxySelector is one of the example, if it is not working or not suitable you can have custom implementation of ProxySelector. So in either case preferred to use ProxySelector to set proxy.

[kulin24]

Thanks Chirag for your quick response.

We will take a look at this option.

[kulin24]

Hi Chirag,

I had a discussion with Jeremy on this and he tried using JVM based proxies specifically ProxySelector as you mentioned. But in some of the bank environments, the authentication of proxy was getting expired and there was a problem in using ProxySelector like solutions.

Therefore, there was this specific need to use the HTTPCommandExecutor parameter when initializing the driver.

There are two things we wanted to know your opinion on:

1. Is it possible for this NTLM proxy based driver creation to be included in QAF?

2. If 1st option is not what can be done then, is there any way we can pass HTTPCommandExecutor object from any listener before initializing the driver and if this object is configured in bundle() properties the we should initialize our driver using this constructor new QAFExtendedWebDriver(executor, urls, desiredCapabilities, reporter);.

It would be great if we allow the support for HTTPCommandExecutor based constructor in QAF.

Looking forward to know your take on this.

[kulin24]

I also tried using normal ProxySelector, it is not detecting any proxy and the proxy type is showing as DIRECT even though I am in a NTLM proxy environment.

I am not able to connect to the remote server because of this.

The initial solution share with HTTPCommandExecutor is working in the same environment.

[cjayswal]

How and Where you are registering ProxySelector? Proxy selector works fine, set it either before driver init or beforeSuite (preferred) I had used that way and it works fine for one of client project running behind proxy.

[kulin24]

Can you share the code sample that you have tried before. I was registering the Proxy Selector before initializing the driver in the main method. I was using the code which was similar to the below code: `private void detectProxy() { try {

		System.setProperty("java.net.useSystemProxies", "true");
		List l = ProxySelector.getDefault().select(
				new URI("http://mineclipse.olympe.in"));

		for (Iterator iter = l.iterator(); iter.hasNext();) {

			Proxy proxy = (Proxy) iter.next();

			InetSocketAddress addr = (InetSocketAddress) proxy.address();

			if (addr == null) {
				config.put("Proxy", "false");
				config.put("ProxyHost", "");
				config.put("ProxyPort", "");
			} else {

				config.put("ProxyHost", addr.getHostName());
				config.put("ProxyPort", "" + addr.getPort());

			}
		}
	} catch (Exception e) {
		e.printStackTrace();
	}
	try {
		Thread.sleep(1500);
	} catch (InterruptedException e) {
		e.printStackTrace();
						
	}

}`

[kulin24]

Hi Chirag,

Can you please reconsider our suggestions here:

1. Is it possible for this NTLM proxy based driver creation to be included in QAF?

2. If 1st option is not what can be done then, is there any way we can pass HTTPCommandExecutor object from any listener before initializing the driver and if this object is configured in bundle() properties the we should initialize our driver using this constructor new QAFExtendedWebDriver(executor, urls, desiredCapabilities, reporter);.

It would be great if we allow the support for HTTPCommandExecutor based constructor in QAF.

We really need to use HTTPCommandExecutor as some of our clients have very secure proxy based environments.

[cjayswal]

I would not recommend it because it will address proxy only for driver connection. For example, in case of perfecto if someone wants to use rest api, it will not work for that and you need to do implementation separately to support rest api. Considering such cases we didn't go in that route in qaf implementation even if it can be one solution!. Like i said the ProxySelector solution works fine, may be i can take a look why it is not working for you or what you are doing wrong.

[kulin24]

We really appreciate your help. Please let me know when is a good time to connect. Thanks, Kulin