About activate Keyboard on Web Browser Search Bar Interaction

[TREEANA]

When using the web browser within the Unity app you've created, I want the keyboard object to activate when touching the search bar (a space within the web browser, not a UI you've separately designed to receive user input). Additionally, I'd like the keyboard to deactivate when the user presses the Enter key on the keyboard. I'm curious if you have any methods related to this

[TLabAltoh]

I think this can be achieved by adding a focus event callback process to the text field inside the web page.

TLabWebView.AddEventListener() can add event callbacks written in javascript syntax to the page content, so you can use this to register a process to send messages to unity.

Specifically, register a process that retrieves the text field on the web page load complete event and adds a focus event callback to the retrieved text area.

function() {
    com.tlab.libwebview.sendMessage(String go, String method, String message)
}

This function calls UnitySendMessage from javascript (Recently added features). You can use this function to send an arbitrary message to Unity informing it that a text area focus event has occurred, and thus get the timing of the keyboard activation.

See this document for the arguments.

Please consider that com.tlab.libwebview.sendMessage has not been tested yet and may not work.

Pull the latest commit as it uses some recently added features.

It is very difficult to convey in words, as it is a complex implementation of a complex process. Anyway, we will add more samples in the future.

[TLabAltoh]

For the enter key, each key on the keyboard uses a button on the uGUI, so you can add any event to the enter key button.

[TREEANA]

Thanks you for response.

To activate the keyboard object, I added two methods to the TlabVRkeyBoard scripts file:

public void HideKeyboard(bool active) { m_keyBOX.SetActive(!active); }

public void ShowKeyboard(bool active) { m_keyBOX.SetActive(true); }

If I want to make the ShowKeyboard() function trigger , would it be okay to create it like this?

function() { com.tlab.libwebview.sendMessage(TLabVKeyBorad, ShowKeyboard(), String message) }

Also, the TLabWebView script does not have the AddEventListener() function, so should I create it myself?

[TLabAltoh]

As you say, add event listener was not defined. My earlier explanation was that I had the wrong function name. Please refer to the following code to add event callbacks with evaluate js . (taken from the sample code)

    public void AddEventListener()
    {
        // Added a listener to retrieve page scroll events

        /*
            const scrollNum = document.getElementById('scroll-num');

            window.addEventListener('scroll',function(){
              scrollNum.textContent = window.pageYOffset;
            });
        */

        string event_name = "scroll";
        string tag_name = "window";
        string callback = tag_name + ".addEventListener('" + event_name + "'), function(){ });";

        m_webView.EvaluateJS(callback);
    }

[TLabAltoh]

For sendMessage, the arguments should be, from left to right, game object name, function name, and message. The example you showed is generally correct.

[TREEANA]

ty for your fast feedback. I will try it

[TREEANA]

I have a further question I'd like to ask. The event listener code you provided as an example is designed to respond to scroll events. I'm curious about how to modify the code to react when a text box becomes highlighted. Additionally, I'm interested in whether it's viable to use GameObject.Find() and otherObject.GetComponent<OtherClass>() instead of com.tlab.libwebview.sendMessage.

[TLabAltoh]

I wrote a few days ago that I would add samples to the repository, but I haven't even started yet (sorry). I haven't checked the actual operation, but here is a summary of the possible methods I have investigated.

References

• mdn document (focus event)
• mdn document (get elements by tag)

Sample for executing an event when the input element is focused

Sample 0

• Here is a sample for acquiring an element with the desired tag and registering a focus event. Please check the operation using Playground or other applications.

<!-- >>>>>> this is HTML code -->

<form id="form">
  <label>
    input 0:
    <input type="text" placeholder="please type something ..." />
  </label>
  <br/>
  <label>
    input 1:
    <input type="text" placeholder="please type something ..." />
  </label>
  <br/>
  <label>
    input 2:
    <input type="text" placeholder="please type something ..." />
  </label>
</form>

<!-- <<<<<< this is HTML code -->

// >>>>>> this is .js code

const inputs = document.getElementsByTagName("input");

for (const input of inputs) {
  input.addEventListener("focusin", (event) => {
    event.target.style.background = "pink";
    console.log("foucus in !");
  });

  input.addEventListener("focusout", (event) => {
    event.target.style.background = "";
    console.log("foucus out !");
  });
}

// <<<<<< this is .js code

• I'll leave a screenshot as well.

Sample 1

• Sample 0 shows how to retrieve a tag element from the page content and how to add a focus event to the retrieved tag.
• By executing javascript from Unity to retrieve all input tag elements in the page content and registering a process to send a message to Unity when the page is focused, Unity should be able to catch focus events.

// >>>>>> this is .js code

var action = function() {
	// Processing to be performed
	// Free to name game objects and methods according to the scene of your project.
	com.tlab.libwebview.sendMessage("gameobj", "onfoucus", "focus has occurred on the input element !");
}

// <<<<<< this is .js code

Note

• Currently there is only sendMessage to send messages from plugins to Unity (The ability to call Gameobject.Find() etc. from plug-ins is currently not implemented).
• m_webView.EvaluateJS(js) is a function to execute javascript in the page content, and should be used as if you were passing the contents of an arbitrary .js file as an argument.

[TREEANA]

Thanks for your detail explanation.

I updated TLabWebViewVR-OculusIntegration-2022.

and made TLabWebView.AddEventListener() like below.

    public void AddEventListener()
    {
        string jsCode = @"
    const inputs = document.getElementsByTagName('input');
    for (const input of inputs) {
        input.addEventListener('focusin', function() {
            com.tlab.libwebview.sendMessage('SimpleKeyboard', 'ShowKeyboard', 'Input focused');
        });
    }";

    
        m_webView.EvaluateJS(jsCode);
    } 

But I got this errror message can you help me ?

[TLabAltoh]

In the code sample, m_webview was treated as a variable of type TLabWebView, and EvaluateJS() is a method defined in TLabWebView, so it can be called directly when a function is defined inside.

// >>> I think it would work this way. >>>
public void AddEventListener()
{
    string jsCode = @"
    const inputs = document.getElementsByTagName('input');
    for (const input of inputs) {
        input.addEventListener('focusin', function() {
            window.TLabWebViewActivity.unitySendMessage('SimpleKeyboard', 'ShowKeyboard', 'Input focused');
        });
    }";
    
    EvaluateJS(jsCode);
} 
// <<< I think it would work this way. <<<

Also, com.libwebview.sendMessage() was not working, so we fixed the plugin. We have only updated this repository (Unity2021), but will update the repository for Unity2022 later.

// >>>
Function() {
    window.TLabWebViewActivity.unitySendMessage(String go, String method, String message)
}
// <<<

[TREEANA]

Does this mean that to execute the above method, I need to use the TLAB folder from the TLabWebViewVR repository? I would also like to know if the TLAB asset will work properly when run on the Oculus Quest2.

[TLabAltoh]

Does this mean that to execute the above method, I need to use the TLAB folder from the TLabWebViewVR repository?

I assume from the attached photo, AddEventListener() is defined in the class TLabWebView in the Assets\TLab\TLabWebView\ Runtime\TLabWebView.cs. Since this TLabWebView class is the class in which EvaluateJS() is defined, TLabWebView.AddEventListener() can call EvaluateJS() directly without using m_webview.

I would also like to know if the TLAB asset will work properly when run on the Oculus Quest2.

The reason com.libwebview.sendMessage() did not work is that there was a problem with the java plugin that runs WebView. This problem would have occurred on any device/platform (I fixed it this a.m). Regarding the question of whether it works correctly with Oculus quest 2, one problem that has been identified to date with TLabWebView-OculusIntegration2022 is that it sometimes crashes when the application starts. If you experience similar problems, we recommend using Unity2021.

I mentioned earlier that I would update the repository for Unity2022 later, and I just updated TLabWebViewVR-OculusIntegration2022 so that window.TLabWebViewActivity.unitySendMessage(String go, String method, String message) can also be called in TLabWebViewVR-OculusIntegration2022.

[TREEANA]

Thanks you for your detailed explanation. I updated TLabWebView-OculusIntegration2022 and I added AddEventListener() in TLabWebView like below public void AddEventListener() { string jsCode = @" const inputs = document.getElementsByTagName('input'); for (const input of inputs) { input.addEventListener('focusin', function() { window.TLabWebViewActivity.unitySendMessage('SimpleKeyborad', 'ShowKeyborad', 'Input focused'); }); }";

      EvaluateJS(jsCode);
  }

However, It seemed like window.TLabWebViewActivity.unitySendMessage() is not worked.

I attached the video below.

https://github.com/TLabAltoh/TLabWebViewVR/assets/76998594/49227c43-fb74-4179-bda9-a4e9c5fcac2e

[TLabAltoh]

I have again checked the code you presented earlier. I think one of the reasons why the callback does not work is that the destination function of unitySendMessage() specifies a function with a bool type argument.

public void ShowKeyboard(bool active)
{
   m_keyBOX.SetActive(true);
}

unitySendMessage() should only work with functions that take character types as arguments. Prepare a function that accepts a character-type argument and specify it as the destination of unitySendMessage().

public void OnMessage(string message)
{
   // Receive and handle messages from webview here
   // You need to define such a function yourself.
}

I myself overlooked this.

[TREEANA]

I'm sorry, but I'm beginning to understand less and less. In which script should the OnMessage function be included? Is it correct to use it within the TLabWebView script?

Also, how does Unity recognize the OnMessage(string) function? For instance, when executing unitySendMessage('SimpleKeyboard', 'ShowKeyboard', 'input_string'), does the 'input_string' value automatically get passed to the OnMessage(string) function?

If that's the case, should I create the OnMessage function like this?

public void OnMessage(string message)
{
    if(message == "show_keyboard")
    {
        ShowKeyboard(true);
    }
    else
    {
        ShowKeyboard(false);
    }
}

So, do I just need to add a new OnMessage function and input a string value as the third element in unitySendMessage, leaving everything else as it is?

[TLabAltoh]

In which script should the OnMessage function be included? Is it correct to use it within the TLabWebView script?

The OnMessage function can be included in any script as long as the following conditions are met

1. the class defining OnMessage must inherit from Monobehavior

2. the class that defines OnMessage must be attached to the destination game object

Since TLabWebView satisfies the above conditions, TLabWebView would be one of the candidates to define an OnMessage function (note, however, that TLabWebView is the part that I will make changes to in the future, so there will be conflicts with your own changes when incorporating repository changes in the future, well, but this is about git, not unity, so don't worry about it now.)

Although i have described the name of the function that receives messages as OnMessage, it does not mean that it must be defined as OnMessage to receive messages. Define the function with whatever name you deem appropriate.

Anyway, what I want to convey is that OnMessage can be defined in any class as long as the conditions are met.

how does Unity recognize the OnMessage function?

unitySendMessage sends a message to a function when the class attached to the game object has a function with the specified name.

Therefore, to send a message to OnMessage, it must be changed from ShowKeyboard to OnMessage. (i.e., the second argument should be the name of the function to which you are sending the message)

If that's the case, should I create the OnMessage function like this?

As for the process to be performed by OnMessage, I think the code you have shown will generally work fine.

So, do I just need to add a new OnMessage function and input a string value as the third element in unitySendMessage, leaving everything else as it is?

Specify the function name of the OnMessage function (or another named function that replaces OnMessage) as the second argument.

Also make sure that the first argument is also a game object with the class in which the desired function is defined.

[TREEANA]

Certainly, here's your revised sentence with the added apology:

"Thank you for the detailed explanation.

However, it seems to not work.

I attached photos below.

Also, I apologize for asking questions about this for almost 3 weeks."

[TLabAltoh]

Don't worry about it. I can't find anything particularly wrong with the code and it is odd that it is not working as it should.

What is the commit id in the TLabWebViewVR repository you are using? Since I only recently fixed the unitySendMessage issue, I think it's possible that you haven't pulled the fixed repository yet.

Please try to pull the latest repository once. Also, as for the code in the attached photo, I recommend caching it in a separate file somewhere, as your changes may be lost in the course of git operations.

In particular, make sure that the submodule Asset/TLab/TLabWebView/ in the repository is the latest commit. My modification of unitySendMessage is in this section.

[TLabAltoh]

Another possibility that is not working as it should is that there may be a problem with the javascript code. You would need to check and see if you can receive a message when you run a simple javascript that just calls unitySendMessage, etc. If you can receive a message with a simple javascript, then there may be a problem with the javascript code.

[TREEANA]

I am using TLabWebViewVR-OculusIntegration-2022. commit id (2a096a78242931da9aad54c6fd0168d5d9d258cb), I found there is recently update today, I will try it. Is unitySendMessage available in TLabWebViewVR-OculusIntegration-2022 repo?

[TLabAltoh]

Okay, it looks like the repository is using the modified version. So it doesn't seem to be a repository version issue...

You may need to check and see if you are calling AddEventListener()...

[TREEANA]

I just added AddEventListener() to the tlabview script, do I need to do anything else?

[TLabAltoh]

AddEventListener is not automatically called, so you must call it yourself.

[TREEANA]

How do I call it? I just made it. I actually thought it worked automatically and didn't know.

[TLabAltoh]

Since it is difficult to know when a web page is loading, it is better to execute it when the pointer is held over the canvas.

However, implementing such a process would be an additional effort to the current work, so for now, it is better to place a button on the uGUI and execute AddEventListener from there.

The process of executing AddEventListener without the uGUI's Button should be done after confirming that unitySendMessage has been executed for now.

[TLabAltoh]

I think this can be achieved by adding a focus event callback process to the text field inside the web page.

TLabWebView.AddEventListener() can add event callbacks written in javascript syntax to the page content, so you can use this to register a process to send messages to unity.

Specifically, register a process that retrieves the text field on the web page load complete event and adds a focus event callback to the retrieved text area.

function() {
    com.tlab.libwebview.sendMessage(String go, String method, String message)
}

This function calls UnitySendMessage from javascript (Recently added features). You can use this function to send an arbitrary message to Unity informing it that a text area focus event has occurred, and thus get the timing of the keyboard activation.

See this document for the arguments.

Please consider that com.tlab.libwebview.sendMessage has not been tested yet and may not work.

Pull the latest commit as it uses some recently added features.

It is very difficult to convey in words, as it is a complex implementation of a complex process. Anyway, we will add more samples in the future.

At that time, I originally wanted to tell you that EvaluayeJS allows you to add event listeners.

After confusing EvaluateJS and AddEventListener, I presented the sample code using AddEventListener as the function name, which may have triggered a misunderstanding.

[TREEANA]

I did not understand "TLabWebView.AddEventListener() can add event callbacks written in javascript syntax to the page content, so you can use this to register a process to send messages to unity."

How to add event callbacks written in javascript syntax ?

if I add callbacks , do I need to call AddEventListener() ?

[TLabAltoh]

I did not understand "TLabWebView.AddEventListener() can add event callbacks written in javascript syntax to the page content, so you can use this to register a process to send messages to unity."

I wanted to write that EvaluateJS can be used to embed callbacks defined in javascript in a web page, but I had misplaced the function name itself: I had mistakenly written AddEventListener instead of EvaluateJS.

In this code sample, i wanted to convey that you need to define your own processing to add event listeners to the web page using EvaluateJS.

How to add event callbacks written in javascript syntax ?

This is shown in the code sample. I don't think your own definition of AddEventListener is syntactically incorrect either (I haven't run it yet, so I can't say it won't cause errors, ).

if I add callbacks , do I need to call AddEventListener() ?

AddEventListener is a function that you yourself have defined based on the code sample, so you must call it yourself.

[TREEANA]

It seems that despite many attempts, it appears that JavaScript is not being recognized. Within Unity itself, isn't there a basic principle for detecting events when an input field (for example, a text field or text area) receives focus?

[TLabAltoh]

The WebView UI is not compatible with the Unity GUI, so the only way to catch events is to use javascript or other means.

I tried to see if I could add a sample that detects interactions with input and textarea tags, but depending on the site, I could not get the events as I intended.

On google.com, both focusin and focusout can be obtained, but on youtube.com, the behavior was not as intended, probably because something is reset each time an operation is performed.

I have not checked the implementation of each site, so I may be wrong.

The following is a sample of the behavior on google.com, which may not be perfect, but please use it as a reference.

Code Sample

using UnityEngine;
using TMPro;
using TLab.Android.WebView;

public class SearchBarInteractionSample : MonoBehaviour
{
    [SerializeField] private TextMeshProUGUI m_text;
    [SerializeField] private TLabWebView m_webview;

    /// <summary>
    /// Press button to execute
    /// </summary>
    public void AddEventListener()
    {
        string jsCode = @"
            var inputs = [];

            function querySelectorAllForAllTree(node, id) {
                if (node == null) {
                    return;
                }

                if (node.shadowRoot != undefined && node.shadowRoot != null) {
                    querySelectorAllForAllTree(node.shadowRoot, id);
                }

                for (var i = 0; i < node.childNodes.length; i++) {
                    querySelectorAllForAllTree(node.childNodes[i], id);
                }

                if (typeof node.querySelector == 'function') {
                    var element = node.querySelector(id);
                    if (element != undefined && element != null) {
                        if (!inputs.includes(element)) {
                            inputs.push(element);
                        }
                    }
                }
            }

            querySelectorAllForAllTree(document, 'textarea');
            querySelectorAllForAllTree(document, 'input');

            function focusin() {
                window.TLabWebViewActivity.unitySendMessage('SearchBar Interaction Sample', 'OnMessage', 'Foucusin');
            }

            function focusout() {
                window.TLabWebViewActivity.unitySendMessage('SearchBar Interaction Sample', 'OnMessage', 'Foucusout');
            }

            for (const input of inputs) {
                input.addEventListener('focusin', focusin);
                input.addEventListener('focusout', focusout);
            }

            window.TLabWebViewActivity.unitySendMessage('SearchBar Interaction Sample', 'OnMessage', 'Executed');
            ";

        m_webview.EvaluateJS(jsCode);
    }

    public void OnMessage(string message)
    {
        Debug.Log(message);

        m_text.text = message;
    }
}

https://github.com/TLabAltoh/TLabWebViewVR/assets/121733943/4719b291-6d90-4865-a239-a9a25c225186

youtube.com did not reset the event listeners as well as google.com when I was checking the javascript behavior on PC. Maybe if WebView also uses the PC user agent, the event listeners will not be reset and focus events can be retrieved correctly ...

I think this issue would be helpful to change the user agent.

m_webView.SetUserAgent("Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0");