Show Current Cursor Position In Selenium

January 30, 2024 Post a Comment

Is there a way to show current cursor position or something like this? I have action chain that should click on the exact point on some object but I guess I've picked up wrong coor

Solution 1:

I had an opaque HTML5 canvas where I had to check values of tooltips at various offsets, and what trouble! After lots of banging my head, I learned several important things:

move_to_element_by_offset has issues prior to 2.42. See also https://code.google.com/p/selenium/issues/detail?id=4215
Even after an update, move_to_element_by_offset still seems persnickety. I switched to a combination of the ActionChains move_to_element hover (defaults to the center of the element) and a second ActionChains with move_by_offset
Now that you're in position, re-request the position-sensitive element. In my case, that's a tooltip object. This selenium.webdriver.remote.element will have the .location['x'] and .location['y'] members you need to sanity check your position.

Things that don't work for grabbing cursor position after a perform'd ActionChain to hover, to save everyone some time:

.location['x'] and .location['y'] on the element you pass into move_to_element. These values are not updated by ActionChains.
selenium.webdriver.remote.webdriverswitch_to_active_element; I'm not clear what elements qualify as "active", but it doesn't acknowledge our tooltips as active elements and gave me back [0, 0] coordinates.
selenium.webdriver.remote.webdriverget_window_position; the documentation was a bit vague, but this does just what the function suggests: a window rather than a cursor position.

Solution 2:

Yeah, this problem is frustrating.

This method will allow you to see where the mouse is by painting a red dot on it: Start by adding the following CSS to your webpage, temporarily:

// TODO: Remove me after you've figured out where the mouse is moving to..dot {
  background: red;
  position: absolute;
  width: 2px;
  height: 2px;
  z-index: 10000;
}

Then, in your code, just before you're calling the *.moveTo() method, add a pause to allow you to inject code into the browser's console. This code is in JavaScript (JS / webdriver.io) but it's easy to add in any language:

// Other test stuff...

console.log("This gives you 10 seconds to inject the code..."); // <-- Add this and the next line
browser.pause(10000);

browser.moveTo(...gawdKnowsWhere);

Now, when you see the test command line console is waiting 10 seconds, open up Chrome/Firefox's developer tools (F12) and paste this snippet into the browser's "Console" tab:

  (function() {
    "use strict";

    document.onmousemove = handleMouseMove;
    functionhandleMouseMove(event) {
      var dot, eventDoc, doc, body, pageX, pageY;
      
      event = event || window.event; // IE-ism// If pageX/Y aren't available and clientX/Y// are, calculate pageX/Y - logic taken from jQuery// Calculate pageX/Y if missing and clientX/Y availableif (event.pageX == null && event.clientX != null) {
        eventDoc = (event.target && event.target.ownerDocument) || document;
        doc = eventDoc.documentElement;
        body = eventDoc.body;

        event.pageX = event.clientX +
          (doc && doc.scrollLeft || body && body.scrollLeft || 0) -
          (doc && doc.clientLeft || body && body.clientLeft || 0);
        event.pageY = event.clientY +
          (doc && doc.scrollTop  || body && body.scrollTop  || 0) -
          (doc && doc.clientTop  || body && body.clientTop  || 0 );
      }

      // Add a dot to follow the cursor
      dot = document.createElement('div');
      dot.className = "dot";
      dot.style.left = event.pageX + "px";
      dot.style.top = event.pageY + "px";
      document.body.appendChild(dot);
    }
  })();

Now, go hunting for a red dot. Wherever the mouse is moved to, it will not be able to click there because there's a red square in the way now. But at least you can see where it's trying to go.

Can't see the dot?

Some tips:

Make the dots bigger in the CSS by modifying the width & height.
Move your mouse across the screen and watch all the red dots. Are they showing up when you move your mouse?
Make sure you can paint a red dot where you expect it to be. The z-order might not be high enough, etc.

Solution 3:

If all you want to confirm is whether Selenium is clicking on the right element, instead of click()-ing on the element:

actions.move_to_element(my_element).click().perform()

context_click() on it:

actions.move_to_element(my_element).context_click().perform()

Selenium will right click on your element, which will result in the context menu being displayed at the bottom right corner of your cursor. Once you see the position of the context menu, you will know whether Selenium is clicking on the right element.

Solution 4:

I don't see any possible methods for changing the resolution directly as we can not use any locator for this.

For your example of JWPlayer, I am sure that there might be some javascript snippet which will allow changing the resolution. Using the Selenium JavascriptExecutor you can run that javacript code to simulate the resolution change.. although not by clicking the cursor.

WebDriverdriver=newChromeDriver();

if (driver instanceof JavascriptExecutor) {
    ((JavascriptExecutor) driver).executeScript(YOUR_JAVA_SCRIPT_STRING);
}

I noticed you mentioned you don't want to use any native JWPlayer API. I see this as the only possible solution.

Solution 5:

It was a little hard for me as a selenium's newbie to understand that it just can't generate clicks on flash objects. The only possible solution I see here is to use sikuli-script.

I've found the project on GitHub: https://github.com/kevlened/sikuli_cpython. It allows to use sikuli scripts (written in jython) in pure python so I guess I will be able to write some sikuli scripts and integrate them with selenium tests.

Python Dictionary