Controlling the desktop
The endia desktop commands act in the signed-in user's session, where the screen is. Most commands name a process by its ID, and the keyboard and mouse commands first bring that process's main window to the front. desktop pointer --screen instead acts at a point on the screen, without naming a process.
The examples read a value from JSON with jq, which is a separate program that must be installed on the Linux side where the examples run.
pid=$(endia desktop launch orders-dev-dev-1 C:\\app\\Orders.exe | jq -r .processId)
endia desktop wait orders-dev-dev-1 "$pid" "Window[Orders]/Edit#CustomerName"
endia desktop set-value orders-dev-dev-1 "$pid" "Window[Orders]/Edit#CustomerName" "Ada"
endia desktop invoke orders-dev-dev-1 "$pid" "Window[Orders]/Button[Save]"
endia desktop click --button right orders-dev-dev-1 "$pid" "Window[Orders]/List/ListItem:3"
endia desktop keys orders-dev-dev-1 "$pid" "ctrl+s"
endia desktop keys --text orders-dev-dev-1 "$pid" "Order 42"
endia desktop pointer orders-dev-dev-1 "$pid" click 120,45
endia desktop pointer --screen orders-dev-dev-1 double-click 960,540
Commands
| Command | What it does |
|---|---|
desktop launch <machine> <program> [args] |
Starts a program on the desktop and prints, in JSON, its process ID (processId), start time, session, account, integrity level and windows. |
desktop wait <machine> <pid> <element> |
Waits until the element exists, then prints it and its window. With --gone, waits until it no longer exists. --timeout <seconds> sets the longest wait; the default is the service's desktopActionTimeoutSeconds, 60 seconds unless changed. |
desktop invoke <machine> <pid> <element> |
Presses a button or other element through UI Automation, without moving the mouse. |
desktop click <machine> <pid> <element> |
Brings the element's window to the front and clicks the centre of the element with the mouse, for an element that responds only to mouse clicks. --button right and --double change the click. |
desktop set-value <machine> <pid> <element> <value> |
Sets a text box or similar element, and prints the value read back. A password field accepts only --secret-stdin, which reads the value from standard input and never prints or logs it. |
desktop keys <machine> <pid> "<chords>" |
Presses key chords such as "ctrl+s" or "alt+f4 enter". With --text, types the text as written. |
desktop pointer <machine> <pid> <action> <x,y> |
Moves the mouse to a point in the process's main window and clicks, double-clicks or scrolls there. --button right and --button middle change the button; --delta -3 scrolls three notches down. It prints the screen point, and the window there with the process that owns it. |
desktop pointer --screen <machine> <action> <x,y> |
Does the same at a point on the screen, without choosing a window or bringing one to the front. |
desktop windows <machine> |
Lists every visible window on the desktop, with the process that owns it, its position and size, and its title. |
desktop tray ls <machine> |
Lists the icons in the notification area beside the clock, including those hidden behind "Show hidden icons". |
desktop tray click <machine> <icon> |
Clicks the icon whose name starts with <icon>, opening the hidden icons first when it is there. --button right opens most programs' menus. |
desktop uac --program <exe> --consent <machine> |
Answers the Windows elevation prompt for that program. --consent accepts it and --cancel refuses it. --user <admin> --password-stdin enters an administrator's name and password. |
Elements
An element is named by a path of segments separated by /. The first segment always names one of the process's windows; each later segment is searched for anywhere inside what the segment before it found. So Window[Orders]/Button[Save] is the Save button in the window called Orders, while Button[Save] on its own looks for a window that is a button, and fails with ELEMENT_NOT_FOUND. When you do not know the window's title, Window/Button[Save] works while the process shows one window.
A menu that a program opens is a window of its own. A Windows menu's window has the control type Menu: Menu/MenuItem[Exit]. A menu drawn by another framework may be an ordinary Window, as Avalonia's is: Window/MenuItem[Exit]. desktop inspect shows which. When the first segment matches no window, the tree in the error shows each window without its children, so an empty children there does not mean the window is empty.
Each segment may give, in this order:
- a control type (
Window,Button,Edit); - a name in square brackets, compared exactly and with case (
[Save]); - an automation ID after
#(#CustomerName); - a position after
:, counting from 1 (:2).
It must give at least one of the first three. Window[Orders]/Edit#CustomerName names an edit box in the window called Orders, and Window[Orders]/Button:2 the second button in it. desktop inspect shows the control types, names and automation IDs to use. Naming an element is sturdier than clicking a position, because it does not depend on the window's size or layout.
Waiting, passwords and tray icons
Waiting. A program shows its windows some time after it starts, and a window changes after a click. Wait for the element the next step needs with desktop wait instead of sleeping or polling desktop inspect: it returns as soon as the element exists, and fails with TIMEOUT, showing the window's elements, if it never does. A one-segment selector such as "Window[Orders]" waits for the window.
Passwords. set-value, keys and pointer refuse a password field, so that a password cannot be typed by accident or end up in a log. To enter one, pass it on standard input: printf '%s' "$password" | endia desktop set-value --secret-stdin orders-dev-dev-1 "$pid" "Window[Sign in]/Edit#Password". East India writes it into the field and never prints, logs or reads it back.
Notification-area icons. A tray program's icon belongs to the Windows taskbar, not to the program, so the commands that take a process ID cannot reach it; desktop tray can. Icon names are the programs' tooltips, and many programs add their state to them, so desktop tray click takes the start of the name: endia desktop tray click --button right orders-dev-dev-1 "Orders" clicks Orders: connected. The menu that opens is a window of the tray program, so continue with its process ID: desktop wait and desktop invoke with "Menu/MenuItem[Settings]" for a Windows menu. Avalonia's menu is a Window whose items do not answer invoke: use "Window/MenuItem[Settings]" with desktop click.
Positions
Every position is in physical pixels, and there are two origins:
- Relative to a window. With a process ID, the position is measured from the top-left corner of the process's main window's client area: the inside of the window, below its title bar.
120,45is 120 pixels right and 45 pixels down from there, wherever the window is. - Absolute on the screen. With
--screen, the position is measured from the top-left corner of the screen.desktop windowsgives window positions on the same scale, so a point inside a window's rectangle there lands on that window, if no other window covers it.
desktop inspect gives each element two rectangles. bounds is measured from the window's outer top-left corner, which is not the client area's corner. screenBounds is measured from the top-left corner of the screen. To click an element, use desktop click with its selector, which brings the window to the front and clicks the centre of screenBounds for you.
Process IDs. Windows reuses process IDs. Every command that takes a process ID also takes --started <time>, the start time process inspect prints, such as 2026-10-05T14:03:22.1234567Z (UTC), and refuses to act if the process with that ID started at another time.
Elevation
desktop launch --elevated starts a program with administrator rights, and is refused when the desktop user is a standard user. In that case, start the program normally and answer its elevation prompt with desktop uac.
Windows of elevated programs. The part of East India that acts on the desktop runs as the desktop user without administrator rights. Windows does not let such a program press buttons, type or click in the windows of a program that runs with administrator rights, whether it was started with desktop launch --elevated or through an elevation prompt. So invoke, click, set-value, keys, pointer, inspect, and wait for an element inside such a window fail at once with ACTION_UNSUPPORTED, and the message says that the process runs elevated. What still works on those windows: desktop screenshot, desktop windows, process inspect, and desktop wait for the window itself, such as "Window[Completely uninstall]". To get past a dialog of an elevated program:
- Start the program without
--elevatedwhen the test allows it; its dialogs can then be driven as usual. - Use the program's own options for running without a dialog, such as a silent, headless or force switch.
- End the program with
endia process stoponce its work is done.
A machine whose image was built over a root image prepared before East India 0.16.122 reports the same cause with older, less clear messages, such as Process <pid> shows no window or Access is denied.
Screenshots
shot=$(mktemp --suffix .png)
endia desktop screenshot orders-dev-dev-1 > "$shot"
endia desktop screenshot --pid "$pid" orders-dev-dev-1 > "$(mktemp --suffix .png)"
endia writes the screenshot as a PNG to its standard output, on the computer where it runs, so you decide where the image goes: a temporary file, as above, or a program that reads the image directly. Nothing is saved on the PC or in the machine. -o <file> writes the image to that file instead and prints the file's full path. Without --pid the command captures the whole desktop; with --pid it captures that process's largest visible window.