Type, Click and Use Menus In Shortcuts

Shortcuts doesn't seem to allow you to select menu items, simulate key presses or click buttons. But you can do those things with some scripting and add them to your Shortcuts that way. It takes some skill and patience to get what you want.

Video Transcript: Hi, this is Gary with MacMost.com. Let me show you how to choose menu items, trigger key presses, and even click elements using the shortcuts app. 
MacMost is brought to you thanks to a great group of more than 1000 supporters. Go to MacMost.com/patreon. There you can read more about the Patreon Campaign. Join us and get exclusive content and course discounts.
Now the Shortcuts App on your Mac is great for automating things. But a lot of times people want to do some very basic things. Like choosing a specific Menu item in an app or triggering a keypress or even clicking a button or some other element that they can see on the screen. Shortcuts doesn't have a straight forward way to do this. But you can use Scripting in Shortcuts and scripting can actually do all three of these things. But I should warn you that this doesn't work as smoothly as you would probably like. Sometimes it is difficult to figure out the right sequence of commands and in some apps they just don't seem to work at all. This also takes some amount of programming ability and a lot of patience so this is a little more advanced than most of the tutorials I do on Shortcuts.
So let's start with a simple task. Suppose I want to open TextEdit and start a new document. To do this without Shortcuts normally I would launch TextEdit and then I would choose File, New. Now it is easy enough in Shortcuts to open an app. But how do you choose File, New? Let's create a new shortcut here. I could use the Open App command here to open TextEdit. But I'm actually going to do that in the script because it is just one more line. So instead I'm going to Search for JavaScript and I'm going to choose Run JavaScript for Mac Automation. JavaScript and AppleScript can both do the same things but I prefer JavaScript. It's a much more modern solution that makes a lot more sense. 
Now here's the script that will actually get this done. If you look at the first line here it just takes the application TextEdit and activates it. So it will launch it, if it is not already launched, otherwise it will bring it to the front. The second line will set a variable called App to Application System Events.processes. Then in square brackets TextEdit. So it will find the TextEdit App that is running and get its process. TextEdit is an app that sits as a file on your Applications folder. But the process is the actual running app and this is what we need to issue commands to it. Now we want to use app here and then look at the Menu Bars. There is only going to be one so we'll get it and of course raise our zero base so zero is the first menubar. Then we're going to look at the menus in the menu bar. Then we're going to get them by name looking for the menu called File. Then we're going to look for the menu item called New. Then we're going to issue the command Click to it. So this should be the same as choosing File, New. Let's run this and see. There is launched TextEdit and we have a New Document. 
Now what if you want to go deeper in a menu. For instance under Edit there's Insert and then Line Break. Let's say we want to have a shortcut that as some part of what it does it's going to use Edit, Insert Line Break. Well, we need to dig deeper into the Insert Menu. Actually there are many steps here because there is the Edit Menu and then there is the Insert Menu Item here and then the Insert Menu here and then the Menu Item line break. So here's the code that will do that. 
The first two lines are the same as before. We're going to start the third line off by looking at the only Menu Bar in the menus there and then looking for the Edit Menu. Then we're going to look at the Menu Items for Insert. The Insert Menu Item has a menu under it also called Insert, as I showed before. Then we'll look for the Menu Item line break and click that. So now when I run this you can see it goes to TextEdit. I had this document scroll open and you can see how it inserted a line break there. That's how you access submenus.
So next let's take a look at Key Presses. Key Presses are a little easier once you realize that they're system-wide. So you're not going to tell an app that there is a key being pressed. You just tell the System. The System is going to apply that Key to whatever window is upfront. Just like if you were to type on your keyboard. So there is really only one line of scripting you need for that. Here's that line. All you need is Application System Event, which is the System, and then you're going to issue the command Keystroke. Then whatever key is pressed. In this case the letter A. So when I run this you'll actually see it types the letter A in the current app, which is in this case Shortcuts. Now what happens if you want a modifier key for that? Well, then you would simply add this, a second parameter keystroke, and you put in curly brackets {using and then in square brackets [a list of modifiers]. In this case the modifier we want is Command down. Then we close the square and and curly brackets there. So, we could add a second item here that is an Option Down as well. Now this will issue a Command A which should select All. So when I run this you could see how it selected all the of text there. If TextEdit had been the app that was running it would have selected all the text there. I can actually add this Textedit activate line right here and now when I run this it will actually launch TextEdit and then it uses Command A to select All.
So now what you have it two ways to actually issue commands. One is to choose from a menu or submenu. The other is to issue keyboard commands and you can use these with modifier keys. In general I find this more reliable to use the keyboard shortcuts for things. So in the previous example instead of choose File, New from the menu I could have simply issued a Command N keypress. Both would have done the same thing. But over different apps and different situations I find Command N is a little more reliable than File, New. 
Now if you ever need to use a keypress that is not a typeable character like say an arrow key then you can use Key Code instead of key stroke. Notice the C is capitalized in key Code. Then you give it a number. Then you could lookup Mac Key Codes online and find a variety of different places that list them. The left arrow key, for instance, is Key Code 123. So if I were to run this you could see how it acted as if I had pressed the left arrow key. I can run it again and you could see how the left arrow key, again, has been pressed. 
So the next thing is actually clicking on interface items. Things like buttons, for instance. This turns out to be much more difficult to do. Mostly because you need to specify exactly what thing is clicked and figuring that out is very difficult. The apps don't just reveal that. You have to test and work to actually figure out a reference for a specific item. For instance, let's go to System Settings. In System Settings when you launch it you are automatically on Appearance. Let's say we wanted to have a shortcut that took us to General instead of Appearance. Well one way to do that would be to click General, just like I did. So how can we get Shortcuts to click General. So here's the code that will do that. 
First we're going to bring System Settings to the front or launch it. Then we're going to get the Process System Settings so we can use it to trigger an action. So here it is down here and then we're going to look for the first window, in the first group, in the first splitter group and then the first group inside that, the first scroll area, then something called an Outline, and then a row, and then we're not going to click it we're going to Select it. Figuring this all out is incredibly difficult. Obviously this element is deep down nested inside of other elements in the user interface. Now let's try running this. You can see it will work. It will bring System Settings to the front and select General. If I want to select Screen Time, I go to Row 10. There's kind of this divider here so I'm going to have to subtract 2 and now when I run it it will go to Screen Time. But I run into a problem if I Quit System Settings and then I try this. It's going to hit me with an Error there because what is happening is that it launches System Settings and then plows ahead to try to do this before the System Settings App is really launched. So if I use Open App instead, put it here, and then choose System Settings and then maybe a wait right here, then I can have System Settings quit. It will Open it up first, wait that one second, then run that JavaScript. You can see it works fine. 
Now how do you figure out that General is buried deep down inside of all those groups and splitter groups, and outlines, and rows, and all of that. Well, one way is to use a script like this. This I'm running in Script Editor because I can just easily see the output.This will basically get an app that is running and go through all of the different objects and all the different windows. So I want to make sure System Settings is running here so then at the bottom here I want to switch to the List View here and go to Messages so I can see the messages that are being outputted by Consult.log. When I run this you can see it is going to run through all of the windows of System Settings and show me all of the different elements and eventually I'll figure out where the elements are and be able to kind of trace it to the back. Again, this is pretty advanced technique. If you're experienced with programming you can probably figure out this and then use that to recreate this route to finding that exact item. 
Here's another example of how that can be used. This will activate TextEdit. It is going to assume that it is already running so it's not going to wait or anything for it to launch. Then it is going to issue the keyboard shortcut Command P which will print. Then it is going to go and look at TextEdit. It's going to look at Buttons and it is going to click this button. Which button is that? Well, it turns out that that's the PDF button. So when I run this you can see it go to TextEdit and then bring up the Print dialogue. Now it is saving as a PDF. So it works its way to bring up the Print dialogue and click this PDF button. Something I know some people have asked for. So then the only trick there is figuring out that it is the first window, first sheet, first splitter group, second group and then first button in that group. That is what is needed to be clicked. It is definitely very difficult to figure that out but it is possible if you have some skill and your persistence. 
Now if you don't have the skills to be able to do this or you just can't get what you want to do to work there is an alternative and that's the highly respected Keyboard Maestro App. I have no affiliation with them. I just love their app and use it for a variety of different things that makes it really easy to find and click buttons and do other automations on your Mac. It's what I usually recommend to people who can't figure out how to do all of the advanced coding that's needed in JavaScript to get things done. I hope you found this useful. Thanks for watching.

Click a menu item

Application("TextEdit").activate();
var app = Application("System Events").processes["TextEdit"];
app.menuBars[0].menus.byName("File").menuItems[ "New" ].click();

Click a menu item in a submenu

Application("TextEdit").activate();
var app = Application("System Events").processes["TextEdit"];
app.menuBars[0].menus.byName("Edit").menuItems[ "Insert" ].menus[ "Insert" ].menuItems[ "Line Break" ].click();

Press a key

Application("System Events").keystroke('a')

Press a key with a modifier

Application("System Events").keystroke('a', { using: ['command down'] })

Press the left arrow key:

Application("System Events").keyCode(123)
// 124=right, 125=down, 126=up, 36=Return

Click General in System Settings

Application("System Settings").activate()
var app = Application("System Events").processes["System Settings"];
app.windows[0].groups[0].splitterGroups[0].groups[0].scrollAreas[0].outlines[0].rows[12].select()
// Note: General may not be the 12th item for you,
// you'll need to experiment

Print and then Click PDF in TextEdit:

Application("TextEdit").activate()
Application("System Events").keystroke('p', { using: ['command down'] })
Application("System Events").processes["TextEdit"].windows[0].sheets[0].splitterGroups[0].groups[1].buttons[0].click()
// Note: The PDF button may not be at this location
// depending on many factors

Script Editor JavaScript to help find user interface elements:

// Note this is JavaScript so make sure your
// Script Editor window is set to JavaScript
// NOT AppleScript

listObjects(Application("System Events").processes["System Settings"].windows(), "" )

function listObjects(uiObjects, indentStr) {
    for (var i in uiObjects) {
		if(uiObjects[i] instanceof Array) {
			listObjects(uiObjects[i], indentStr + '  ')
		} else if(uiObjects[i] instanceof Object){
			console.log('' + indentStr + i + ' Name:' + uiObjects[i].name() + ', Class:' + uiObjects[i].class() + ', Desc:' + uiObjects[i].description() + ', Value:' + uiObjects[i].value());
			listObjects(uiObjects[i].uiElements(), indentStr + '  ')
		}
	}
}

Comments: 2 Responses to “Type, Click and Use Menus In Shortcuts”

Ken Nellis

1 year ago

Very clever, Gary. Thank you! I guess we should presume that those indices (e.g. 12 for the General System Settings selection) might change in future macOS updates.

Gary Rosenzweig

Ken: Not just in future updates, but from Mac to Mac. You will have different items in a MacBook vs a Mac mini, for instance. You may see different things depending on your settings and what you have installed too.

Comments Closed.

Type, Click and Use Menus In Shortcuts

Comments: 2 Responses to “Type, Click and Use Menus In Shortcuts”

Welcome to MacMost

Free Weekly Newsletter

MacMost Online Courses

Keyboard Shortcuts PDF

Connect with MacMost

MacMost Sections

Popular Tutorials

Information