Since I said the mouse needed to be seriously re-examined as the primary device for interacting with the user-interface (see my previous entry), it’s only fair that I give an example of a better way to do it. In this entry I explore one possible way to minimally change the interface to almost remove the mouse entirely, without increasing the difficulty of learning how to use software.
(Note: Click on Images for a full-size view.)
(Here we have an unaltered screenshot of the experimental subject.)
First step: rip out essentially all of the traditional controls. That means drop-downs, buttons, and menus. Notable exceptions include the scroll bars and status bar (both of which provide excellent and frequently needed feedback like what the open file is, and where in the document the user is). Also, I’m going to take some liberties with the status bar to pull out some of the more cryptic (and rarely referenced) information in favor of somewhat more relevant data.
(The closest thing to a decapitation of an application you’ll see.)
Second step: sit a user down (possibly with a close supply of anti-anxiety medication for those less comfortable with change), and tell them that if they want to “Control” the application, they need to press the “Control” key (great name for that key, huh?). When they do, overlay the application window with something like the following:
(Okay, so I’m not a graphic designer, but I bet there are a few around who could pretty this up.)
Notes on the sketch: (1) Yes, this is a lot few functions than OpenOffice writer has. I’m just trying to present proof that all the icons and the most used part of the menu can readily be represented this way. Comprehensive feature lists are better represented by my menu-replacement sketch below. However, the idea is that that should be rarely needed. If it’s used with any frequency, the application designer anticipated the user needs poorly. (2) I know some of the key-bindings are less than intuitive. I blame the 3am restarting of the whole design thanks to a bug that trashed my last design (followed by the same bug killing it a second time at 6am).
Now, there are some subtleties to the design. First, there could be two ways to access the dialog—tapping control, alt, or whatever could toggle the reference screen on until the modifier is tapped again, or, if the user holds down one of those modifiers, the reference screen disappears as soon as it’s released. This makes the use of the control key much more accessible for those of us who haven’t moved it from it’s instant-carpel-tunnel-inducing location at the very edge of what an average-sized hand can reach.
Next, commands can be put in bold if they’ve been used recently. (The definition of “recently” was the subject of extensive debate when I was working with highlighting recently changed items in my last project. I’ll leave “recent” undefined for lack of true resolution of that question for me.) Microsoft’s “adaptive” menu system (also known as “Help! Where did half my menu go?”) tries to address the same problem of adapting to user’s usage patterns. This, however, is a much better way to speed finding of common commands. It doesn’t shuffle items around or hide them, (both of which confound the user’s ability to memorize the interface and wreaks havoc on users trying to use someone else’s copy of a program).
Now, imagine the user’s thought sequence as they try to enter a command. “Hm. I need to save. Hit ‘Control,’ save… ah, ’s.’” Imagine that a few dozen times, and it starts to sound a lot like studying flashcards. For free, just by using the interface! Within weeks (assuming fairly sporadic usage), a user has memorized the shortcuts to all their common commands, obviating even looking as they execute them. Daily users could be fully proficient in even uncommonly-used combinations within days, with the pop-up screen little more than a flicker (at which point the user could set a delay for its appearance or turn it off entirely to keep it from getting in the way).
Of course, there are some potential difficulties with the design. The most glaring is the amount of seeming screen clutter when the user first sees the reference screen. However, the current system requires the person to memorize that the picture of a floppy disk means “Save the file I’m working on, without changing the name” versus the floppy disk with red lines on it, which means “Save the file I’m working on with a new name,” or look at long lists of words which (sometimes) lack any visual cues at all (in the menu system). Careful layout can also minimize this problem. There’s also the relatively computationally expensive nature of the reference screen, being a translucent (possibly dynamically-generated) pop-up. Thankfully all current versions of windowing systems (X Windows (for Linux/BSD/etc.), Windows, and MacOS X) support hardware accelerated eye candy like translucence. It’s not even essential to have it look so fancy, and caching makes generation nearly free.
Now, software like OpenOffice.org tends to have features packed in by the pound, so my button-replacement fairs poorly with those. It would be entirely reasonable to say that menus can stay. They do take minimal screen estate, and can be used with a keyboard.
I’m not a fan of half-measures, though. The menu is hard enough to navigate with a keyboard to encourage the user to reach for the mouse I’m trying to obviate. Consequently, it could be replaced by the following design (again, click for larger versions):
Now, the user accesses the “Menu System” (which, curiously now looks more like a menu than before) by tapping the “Alt” key. The above screen appears over the application. The user then types a few letter from the command they’re looking for (the letters can be from anywhere in the command, and several groups of letters can be searched for by separating them by spaces). Once the user starts typing, entries not matching the string(s) are greyed out, as are columns without any matches. After typing the three letters “sav” the screen looks like the next mock-up (note the underlining of the matching part of the name):
Once the list has been trimmed down enough, the user selects the item they want by hitting tab (or moving with the arrow keys) until the one they want is selected (selection not shown here) and hitting enter. If there is only one match, it is automatically selected so the user need only hit enter once the list has been fully reduced.
This system again draws on the same strengths as the pop-up control window (which I won’t enumerate again), but allows vastly denser commands.
There are several shortcomings of these last two mock-ups (which is fine, they’re not final designs, just proof-of-concepts). First, the OpenOffice.org’s menu system has been copied here, but I excluded the sub-menus for time considerations. Second-layer menus could easily be implemented by restarting the process with the sub-menu’s options once the name of a sub-menu is selected. Also, this design excludes the helper icons that OpenOffice.org has (again, time considerations—cropping, cleaning and placing scores of icons is just too slow work), as well as check marks beside the entries that are toggles. The latter problem is because I didn’t leave myself enough room for those, and they aren’t important enough to the idea of the system to warrant a complete re-layout.
Final thoughts: I hope this design has given you an idea of how easy a re-design of the GUI concept would be, and how powerful common input devices like the keyboard can be when used correctly. HCI/User Experience designers, don’t just live with the status quo.
Update: Jeff Atwood wrote about Vista’s new, more keyboard-friendly, start menu, which it seems, is only a few tweaks away from being comparable to my menu replacement design. Unfortunately, I, being an exclusive GNU/Linux user, I don’t see much of the recent Redmond or Cupertino products.
monopole | 18-Jun-07 at 6:52 am | Permalink
Back in the old days we had a single unified discrete interface device, the Hollerith card.
Given that I go that far back here’s my take on your scheme.
Back before the mouse was prevalent we got by with pure keyboard commands. Hell, I go back to TECO and TECO VT. As such I fully appreciate the advantages of memorizing control sequences. But nowadays, I use the mouse (actually my trackball) heavily.
Why? Mode flipping. If you use one program and only one, EMACS for example, keystrokes are fantastic. But if you have to switch between programs on a regular basis, all that muscle memory works against you. For example over the period of one hour I’ll have had to switch between Word, Firefox, Eclipse, GIMP and DIA. All of which have radically different keyboard controls. It’s like switching between Python and Cg, all reflexive syntax practices go to hell in short order.
Secondly, getting your hands off the keyboard is a goodthing for avoiding carpal tunnel syndrome.
Simon Arthur | 18-Jun-07 at 7:46 am | Permalink
It’s on the right track, but you’re presenting too many choices to the user at once. It’s easy to become baffled by a giant screen full of options.
Chaim Krause | 18-Jun-07 at 10:59 am | Permalink
I like your proposal.
I am thinking that the submenu items could be displayed as the user types, displacing main menu items that do not mean the search criteria. I also like to see incremental searching of all menu items. (Have a look at Jeff Atwood’s post on Incremental Feature Search in Applications.)
I think I will try out these paradigms in one of my own applications.
Clay Barnes | 18-Jun-07 at 12:23 pm | Permalink
monopole:, good point on the mode flipping, but consider this: pretty much all applications use a picture of a floppy to represent saving—couldn’t we use essentially all the same shortcuts for the same actions in all applications, to a point? At the least the common actions should be standardized, and those that differer between programs wouldn’t be a problem for having different controls because we can easily think “bold = Ctrl-B” and “brush = Ctrl-B” without confusing the two, as Bold is generally a text program’s operation and Brush is a graphical program’s operation. I definitely agree that this is potentially a serious issue—which is why, even in icon placement, I’ve long loudly advocated more standardization of interfaces, even tradional ones. (I, too, hit “Esc :wq” or start twiddling hjkl at least 10% of the time I’m in a text box!)
As for carpal tunnel, the mouse can also induce carpal tunnel, so you should be depening on good exercises to prevent it, rather than device switches.
simon arthur: There may be a lot of options, but keep in mind that not all density is bad. Yes, this may be overkill, but it’s really hard to tell how much content it takes to truly disorient a user without actual testing (which I haven’t the money to fund for just fun).
Chaim Krause: I would be ecstatic if someone had something like the implemented to see how well it actually worked! Please keep me posted on anything you find about its actual implmentation. I’ll have to think of a name for the design so people can refer to it without describing it every time. I suppose “Clay’s Great UI Design” is rather narcissistic. :-D
Simon Arthur | 18-Jun-07 at 7:30 pm | Permalink
Forgot something: have you seen how the control key works in Konqueror? Hit it, and a tooltip appears next to every link and form element with a letter you can use for quick access.
Clay Barnes | 18-Jun-07 at 10:02 pm | Permalink
Yes, actually. Perhaps that was subconsious inspriation!
fifilip | 19-Jun-07 at 5:04 am | Permalink
a good start is a window manager. have a look at Ion (http://modeemi.fi/~tuomov/ion) which was developed from the same idea that using the mouse in many cases is suboptimal.
it is light, fast, easy and highly configurable (using lua). i’m using this on my laptop and several workstations for several years. (note: i’m not affiliated, just an enthousiastic user)
Chaim Krause | 19-Jun-07 at 7:18 am | Permalink
monopole,
I understand your point, but believe much of that can be overcome by allowing users to customize the hotkeys/shortcuts/command keys associated with various operations in an application. One of the most common examples of this is customization of text editors. For example, the first thing most do after installing an IDEs is to choose the default keyboard shortcuts from emacs, Brief, WordPerfect, etc. Most feature ripe applications allow this.
Clay Barnes | 19-Jun-07 at 1:08 pm | Permalink
fifilip: I’ve actually long been studying the window manager aspect, and I have one design I really like that lies between the strict tiling Ion uses and the flexibility that floating windows offers. My layout (actually shown in an earlier entry here) is based on primarily moving by “edge” so applications pack well rather than being inched along with greater precision that users really want (despite that they think they do). I’ll actually go into that design in a later entry (possibly even my next one), and I’ll be examining the benefits/difficulties of my openbox3 layout vs. Ion vs. wmii/dwm, etc., and true floating window managers.
Nathan Bell | 20-Jun-07 at 1:35 pm | Permalink
Your proposal also reminds me of an article I read a few days ago by Jeff Atwood. He describes the so-far brief history of ‘incremental search’ (similiar to your menu system) at Microsoft. http://www.codinghorror.com/blog/archives/000887.html
Jeff also links to an article about the efforts of a Microsoft employee to NOT kill incremental search in MS Office (he suggests distributing it as an add-on, but it looks like internal politics have killed it). http://www.istartedsomething.com/20070203/saving-scout/
Zach Nation | 04-Jul-07 at 8:48 pm | Permalink
On the topic of incremental search, I should point out that you can do that in Mac OS X (since whichever version introduced Spotlight) from the keyboard by pressing Command-Spacebar. An even better third-party version called Quicksilver introduces a few more features, but incremental search has been around for quite a long time on Mac OS X and it’s the primary way I launch applications. The first three letters of any app name followed by Enter will usually do it… and if I’m looking for a document or something with a conflicting name, I have a long list of things to browse through.
Ok, looking back at that comment, I realize I sound like an Apple fanboy. I swear I’m not…
The New Interface Advocate » Blog Archive » The misused mouse, part 1: The story of the mouse's decline | 08-Jul-07 at 6:21 pm | Permalink
[...] he’s in the mood for. « Throw out that mouse—you upgraded to a keyboard! The misused mouse, part 2: A proposal for a nearly mouseless interface. [...]
The New Interface Advocate » Blog Archive » The only two interface designs ever conceived: | 08-Jul-07 at 8:55 pm | Permalink
[...] The New Interface Advocate Clay Barnes talks about HCI, the user experience, and whatever he’s in the mood for. « The misused mouse, part 2: A proposal for a nearly mouseless interface. [...]
Evan Erwin | 09-Jul-07 at 8:37 am | Permalink
Have you ever tried mouse gestures? It sounds like what you’re looking for, without having to “devolve” into keyboard only.
I’ve been using mouse gestures for years now. Love love love em. Can’t browse without em anymore, and it SIGNIFICANTLY increases my browsing efficiency.
Mouse gestures in regular programs would be amazing.
Matthew | 09-Jul-07 at 8:47 am | Permalink
This sounds a lot like enso: http://www.humanized.com/ it uses the caps key to get a command prompt
nick danger | 09-Jul-07 at 8:49 am | Permalink
Where is your replacement for tooltips? That’s how most novices learn what stuff is.
s.mitchell | 09-Jul-07 at 8:53 am | Permalink
Keyboard is definitely better for productivity–takes longer to learn, but worth it. If we could just get many apps to agree on a standard basic setup…
I think that if the same effort at making a mouse interface what it is today, were spent on making keyboard interfaces better, we would be a lot further along on this.
Yes I do Cad, and need a graphical pointing device for that, but Cad suffers when a generic mouse is used for a pointer. A 4-7 button puck on a small tablet was much more productive for me than a mouse for Cad. My point is that since the mouse IS, cad suffers. Were a keyboard interface used for much of computing, Hardware for Cad would have developed differently, and better for use in Cad programs.
Dennis Bathory-Kitsz | 09-Jul-07 at 8:54 am | Permalink
Two points…
My wife is a confirmed mouse user. No matter how I’ve presented even the easy Windows keyboard controls (such as CTL-A, -C, -V, -P and -Z in particular), she will always use the mouse. She likes how it feels and how it makes her more comfortable — keyboards are for computer people and typists, and she refuses to be either. Indeed, those are the two most common reasons I’ve come across when I ask friends why they use a mouse for tasks easier with the keyboard.
The second point is that an application changing a common key sequence can totally trash those of us who avoid the mouse. Case in point: In Audition 2, Adobe changed “Save As…” from ALT-F-A to ALT-F-E, and now ALT-F-A is “Save All”. How many projects have I wrecked midway using the unconscious two-decades-learned CTL-F-A with Audition! It’s so bad that I won’t use the program except for recording (version 1 doesn’t support my hardware interface properly).
Dennis
Charlie Hubbard | 09-Jul-07 at 9:07 am | Permalink
First revision looks like heavily borrowed from emacs. I like it, but I feel like this is going floating back and forth between Emacs and your typical windowed UI toolkits.
What about using OO paradigms whereby the menus are filtered based on what the selected object will support. So the popup menu is context aware. Kinda like my IDE when I have a foo object I type foo. and I see all the methods I can call on the type of instance foo is. Emacs was more a functional design paradigm whereby I just needed to find the function that I wanted to run which was divorced from the data it was used against. Making it hard to find the function sometimes.
It’s still very cool to rethink all of this.
cruise | 09-Jul-07 at 9:08 am | Permalink
I recently discovered a wonderful program called Find and Run Robot for Windows (which I still keep around for gaming):
http://www.donationcoder.com/Software/Mouser/findrun/
which has completely replaced my start menu and desktop clutter for launching programs.
The “run” command in the latest CVS versions of Enlightenment 0.17 has similar behaviour, though without re-ordering commands based on recent use.
davepermen | 09-Jul-07 at 9:09 am | Permalink
Office 2007 has a similar feature. While holding Alt, you see a key-tag on every choosable item in the (very nice) ribbon bar (and everywhere else where you can access something). you can navigate that way through the whole system.
I’m against forcing a user to it, but I’m motivating anyone who helps people see the choise.
a good read, and good ideas.
Richard Neill | 09-Jul-07 at 9:09 am | Permalink
This is one place where the IBM trackpoints are so good - I can use the mouse (3 buttons AND horiz/vert scroll) without moving my hands much from typing position. I still mainly use kbd shortcuts, but it’s much easier not to have to move my right arm around so much. (I use an ultranav keyboard on my desktop machine too - they are really very good things).
* Of course, it’s useful to have a “real” mouse as well, for things like PCB layout.
* Many people dislike trackpoints. That’s sad, because they simply don’t know how to use them. The common faults are:
- using a trackpoint not made by IBM (poor build quality, eg HP)
- not having the gain setting high enough (need lots of force to move it)
- not knowing that, after a few mins at high gain,. the TP may start to drift. SImply lift your finger off it for a few secs and it will re-calibrate.
Ed Singleton | 09-Jul-07 at 9:25 am | Permalink
Personally I think a quick step in the right direction would be if every time you clicked on something with the mouse, a tool tip told you what the keyboard shortcut would have been for that action.
That would be enough to get people realising the can use the keyboard.
Nothing will work completely though until you can actually do everything with a keyboard. (Try unplugging your mouse for a day and you’ll see how some things are basically impossible).
Jesse | 09-Jul-07 at 9:30 am | Permalink
Very good proposal.. However, it is already being integrated in Mac OS systemwide.. Now with Tiger you can have something like Spotlight or Google Desktop or Quicksilver (really powerful) to start and manipulate.. With Leopard, the help menu will be extended with a search box.. When you then type something like ’save’ it will actually highlight the ’save’ option in the file menu dropdown.. If you type something which cannot be found in the menu’s it will do a search on the words in the help file of a program.. This will work with every program.. I think that’s very smart cause it will not bother programmers with additional work and it will be available systemwide so no differentiating layouts will appear as well..
However, your idea about the overlays when you press a modifier key, would be an amazing addition to that as well..
ORB | 09-Jul-07 at 9:30 am | Permalink
Well actually software developers do is provide both options, whether the application is suited for keyboard or mouse.
Your CTRL option is entirely similar to pressing ALT and dropping the menus and choosing the options. Also showing all the options at once can really scare anyone, so the hierarchical menu design is really good (No wonder lot of design research has been done on all this)
Also they provide buttons for commonly used options. Thats good for mouse.
So you see, no one forced you to use mouse. Both options are available and people get used to it in accord.
This comes at a cost. Occupancy of the screen. But think about it, whats better? Getting a new bigger display (when displays are getting cheaper and cheaper)? or limiting users to just one input device?
For Users Only » Blog Archive » Misuse of the mouse | 09-Jul-07 at 9:31 am | Permalink
[...] A recently launched blog The New Interface Advocate has posted 2 interesting articles about the misuse of the mouse: The misused mouse, part 1: The story of the mouse’s decline and The misused mouse, part 2: A proposal for a nearly mouseless interface. [...]
Anon | 09-Jul-07 at 9:34 am | Permalink
Your control key overlay proposal is very very similar in functionality to how old genuine WordStar worked in its day. Even including your suggestion for a “delay” before the overlay pops up.
Old WordStar had what they termed “on screen menus” and the user could pick to have them up all the time, or to have them appear with the first character of a key-command was struck. If they were set for “appear upon keyhit” then hitting control+k would pop up the menu for all the commands accessible through control+k. Same for control+Q and Control+p and so forth. There was even a delay setting for the menu popup so that a more knowledgable user could set the menu delay to a second or two, giving him time to keyboard memorized commands e.g., Control+K control+B to begin a block mark, without having to see the menu pop-up, but for less used commands, hitting control+K and lingering a second or two would cause the menu to appear and then you could pick the appropriate command out of the menu.
Additionally, your idea for incremental search in pull down menus is very similar to what Infocom (think Zork) implimented in their dos database program Cornerstone. The “menu” was command line accessable (although arrow keys worked for “positional navagation” and a user selected a command by typing the first few letters of the command word. As each letter was typed, the list of available commands would shrink to only those words that began with that set of letters. Once a single world was selected, hitting tab or return activated that entry (sometimes then bringing up a sub-menu to select from).
I’ve always thought both systems were in many ways superior to the current mouse heavy interfaces. More learning curve, yes, but by far faster to utilize once you overcame the learning curve.
The trouble with current mouse heavy GUI’s is that they are almost all directed at the total novice user, with absolutely no thought into providing easy ability for the power user to make use of them, and with no easy way to “train” the novice to become a power user by use of the program alone. With most, it takes someone who is already “power user” to become a “power user”. The novice simply and always remains a total novice.
mouse lover | 09-Jul-07 at 9:36 am | Permalink
This interface is VERY dependent on people remembering commands. I am a graphic designer by trade so I spend a lot of time in front of the computer. But even so I don’t remember most keyboard shortcuts. I suppose I could take the time to train myself… but why? Everything I need to do I can do quickly and easily with my mouse. I suppose I loose a few milliseconds moving my mouse up to a drop down menu but I don’t notice it. It seems very easy and natural. I don’t know what people have against mouse interaction anyway. There seems to be a whole cult of the keyboard. If i had to make a choice between the two I think I’d choose the mouse - it’s simple and direct.
Achilleas Margaritis | 09-Jul-07 at 9:40 am | Permalink
Here is another solution, potentially better: remove all choices and buttons from the screen, and make everything accessible from the context menu of the clicked object. For example:
save document: right-click on document, select save.
save document as: right-click on document, select save as.
open document: right-click on empty space, select open.
new document: right-click on empty space, select new.
apply new style: select text, right-click on selection, select style.
apply new color: select text, right-click on selection, select color.
The context menu need not be hierrarchical, but a pie menu, where options appear all around the mouse position that was clicked, thus minimizing mouse distance travel.
In order to avoid not hitting the menu, make the mouse locked inside the menu area, until clicked. If the user wants to cancel the menu, let him/her click with the right button again, or press esc, or hit cancel/close.
innovati | 09-Jul-07 at 9:46 am | Permalink
Hey, nice blog, I solved this mouseless problem by using modifier keys and on-screen menus almost a year ago and sent those to Gael Duval, of Ulteo Linux where we discussed the probabliity of it being in a future release, so be careful and be careful those reading this because there is at least prior art here, if not somewhere else as well.
(I’m also the one who influenced the SymphonyOS UI to have a more edge/corner design instead of a taskbar like it’s original mockups)
Feel free to e-mail me at innovati@gmail.com if you are interested in my mockups or discussing some problems I see with the proposed interface.
I find it rather refreshing that the program I used to first demonstrate my interaction idea was *also* openoffice.org and all I was left with was a blank page, true WYSIWYG editing.
Karl G | 09-Jul-07 at 9:48 am | Permalink
This looks basically like what the guys at Humanized are trying to do. I attribute the increasing interest in transient CLI interfaces to Quicksilver. There were app launchers that accomplished the same thing before, but Quicksilver is the first to my knowledge to add additional verbs to the interface beyond ‘launch application’.
The only application I know of that makes pervasive use of the transient CLI style is Google’s Reader app. Mihai Parparita is responsible and it’s an extended derviative of his work on the ‘gmail macros’ greasemonkey hack for gmail. I believe that this style of interface has potential to gain widespread acceptance on the web over the more traditional WIMP controls since web HCI conventions are not yet firmly established. I encourage you to focus your efforts on web apps rather than desktop apps because you’re more likely to make an impact.
I’m one of the very few remaining wmi users. Anselm’s efforts on wmii are a step backwards from a usability perspective. The static nature of WMI allows you to keep your environment fixed and develop a tacit knowledge of where windows are. The dynamic rearranging of windows in early versions of wmii (I haven’t looked recently) and other tiled WMs I’ve tried confounds the development of app location muscle memory.
HikingStick | 09-Jul-07 at 9:50 am | Permalink
For navigating multiple menus from the same screen (while I do like your type-ahead idea), why not use a key to delimit between top-level menus and their submenus? Such a scheme would allow your type-ahead search to work while also allowing easy keyboard navigation to specific menu items. ‘Alt’ would bring up the main menu, ‘I’ could be interpreted as the start of a type-ahead search, but we next type ‘,’ (comma) to instruct the GUI that we are moving to a submenu (in this case, the submenu of the Insert menu). Then, for Indexes and Tables, the user types a seond ‘I’. In all, the sequence would be (I’m avoiding angle brackets so as not to invoke HTML tags):
‘Alt’ ‘I’ ‘,’ ‘I’
In my experience, until people try keyboard shortcuts and realize how effecient they can be, they simply go on mousing. Once introduced to a good shortcut, though, they typically change their habits and use it going forward. For the most part, I’m a keyboarder, unless I’,m doing recreational browsing. I’m also a laptop user, so I’ve enjoyed models that utilize the pointing stick at the center of the keyboard. When I need the mouse, it’s there, without having to move my hands from their primary locations.
Miah | 09-Jul-07 at 9:50 am | Permalink
So what you are saying is that you want a word processor that works like WordPerfect 5.1 did in the old DOS days? I’ll agree with you that applications need a re-look concerning the mouse (and people who are mouse dependant) but I’d start looking at good applications first for ideas instead of stinky old word processors. Try Adobe Illustrator (I’m thinking of version 8, but the new ones have the same keys) for a quick view of an application that puts all of it’s commands right at your fingertips. The left hand stays on the keyboard while the right hand stays on the mouse (sorry southpaws!) and you have complete control over the application.
Really the word processor needs to be drug out in the street and beaten with it’s own shoes. But “people are used to it” is used to refute anybody making a suggestion about how it could be an awesome application…
Brian | 09-Jul-07 at 9:51 am | Permalink
This would work best, I think, for a niche application. One that people use day in and day out where the muscle memory works best precisely because they aren’t in other applications often. For example, data entry applications - like medical record or banking software.
Stephen | 09-Jul-07 at 9:52 am | Permalink
I like the idea, but I could argue the mouse is under-utilized. If you program your mouse buttons to trigger the keyboard short cuts, you would fine the mouse can be your friend.
Personally, I use the keyboard short cuts quite a bit in combination with my mouse and I find my time is being spent more effeciently than most. Why create something new, when most of the shortcuts currently in place are not being used?
Alex Andronov | 09-Jul-07 at 9:53 am | Permalink
Just a quick thought. I think when there are fewer than 10 options left on the menu pressing 1 - 9 should select that option (as well as tabbing if you prefer). I think I would remember Sav3 for Save-All rather than SavTabTabEnter and it’s fewer key presses.
Penguin Pete | 09-Jul-07 at 9:55 am | Permalink
For love of Moses! You don’t need to re-design anythng!
Go find almost any Linux desktop and hit Ctrl-Shift-NumLock to toggle the keyboard mouse on and off. You can guide the mouse with the number-pad and use the /*- keys to pick left-middle-right clicks and the 5 to click. Menus can be manipulated with enter, esc, and the arrow cursor keys.
Beyond that, users can learn simple keyboard shortcuts. They exist everywhere. Beyond that they can define their own in any window manager I ever heard of.
Beyond that, they can use the RatPoison window manager: http://www.nongnu.org/ratpoison/
(the ‘rat’ they refer to is the mouse)
Beyond that, they can Ctrl-Alt-F* to the console. Console development has kept pace with the desktop on many fronts - you’d be amazed what you can do there and how much faster you can do it. Oh, and have you heard of Emacs and vi?
Believe it or not, computers existed for quite some time before mice, and we’ve been having this mouse/no-mouse debate since Usenet. You have not only re-invented the wheel here, but proposed a square one with an off-center axle.
Stewart Dean | 09-Jul-07 at 10:05 am | Permalink
First up what you’re proposing is returning to the speed of use that programs like Lotus 123 had but these interfaces become ’skill’ based. If you are going to use something on a regular basis then a non intuitive interface is fine providing it leads to ease and speed of use. The problem comes if you occasionally have to use an application. You schema for those who only use the program sometimes will get in the way rather than help them. There is a common mistake in many user interfaces where the assumption is that the user lives in the application instead of just occasionally visiting, something that many websites also make the mistake of when including ‘my’ or ‘your’ sections.
Christopher | 09-Jul-07 at 10:09 am | Permalink
Zach: No need to worry about sounding like an Apple fan-boy, Linux has had that capability for a year or so thanks to the Beagle project. I’m not too sure on the specifics of the Mac implementation of the idea, but I’m sure they’re pretty similar. It’s very handy! One less thing I need to use the mouse for!
robustyoungsoul | 09-Jul-07 at 10:16 am | Permalink
Brilliant.
WANT
The Othe Joe | 09-Jul-07 at 10:17 am | Permalink
Good ideas but willl still end up duplicating a lot of the same work, for multiple programs. Look at this program: http://www.humanized.com/ it’s a nice start. It ought to be more shell level. Another improvement i would like to see is a new file indexing system. Why in the hell are we still using folders when we could be using search and categorization
Chris Brandon | 09-Jul-07 at 10:54 am | Permalink
Keyboard shortcuts:
For my hands, the left Alt key works much better than the Control key for keyboard shortcuts. It sits just to the left of the spacebar, easily accessible to the left thumb. I still use some Ctrl-key combinations (Wordstar habits are hard to break), but I can move the cursor all over the place with Alt-j, Alt-k, Alt-l, and Alt-; (cursor left, right, up, and down, respectively), and scroll text mouselessly with Alt-u and Alt-m. Most of these assignments can be made within my text editor (Kate) and word processor (Textmaker); others can be implemented using KDE’s “keyboard actions” utility. In part, this works because the thumb is stronger, and has better nervous innervation, than the little finger.
Mouse shortcuts:
Mousing would be much more efficient if window managers did a smarter job of managing windows. One big annoyance is that dialogue boxes often appear in the middle of the screen by default, instead of where the mouse cursor happens to be. As an example: if I click the “close program” box at the top right of a window, the program’s “confirm” box appears in the middle of the screen; on a big monitor, I have to move the mouse a fair distance just to choose “OK”. Why is this? Why can’t we have a context-specific mouse cursor, so that the window manager makes the dialogue box appear close to the mouse cursor?
Hugh Wish | 09-Jul-07 at 10:59 am | Permalink
The first mistake that HCI people make is that there is a “best way”.
The fact is that there are visual people, audile people and kinesthetic. Each must be approached in a different way.
Each perceives of the world in a completely different mode.
Most people gravitate toward more visual interfaces because they are visual people. The kinesthetic mode is the most native / most powerful mode for computers. As soon as the visual bias is gone from interfaces then silly problems like poor mouse interaction will disappear by themselves.
The keyboard based “all at once” style of this interface would suggest that it is an audile approach with perhaps some kinesthetic elements.
Read Marshall McLuhan’s “Understanding Media” if you really want to know the science behind these HCI decisions.
Mumblings of a Web Developer » The keyboard and mouse as input devices | 09-Jul-07 at 11:07 am | Permalink
[...] I just ran across a couple of interesting articles about keyboard vs mouse input: http://hci-matters.com/blog/?p=8 and the followup: http://hci-matters.com/blog/?p=9 [...]
Julián | 09-Jul-07 at 11:13 am | Permalink
i’m starting my studies as a graphic designer here in argentina (UBA, Universidad de Buenos Aires) and i think you pretty much have hit the nail maybe not for everybody (that should be tested) but i think it’s a pretty useful thing, i for one use and abuse the “j” in winamp just to give an example or seraching files in almost any file manger by writing it’s name
it’s just so easy and comfortable….
hope to hear more coming from your way
Julián Lorenzon
David | 09-Jul-07 at 11:18 am | Permalink
What you’re describing in the first 3 screen grabs, is Apple’s “Pages”, with the little “lozenge” button clicked to turn off the toolbars.
Check out an OS X app called KeyCue to do exactly the kind of pop-up you’re showing in the 3rd slide.
Basically, everything you’re talking about, I have now, in almost every application, on OS X.
Ken | 09-Jul-07 at 11:23 am | Permalink
Hm, I’m doubtful. Accessing everything through memorized keyboard shortcuts may work great for you and me, but many people don’t have that same aptitude (lacking in memorization or keyboard facility). Visual UI design provides guidance, exposing the most primary functions so newcomers and non-shortcut-memorizers have an idea of how the interface is intended to function. And for those without great keyboard skills, clicking on a menu item can be much easier than hunting for the left bracket symbol. Your design might work for specialized systems (e.g. call center operators) but it’s not a great fit for the diverse general public.
Seth | 09-Jul-07 at 11:53 am | Permalink
Have you thought about combining this technique with a similar mouse-based version? Autodesk Maya (a 3-d application) uses a “hotbox” control — essentially, holding down the spacebar brings up ALL the menus (or right-mouse button for context-sensitive menus) in a region surrounding your mouse pointer. To select a menu item, you merely swipe the mouse in the direction that leads to the menu you want. Muscle memory comes into play since the various options are ALWAYS in the same place in relation to the mouse pointer, so swiping upwards or to the bottom-right will always execute the same command. As you get used to where menus appear, you can swipe so quickly that selections are made almost instantaneously.
Your control scheme could definitely increase productivity in applications that are keyboard-heavy, like word-processing or spreadsheets. However, for “creative” types like me, having both hands on the keyboard is something to be avoided, since 80 % of my interaction with applications is through a continuous input (mouse / tablet). However, I could see a combination of your design with a mouse hotbox making my job easier. Or at least reducing the learning curve for frequently used options.
Tak | 09-Jul-07 at 11:56 am | Permalink
I appreciate visionaries like yourself, but… It sounds like you’re re-inventing vi and vim. Gvim is also much like your idea.
happy_coder | 09-Jul-07 at 12:05 pm | Permalink
Mr. Barnes, I suspect you are too young to remember the early days of word processing on PCs (or even on mainframes). When I saw your proposals, I immediately thought of past programs. WordPerfect and WordStar used all keyboard commands, with similar menus displayed upon request. Yes, users can learn to memorize sets of commands, but they must first learn the concept of what that command will do. The visual displays shown in Office 2007 in the menu “ribbons” are extremely helpful for learning, as well as for remembering.
Most memory improvement practices involve using visualization of objects to associate with names or words. The human brain is better at remembering images than lists of words. Studies have shown that if you speak a list of 20 words to a group of people, they will remember only about 7 of them by the time you reach the end of the list.
As the number of available software features increases, so does the number of available commands. Everyone wants software to do more and more, so we should not expect any decrease in this. Office 2003 had over 1500 different commands available - thankfully not all of them are displayed in menus!
As a graying programmer/university instructor/PhD student who also goes back to punch card days and who has spent a fair amount of time researching alternative input methods for PDAs, I’ve seen many ideas and theories come and go. Researchers should always be steeped in the history of their field to avoid simply recycling old ideas.
john smith | 09-Jul-07 at 12:28 pm | Permalink
It seems your entire argument is that reaching for the mouse is a waste of effort and pointless movement. This is true.
This is also why millions of us use IBM trackpoint keyboards, which have a mousetip directly in the middle of the GBH keys. You don’t even have to take your fingers off the home position to move the “mouse”
The problem is that IBM holds the patent and charges for its use, AND that it takes at least a week of use to become competent, AND worst of all the rubber tips wear out after 6 months or so and must be replaced, however 99.999% of the people I’ve ever seen that have older notebooks with the trackpoint say they hate it because their finger slides off- ie they have no idea the caps can be replaced at a price of 10 for a few bucks.
I never use a mouse unless I’m playing a 3d shooter. The IBM trackpoint is the PERFECT solution. Better than your convoluted control key popup window effort.
Bill | 09-Jul-07 at 12:46 pm | Permalink
I’m sorry, but I really don’t see anything here that I’ve not seen before. Let’s toss the Dvorak keyboard, in too! .
I remember an article in Byte magazine that came out about the time the Macintosh was first released. This article ‘proved’ that using a mouse was much less efficient than using a keyboard. Sure, a keyboard was more efficient than a mouse *if* you knew the keyboard commands back then, but this was also at a time when the word processor you used came from one company, the spreadsheet from another, and the user interface of both had nothing in common between them. We really haven’t progressed in our understanding of what needs to be done since that article was written, about 23 years ago.
FWIW, I first read about ‘incremental search’ in an Apple user interface manual for the Apple //e. I was developing software for the Apple //e at the time, and the book was an interesting read. Also, one of the Lisa/Mac designers wrote up a proposal somewhat similar in nature to Clay’s proposal in Byte magazine quite a while ago.
Consistency (as others have noted) is key, but it’s difficult to achieve consistency when having to deal with different application types. It’s even more difficult to achieve with a ‘feature-rich’ applications that have commands for many, many options, many, many of which a given person will never use.
There is also another aspect that is worthy of discussion, too. The frequent user versus the infrequent user.
The frequent user of an application will have more of an incentive, and more of an opportunity to learn and practice the keyboard equivalences of a given application than the infrequent user. The infrequent user will forget just about everything they have learned about an application in between uses, and has no motive to learn keyboard sequences beyond the most basic. An explorable interface is much better for them than one that is at least somewhat hidden.
I agree that a new interface paradigm is a worthy thing to do. However, I think we’ve gone down a long way down the wrong path, not only in user interface design and tools, but also in application and OS design. We’re not going to solve this problem by simply tweaking the UI. I believe the problem goes deeper than that, to how we’ve come to design and think about applications.
And like the Dvorak keyboard, as good of an idea it might be, it may be too late to change.
— Bill
Mohamed Ameen | 09-Jul-07 at 1:13 pm | Permalink
I love it !
Tom S | 09-Jul-07 at 1:45 pm | Permalink
I really like the idea of the popup/overlay when a user presses the control key. I still love my mouse, but that would be a great addition to any non-trivial program which is somewhat keyboard centric. Very nice.
minimal design | 09-Jul-07 at 2:14 pm | Permalink
Most people out there go by the “good enough” philosophy. They don’t care about using their computer more efficiently. They like to click on big juicy icons, and they don’t even use some of those applications often enough to make any kind of learning curve worth it.
Also, the difference of shortcuts from one OS/App to the other makes it impractical to memorize shortcuts for all of them.
Here’s another one: could you seriously memorize every single shortcuts for every function within an app like say… Photoshop? Your “modifier Key UI” thinggy would crumble under the weight of Photoshop’s functionality. And half of the dialog boxes in Photoshop need mouse interaction…
I’d love to see someone using beziers curves in Illustrator with keyboard shortcuts… ;)
And your “new” UI just hides the icons and makes them accessible with modifier keys… Hardly a breakthrough in usability.
VI and its keyboard based paradigm is great for geeks, but Illustrator can’t work without some sort of physical input device. I personally use a Wacom tablet as my primary input device, but I use almost exclusively use the keyboard when coding in TextMate (The ultimate shortcut app… there’s one to scratch your back and another one to make coffee).
This article does not introduce anything new…
Nathan Curry | 09-Jul-07 at 2:17 pm | Permalink
@monopole: I think that while getting your hands off the keyboard is good for carpal tunnel syndrome, CTS is a disease of a specialist (ie, all day computer user). These people (myself included), can and should spend a little time customizing their environment–from chairs to desks, light levels to UI configuration. Plus, you could just make it legal to click on the menu after pressing Alt-S-P-E to visually filter down to the spell check feature.
Having a default interface that benefits most users (the casual ones) is a great idea. Kind of like how generally the default download location is the desktop. Annoying? Yes, but if you’re savvy enough to be annoyed by your new downloads, you can probably figure out how to move them. Tell a random person to open up c:\Downloads, and tell me how they do.
My life as a technician was easier when My Computer was on the Desktop by default (because it’s harder to explain to muggles, not because I’m the Worst Computer Technician Ever)
Randolpho | 09-Jul-07 at 2:32 pm | Permalink
Clay,
I like the idea, and I may (read: if I ever have time) try to implement a framework for doing something along those lines.
If I may suggest, however… it’s not a good idea to do away entirely with buttons and menus. It would be better instead to offer such an interface in addition to classic (or even MS Office 2007) input techniques.
In many cases it will be better to do mouse/button combinations than use the keyboard. For example, when formatting a document one will frequently highlight and format, and if those formatting commands are only available through the keyboard, productivity will slow. That’s not to say that we can’t get rid of a lot of toolbar buttons — a floating context menu that appears above a recent selection with common commands as buttons should do the trick. One of Office 2007’s niftier new features does exactly that.
In the end, I think this would be an excellent supplement to current interfaces, but I would argue against a complete replacement of classic (or, again Office “Ribbon”-style) GUIs.
JG | 09-Jul-07 at 3:27 pm | Permalink
I find it ironic that an argument encouraging keyboard use instead of mouse use asks readers to “click” on thumbnails to see full-size images. Any thoughts on efficient mouseless web page interaction?
Paul Lalonde | 09-Jul-07 at 3:32 pm | Permalink
You really need to try the exact opposite of your approach. The fundamental problem with most mouse-based interfaces (as you point out) is the buttons and menus. What about a mouse-based interface without those? One where the underlying operating system and scripting interact efficiently with text selection? Where the mouse does the selection operations and the text does the actions?
Try acme: http://en.wikipedia.org/wiki/Acme_(text_editor)
R. Bassett Jr. | 09-Jul-07 at 3:54 pm | Permalink
Like most good ideas, what you are describing is not a new concept.
The best program I have ever used for writing was the first one I learned for the PC: Word Perfect 5.1 for DOS. It did have mouse support, but it was so much faster to use the keyboard for all functions - just like it is still so much faster to use the keyboard for just about all functions when writing a document in Open Office (or vi or nano or Midnight Commander Notes or Treepad and even the good old MS apps).
The keyboard is an excellent human interface device for people with full or partial motor control of their hands. The mouse is an excellent companion to the keyboard. And, the gamepad, stylus, and joystick greatly increase the functionality of computing systems.
I have spend considerable time over the last fifteen years thinking of a “better” way to interface with a computer, ever since I had a crazy idea or two, but in the end it seems to me that what we have will get us through until mental controls are prefected and commercially available.
The “better user interface” is already there in most programs. You just need to use it. That is not to say that there are not a slew of aweful interfaces out there.
Paul | 09-Jul-07 at 4:13 pm | Permalink
Chaim Krause:
I’ve always avoided changing shortcut keys immediately when using a program because even though you can do it, doesn’t mean you should. I often get key conflicts and getting help looking at reference material for the app is now mostly pointless. Most programs use fairly standard key sequences or (hopefully) they have a good reason to stray. Although, I do sometimes change them slightly after knowing a program really well. Usually, I’m adding new keys or removing potentially dangerous shortcuts (like the save-all mentioned above).
Seth:
I love Maya’s approach. Unfortunately, I’m quite sure it’s patented. It’s also nice that it’s context sensitive. I wouldn’t suggest “Modes” for other apps, but keep the context sensitivity. You can also see that they used this idea in Alias/Autodesk Sketchbook which only uses a stylus for input. I think there’s more innovative ways to integrate the mouse/keyboard and alternative pointing devices (many mentioned nub and stylus above). I HATE switching between mouse/keyboard. I would like to be able to do everything with one or the other, but benefit from using both together.
I feel the best way to deploy and get something like this to be adopted is by converting a major application or two (word processor or web browser–I saw a suggestion for Firefox using alternative input methods). Standardize it and then create libraries so other applications can easily add this uniformly. I feel this is more likely to be adopted in something like Linux. Or if it’s an external package that’s supported via plugin (like Quicksilver or Growl) it could easily be adopted in OS X (since you can already hide the toolbars with the “lozenge” and it seems to already support plugin apps).
Continue innovating UI design!
Millsy | 09-Jul-07 at 4:32 pm | Permalink
Lets see, something with 104+ buttons to control things, that a device with 3 buttons and a scroll wheel can control, one handed?
Instead of challenging people to find IT guys who use a mouse on systems that have been jury rigged into menu systems from the 80’s, why not challenge more IT people to properly use context sensitive menus with quick 1 level access to the functions required.
Computer games have been getting smarter about using the mouse for quicker access to commands, 1 click and a flick of the wrist is faster than typing “sav” then having to select 1 of the 3 options, after already hitting the “command” key.
Greg Kise | 09-Jul-07 at 5:19 pm | Permalink
You are right about the overuse of the mouse.
However your argument can easily be shortened to one idea…every application should learn user commands like Quicksilver. Simple.
Allow the user to determine the level of complexity, even add plugins to customize the menu system. Conceivably you could have a unified interface tool for all applications.
-gk
Jarvis Cochrane | 09-Jul-07 at 5:55 pm | Permalink
Two apposite remarks:
We’re going back! Back to the Future!
and
Wordstar!
Chui | 09-Jul-07 at 6:23 pm | Permalink
Wordperfect. That brings back memories. Thank goodness we no longer have to remember the strange function keys. F7 to quit, and what about those lovely cardboard strips we stick on top of our keyboards?
Chip | 09-Jul-07 at 8:59 pm | Permalink
This design reminds me a lot of of editor software I used when I was younger. If I recall, it was called Personal Editor II from IBM. It always struck me as being much more rationally oriented than other text-based editors that I’ve used before or since (WordPerfect for DOS, Emacs, vi, or even nano for Linux). The big difference was the fact that the ctrl, alt, and paired shift keys would pop up a contextual menu of all the associated keys. If you added mouse control for text selection, etc., it would pretty well stand up to most of the basic modern editors.
stefoid | 09-Jul-07 at 11:05 pm | Permalink
I suppose decent voice recognition cant be too far off.
some sort of mouse button and or keyboard key that made the computer ‘listen’ for commands, and those commands operating the drop down menus ons-creen would be nice.
like:
a) hold down ‘control’ key to tell computer to listen
b) say “save” and the save menu drops down or appears or whatever, visually revealing your options from that point in the command heirarchy
c) say “as desktop slash work slash Mydocument”
d) let go of ‘control’ key to activate command to save to Desktop/work/Mydocument.doc
Westley | 09-Jul-07 at 11:26 pm | Permalink
A few issues and ideas:
1. Using Control and Alt in the way your suggest disables their “continuous” functionality when used with the mouse. For example, you can select text using the mouse, then copy or move it to another place in the document using Ctrl or Alt (in Explorer, you can also create a shortcut to a file or a group of files using drag-Shift - not highly intuitive, but I’m just trying to demonstrate the “continuous” ability of discrete controls).
2. You forgot context-sensitive menus. This is something that is actually doable with a keyboard - those menu keys, usually between the right-Alt and right-Ctrl - but I haven’t seen anyone ever using them. Should be good as food for usability thought.
3. Tip for your menu-accessing suggestion: Apart from graying-out menu commands, you can also use auto-complete to show and access the ones that are left. Browsers have been doing this since the dawn of history, and it’s very intuitive to all users today.
4. The problem with “muscle memory” in today’s life is that it is actually easier to find the Save button - it’s always on the top-left - than it is to find the Ctrl-S combination. I have a PC with ergonomic keyboard at home, a standard keyboard for my desktop at work, a tablet - that’s miniature keyboard plus pen plus on-screen keyboard, and when I go to other people’s machines I use their keyboards. Apart from the different keyboard sizes and layouts, there’s the hand locations that is completely different (hands on tablet’s “body”, hands on desk reaching up to the keyboard, hands on desk reaching straight to a lower keyboard, hands on special palm-rest, etc.)
I guess that’s all going to change once we have perfect voice recognition :)
WhiskeyJuvenile | 09-Jul-07 at 11:28 pm | Permalink
Two words: Word 2007
They’re making baby steps.
Westley | 09-Jul-07 at 11:28 pm | Permalink
Forgot something: Mouse gestures. The way to simulate discrete actions with continuous input device, plus the muscle memory. Win-win situation?
Helmey | 10-Jul-07 at 12:08 am | Permalink
Actually, the general principle of this is already being approached a different (and I think better) way: the Optimus keyboard. Too bad it’s so damn expensive.
Mohammad Bahathir Hashim | 10-Jul-07 at 3:07 am | Permalink
GNU/Emacs or vi + LaTeX can do the same thing and better, with an excellence printout quality :). I am able to create complex math equations + matrix + tables, change/reize/resize fonts, without touching mouse at all. Even better, I also can create macros to automate some repeatative tasks.
Try and feel it yourself., the power fo GNU Emacs :)
OS : GNU/Linux distro Slackware
X11 : X11R7.2
WM : evilwm
Editor : GNU Emacs v21.1.1 + AuTeX
teTeX distro.
White Guy | 10-Jul-07 at 5:46 am | Permalink
Has anyone asked the folks on the Asian continent if they favor mouse-fingering or CTRL-F-U keys… or do we still just ignore them… i.e. the world’s majority… because they are short and talk funny?
Sherman | 10-Jul-07 at 6:04 am | Permalink
Take a look at Blenders UI. It teaches you to have one hand on the keyboard and one on the mouse. It is one of the most effective use of ui I’ve ever seen. http://www.blender.nl
Sherm
Sherman | 10-Jul-07 at 6:05 am | Permalink
oops http://www.blender.org
James | 10-Jul-07 at 7:33 am | Permalink
I fully understand your intent. I sit in front of a CADD station all day, and thank God it is customizable. While it doesn’t do what you are proposing, I have a five button mouse. I have ctrl, alt, and shift mapped onto the mouse. With the combinations afforded me in those three keys and the 12 Fkeys up top I can go weeks without having to pick a tool off a menu or use a key-in. I have 96 function keys defined. Good luck with your ventures. If you need a guinea pig let me know.
Clay Barnes | 10-Jul-07 at 5:27 pm | Permalink
Before more people ask me:
Yes, I have used Emacs, vim (my only text editor), LaTeX (my only “word processor”—I had to install OO.o for the screenshots), ratpoison, Ion, wmii (in use at work), openbox3, and a host of other Linux software, as that’s my exclusive platform.
Also, I am aware that there are no new ideas. I appreciate those of you who point me to implementations similar to my propsal, as they are very educational. Those of you who seem to be hinting that I “stole” from said projects, I attempt to cite any examples I can think of that influence me. I may not hit everything for absence of perfect memory, but I’ll revise as I’m reminded of influences.
Finally, as I said in my other entry, I am by no means saying we should get rid of the mouse, but I am saying that almost everything it’s currently used for can be replaced by the keyboard, hopefully freeing the mouse for the tasks that it’s better suited for.
Alex | 11-Jul-07 at 3:00 am | Permalink
Made me think that (especially considering what Clay put) an independent overlay application would be good. By that I mean certain keys like Ctl, Alt are intercepted and the separate interface app pops up a different overlay depending so you might have fast commands, all commands, verbose commands where fast are things like save that you do often, all commands is something like the full navigable menu system and verbose is a type in keywords and it shortens a list based on likelihood of it being one of the specific commands with each having a short cut key to use it off that list.
The advantage of the overlay application is you can then remap things between applications so that you have a consistent application interface as far as possible between different applications.
mace | 11-Jul-07 at 9:17 am | Permalink
Yes, like very many other commentors i cannot stop thinking about Emacs while reading through your post. Of course one of the reasons for that is that i use it daily, though i’m not much of a heavy-user, less than i used to really.
But are you familiar with the Plan 9 From Bell Labs[0], a son of Unix if you will. The original Unix ideas have been taken further, for example “everything is a stream” (including the elements of the GUI which is marvellous).
Plan 9 is more of a laboratory project, though some enthusiasts actually use it too. The interface of the Acme text editor is fascinating, a GUI for heavyusers. Well, the whole project worth a good look. :)
[0] http://netlib.bell-labs.com/plan9/
Not Just Another Human Being » Requirements for Awesomeness (In Websites) by Josh Simmons | 12-Jul-07 at 11:31 pm | Permalink
[...] Short Cut Keys The mouse is very effective device for certain types of input, especially when continuous streams of data are helpful (ie, scrolling or dragging something on the screen), however, for executing commands it is not the best option: the keyboard is. If you know any technophiles, you probably notice that the longer they’ve been in the industry the more they use the keyboard. It’s an efficiency tool and it’s extremely effective at executing commands. It’s my opinion that a good website, an interactive website that is, would enable short cut keys. I would like to be able to navigate and use websites entirely with my keyboard, maybe hit CTRL + S to set the focus to a search box, after which I type in a query, results are displayed, and I can hit CTRL + 1 or something to that effect to highlight or select the first search result, etc. Of course, for some websites this is complete overkill (ie, websites for most mom and pop operations or brick and mortars). [...]
Philip Ganchev | 13-Jul-07 at 7:45 pm | Permalink
@JG - efficient mouseless web navigation is achieved with Firefox by turning on find-as-you-type, and type the names of links to highlight them.
What argument are you talking about?
Philip Ganchev | 13-Jul-07 at 8:29 pm | Permalink
I have been developing a very similar UI proposal, where the user invokes command mode and types words and commands whose descriptions match the words appear in a drop-down list. Matching words are highlighed in the list, and the list gets shorter as the user types. Matching the command descriptions allows users to access a command without knowing its name. The list widget has a maximum size so as not to obscure too much content in the window — the descriptions take up space, and unlike your proposal, does not help with positional memory. My idea was to implement it in GTK, so that it automatically works with any GTK program.
I also wanted to extend this to button and tab labels, so that users can conveniently select the button they want without looking for an accel (which may not even be specified by the programmer), and without having to “tab” through buttons on the window. But I don’t know how to integrate it with the menu search.
I am astonished at how most commenters have misunderstood your proposal and its implications. Most are completely off-topic. The proposed UIs (Clay’s and mine) are orthogonal to keyboard control of windows, program invocation and file search. It is a way to issue commands to an application. You can’t do that with Ion, Spotlight, Quicksilver, Enso, “start menu” search, or a “run” dialog. Unlike ribbons, KeyCue, search under the Help menu, and easy access to standard menus, its incremental search avoids hierarchy. This avoids having to hunt for the item in the menu hierarchy, or remember where itis.
It removes the burden of learning key combinations, the reason why regular shortcuts are rarely used. It supports exploration. And it works for all kinds of apps — desktop apps, web apps and niche apps — and benefits all users — occasional and frequent, beginner and “advanced”, visual, auditory and kinesthetic. But it is much more useful for native apps because users spend much less time using Web apps.
We are not trying to remove the mouse entirely, just use the best device for each task. The keyboard skill required for the tasks described is much less than the mouse skill, even with a trackpoint. Invoking a pie menu and selecting an item requires much more coordination, even if the mouse is constrained. You have to make sure you click on “empty space” and that you select the right item; this reauires you to watch the screen and requires more coordination. And, you cannot habituate across apps, unlike a command like “Alt+save as” in the new UI, which means the same in all apps that have it. All Gimp’s commands cannot fit on a pie menu anyway, nor are there enough mouse button combos to go round. Mouse gestures also require a lot of skill and remembering (how do I invoke italic font?), and are less efficient for tasks described in the post. Typing shortcut commands like “sav3″ instead of “save as” is much harder to remember and explain, and this is too big a price to pay for saving several keystrokes. And using the keypad to direct the mouse pointer misses the point altogether. The Optimus keyboard is expensive, only works on one computer, cannot accomodate all functions of a large app, and requires mode switching for different apps’ diffrent key combos.
Unlike a toolbar, the command list can show all the functions the app supports, including their icons and descriptions. And, users can specify the command they want, instead of hunting for it. I have not seen evidence that people remember images seen better than words heard, but I have that people visually distinguish words better than images when there are more than a dozen or so. Anyway, the former is irrelevant because the new UI includes icons, and because icons alone are imprecise and ambigouous — see http://www.humanized.com/weblog/2007/06/25/the_end_of_an_icon/ for example.
The problem that different apps have different key combos is solved by use of commands. Instead of Control+Shift+s or ALT+F-A, type “Alt+save as”, which will execute the same command in all apps. Of course, app shortcuts should be standardized, but changing apps requires more work, coordination and cooperation among developers. Customization is not a solution either, it carries problems.
It would be great to re-implement the whole system “correctly”, like Archy (http://www.raskincenter.org), but that is a lot of up-front effort, and throws out everything that works already.
@Alex - It is not good to have different overlays (fast commands versus all commands) because that interferes with habituation.
@Paul Lalonde, @mace - how do Plan9 and Acme GUIs help efficiency?
@Sherman - Having one hand on the mouse does not work for typing, such as emails and papers. I go further than Clay — the keyboards is also better for selecting text and positioning the cursor during text editing.
Philip Ganchev | 14-Jul-07 at 12:03 pm | Permalink
Clarification — I wanted to implement the command UI into GTK, so that any GTK program that has menus would have that functionality.
n[ate]vw | 25-Jul-07 at 8:41 pm | Permalink
This looks like a good implementation. Why not make it visible for new users by putting two fairly unobtrusive icons in the upper left corner depicting the “toolbar” and the “menu” HUDs. A rapidly appearing caption/tooltip will say “Press Control/Alt”.
Let the mouse still work, as well as for clicking the icons in the HUD, but this design should naturally encourage users to migrate quickly to the keyboard.
mike | 01-Sep-07 at 11:43 am | Permalink
the gimp (at least under linux) provides for this: use the mouse to navigate though menus. once the desired item is found, leave it highlighted and issue ctrl- to assign a keyboard short cut to that item.
The New Interface Advocate :: The only two interface designs ever conceived: | 12-Nov-07 at 4:00 pm | Permalink
[...] as a dichotomy, that is more for historical than strictly technical reasons. For example, this mouseless GUI design shamelessly masquerades as a search design while subconsciously teaching the user memorized [...]
The New Interface Advocate :: The misused mouse, part 1: The story of the mouse’s decline | 12-Nov-07 at 4:01 pm | Permalink
[...] Without further ado, I give you a proposal for a mouseless graphical user interface. [...]