Natural Language CLIs?

snuf23 asks: "Altavista has a report on the future of Windows as presented by Bill Gates at Microsoft Professional Developers Conference. Curiously, one of the touted features is called "type in-line." Essentially, it's a text based interface to the computer which uses a natural language interface. Having worked at a translation software company for three years, I am familiar with the complications of parsing meaning between human languages. It seems that in computer to human you would have somewhat less complexity, at least in terms of general use. Have any natural language interface CLIs been built? Voice recognition software comes to mind ("Open the file, HAL") but what attempts have there been to replace shell interfaces with natural language interpreters?" While I'm all for making computers easier to use, would typing "move all files beginning with the letter a to the directory called 'foo'" be any improvement over "mv a* foo" (or "move a* foo" for that matter)?

Very simply-

Forcing the computer, a very brittle, nonadaptable thing, to do all the adaptation in the computer-human relationship, is not simply problematic- it is very, very inefficient. I don't mean 'get a 2Ghz CPU' inefficient- I mean people's workflow will be absolutely crippled by essentially playing a game of 'telephone' with their computer. Try this: spend an hour of your day working with your computer by getting a friend who knows you and what you're trying to accomplish, and having the friend at the mouse and keyboard- while you sit next to him and explain all actions verbally.

Sound like a nightmare (or tech support hell?) You're getting the picture. Even with an AI of human scope that is intimately familiar with your work, you will constantly be hitting inefficient areas- you'll get basically squat done and might end up very frustrated. Even if you don't end up frustrated- your productivity will be in the toilet.

This is a self-correcting development- anyone who gets heavy into using it will be removing themselves from the world's cutting edge. It is like a stagnant backwater, like a mechanism designed to take the non-geeky and hinder their ability to compete in a newly technological world by setting them up to be grossly less effective than the geeks.

For that reason it would be better if it did _not_ become a reality, but most likely Microsoft will figure out some way of doing it as it's very in line with their preferred approach. The ideal counter to this would be to continue to develop more efficient systems that require some user learning, so that the MS natural language users can occasionally be challenged by the sight of somebody accomplishing tasks many times faster- or faster by several orders of magnitude, which is not unthinkable. The best response to give when asked 'How do you do that?' is 'You learn how...' with more details if desired.

What about highly inflected languages?

Just out of curiosity, I wonder if this problem would extend as badly to highly inflected languages like German, Latin, or (the ultimate in inflection) Greek.

For those not up on it... In English (which is hardly inflected) verbs are conjugated, but everything else is pretty much left alone. In an inflected languages, verbs are conjugated, but so are nouns and (in some cases) adjectives, adverbs, etc. The net effect is that it is possible to say with much greater precision what the reference of a word is.

The classic example would be where Jesus says (in English) "I tell you, this generation will not pass from the earth until all these things [the end times] are fulfilled." The thing that is never really adequately expressed in English translations is that "this generation" doesn't refer to the generation living at the time of Jesus: it refers to the generation that will exsperience an assortment of signs and wonders.

--

About human interface

There was another computer language which tried to be english-like in syntax. It was called HyperCard and it was originally for the mac (although later ported to windows 95).

To do simple things it wasn't so bad.. you'd drop a button onto the card and script it like "GO TO NEXT CARD". However, things started to break down rapidly if you wanted to do complex things. For example, creating something as simple as a calculator was a monumental feat because every variable was called "IT" so "put IT into clipboard" could be seen all over the place. The language was definately english-like, but that didn't help its comprehensibility.

You see, human language is dynamic. Computer languages aren't. A computer has no way of knowing whether "delete that" means "delete the currently selected item" or "delete the item I'm thinking of". The result might be you delete C:\WINDOWS instead of C:\legal.doc

Our language is dynamic.. it has a very complex ruleset which AI people have been struggling with for a long time.

Oh, and then you have phonetic problems. "delete temp" might mean delete /tmp/*, delete the directory /tmp, or even the file /Temp. I don't want to be around when the computer makes the wrong choice!

Oh, stop whining. You're all trolls and BOFHs.

Why should basic end-user interaction with a computer be arcane and require training or hours with a shelf of books?

As a lone voice up there said, you should be able to view your wedding photos by saying, typing, writing or thinking "show me my wedding photos". Not "see-dee slash-home-slash-images-slash-wedding, semicolon ee-ee dot-slash-star-dot-jay-peg".

This is about natural-language recognition. A Unix, DOS, CP/M or VMS shell is not natural-language.

It's about time we got back to this. The Apple Lisa and the Macintosh did untold damage to progress in this area when it made the WIMP GUI the new standard way for an end user to interact with a computer. Nobody wanted to work on refining the command-line interface for end users, so the command line became an ever-more byzantine interface solely for programmers and administrators.

I remember back in the mid-1980s using dialup BBSes that had natural language interfaces. They ran on humble PC ATs, and set Zork and the other Infocom text adventures as their benchmark for success. These bulletin boards worked just fine with commands like

• go to the library and download "BLUEBOX.TXT"
• list the forums
• go to the cafe and read the new messages
• post a message
• how long have i been connected?

If a BBS running on a system with 320K of RAM could do that in 1985, imagine what a WebTV should be capable of today, never mind a current-day traditional PC. It's about time companies and organizations doing UI research came back to this; the Unix CLI hasn't evoled significantly since the mid 1980s, and the windowing GUI hasn't changed significantly since around 1988.

It doesn't surprise me that /. regulars are dismissive about this with the usual nonsense about making the average person who wants to surf the web and write email learn Unix shell programming. Phooey. Most people use computers as an appliance, not as the center of their lives or as an end in themselves, just as most Linux heads eat pizza without knowing how to make cheese from raw milk.

Nobody's going to take good ol' /bin/bash away from you. Stop begrudging non-programmers ease-of-use.

Re:About human interface

Such a natural-language shell wouldn't have to reproduce the semantics of English (or whatever other language, whether natural or synthetic like Lojban), but only enough as deemed useful for expressing tasks to the computer. It would, in essence, still be a programming language, only with a syntax which is less arcane.

As for the "dynamic" part... well, let's take Perl as an example. Perl is nowhere near as expressive as natural language, and it definitely doesn't resemble any of those syntax-wise. But it's got a lot of shortcuts. For example, the word "it" is represented by the variable $_. Many built-in functions, when they receive no argument, will assume that $_ is to be manipulated. For example, $o = int; is equivalent to $o = int $_;. As an extreme case, print; is equivalent to print STDOUT $_;.

This kind of thing, only extrapolated a bit, and with some added heuristics and user interaction history management, will make a NL shell perfectly useable most of the time, without having to reproduce the semantics of natural languages in full.

Dumbing down computers

Frankly, I dont want my computer to be dumbed down as much as Microsoft thinks it should be.

When a computer is dumbed down, it is bad for everyone. Newbies dont learn anything from a dumbed down computer, and they cant use the stuff they figure out in a big company or most places on the net. Its bad for power users because a dumbed down computer is not powerful at all.

What happens when that newbie goes to use a computer from another vendor, and has to double click rather then single click on an icon, and has a heart attack because its too 'hard' for them? (On a side note, I actually did have a customer call up once and cancel her service with me because she had to double click on the Dialup Networking icon then double click Netscape before she could browse the web. She claimed 'why cant it just dialup to the internet when I want to use it and open netscape automatically, why do I have to do anything?')

When you force a newbie to learn how to use a computer, the right way, they can take the information they learned and use it in a company, on the net, anywhere they find a computer.

EFnet is a good example of where this dumbed down software has caused a problem. Large channels, which are inhabited by long time *NIX geeks who use the tried and true methods of chatting with ircII or some variant, have to deal with newbies who think that color and bold and all sorts of stuff in mIRC is cool (its even worse hearing them call IRC, mIRC).

So now I'm gonna stop blabbering on, and just ask one question, Is dumbing down computers with things like this really worth it? Do we want people so computer illiterate suring the net, opening themselves to attacks, etc?

Re:My second try . .

msCLI> Make an email virus.
msCLI> Done. File created = iluvyou.vbs

Wow, that was simple.
___

natural language CLIs = Clippy Commander

This is just another excuse for ms to pop up that obnoxious paper clip. Parse out the meaningfull word in the sentence and little clippy shows you a menu of choices. It's twice baked Ask Jeeves.

Now, if ms would just devote 1/2 of the money and time they spend on uptime, interoperability, compatibility and, conectivity they might have something usefull.
___

My first try . .

msCLI> mount linux file system.
msCLI> unable to comply.

this thing doesn't work!
___

So...

&gt Take all but the blue file and then go up.

You are in a twisty maze of directories, all different.

Query, yes; command, no

From the brief Microsoft press release, it sounds like a natural language query system, not a command system. Natural language query systems have been around for a decade or two; they don't work much better than "grep", but that's OK, because the cost of errors is low. Ask Jeeves [askjeeves.com] is probably the best known of the Internet era, and it's a good example of how these things work. If you ask a question that's been anticipated by the designers, it's great; if not, you'll probably get some totally bogus result. That's not good enough for a command system.

Microsoft has put considerable research effort [microsoft.com] into natural language parsing. That work resulted in the grammar checker in Microsoft Word, which really does parse and diagram sentences. So they should be able to produce a client-based query system without any trouble. They could probably make it launch the appropriate Microsoft product to do something for command-like requests, too. I wouldn't expect anything complicated.

This sort of thing has more promise for voice input. It makes a lot of sense for portable devices. Typing text into tiny keyboards has go to go.

From a fellow HyperTalk coder...

For example, creating something as simple as a calculator was a monumental feat because every variable was called "IT" so "put IT into clipboard" could be seen all over the place.

Before I found QBasic, I was writing HyperCard games. HyperTalk didn't call all variables it. The variable it was one of two global variables that held the result of some functions that were declared as procedures (this was a common pascal practice back when functions were thought of as having NO side effects). (The other was named theresult (yes, it did have a space). There were shortcuts to put variable contents into it and the result: get foo = put foo into it or in C, it = foo. return foo = put foo into theresult or in C, the_result = foo.

<O
( \
XGNOME vs. KDE: the game! [8m.com]

Have any been built?

Sure there have! While it's perhaps an overlooked or under appreciated area of coding, many MUDs (Multi-User Dungeons, text-based Massive Multiplay Online Role Playing Games) have *excellent* natural language parsers. One example of this is Zork's, though while primitive (and single player) compared to some that exist today, still had the idea.

Many of these MUDs parse for hundreds or THOUSANDs of commands, and do it quite well. Anyway, I'd just like to present an example of how it can be done.

If you'd like to check out some of these look at http://www.mudconnector.com [mudconnector.com]. Note that not all muds have natural language parsers, but I'm pretty sure you can search for ones that do at Mud Connector.

- Wedg

Savvy

Am I the only one to remember "Savvy," the natural language command line interface that worked on Apple // and IBM PCs?

There were two basic parts to it. A Forth interpreter came with predefined "pages" for standard office procedures such as word processor, spreadsheet, database, and so on. A ROM on a card contained some logic to interface between whatever you typed and the Forth system. Since Forth is extensible, it was simple to add new commands.

For example, if the system already had built into it a command to "list all paychecks in the last week" and you typed "lsit..." instead, the system would ask you what you meant, give you some previous times you typed in something like that, and ask you to either select one of them, or define your new command (in terms of old commands, or in a simple programming language that was near to English and not even related to Forth).

I rather liked the Savvy system, since it gradually learned from the user instead of always forcing the user to learn the correct way to do things. And it was pretty amazing to have such a system with virtual memory on an Apple //. With a hard disk it was fast and easy to get common jobs done.

Of course it was too expensive. The idea came out of the space program and was sold for something like $1,295 at first. Then they moved it to the IBM PC and eventually the price fell to less than $100. Excalibur Systems sold it; last I heard the company was doing document management systems.

On the down side, it was completely incompatible with any other software. It didn't even have a communications program to be able to use a modem to import data. And when the Macintosh came along a lot of users thought that a windowing system with a mouse was better than a keyboard.

But a lot of people have used command line interfaces as in Unix and the power is attractive for experienced users. Even with Unix shells one can get some of the same power as Savvy simply by creating aliases or small scripts.

Maybe voice recognition or some other form of natural language system or pattern recognition will be invented. But Savvy proved that you could do a lot of it in just 64KB, on a 1MHz CPU. It didn't try to do everything, just accommodate the user as best as it could. Neat idea. Will we learn from our past failures? Don't count on it!

MUD, MOO,and MUSHes Interface?

I think a handy interface might be that of the MU* style games of the past...

"look"

You are standing in your office, a filing cabinent is to your right, and your desk is in front of you.

"sit at desk"

You are now sitting at your desk. On your desk is your check book register, a typewriter and a notepad.

"use typewriter"

(at this point, a typewriter program would open (i.e. word processor).

It would also accept text input at the bottom in a sort of "chat window"

"Insert new document"

"Throw document away"

"Get document"

Say you "get document"

You take the document. You should write a name on it.

"name document My Post to Slashdot"

Done.

"stand"

You stand up from the desk.

"look at filecabinet"

It's an ordinary 5 drawer filing cabinet. The drawers are labeled "Bills", "Letters", "Charts" , "Graphics", "MP3s"

open letters

I don't know what you mean.

Open drawer named Letters

You open up the drawer name "Letters"

File "My Document to Slashdot"

You store the file under the folder marked "M"

File "My Document to Slashdot" under "S"

You move the file to the folder named "S"

---------

Anyway, you get the point. Is anyone working on something like this?

Consider the audience.

In my spare time, I help some people in the area use their computers. One of my clients is an elderly woman with some limitations on the use of her hand. Selecting anything, especially the tiny text links and widgets used in many situations today, can be quite difficult. I didn't even try to do tasks using the command line, since she wouldn't be able to recreate them without calling me up and asking for assistance.

Sure, hardcore *nix hackers will never need a natural language anything -- in fact, one might argue that the standard suite of commands and GNU utilities is their natural language. But for people who don't work with computers all day long, saying something like "Enter a check," "Open a new document," or "Send an e-mail" will do just fine. Each of those documents would require several windows and mouse clicks even with the most intuitive of GUI's.

this is pointless in the general case

I studied "computational linguistics" for 5 years, and if there's one thing that I got out of it is that this whole endeavour (NL parsing + understanding) is hopeless.

Natural language is ambiguous way beyond peoples imagination, and if there's anything we don't need it is ambiguity in giving commands to a computer. NL doesn't _seem_ ambiguous because we are so good at disambiguating it (most of the time, anyway) using our own extensive knowledge base, about what is "reasonable". For a computer to have access to a similar knowledge base (simulating a brain, in short), is a pretty impossible task at this point in time.

Yes, we can get away with simple hacks and partial functionality, but what good is that? it will still be ambiguous. If you want a CLI interface and you want to move away from programming language style stuff, the least you have to do is define a language that can only be parsed and interpreted in one way. This won't be natural language, so user will have to learn its peculiarities. It's a shame, but deal with it.

If I can generalise for a moment, this whole idea that we need an UI that is closer to people to make computers easier to use for computer illiterates is very shaky. The fact that we can talk to it (through the proposed CLI, or speech) isn't making the computer any easier! Do you think that because I can now say "view attachment" instead of clicking on a button, that this will help Joe AverageUser understand any better why part of his HD was wiped, and why all his email contacts got spammed with a virus? Does it help him understand where his file is stored after he uttered "write file to disk"?

There are tons of (relatively easy) things we can do to make a computer easier to use, but this particular one won't bridge the gap once single bit.

This is NOT pointless in the general case

I've been working in "computational linguistics" for the last two years.

I guess there were a few things you didn't study :)

The past 10 years or so a new field - statistical natural language processing (SNLP) has shown a _lot_ of promise.

Right now, if you throw a SNLP system a bunch of parsetrees, it's able to induce a grammar - even in sufficiently complicated languages. (For simple languages, you can even induce a reasonable grammar just by giving syntactically correct string. Impressive!)

The next stage after inducing syntax from training examples with tagged syntax is to induce semantics from training examples with tagges semantics.

Yes, this is still a research topic, but it is by _no_ means pointless. One day computers will be able to do anything humans can, and more.

Natural Language needs a new model

While I'm all for making computers easier to use, would typing "move all files beginning with the letter a to the directory called 'foo'" be any improvement over "mv a* foo" (or "move a* foo" for that matter)?

Sure, if you stick with the existing model of computer interaction and wedge a natural language interface on it.

However, that would be as useless as having a present-day CLI to low-level system functions only. "read disk sector 1023 from disk 4 via scsi interface 0 number 0 into memory address 1e75FOO" is a waste of even the simple CLIs we have now.

We need to go in the other direction. "Open my wedding photos." "Spell-check the latest draft of my current novel." "When did I receive an e-mail from my publisher with 'foo' in the subject?"

Yes, this means even more work than just parsing a natural language and would require a pretty sophisticated model of interaction, but isn't that the kind of challenge that produces revolutionary advances in computing?

Re:Consider the audience.

Although it doesn't necessarily replace a CLI, the nice folks at MIT's Lab for Computer Science have set up Jupiter [mit.edu], a voice interface for weather information. Give it a call at 1-888-573-TALK if you're in the US and have nothing better to do on a Saturday afternoon. You can ask it simple questions like "What is the weather in Seattle today?" and "Will it rain tomorrow in New York?" and it will respond after a couple of seconds with the answer. Usually it's pretty good, if you're used to the back-and-forth interface. You have to know how it works, though -- you can only speak after the tone it gives, you can't cut it off, and its English isn't that good.

Here's what I heard when I handed the phone to a relative of mine who is hardly computer savvy:

"What's the weather in Baltimore today?" (pause, gets answer) "What about in New York?" (short pause) "New York. NO, not Utica, just New York City. Hello? I don't want to hear the weather in Utica, I don't LIVE in Utica! Just give me the weather for New York City! Oh, I give up." (click)

With all due respect, Artificial Intelligence has a long way to go before it catches up with natural stupidity. :)

Maybe...

Slashdot asks:

'' would typing "move all files beginning with the letter a to the directory called 'foo'" be any
improvement over "mv a* foo" (or "move a* foo" for that matter)? ''

Probably not, but how about "Move all files I changed in the last week onto my zip disk."? I know I'd need to check the 'find' man page for that.

Damn, MS beat me to it

I've been working off-and-on for the last year on an Eliza-type interface for Xf86config. It still has a few bugs, but the gist is there. It's basic interface is:

XEliza What sort of video card do you have?
user it's a rage 128
XEliza Tell me more about this rage 128
user Well, uh, it's got 8 megs of ram
XEliza How does 8 megs of ram make you feel?
user okay, I guess. I wish I had more of course.
XEliza Does it bother you that you wish you had more?
user Well, I don't really play a lot of games, so maybe I'm just being greedy.
XEliza Do people often say you are greedy.
user Not really, I mean, they say I spend too much time and money on my computer
XEliza Who says you spend too much time and money on your computer?
user Well, my mom does.
XEliza Tell me about your mother
error: cannot parse "mother" in module "freud":
no output written

Re:Consider the audience.

But for people who don't work with computers all day long, saying something like "Enter a check," "Open a new document," or "Send an e-mail" will do just fine.

"Enter a check", huh?

The chess 'bot would wake up and tell you

Sorry Dave, I can't let you do that. Chess rules specifically forbid entering a check.

Seriously, ambiguity is a big problem with natural-language interfaces. How is the computer to know if you wanted to do something involving your checking account or if you want to add an integrity constraint to a computer program you are writing? Or if you are playing a chess game and trying to make an illegal move ...

Natural language comprehension is an AI-complete [earthspace.net] problem. That doesn't mean useful approximations can't be done. But they only work if you keep commands simple and adhere to a computer-friendly style of expression. With requests of any complexety the risk of a misunderstanding is too great to trust interface to do anything that can't easily be undone. In this sense it is similar to DWIM [earthspace.net] interfaces (follow the link for a good anecdote).

No doubt natural language interfaces will find its niches, and someday many people may even be using natural language interfaces exclusively. But when these people need to do something more complex or risky, they will need to turn to a hacker who masters some other arcane but concise command language.

/A