Creating a rich web app by talking to my computer

Edited –¬†In the video below, I demonstrate the latest addition to Phosphorus Five; The Magic Menu. Using the this module, I am able to speak plain English to my computer, and have the computer translate my phrases to computer code, saying things such as “add a paragraph”. At which point the computer adds a “p” HTML element/widget to my web app. The concept works by creating associations between Hyperlambda code and spoken phrases and words, which allows the computer to automatically create code, once some word or phrase is spoken.

I realise this might be difficult to believe in, but I am actually now capable of creating highly rich and interactive web apps, using exclusively my voice, to tell my computer what I want it to create on my behalf – At least in theory. Watch the video below for a demonstration …

Basically, what this implies, is that you can speak natural language to your computer, and have your computer create complex and rich web apps, as a result of understanding what your intentions are. The Magic Menu is a module for Phosphorus Five, and allows you to create your own voice commands, and also serves as a plain navbar, if you don’t want to use your voice, or don’t have a supporting browser. Read more about the Magic Menu here. Although, documentation could probably have been more emphasised.

Disclaimer

Speech recognition on the web is a highly experimental technology, and the speech parts of the Magic Menu only works in Google Chrome at the moment. However, since speech actually is just an option, and you can use the magic menu as a plain navigation menu – This is technology that is currently available for you to use, in your own web apps, today. This is not some obscure research technology, but something that’s available for all web developers, wanting to create web apps, using .Net and C#. Everything needed to get started is Open Source.

The Magic Menu also plugs automatically into all your existing Phosphorus Five apps, and hence allows you to use this type of navigation, in all the apps you create yourself in P5.

If you wish to try this out for yourself, you can check out the installation instructions at the project’s GitHub site. It requires Visual Studio, MySQL, in addition to optionally GnuPG to store PGP keys. If you want to try the source, instead of using the installation script, you must in addition to downloading Phosphorus Five itself, also make sure you’ve got Micro, Magic Menu and Hypereval installed. If you install it using the installation script, you’ll need to install both Hypereval and the Magic Menu through the bazar. You can find all relevant projects through my GitHub account.

Notice, for some odd reasons, Google Chrome refuses to accept speech recognition input, if you try it directly towards an IP address, instead of a domain. Although, the loopback (localhost) seems to work though.

Advertisements

A million menu items

Navbars or menus are funny. On the one hand, they’re the source of confusion for our users. On the other hand, they’re what empowers our users as they use the applications we are creating. Hence, there seems to be a tradeoff between power and simplicity when it comes to creating navigation for our apps. How would you feel if I told you that I had “invented” a way to allow you to create a million menu items, without significantly adding to your app’s complexity?

First things first, when you open your File menu, and you see the text “Load”. For you this might seem easily understood. However, for a computer-illiterate, it might as well have spelled out “Warpdrive”. And what’s a “File” anyway? How about “Open a document”. Obviously such a wording is better. However, historically this have been impossible to use, simply since it would end up creating “walls of text” for our users, as they navigate the systems we create.

The screenshot for this blog, suggests a solution for the above dilemma. It will never show your users more than 5 menu items, it features a search interface, and it allows you to use the same menu in all of your apps. First of all, this implies that you don’t have to teach your users a different set of menu items, for all of your apps. Hence, as they have learned how to use one menu, they have learned how to use them all. Secondly, it allows you to have literally a million menu items, without cluttering your UX with “walls of text”. It does this, by showing the user only his 5 most recently used menu items. Implying the user has access to all of his million menu items, but will never see more than 5 items, and the 5 items he will see, are the 5 items that was most recently used. Seriously, do I really need to explain the advantage of this …?

As an additional bonus, the user can choose to turn on speech recognition, allowing him to choose menu items, by literally speaking to his computer. Since there are never more than 5 items, you can display what the menu item does, with a complete phrase, and not only a single word, such as “Load”.

This happens to much more closely resemble the way our minds work, and creates an intuitive understanding of how to interact with your apps’ menu items. The above screenshot is an actual working module for Phosphorus Five, referred to as its Magic Menu. It allows you to create a single entry point UI, allowing you to control any aspect, of all your Phosphorus Five applications, seamlessly integrating your apps together, such that they become a uniform whole.

Yet again, seriously, do I really need to explain this …?

For the record, the Magic Menu is still in BETA, but will become an integrated parts of the next Phosphorus Five release. If you wish to play around with it today though, feel free to fork the GitHub repository, which you can find here

Agile software development is all about the tools

The above statement, is the equivalent of putting my foot into my mouth, and committing social software suicide. The reason is that the Agile manifesto, repeats¬†“Agile is not about the tools”, in literally every single paragraph. So when I claim what I do in the header of this article, I literally disagree with every single software development theory, that has ever been put forth – And every single software architect guru, that has walked this Earth in recent times. However, I can prove that I am right, and I intend to do so in this article.

At the heart of “being Agile”, is the ability to be customer driven. This implies that the customer is allowed to make all the important decisions of what features the end software should have. In fact, this is the sole motivation for why system developers seeks to become “more Agile”. When the customer gets to decide how the end product ends up, the software is able to reach its goals. When the customer is not able to influence the end result, we might just as well start flipping burgers, since we don’t provide the expected value for our customers – And our customers will end up leaving us, and go to somebody who is able to deliver what the customer actually wants.

If you wanted to buy a car, and the seller gives you a horse, how would you feel?

The above is such a self evident fact, that arguing about it, is meaningless. If you create a statue out of granite, then once it starts taking shape, there is no way to really modify it. If you instead start out with building blocks, or “clay” to use an analogy – You can always go back and change it. Hence arguably, we have already proven that “Dynamic programming languages is a better tool for being Agile than statically compiled languages, and since programming languages are tools, well, you get the point” …

The Agile manifesto, and its different methodologies, seeks to “facilitate for change”, by creating all sorts of theories and methodologies about how to manage your code. For instance, eXtreme Programming taught us that we need Unit Tests, since this allows us to change our code, without introducing bugs. The objective of course, is to change your system. The Unit Tests from this perspective, is simply a tool, that allows you to safely apply your change.

Hence, arguably, we have already implied that XP is all about the tools

However, we can bring this much further. Imagine a system, where its core developer only creates the core building blocks, and the process of tying these together, is left for the customer – Under guidance from the main developer of course. Such systems facilitates for becoming Agile, to an extent impossible to imagine for most system developers. Then imagine a system where its core developer, can apply most changes, faster than the customer can communicate these changes – Facilitating for that its core developer(s) can modify the system, in a conversation with the customer, showing the results of the customers needs, seconds after they have been communicated? Would that be “Agile”?

Well, let me show you a tool which facilitates for that.

Then come back to me and argue that “Agile is not about the tools”

Facts are, Agile is only about the tools. And the better tools you start out with, the more Agile you allow yourself to be. And the more Agile you allow yourself to be, the more happy customers you will have. And the more happier customers you have, the more happy you will become, and so on …

Agile is all about the tools!

Magic menu is a new way of communicating between computers and human beings, allowing humans to rapidly “configure” what they want the computer to do, once some “button” or “menu item” has been selected. Read the following article, and try to explain to me how the Magic menu is not a tool, and how it does not facilitate for becoming more Agile – And I will eat the foot that is now in my mouth, if you can …

Magic menu, one navbar to rule em’ all

How would you feel if I told you that you’d never again have to create another navbar, toolbar, or menu – And that you’d never again need to worry about how to have your users navigate your apps? Sounds like science fiction? The Magic menu arguably is that thing. It allows you to use the same navbar, throughout all your apps, and arguably allows you to seamlessly integrate, all your Phosphorus Five apps, resulting in that the user would experience these apps, as if they were simply modules, in an application suite.

The Magic menu from one level is a plain old menu, or navbar if you wish. It allows you to declare your menu items as a tree structure, having an infinite number of dropdown items, creating any number of menu items you wish. However, it features a couple of unique properties, which once you’ve gotten used to, will highly likely feel very comfortable.

First of all, it will never show more than 5 menu items at the same time. The items it shows at any point in time, will also just so happen to be the 5 most recently used menu items. It comes away with this, by displaying its menu items in a pageable grid, which allows the user to easily search for items he’s looking for. See the screenshot above to understand how it will look like.

Secondly, it will automatically plug into any Phosphorus Five apps. Any P5 apps you have, that has a widget on its page, with the CSS class of “toolbar”, will automatically have an additional button injected, which allows the user to launch the Magic menu.

Third of all, it allows you to associate some piece of Hyperlambda to a phrase, word, or sentence, describing your menu item. Later when this phrase or word is selected by the end user, that Hyperlambda will be evaluated. This allows you to easily create any number of menu items, and have any arbitrary Hyperlambda evaluated, once the menu item is later clicked.

And finally, it even allows the user to select menu items, using natural language, by literally speaking to his computer, to select menu items. The last feature unfortunately only works in Google Chrome at the moment, since it’s dependent upon some of the bleeding edge Speech API standards, only currently available in Google Chrome. If you enable it though, then when some menu item has been selected, you can have the Magic menu read out loud some piece of text, using Speech synthesis. This feature, literally allows you to control every single aspect of your web apps, using nothing but natural speech, while listening to the feedback your computer is giving you, which depends upon what you tell your computer. Hence, you can entirely replace your GUI with a “VUI”, or Voice User Interface, no longer making it necessary to even look at your phone.

The last feature, allows you to for instance compose an email, while driving your car, or do anything else you would normally need your eyes and hands to do, without anything but a loudspeaker and a microphone.

A friend of mine, which tested the Magic menu said to me; “This is pissing on Siri”. I probably wouldn’t have used such strong words, but once you enable speech recognition and speech synthesis in your Magic menu, Siri from Apple basically appears like technology from the stone age. The Magic menu also features a very rich administrative dashboard, allowing you to easily configure and manage your items. You can see a screenshot of its backend below.

From its backend administrative interface, you can browse, view, delete and edit existing menu items, and create new items if you wish. You can also easily filter items, finding the exact item you’re looking for. And each menu item, can be localised and translated, into any language you need to support for your end user. Below you can see an example of editing an item, which happens to be localised for both English and Norwegian.

In the above screenshot, you can also see the code editor, for associating Hyperlambda with your menu items.

A menu item can be configured to be either a “Global” command, a “Private” command, or both. A global command, is a menu item that will be “sticky”, and always be available for the end user to choose. Regardless of which “dropdown” level the user has chosen to enter. These are commands that the user needs access to, regardless of where he is in his menu structure, and where he has navigated to. A private command, is a command that is only available for the currently logged in user. The latter typically includes administrative commands, that should not be available for all users, but only a “root” account in your system.

If the Hyperlambda for your menu item returns “false”, then no new stacking context is opened when the item is selected. If it does not return false, a new stacking context will be created, which becomes the equivalent of a “dropdown menu item” in a traditional menu system. The fact that the Magic menu is capable of creating such “stacking context”, happens to basically facilitate for it being an execution tree, or a decision tree, allowing the end user to literally have “conversations” with his computer, such as the following illustrates.

  1. User; Open a page!
  2. Computer; Which page?
  3. User; The one I created yesterday!
  4. … computer opens page …

Configuring the Magic menu can also be done during normal usage of your app. If you search for a phrase or sentence which yields no results, or speak a sentence or phrase if speech is enabled, and your phrase yields zero results – You will be given the option of creating a new command, if you are logged in as “root” in your system. Below is a screenshot demonstrating this feature.

This allows for instance a sys-admin, with some basic knowledge about programming, to configure the menu items, literally with his voice. Hence, resulting in that the burdening of configuring navigation, is taken away from the main developer, and left as an exercise to the customer. Effectively making all your apps become extremely “Agile”, and the process of creating your apps, ends up being completely “customer driven”. Imagine the following scenario.

Customer; I need a new button.
Developer; Speak to your computer!

The video below demonstrates one of the early versions of the Magic menu, and how it is interacting with the user, when speech is enabled.

I know that the Agile Manifesto explicitly claims that “Agile is not about the tools”. However, if you use the Magic menu, your apps literally becomes Agile by default – Allowing your customers, to easily modify almost any aspect of your apps, using nothing but their voice. See the video below for a demonstration of how this might occur.

The combination of the features shown in the above video, literally makes your apps “become alive”, and facilitates for your customers, being able to tie together the UX, such that they can decide themselves to a large extent how the app should behave, feel and interact with its users.

That’s Agile!