24 September 2016

Sharing download links to (hidden) HoloLens apps

The Windows store has a very neat feature. You can send out direct http links to people that, when entered, show and app directly in the web version of the store, with a neat button next to it to initiate install. For instance, if you hit this URL: https://www.microsoft.com/store/apps/9NBLGGH08D4P

It will take you directly to my app Map Mania.image

Even more handy is that this also works with hidden apps, so you can submit early versions of your app to the Store as hidden - and only hand out the link to a limited number of people. The store has more advanced methods for distributing betas these days, but as a low friction and easy way to send your POCs to customers this direct link feature is still very useful.

Unfortunately, this little trick does not work for HoloLens. For instance, my very first HoloLens app in the Store - “AMS HoloATC” - is accessible via URL: https://www.microsoft.com/store/apps/9NBLGGH52SZP

It will actually show you the app, but it will also say “This app does not work on your device”. Even on a HoloLens.

image

The solution for this is pretty simple – don’t use the http link, but use a store protocol link. Thus, you enter in Edge: ms-windows-store://pdp/?ProductId=9NBLGGH52SZP

And this will open the Windows Store App at the right place. And a button to install the app:

image

So you simply paste the product id behind “ms-windows-store://pdp/?ProductId=” and you can once again share links to selected audiences.

Credits go to my fellow MVP Tom Verhoeff who suggested trying this in an online conversation this afternoon, when I wanted him to try and download my app to see if it was available already. Incidentally, feedback on the app is also appreciated. In it’s current form it’s a one man spare time project. A video will appear shortly, and I will also document in detail how it’s built. Stay tuned.

09 September 2016

How shipping an UWP app update can make your app unavailable

First of all – don’t do this. You might regret it dearly.  

One might consider the fact that this is possible at all to be a bug in the Store. For the moment I am just operating on the assumption that I, a Windows Development MVP, actually managed to mess up my best paying app’s availability by being distracted and not reading the wording in the submission carefully. The sole purpose of this little blog post is to prevent you falling into the same trap.

I happily created a new version of my app now supporting the Anniversary Update, so it would be available on all clients (most notably the XBox One!) In the package page, for some reason I misread the line next to the top checkbox:

image

It says, now in gray print because the submission is already done:

“Let Microsoft decide whether to make this app available to any future device families”

The key error I made here was missing the word future. I had played a little with the check boxes above the platforms and then I noticed the text above, got distracted for some reason, misread it, and unchecked all the boxes thinking “better let Microsoft handle this choice”.

Wrong. Very wrong. Microsoft handles future device families, not current device families. Set the checkboxes like this – and what you end up with is a submission that is not eligible for any of the mentioned devices. And it says so if you read the legend below the package. What struck me in hindsight as odd is that the app was certified and published without a hitch with what might be considered as a completely senseless set op options. Anyway, the net result was - when you tried to search in from the store, it never showed up, and if I used the direct link ($0.99, free trial included, thank you for supporting your faithful developer) it said, in the browser:

"This app does not work on your device"

On every device.

Fortunately the good people of the Windows Store were nice enough to point out my error, so I resubmitted (just the same package, just all checkboxes checked now!). So this is how a submission is supposed to look, and it will look that way if you don’t mess with those checkboxes to begin with:

image

Then the Store Team still had to do something to boot my previous submission out of the queue. When that was done, the app became available again. After five days of absence.

So I guess I stumbled upon an edge for which case the people of the Windows Store could not even imagine some stupid enough to actually stumble upon. ;) Quite a sobering experience for me, both as a user and a developer – even when your users are intelligent people. somehow, some way they will find a way to click on the wrong button and mess up. Usually I am on the other side of the fence. I will never say again “how can anyone be so stupid to do XYZ” because th8ings like this happen.

This is not the proudest article of my blog, but I figured that if I could fall into this trap, there is a remote possibility other people would do so as well. Bottom line: be careful with these checkboxes, read carefully what they mean, and don’t get distracted during a submission ;)

10 August 2016

Why servicing UWP IoT apps via the store is such a Big Deal

In the past I have been dealing with IoT equipment made by a manufacturer that shall remain nameless here, but their solution was - like a lot of IoT solutions are today - based on Linux. Now granted, they had a quite nifty data exchange option via LoRa meshing. But updating both firmware and apps was a nightmare. You were constrained to your own network, so pushing out updates was your own responsibility, you had to do that per (sub) network, one by one, app by app.

Now this was (semi) professional sensing equipment, not intented for use by Joe and Jane Sixpack. The stuff that actually can be purchased by Joe and Jane is even in more dire straits. See for instance this horror story about smart locks. Or actually, dumb locks, as it turns out

Now first of all, a lot of these manufacturers are at fault for delivering essentially insufficient safe equipment. What's even worse is that they refuse to fix it. But in the long run, they are actually right about updating the lock software. It has very little sense, as most of the users won't update the lock (or whatever smart device they may have purchased anyway), either because they don't know how to do it, or because they rather watch the Olympics or some other sports event du jour in stead of reading obscure websites about security to keep up to date on the status of their smart lock, light bulb or whatever other gadget they bought (or got from a well intending friend or relative).

The only solution to this, of course, is that both the lock software itself and the firmware could be serviced remotely, without requiring the user to do something. This would of course require some kind of secure communication protocol, centrally guarded... kind of like how a computer or a phone and it's apps are updated. And wouldn't you know it, that is exactly what Microsoft are doing. Rather too quietly in my humble opinion. Maybe because it’s still in preview. But this is a big deal, and I think it deserves a lot more fanfare.

Windows 10 IoT Core can already update itself, so whenever Microsoft adds new features or improves overall security and stability, the base software can be updated without affecting what is running on top of it. Now by making the apps running on it servicable as well, Microsoft are providing the ultimate solution for making IoT devices servicable remotely and securely, without the user having to do anything.

Drawbacks? What if your lock is just about rebooting when you want to go out (or in)? And then there's the age old "quis custodiet ipsos custodes" - who guards the guards? You (and manufacturers) will have to decide whether or not they want to trust Microsoft - a company that has decades of security expertise, a enormous cloud infrastructure, and basically runs on selling trust - or just hope some random hardware dude does a better job and do it right the first time, because they cannot update their stuff once they have sold it.

I think the time is ripe for Smart IoT, and I applaud Microsoft for making this move. I once dreamed about it in a closed conversation with some Microsofties, and now it's coming true. Not doubt my dreaming has nothing to do with it, but the fact that it does come true indeed, is not the less awesome

The second blog post in a row without any code attached to it. My apologies, I will return to code next time ;)

03 August 2016

Why you should update your apps to UWP

Woe is us

No code this time, but a kind of a rant. Or some advice. Whatever you want to call it.

So we all saw the stories. Several pundits all over the world claim Windows Phone is going down, the Windows Store is going nowhere, it’s all going bust, blah blah blah doom and gloom, woe is us. I must admit I’ve been lazy converting my apps from Windows 8.x / Windows Phone 8.x to UWP too. Not out of defaitism, but, well, I got distracted. And for good reason too – there’s now so much fun stuff to play with. First there was IoT Core and some great IoT Azure features, then came HoloLens and UWP on XBox One - it’s hard to set priorities. Especially if you suffer from the shiny new toy syndrome like me ;).

A little confession

So far I have updated only one app to UWP, and submitted it to the store – good old Map Mania. It was my first serious Windows Phone application, dating back to the Windows Phone 7 days; it made it’s transition to 8, 8.1 and was languishing in the store. You can buy it for €0.99, although buying apps is an outdated model in these freemium days, right? And it was not even my most downloaded app – a little over 8000 downloads does not even come close to my more-or-less hit 2 Phone Pong. So why did I chose to update this app specifically? Well, mainly because I use it myself. It’s ability to show Open Street Maps is a great boon when making hikes in more rural places of Europe where the wife and me tend to go on holiday (although it was also great in finding spots around Rome). In addition, I think its ability to show WMS maps is also fun, a throwback to my years as a GIS programmer. And there’s sentimental reasons as well. Anyway, May 11th 2016 Map Mania Universal passed certification. Mission accomplished. I announced its existence with one tweet and forgot about it – because a HoloLens was on it’s way. That is why I am an engineer, not an entrepreneur– I know nothing of marketing, nor am I particularly interested in it.

Money talks. Numbers too

Now three months in, I browsed around some new Dev Insiders Pages on the Dev Center, and found these rather unexpected numbers in the payout pages:

image

Apparently, for every copy of Map Mania for Windows Phone 8.x, I am selling a little over 1.4 UWP copies. But well, ho hum, that’s nice. It’s not exactly breaking the bank.

I am going to share another number with you. Not one that I am very proud of, but what the heck:

image

The point is, look at the latest payout date. June 2015. In fourteen months I have sold for like €10 Map Mania 8.x. copies. But in three months I have sold UWP copies for an amount of  €14.69 . Ergo: I net about 71 cents per month on 8.x, and €4,90 per month on the UWP app. That’s almost seven times as much. It’s still quite not time to call the boss and give my month’s notice, but still - seven times as much. And these are new users - because if you upgraded from an older version, you keep owning it.

What the [redacted]?

How is possible in the light of all the doom and gloom stories? If I check the new download reports I see quite a few Windows 10 mobile acquisitions, a little less tablet acquisitions – but both of those are outnumbered by a bucket load of PC acquisitions. Whatever “PC” may be these days – since Surface saw the light of day, the borders between a laptop and a tablet have become quite hazy.

I’ve been discussing this on Slack with a few people, and with my fellow MVP Ginny Caughey, and we think there are apparently three things at work

  • Relative small store = relative high visibility
  • The huge number of Windows 10 on PC installs is apparently kicking in
  • Those new users are apparently willing to put down money.

Long story short

Update your apps! There is life (and money) in the UWP market. Learn new skills, earn a few bucks, and who knows, maybe you will sell an app to a HoloLens user. But an ‘ordinary’ PC user’s euros, dollars or whatever are currency too, right? There quite a lot more of those.

…and they are willing to pay, apparently.

26 July 2016

HoloLens CubeBouncer application part 5-using gravity, adding more speech & sound, and app icons

Preface

imageIn the previous post I showed you how to integrate speech recognition. This post mostly builds upon stuff we now already know. It introduces actually one new concept. From the very first post you might recall I instructed you to turn off gravity in the rigid body component. Hence, the cubes float in the air. But they still have mass, so like any body a cube has inertia. You will notice that when you bounce two cubes – they keep on moving, although they slowly lose speed (because we also defined drag). If you were to set drag to zero, you get the effect as if the cubes move through vacuum – they just keep on moving, as Sir Isaac Newton described in the first of his three laws of motion. So if you were to suddenly switch on gravity, the floating cubes would drop to the floor. Of course, in real life you would not be able to switch gravity on and off at will, but in AR magic is just a matter of setting the right property. And that is exactly what we are going to do.

Down she goes!

We open the CubeManipulator script, and add a one line method:

public void OnDrop()
{
  _rigidBody.useGravity = true;
}

Did I mention using Unity feels a lot like cheating? I did, right? We also need to add a line to the top of the OnRevert method, or else the cube will move to it’s old position when we call “go to start”, but after that it will immediately drop to the floor again. That’s not what we want. when the cube returns, we turn on gravity off for that particular cube.

public IEnumerator OnRevert()
{
  _rigidBody.useGravity = false;
  //rest of method

And then it’s a matter of adding a new command to the SpeechManager:

if (cmd == DropCommand)
{
  if (GazeManager.Instance.Hit)
  {
    GazeManager.Instance.HitInfo.collider.gameObject.SendMessage("OnDrop");
  }
}

Of course you have to define “DropCommand” and add it to the the keyword recognizer like in the previous post, but that’s all. Rebuild the project from Unity, deploy the Visual Studio solution, look at a cube, say “drop” and it falls to the floor. And if you say “go to start” it will still go back to it’s old position and stay there. Easy, right?

Dropping and returning all cubes

By now I think you won’t find this very hard to understand anymore. First, we need actually implement the code in the MainStarter, as this is the object that knows which cubes are available:

public void RevertAll()
{
  foreach (var c in _cubes)
  {
    c.SendMessage("OnRevert");
  }
}

public void DropAll()
{
  foreach (var c in _cubes)
  {
    c.SendMessage("OnDrop");
  }
}

Every cube knows where it came from, so we only have to call “OnRevert” on every cube. In the same line, calling “OnDrop” for every cube will drop all cubes. And in the SpeechManager we just add two commands to recognize: “drop” and “total recall”, or whatever for phrases you choose for these commands. When the phrases are recognize, simply call the right method:

if (cmd == RevertCommand)
{
    _mainStarter.RevertAll();
}

if (cmd == DropAllCommand)
{
   _mainStarter.DropAll();
}

Of course you have to define the RevertCommand and the DropCommand fields again and add them to the KeywordRecognizer, but once again – very little code, powerful functionality. Clever use of and already very clever the SDK goes a long way.

Adding some missing sounds

imageSo in the video I showed in the first post, “create new grid” gave a kind of ping-sound, a returning cube a whistling sound, and “total recall” made a kind of  “tadadaah” sound. I find these kind of affirmative sounds very useful for giving feedback to the user that the app understands you, although in practice you might go for a little less garish sounds than I did ;). 

I added three sounds to Assets/Custom/Audio in Unity, then added an audio source to the Managers game object. I only changed the settings “Play on awake” (to off) and Volume (to 0.5)

Then we proceed to MainStarter.cs, and we add two public AudioClips fields and a private AudioSource clip:

public AudioClip ReadyClip;

public AudioClip ReturnAllClip;

private AudioSource _audioSource;

The Audio source of course needs to be initialized in Start:

_audioSource = GetComponent();

At the top of the CreateGrid method we add one line:

private void CreateGrid(Vector3 hitPosition)
{
  _audioSource.PlayOneShot(ReadyClip);

And we do something similar at the top of the RevertAll method

public void RevertAll()
{
  _audioSource.PlayOneShot(ReturnAllClip);

Go back to Unity, drag the Audio assets “Ready” and “Return all” on top of the Main Starter Script in Managers. We have done that a couple of times already but the be sure, one more time a picture that shows what I mean here:

image

Then rebuild the project, re-deploy from Visual Studio, and sure enough you will hear the sounds the “pringgg!” sound when the grid is created (at app start-up, and when you say “create new grid”), and the “tadadaah” when you say “totall recall”

Then for the final sound – the whistling sound the cube makes when it returns. Well, that’s almost the same, only now we go to the CubeManipulator script, and add a field:

public AudioClip ComeBackClip;

And on top of the of the OnRevert method we just need this?

public IEnumerator OnRevert()
{
  _audioSource.PlayOneShot(ComeBackClip);

Go back to Unity, select the WortellCube prefab and drag the “Return” Audio assets on top of the field “Come Back Clip” that now has appeared on the CubeManipulator script component. Rebuild your project, deploy the Visual Studio solution and indeed, the cube now returns with a whistling sound. when your say “go to start”.

The problem now only is – when you say “total recall” now, the app will play the “tadadaah” sound and a lot of time the whistling sound simultaneously, resulting in quite a cacophony. After all, we are calling the OnRevert method for every single cube. This is not a good user experience. So we will have to change a few things.

Some tweaks to improve the sound experience

We need some way to tell the cubes not to play their sound on “total recall”. To that end, we first need to change the OnRevert method as follows:

public IEnumerator OnRevert(object doPlaySound)
{
  if ((bool) doPlaySound)
  {
    _audioSource.PlayOneShot(ComeBackClip);
  }

Then we go back to the SpeechManager, and in the method KeywordRecognizer_OnPhraseRecognized we add “true” to the OnRevert message that is fired when “go to start” is recognized

if (cmd == GoToStartCommand)
{
  if (GazeManager.Instance.Hit)
  {
    GazeManager.Instance.HitInfo.collider.gameObject.SendMessage("OnRevert", true);
  }
}

In contrast, in the MainStarter script we need to add false in OnRevert:

public void RevertAll()
{
  _audioSource.PlayOneShot(ReturnAllClip);
  foreach (var c in _cubes)
  {
    c.SendMessage("OnRevert", false);
  }
}

imageAnd now if you say “totall recall” you will once again only hear the “tadadaa”, while a “go to start” still lets one cube return with the whistling sound.

Tiles and naming

I have added 5 files to the “Assets/Custom/UWPAssets” folder in Unity (see right). These are all 200% assets. So what you need to do is:

  • Hit File/Build settings/Player settings
  • Go to the right, to the inspector.
  • Expand the Icon pane
  • Scroll all the way down to “Universal 10 tiles and Logos”

image

  • Expand the Square 44x44 Logo pane
  • Scroll down to the “Scale 200%, (88x88 pixels)”
  • Hit the “select” button in the square on the right
  • Select “Square44x44Logo” from the popup

Your net result should be this.

image

image

Repeat this for the other for logo formats. When you are done and you collapse all the image panes, they should all have the suffix”(200)” now.

 

 

Finally, we scroll all the way to the top again, and change the following settings:

  • Change the company name into whatever you like – I took “LocalJoost Ltd.” This not not mandatory
  • Change “Product name” to “CubeBouncer”
  • Hit the “Select” button on the “Default icon” square and once again select the 310x310 square logo
  • Under “Icon”, change short name into “CubeBouncer”
  • Check “Large tile” and “Wide tile” under “Show name on”.

Net result:

image

At this point it’s best to close Visual Studio, and delete the generated solution from the App folder. We have effectively changed the app title, yet you will notice that it does not have any effect if you keep the solution. If you generate the solution anew, you will see it’s name is different too – it’s now called CubeBouncer.sln, no longer CubeBouncerDemo.sln. If you deploy it to a HoloLens, you will see either this in the all apps menu

image

and this one when you have pinned it:

image

And that’s it! Were done!

Concluding remarks

After setting up Unity, adding air taps, force, spatial sound, voice recognition, gravity and some imagery we have a simple but functional HoloLens app that demonstrates a lot (but not nearly all) of the HoloLens interaction model. I hope you enjoyed this trip, I sure had a lot of fun building and documenting it. It was a very educating experience for me and I found it a very fitting subject for the 256th post on this blog ;). Should you have any remarks, questions or improvements, let me know.

The source code for the final project can be downloaded here.

Oh… There is still a very minor issue with it, though. I wonder if anyone actually finds it. András Velvárt is excluded from competition ;). Let me know if you find it. 

23 July 2016

HoloLens CubeBouncer application part 4-integrating speech commands and moving cubes by code

Preface

In the previous post I showed how you could interact with the cubes using air taps, utilizing the physics engine. In this blog post I am going to show how to move the cubes by code (bypassing the physics engine) – and doing so using speech commands.

But first…

The funny thing is - when retracing you steps and documenting them, you are found out things can be done in a smarter way. It’s like giving a code review to your slightly younger, slightly less knowledgeable self. In post 2, I state you should drag the MainStarter script onto the HologramCollection game object. Although that works, in hindsight it’s better to put that under the HologramCollection/Managers object. So please remove the script from the HologramCollection object, drag it anew from your assets on top of the Managers object, then drag the WortellCube prefab on top of the Cube field again.

HoloLens speech recognition 101

The speech recognition API for HoloLens in Unity3D is so simple that there’s basically not much else than 101. I had a look at Rene Schulte’s HoloWorld speech manager and was like… is that all? Well, apparently it is. So I created my own SpeechManager script, and added it to the Managers object. It’s basically a modified copy of Rene’s. Why re-invent the wheel when people smarter than yourself already have done so, right?

The speech manager at this point implements only two commands:

  • “create new grid”
  • “go to start”

and looks like this:

using UnityEngine;
using HoloToolkit.Unity;
using UnityEngine.Windows.Speech;

public class SpeechManager : MonoBehaviour
{
  public string GoToStartCommand = "go to start";

  public string NewGridCommand = "create new grid";

  private KeywordRecognizer _keywordRecognizer;

  private MainStarter _mainStarter;

  // Use this for initialization
  void Start()
  {
    _mainStarter = GetComponent<MainStarter>();
    _keywordRecognizer = new KeywordRecognizer(
      new[] { GoToStartCommand, NewGridCommand });
    _keywordRecognizer.OnPhraseRecognized += KeywordRecognizer_OnPhraseRecognized;
    _keywordRecognizer.Start();
  }

  private void KeywordRecognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
  {
    var cmd = args.text;

    if (cmd == NewGridCommand)
    {
      _mainStarter.CreateNewGrid();
    }

    if (cmd == GoToStartCommand)
    {
      if (GazeManager.Instance.Hit)
      {
        GazeManager.Instance.HitInfo.collider.gameObject.SendMessage(
          "OnRevert");
      }
    }
  }

  private void OnDestroy()
  {
    if (_keywordRecognizer != null)
    {
      if (_keywordRecognizer.IsRunning)
      {
        _keywordRecognizer.Stop();
      }
      _keywordRecognizer.Dispose();
    }
  }
}

In short – when the keywords “go to start” are recognized, “OnRevert” is called on the game object that you are looking at. We have seen this kind of message sending in the previous post already.If you say “create new grid” the CreateNewGrid method is called. This tries to find the MainStarter class as a component (it being at the same level in the Managers game object, it fill find it) and call the method directly. But neither methods are implemented, you will even notice the squiggly lines under CreateNewGrid. So let’s tackle that first, because now our project does not even compile.

(Re)creating a grid.

Creating a new grid is, simply put, deleting the old cubes are creating a new set. This implies that we must know which cubes are present now, something we don’t know now. This actually requires very little extra code:

private readonly List<GameObject> _cubes = new List<GameObject>();

public void CreateNewGrid()
{
  foreach (var c in _cubes)
  {
      Destroy(c);
  }
  _cubes.Clear();

  _distanceMeasured = false;
  _lastInitTime = DateTimeOffset.Now;
}

Very simple – all cubes created are stored in a list. So we need to destroy them one by one, and then we set the _distanceMeasured and _lastInitTime back to their start value – and the Update method that is called once per frame will do the rest. We also have to add one extra line to the CreateCube method in order to collect the created cube, at the end:

private void CreateCube(int id, Vector3 location, Quaternion rotation)
{
  var c = Instantiate(Cube, location, rotation) as GameObject;
  //Rotate around it's own up axis so up points TO the camera
  c.transform.RotateAround(location, transform.up, 180f);
  var m = c.GetComponent<CubeManipulator>();
  m.Id = id;
  _cubes.Add(c);
}

imageSo now the only thing you need to do is drag the SpeechManager script on top of the Managers object too. When you are done, the Managers object should look like you see at the right.

Rebuild the UWP app from Unity, and deploy it using Visual Studio. And there we are. When you said “create new grid” the grid is immediately updated. Still without the ringing sound that shows in my initial video, but we will add that in a later stage.

Sending a cube back to where it came from

As we saw, we already implemented a call to an OnRevert method in the Speech Manager to recall the cube we are looking at to its original position, but we have not implemented it. To this intent, a cube needs to know where it came from and how it was rotated when it was created. So we add the following private fields to the cube:

private Vector3 _orginalPosition;

private Quaternion _originalRotation;

And we set those fields in Start by reading the information from the transform:

void Start()
{
  _rigidBody = GetComponent<Rigidbody>();
  _audioSource = GetComponent<AudioSource>();

  _orginalPosition = transform.position;
  _originalRotation = transform.rotation;
}

So now we need to implement the “OnRevert” method itself like this:

public IEnumerator OnRevert()
{
   var recallPosition = transform.position;
   var recallRotation = transform.rotation;

  _rigidBody.isKinematic = true;

  while (transform.position != _orginalPosition &&
         transform.rotation != _originalRotation)
  {
    yield return StartCoroutine(
      MoveObject(transform, recallPosition, _orginalPosition,
                 recallRotation, _originalRotation, 1.0f));
  }

  _rigidBody.isKinematic = false;
}

So first we we make sure we retain the current position and rotation. Then we set the isKinematic property of the rigid body to true. This effectively turns off the physics engine, so now we can move the cube ourselves. And then we loop over a so-called coroutine until the cube is back to it’s original position and rotation. I think of it as Unity’s equivalent of an async operation. Basically it says – animate the current transform smoothly from the current position to the original position, and at the same time rotate it from it’s current rotation to the original rotation, in 1.0 second. The coroutine itself is implemented like this:

// See http://answers.unity3d.com/questions/711309/movement-script-2.html
IEnumerator MoveObject(Transform thisTransform, Vector3 startPos, Vector3 endPos, 
  Quaternion startRot, Quaternion endRot, float time)
{
  var i = 0.0f;
  var rate = 1.0f / time;
  while (i < 1.0f)
  {
      i += Time.deltaTime * rate;
      thisTransform.position = Vector3.Lerp(startPos, endPos, Mathf.SmoothStep(0f, 1f, i));
      thisTransform.rotation = Quaternion.Lerp(startRot, endRot, Mathf.SmoothStep(0f, 1f, i));
      yield return null;
  }
}

How this works, is – as you can read in the source – is explained here. That sample only applies to moving an object - I have applied that knowledge to both moving and rotating. It basically is a way to smooth out the animation – if you  pay close attention, you will see that the cube starts slows, speeds up very fast, and then slows down again. I must admit that I am quite missing some of the finer points myself still, but this is how it can be done. Important, by the way, is that isKinematic gets set to false again once the cube is back on its place. Fun detail – if a cube that is moving back to it’s start position hits another cube, it is bumped out of the way, because for the cube that is hit, the physics engine is still working ;)

Finally, at the top of the OnCollision method we need to make sure returning objects don’t interfere with other cubes in terms of making sound and other stuff.

void OnCollisionEnter(Collision coll)
{
  // Ignore returning bodies
  if (_rigidBody.isKinematic) return;

And now, if you say “go to start” when you are looking at a specific cube, it will move to the location it came from.

Concluding remarks

In the previous post I have showed you how to move objects using the physics engine - this post has showed you have how to integrate speech commands and move objects via code – remarkably little code, yet again. Once again, you can see the code of the project so far here.

13 July 2016

HoloLens CubeBouncer application part 3-air tapping the cubes away and adding spatial sounds

Preface

In the previous post I showed you how to create a neatly arranged grid aligned to your view in your HoloLens app. Now, it’s time to make a mess of it – I am going to show you how to employ air tap on the cubes to move the them around, and gaze to determine which way they go. Plus, we are going to add some sound to them – when they bounce against each other, and against the wall.

Steal some sounds

image

First of all, we need two short sound clips. One for two cubes hitting each other, one for a cube hitting a wall. Any clip will do, as long as it’s short. I took two from the “Free Casual Sounds SFX Pack” that you can find in the Unity Asset Store (Click Windows/Assets Store or hit CTRL+9)

 

 

imageBeware: importing directly from the store can get you more than you have bargained for, bloating your project with unnecessary stuff. I tend to create a new Unity3d project and import the package there first, to see what happens. And in this case I just cherry picked two sounds, that I called BounceCubel.wav and BounceBall.wav. Then I dragged into the Assets/Audio folder in Unity, as displayed here to the left.

Now it’s time for coding again. That is to say …

Steal some scripts

Writing code is awesome – not having to write code is even better. In the HoloLens toolkit there’s a script that almost does what we want. It’s called “GestureManager” and it’s in Assets\HoloToolkit\Input\Scripts. It’s a script that handles the air tap and sends a kind of message to a selected object. That is almost what we want. We need to copy and adapt it a little.

The next few steps are best done when Unity3d is not running, as it starts to parse scripts as soon as they appear in the folder:

  • Copy Assets\HoloToolkit\Input\Scripts\GestureManager.cs to. Assets\Custom\Scripts
  • Rename the file to RaySendingGestureManager.cs
  • Open RaySendingGestureManager.cs and change the class name to RaySendingGestureManager
  • Change “Singleton<GestureManager>” into “Singleton<RaySendingGestureManager>”
  • Remove the namespace namespace HoloToolkit.Unity
  • Add “using HoloToolkit.Unity” on top.
  • Find the method GestureRecognizer_TappedEvent
  • Change it to the following:
private void GestureRecognizer_TappedEvent(
     InteractionSourceKind source, int tapCount, Ray headRay)
{
    if (focusedObject != null)
    {
        focusedObject.SendMessage("OnSelect", headRay);
    }
}

Basically, the only thing you do, is pass on the headRay “Ray” as extra parameter to the focusedObject.SendMessage method. This means that Unity3d should search for a Script component in the selected GameObject, find a method “OnSelect” in there with a a single parameter of type object - and invoke that. This feels a bit Javascripty, only even less typed, and there is no punishment, either: if the method is not found, nothing happens. No error – just nothing. It that sense it’s like sending a message indeed – if no-one is listening, we don’t care. Talk about loose coupling. You can’t get much looser than this ;)

Why do we need to send the headRay? Because we want the selected object to know from which direction it’s being looked at when you tap. If you want to know the details of air tap, have a look at the rest of the script. For now, we should be content that there is a method called on the selected object – i.e. the WortellCube – when there is an air tap while the cursor is on it.

One more thing – we need to apply the script. Select the HologramCollection/Managers object in the Hierarchy pane, then hit the “Add Component” button, Type “Ray Sending” in the search box and add the Ray Sending Manager component. Net result should be this:

image

Message sent – now catch it!

Okay, so whenever someone air taps while their gaze is at ‘something’, Unity3D tries to call on “OnSelect” method. Let’s make sure that if that something is a WortellCube, it listens to it. Start as follows:

  • Select the “Assets/Custom/Scripts folder
  • Right click, hit Create/C# script
  • Call it “CubeManipulator”
  • Select the WortellCube Prefab
  • Drag the new script on top Inspector tab, below the “Add Component” button.
  • Hit File/Build Settings/Build and build the solution
  • Open the Visual Studio solution (or reload it)

Open the CubeBouncer.cs script, and change it so it looks like this:

using HoloToolkit.Unity;
using System.Collections;
using UnityEngine;

public class CubeManipulator : MonoBehaviour
{
    private Rigidbody _rigidBody;

    public int ForceMultiplier = 100;


    // Use this for initialization
    void Start()
    {
        _rigidBody = GetComponent<Rigidbody>();
    }

public void OnSelect(object ray) { if (!(ray is Ray)) return; var rayData = (Ray)ray; _rigidBody.AddForceAtPosition( new Vector3( rayData.direction.x * ForceMultiplier, rayData.direction.y * ForceMultiplier, rayData.direction.z * ForceMultiplier), GazeManager.Instance.Position); } } }

Since the script is now part of a compound object, I can easily get a reference to other components in that objects. In this case, we can get a reference to the RigidBody component. For performance reasons (and because I find it more tidy looking), we store that in a private field that is initialized in the Start method.

And then there’s the OnSelect method. After first checking if we indeed get a Ray supplied in the ray parameter, we simply call the RigidBody’s AddForceAtPosition with a Vector3D made out of the direction of the Ray, times 100, and the position where the cursor is now. And that’s it. Unity gives the Cube a push in the direction you are looking on the place where your gaze is locked on the cube. And off it goes, spinning into the void, bouncing off other cubes and walls. Notice the cubes slowly slow down by themselves. That is all caused by the physical characteristics – the bouncyness of the Physic Material, the weight and drag of the Rigid Body. Having written software that calculates bouncing at angles myself, it almost feels like cheating. But this is the awesome power of a fleshed out physics engine.

To see this stage you actually have to rebuild the project from Unity first again or you will get an “[Position out of bounds!]” error when you try to run the app on a HoloLens or the emulator. This is because we have added a public field again, which translates into a property that can be filled by Unity. Every time you add public field to a script you will have to rebuild the project from Unity.

There is a still a thing missing. The cubes bounces off each other in dead silence. Time to fix that, and make the experience more immersive.

Let’s make some noise!

First, we need to add three public and one private field to the CubeManipulator script:

private AudioSource _audioSource;

public int Id;

public AudioClip BounceTogetherClip;

public AudioClip BounceOtherClip;

_audioSource gets initialized in Start as well:

void Start()
{
  _rigidBody = GetComponent<Rigidbody>();
  _audioSource = GetComponent<AudioSource>();
}

And then we have to act on a collision. That is just another simple private method that is automatically called by Unity. And wouldn’t you know it, it’s called “OnCollision” – duh ;). So let’s add that to the script:

void OnCollisionEnter(Collision coll)
{
  // Ignore hits by cursors
  if (coll.gameObject.GetComponent<CursorManager>() != null) return;

  // Play a click on hitting another cube, but only if the it has a higher Id
  // to prevent the same sound being played twice
  var othercube = coll.gameObject.GetComponent<CubeManipulator>();
  if (othercube != null && othercube.Id < Id)
  {
    _audioSource.PlayOneShot(BounceTogetherClip);
  }

  // No cursor, no cube - we hit a wall.
  if (othercube == null)
  {
    if (coll.relativeVelocity.magnitude > 0.1)
    {
      _audioSource.PlayOneShot(BounceOtherClip);
    }
  }
}

So if a cube collides with a GameObject that has a CursorManager – it’s a cursor, so ignore it. But if it has a CubeManipulator - it’s a cube too! Then play the BounceTogetherClip AudioClip. But only if the Id of this cube is lower than the cube we hit – to prevent double sounds. If it was not a cursor or another cube, it was most likely a wall – so play the BounceOtherClip, but only if the cube hit the wall ‘hard enough’.You do this by checking the collision magnitude. I took a quite arbitrary value. Once again – it feels like cheating. Unity takes care of mostly everything.

Aside – don’t try to be smart and change “if (othercube != null && othercube.Id < Id)” into
if (othercube?.Id < Id)”. It will compile, it will run, and it will also mess up Unity3d. Avoid C# 6 constructs.

We only need to do a couple of things more to not only see but also hear our app at work. too. Save the script, then go back to Unity. Open the WortellCube prefab again, scroll down to the CubeManipulator script component. You will see – like expected – the script has three extra properties now: Id, Bounce Together Clip and Bounce Other Clip. So drag the Audio asset “BounceCube” on top of the Bounce Together Clip field, and BounceWall on top of Bounce Other Clips. Net result should be this:

image

Build the project again, and go back to Visual Studio. There is that tiny thing about the Id we have to solve. Because now we have an Id, but nothing is set yet. Remember that in the previous post the method CreateCube in MainStarter.cs had an id parameter that was not used? Well, now we are going to use it. We change that method a little by adding two lines:

private void CreateCube(int id, Vector3 location, Quaternion rotation)
{
  var c = Instantiate(Cube, location, rotation) as GameObject;
  //Rotate around it's own up axis so up points TO the camera
  c.transform.RotateAround(location, transform.up, 180f);
  var m = c.GetComponent<CubeManipulator>();
  m.Id = id;
}

And thus we assign the Id we need to make only one cube play the bounce together sound when two cubes hit. Deploy the project to a HoloLens or an emulator, and the cubes click when they bounce together, or emit a kind of boom when they hit the wall:

Also, if you have an actual HoloLens, try turning your head after launching some cubes – you will hear the sound actually coming from the right direction. This is because we have included an AudioSource component configured for Spatial Sound in the WortellCube prefab - and that travels along with the cube. Once again – it feels like cheating, but it works.

Concluding remarks

We have learned to intercept air taps, and apply directed force to a rigid body – making cubes all bounce by themselves (or actually, by Unity. We have also learned to act on collisions, and play spatial sound. Still by writing very, very little code. Next time – speech command and moving stuff yourself!

You can find, as always, the completed project so far here.