Developing for Oculus Rift DK2

I've purchased an Oculus Rift Development Kit 2. In a sense it feels like going back to my thirties. Much of my professional career was in Virtual Reality. I can't remember for how long exactly, but I think it was for 8 years or so, that I developed software and ran VR projects for SARA's CAVE. See below for a picture or me, mining genomics data that we visualized for Johnson&Johnson.

Since 2007, my career has been in Video Games. Even though I had been making games since 1982, it took me 25 years of hobby game development before I decided to do this for a living. And since 2010 I do this independently as a private enterprise. Back to 2014: my two careers have crossed, and I find myself actually developing Games for Virtual Reality. A younger me would have labeled this as heaven, but the years at SARA have made me cautious about the adoption rate of VR. This current VR revival could be another one that is destined to go out like a candle.

This web page will function as a development log. I intend to document peculiarities, problems and solutions that are bound to pop up during DK2 development. If you just want to play my VR game, download The Little Plane That Could.

Windows Driver issues

• When using the 'Direct' Rift Display Mode, I get extreme chromatic shifts. The different channels R/G/B are rendered all over the place, not converging what so ever. 'Extended Desktop' display mode works better.
• The SDK precompiled example runs badly, the jitter is horrible and is only alleviated a bit if I toggle multi sampling. It's hard to say which mode (ON or OFF) reduces jitter, because both look exactly the same, with the same level of aliasing.

Development under Windows

• After installing windows runtime and sdk, the latest firmware can be flashed to the device.
• The SDK example 'Oculus Room Tiny' is intended as the minimalistic code to base your first VR app on. It comes with a Visual Studio 2013 solution. Unfortunately this does not build out of the box. But what is worse: it is a D3D app, and not an OpenGL based app. The SDK seems to support both, but unfortunately the sample is in D3D only. Bummer!
• Building Oculus Room Tiny yields a: Cannot open include file: 'AtlBase.h'
• Linking the sample also fails: fatal error LNK1104: cannot open file 'atls.lib'
• To fix these compiler and linker errors, you need to install a Windows Driver Kit, as described on Stack Overflow.
• If you want to link your own app against libOVR, you need to add these to the linker input: libovr.lib (libovrd.lib for Debug), ws2_32.lib and winmm.lib.
• The Rift display mode (Extended Desktop vs Direct) seems to mess with OpenGL context creation. When 'Extended Desktop' is enabled, SDL_GL_CreateContext() can create a 3.2 Core profile context. If I ask for a 3.2 Core profile when the mode is set to 'Direct', then SDL_GL_CreateContext() crashes with an access violation at address 0.

Development under GNU/Linux

• Let's start with some good news: Oculus intends to release DK2 SDKs for Windows, OSX and Linux. The bad news: so far no Linux SDK for the DK2 has appeared. It was labeled as 'soon' but has been delayed quite a bit now.

General Development issues

• You can create the projection matrix for each eye with a call to ovrMatrix4f_Projection() but you need to transpose the result before using.
• You can build the view matrix for each eye using the eyePose from ovrHmd_GetEyePose(). Strangely enough you still need to shift the position of the eye with the intra occular distance. Like the project matrix, the view matrix need transposing as well.

Routers with restrictive NAT spoiling online gameplay.

I have published an online tank simulator for both iOS and Android. On the app store it is called Buggy Bang Bang! and was renamed as Six Wheels and a Gun on Google play. Players on iOS and Android can even play each other, as they are all pooled on the same lobby server. I wrote the lobby server code myself using Python.

A neat thing about my networking code (C for the clients, Python for the lobby server) is that it is 100% UDP based. There is not a single TCP connection used, just some UDP packets that get exchanged with both server and opponent's client. Two players playing my game are typically on a LAN behind a router that does NAT (Network Address Translation) and this complicates things. It means that the only way they can connect to each other is to have them both connect to a server on the internet, with a public IP. This server will then see datagrams coming in on some random ports, and can tell both parties what these ports are. This is the concept behind UDP hole punching.

Things go wrong if your router is too restrictive. It may do Symmetric NAT, or Restricted Cone NAT. If this happens, the players will not find each other on the ports as described by their common server. In my game, it will result in two buggies that refuse to move and after 12 seconds, the match will time out and record this as a 'forfeit' by the other party. Moral of the story: if you play my game and the HUD controls don't show up with the buggies just sitting there, then your router or your opponent's router will not play ball. So far my approach to this is: too bad it happens, better luck next time. If you want, read the manual of your router and change the NAT type.

UPDATE

I've been adding a new feature to S.W.a.a.G. that detects restrictive NATs. Instead of pairing up a user with a bad NAT, we nominate two players for a session. If one of them fails to punch a hole in the firewall, the match will not be started. The players are informed of the situation with either of these two messages:

• YOUR NAT SEEMS TOO RESTRICTIVE.
• OPPONENT NAT SEEMS TOO RESTRICTIVE.

So at least the players will know what is going on. If it is your opponent, you can just retry, and hopefully get paired with another user.

OpenAL with Android NDK.

This posting is for future reference, if I ever have to get OpenAL working again under Android's NDK. It will also be useful to my peer game developers who intend to do the same.

Step 1

Get an Android port of openal-soft.

Step 2

I like to build static libraries for android game dev, instead of dynamic ones. So I replaced the jni/Android.mk with my own:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

OPENAL_DIR := OpenAL

AL_SOURCES := \
  $(OPENAL_DIR)/Alc/android.c              \
  $(OPENAL_DIR)/OpenAL32/alAuxEffectSlot.c \
  $(OPENAL_DIR)/OpenAL32/alBuffer.c        \
  $(OPENAL_DIR)/OpenAL32/alDatabuffer.c    \
  $(OPENAL_DIR)/OpenAL32/alEffect.c        \
  $(OPENAL_DIR)/OpenAL32/alError.c         \
  $(OPENAL_DIR)/OpenAL32/alExtension.c     \
  $(OPENAL_DIR)/OpenAL32/alFilter.c        \
  $(OPENAL_DIR)/OpenAL32/alListener.c      \
  $(OPENAL_DIR)/OpenAL32/alSource.c        \
  $(OPENAL_DIR)/OpenAL32/alState.c         \
  $(OPENAL_DIR)/OpenAL32/alThunk.c         \
  $(OPENAL_DIR)/Alc/ALc.c                  \
  $(OPENAL_DIR)/Alc/alcConfig.c            \
  $(OPENAL_DIR)/Alc/alcEcho.c              \
  $(OPENAL_DIR)/Alc/alcModulator.c         \
  $(OPENAL_DIR)/Alc/alcReverb.c            \
  $(OPENAL_DIR)/Alc/alcRing.c              \
  $(OPENAL_DIR)/Alc/alcThread.c            \
  $(OPENAL_DIR)/Alc/ALu.c                  \
  $(OPENAL_DIR)/Alc/bs2b.c                 \
  $(OPENAL_DIR)/Alc/null.c                 \
  $(OPENAL_DIR)/Alc/panning.c              \
  $(OPENAL_DIR)/Alc/mixer.c                \
  $(OPENAL_DIR)/Alc/audiotrack.c           \
  $(OPENAL_DIR)/Alc/opensles.c


LOCAL_MODULE    := openal
LOCAL_SRC_FILES := $(AL_SOURCES)

LOCAL_C_INCLUDES := \
  $(HOME)/src/openal-soft/jni/OpenAL \
  $(HOME)/src/openal-soft/jni/OpenAL/include \
  $(HOME)/src/openal-soft/jni/OpenAL/OpenAL32/Include \


LOCAL_CFLAGS += \
  -DAL_ALEXT_PROTOTYPES \

MAX_SOURCES_LOW ?= 4
MAX_SOURCES_START ?= 8
MAX_SOURCES_HIGH ?= 64

LOCAL_CFLAGS += -DMAX_SOURCES_LOW=$(MAX_SOURCES_LOW) -DMAX_SOURCES_START=$(MAX_SOURCES_START) -DMAX_SOURCES_HIGH=$(MAX_SOURCES_HIGH)
LOCAL_CFLAGS += -DPOST_FROYO

include $(BUILD_STATIC_LIBRARY)

Step 3

Get an Android port of freealut.

Step 4

Add an Android.mk file to src/ directory containing:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE := alut

LOCAL_SRC_FILES := \
  alutBufferData.c \
  alutCodec.c \
  alutError.c \
  alutInit.c \
  alutInputStream.c \
  alutLoader.c \
  alutOutputStream.c \
  alutUtil.c \
  alutVersion.c \
  alutWaveform.c

LOCAL_C_INCLUDES := \
  $(HOME)/src/freealut/include \
  $(HOME)/src/openal-soft/jni/OpenAL/include \

#LOCAL_CFLAGS +=

include $(BUILD_STATIC_LIBRARY)

Step 5

Some config.h stuff does not properly make it into the source. Not sure why, so I manually edited alutInputStream.c alutInternal.h and alutUtil.c to overcome this.

Step 6

Now you can pull these static libraries into your build by adding to your project's Android.mk file something like this:

LOCAL_STATIC_LIBRARIES += alut openal
...
$(call import-module,openal-soft/jni)
$(call import-module,freealut/src)

UPDATE #1

When porting to MSWindows, I found that OpenAL is pretty tricky there as well. Some info for future reference:

• You need to link against libOpenAL32.dll.a
• You need to supply soft_oal.dll with your game, renamed as OpenAL32.dll
• As with Android and OSX, the CMake stuff of freealut is horribly broken, so you need to pull in the sources to your project and hack around the platform specific stuff.

UPDATE #2

When building for 64bit arm, or arm64-v8a as it is known, I get crashes (SIGSEGV) in alSourcePlay. Maybe openal-soft is not 64 bit safe? The android port is pretty old, as the last change is from 2012.

Flight Simulation on touch screens.

My current project is biplane combat simulator. I have been developing it under GNU/Linux, and porting it to Android. It needs a commercially viable platform to run on, and I am pretty sure GNU/Linux games are not making money at the moment. This may change with the Steam box, but for now, I have to look at iOS, Android or maybe a console platform.

After porting my biplane game to Android, I learned a surprising fact: When it comes to flight simulators, there is absolutely no substitute whatsoever for a physical analog joystick. You really need the self-centering stick with plenty of instant feedback if you want to control an airplane.

The first substitute I tried was touch control, with a virtual joystick on the screen. I added snapping-to-center when you release your touch, so that there is some sort of self-centering going on. But this was not enough to make the plane controllable. The lack of tactile feedback means that you have little awareness of the absolute position of the virtual joystick without looking at the control graphics.

I may need to redo this touch control experiment with iOS though. Android is notoriously bad when it comes to responsive touch control. Try this for fun: swipe some content (web page, e.g.) up and down at roughly 5Hz or so. You will see that the motion of the content is opposite of the motion of your finger. If your finger goes up, the content still goes down from your swipe 100ms ago.

I thought that tilt-control (changing pitch and roll of the phone to directly control the virtual flight stick) would fix the awareness issue. But it comes with a loss of snap-to-center functionality. To center the control, you actually have to carefully move the device orientation back to its neutral position. Android now has sensor fusion and can use both gyroscope and accelerometer to accurately determine the gravity vector. But even using this (via Android sensor type 'Gravity') was not enough to constitute a usable control system.

So for now I will make a bold statement: Android and flight simulators do not mix. I guess it means I need to use another platform to release this for. Maybe PC, but then, so far I've not been able to get a project greenlit for Steam, and PC gaming seems to be synonymous to Steam gaming these days.

Recording Game Play Videos

This is mainly a note to future self, and also others that need to record a video of game play under Ubuntu. First off, it is of no use to use non-standard resolutions when uploading to YouTube. I recorded some videos with a vertical resolution of 608 lines, which ended up as 480p encoded on YouTube. So it's best to use a window size of 1280x720 to get a quality YouTube video.

My tool of choice for recording a video under Ubuntu is the tool called 'RecordMyDesktop'. I thought I had a very fast (Haswell) CPU, but when using 'encode on the fly' the output video will be full of faults. (Typically frozen video, or image lagging behind the sounds.) So under the Advanced tab, it is essential to set the 'encode on the fly' to disabled. It turns out that my SSD drive is fast enough to store this raw 1280x720 stream, so that's fine. You do need to wait for encoding after you stop recording the video though. Also in the advanced settings make sure you don't have 'Record Window Decorations' set, and that the sound device is 'Pulse'.

There is an annoying bug in RecordMyDesktop that will save the video under the wrong file name. When you select 'Save As', and choose a file name, it will actually save the video under the previously used file name. And next time you save a recording, and pick another name, it will use the old name again.

Reoccurring programming bugs

Ugh, how I hate it. That feeling that you are struggling to fix an issue with your code, and you know you have battled with it before on a different project. Of course you can't remember what the fix was, and the two code bases are too different for easy compare. In the Netherlands, we have a saying: "A donkey will not stumble on the same stone twice." Meaning that if you do, you are dumber than a donkey. Here is the measure I take to make sure I do not get hit a third time: I blog about it for future reference.

My issue involves closing the window of an OpenGL app under Mac OSX by clicking the red window close button. My app would then crash in the OpenGL drawing (triggered by a CADisplayLink.) Stopping a display link is a bit of a mess, as I always seem to be to late with it. Let's see the hooks where I can close down the display link:

My AppDelegate gets callbacks about OSX's intention to close down. We have a lot of these callbacks going on, as there are:

-(void)applicationWillResignActive:(NSNotification *)notification
-(void)applicationWillTerminate:(NSNotification *)notification
-(NSApplicationTerminateReply)applicationShouldTerminate:(NSApplication*)sender
-(BOOL)applicationShouldTerminateAfterLastWindowClosed:(NSApplication *)sender

Astoundingly none would be fast enough to close down the display link before resources get torn down. It turns out only this one is called early enough:

-(BOOL)windowShouldClose:(id)sender

However, my AppDelegate was not getting called with this. It turns out I had not made my app delegate the delegate for my window:

[ window setDelegate:self ];

This requires the AppDelegate to be a NSWindowDelegate:

@interface DigAppDelegate : NSObject <NSApplicationDelegate, NSWindowDelegate>

Car engine

I like car racing simulations. I spent a lot of time playing Gran Turismo Prologue on my PS3 years ago. So I thought it would be time for me to try making my own racing game. After coding a Hover Bike and all sorts of Heavy Machinery that should be easy, not?

Well, it turns out to be a lot more difficult than I expected. Just some high school physics will not cut it. The image above is my prototype in action. I'm pleased with the suspension, the steering and the traction models. Most of which you get for free by using the OpenDE physics engine. But how can I create a simulation model for the engine?

OpenDE lets me speed up/dn the wheels by simply specifying:

• Desired wheel velocity.
• Torque available to achieve this wheel velocity.
• Mass of the wheel.
• Moment of inertia of the wheel.

So how do I go about this? How do I determine what speed the wheel wants to turn, and how much torque is available for this?

In real life, things are more complicated. The engine primarily speeds up the fly wheel, not the car wheels. Depending on the state of the clutch, the gearbox, it will result in wheel acceleration. Do I need to model a flywheel in my simulation? And somehow transfer kinetic energy between flywheel and car wheels depending on clutch and gearbox? How does the accelerator fit in here? And what about the torque curve of the engine, and the rpm of the engine? I am beginning to think a few simple equations are not going to suffice.

How does the accelerator affect the engine RPM? Well, that depends on engine load. How do I calculate that? It seems that there is also a circular dependency in here somewhere:

With an engaged clutch, you can only increase wheel velocity if you increase engine RPM. But engine RPM can only go up if wheel velocity goes up. I think I need to study up on what is actually engine load, and how I can measure it in my sim.