Using the OpenAL library in Xcode for iOS.

The last two days I have been porting The Little Plane That Could to iOS. It was already running on OculusRift(Windows), Linux, Android and Mac OSX. When porting it, I struggled with the poor OpenAL implementation that comes with the iOS SDK.

I knew that there was a very low limit of 32 sound sources for iOS. I figured that this limit only applied to sound sources that were in 'AL_PLAYING' state. And sure enough: you can create much more than 32 sources, without getting errors back from the OpenAL implementation.

In my game, I have 48 planes flying around, each with engine sounds. If we forget about gun sounds and explosion sounds for a while, these 48 engine sounds already exceed what the iOS implementation of OpenAL can handle. So this is why I adopted a method of only playing the four closest engine sounds. If an engine sound was further away, I would pause the playing of this sound. This worked well on all platforms, except iOS.

The problem I experienced was the following: alGetError() would start returning a non-sensical value of -1. And the sounds would no longer get started. The return value of -1 for alGetError() violates the OpenAL specification standard though. If you study the header files of the SDK, you see that the only acceptable return values are 0 (AL_NO_ERROR) or:

/** 
 * Invalid Name paramater passed to AL call.
 */
#define AL_INVALID_NAME                           0xA001
/** 
 * Invalid parameter passed to AL call.
 */
#define AL_INVALID_ENUM                           0xA002
/** 
 * Invalid enum parameter value.
 */
#define AL_INVALID_VALUE                          0xA003
/** 
 * Illegal call.
 */
#define AL_INVALID_OPERATION                      0xA004
/**
 * No mojo.
 */
#define AL_OUT_OF_MEMORY                          0xA005

After struggling to find the cause of the problem for half a day, I finally solved this issue. It turns out that the 32 source limit applies to BOTH playing and paused sources. Only sources that have not started playing yet (AL_INITIAL) or have been stopped (AL_STOPPED) do not count against the limit. So the fix was relatively easy: sources that are not nearby should not be paused. They should be stopped instead. This does not take away the fact that the alGetError return value is not according to spec. Apple should fix this.

What has Bram been up to?

So, it's been quiet for too long on this blog. It is about time to report on what I have been up to. Why not start with a small snippet of video?

Since July 2013 I have been mulling over the next step for my crane sim game. And as the holy grail, I have put forward an ambitious development target: deformable terrain. Frankly, I have not yet seen a single game getting this right. I most likely will not achieve completely realistic, sandbox-style, unlimited deformable terrain either. But I will surely try. The video above show a bulldozer at work in my terrain simulator.

Currently I have achieved the following:

• Proper 3D terrain, not those crummy height-fields. So you can have overhangs.
• Universally deformable terrain: you can dig anywhere and not just at designated spots.
• Tunnelling! Yes - you can dig your own tunnel.
• Soil mixing. You can dig up grey coloured dirt in the west, and deposit it on a brown coloured sediment in the east.
• Near infinite size. I bound the height and depth (Z) of the plane. But in the long/lat (X/Y) directions, you can drive near infinitely far. And deform the terrain at any place. Distances are only limited by MAX_INT and the size of your disk, because any modifications you do to the terrain are stored to disk.
• Procedural definition of the virgin grounds. I only store disturbances of the virgin ground. So you can dig a hole, then drive 8 hrs going in any direction, and drive 8 hrs back to the original location. Your hole in the ground will still be there! (This is a big thing!)
• Tracing back your steps, BTW, is easy. Because while driving, depressions are made in the soil which are permanent. They will still be there, even after 100s of hrs of play. Everything persists.

I think this technology is pretty unique and not found in any other game. The basic tech is all implemented, and nearly good enough. What is not there at all, is game-play. Wonderful game technology is one thing. Making a interesting, fun to play game is another. There are some routes I could take w.r.t. the game-play:

• Make it into an arcade style game with a lot of KATCHING! noises, and bright graphics every time you dig up a gem from the earth.
• Do it like the original Little Crane game: break up the gameplay in a number of puzzle-like levels with simple goals. Digging for treasure, unearthing a temple, burying an item, digging a tunnel, building a dam and such, could all be simple objectives in such a game.
• Make a full blown gold-mining simulator. Process earth by moving it into a trommel.
• Make a full blown gold-mining simulator plus economic sim: borrow money from bank, buy claims, buy diesel, etc.
• Make a full blown gold-mining simulator MMO. Buy and sell claims online. Bid for scarce resources (land) against other players.

The smaller the scope, the higher chance of succeeding of course, so I should avoid the latter ones.

Things I am doing at this moment include added more vehicles, like dump truck and excavator to the simulation. I also have to trace a bug where soil simulation goes hay wire at negative world coordinates. As my world is procedurally defined, I could cheat and spawn all action at large positive coordinates, but that feels wrong.

Developing for Oculus Rift DK2

I've purchased an Oculus Rift Development Kit 2. In a sense it feels like going back to my thirties. Much of my professional career was in Virtual Reality. I can't remember for how long exactly, but I think it was for 8 years or so, that I developed software and ran VR projects for SARA's CAVE. See below for a picture or me, mining genomics data that we visualized for Johnson&Johnson.

Since 2007, my career has been in Video Games. Even though I had been making games since 1982, it took me 25 years of hobby game development before I decided to do this for a living. And since 2010 I do this independently as a private enterprise. Back to 2014: my two careers have crossed, and I find myself actually developing Games for Virtual Reality. A younger me would have labeled this as heaven, but the years at SARA have made me cautious about the adoption rate of VR. This current VR revival could be another one that is destined to go out like a candle.

This web page will function as a development log. I intend to document peculiarities, problems and solutions that are bound to pop up during DK2 development. If you just want to play my VR game, download The Little Plane That Could.

Windows Driver issues

• When using the 'Direct' Rift Display Mode, I get extreme chromatic shifts. The different channels R/G/B are rendered all over the place, not converging what so ever. 'Extended Desktop' display mode works better.
• The SDK precompiled example runs badly, the jitter is horrible and is only alleviated a bit if I toggle multi sampling. It's hard to say which mode (ON or OFF) reduces jitter, because both look exactly the same, with the same level of aliasing.

Development under Windows

• After installing windows runtime and sdk, the latest firmware can be flashed to the device.
• The SDK example 'Oculus Room Tiny' is intended as the minimalistic code to base your first VR app on. It comes with a Visual Studio 2013 solution. Unfortunately this does not build out of the box. But what is worse: it is a D3D app, and not an OpenGL based app. The SDK seems to support both, but unfortunately the sample is in D3D only. Bummer!
• Building Oculus Room Tiny yields a: Cannot open include file: 'AtlBase.h'
• Linking the sample also fails: fatal error LNK1104: cannot open file 'atls.lib'
• To fix these compiler and linker errors, you need to install a Windows Driver Kit, as described on Stack Overflow.
• If you want to link your own app against libOVR, you need to add these to the linker input: libovr.lib (libovrd.lib for Debug), ws2_32.lib and winmm.lib.
• The Rift display mode (Extended Desktop vs Direct) seems to mess with OpenGL context creation. When 'Extended Desktop' is enabled, SDL_GL_CreateContext() can create a 3.2 Core profile context. If I ask for a 3.2 Core profile when the mode is set to 'Direct', then SDL_GL_CreateContext() crashes with an access violation at address 0.

Development under GNU/Linux

• Let's start with some good news: Oculus intends to release DK2 SDKs for Windows, OSX and Linux. The bad news: so far no Linux SDK for the DK2 has appeared. It was labeled as 'soon' but has been delayed quite a bit now.

General Development issues

• You can create the projection matrix for each eye with a call to ovrMatrix4f_Projection() but you need to transpose the result before using.
• You can build the view matrix for each eye using the eyePose from ovrHmd_GetEyePose(). Strangely enough you still need to shift the position of the eye with the intra occular distance. Like the project matrix, the view matrix need transposing as well.

Routers with restrictive NAT spoiling online gameplay.

I have published an online tank simulator for both iOS and Android. On the app store it is called Buggy Bang Bang! and was renamed as Six Wheels and a Gun on Google play. Players on iOS and Android can even play each other, as they are all pooled on the same lobby server. I wrote the lobby server code myself using Python.

A neat thing about my networking code (C for the clients, Python for the lobby server) is that it is 100% UDP based. There is not a single TCP connection used, just some UDP packets that get exchanged with both server and opponent's client. Two players playing my game are typically on a LAN behind a router that does NAT (Network Address Translation) and this complicates things. It means that the only way they can connect to each other is to have them both connect to a server on the internet, with a public IP. This server will then see datagrams coming in on some random ports, and can tell both parties what these ports are. This is the concept behind UDP hole punching.

Things go wrong if your router is too restrictive. It may do Symmetric NAT, or Restricted Cone NAT. If this happens, the players will not find each other on the ports as described by their common server. In my game, it will result in two buggies that refuse to move and after 12 seconds, the match will time out and record this as a 'forfeit' by the other party. Moral of the story: if you play my game and the HUD controls don't show up with the buggies just sitting there, then your router or your opponent's router will not play ball. So far my approach to this is: too bad it happens, better luck next time. If you want, read the manual of your router and change the NAT type.

UPDATE

I've been adding a new feature to S.W.a.a.G. that detects restrictive NATs. Instead of pairing up a user with a bad NAT, we nominate two players for a session. If one of them fails to punch a hole in the firewall, the match will not be started. The players are informed of the situation with either of these two messages:

• YOUR NAT SEEMS TOO RESTRICTIVE.
• OPPONENT NAT SEEMS TOO RESTRICTIVE.

So at least the players will know what is going on. If it is your opponent, you can just retry, and hopefully get paired with another user.

OpenAL with Android NDK.

This posting is for future reference, if I ever have to get OpenAL working again under Android's NDK. It will also be useful to my peer game developers who intend to do the same.

Step 1

Get an Android port of openal-soft.

Step 2

I like to build static libraries for android game dev, instead of dynamic ones. So I replaced the jni/Android.mk with my own:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

OPENAL_DIR := OpenAL

AL_SOURCES := \
  $(OPENAL_DIR)/Alc/android.c              \
  $(OPENAL_DIR)/OpenAL32/alAuxEffectSlot.c \
  $(OPENAL_DIR)/OpenAL32/alBuffer.c        \
  $(OPENAL_DIR)/OpenAL32/alDatabuffer.c    \
  $(OPENAL_DIR)/OpenAL32/alEffect.c        \
  $(OPENAL_DIR)/OpenAL32/alError.c         \
  $(OPENAL_DIR)/OpenAL32/alExtension.c     \
  $(OPENAL_DIR)/OpenAL32/alFilter.c        \
  $(OPENAL_DIR)/OpenAL32/alListener.c      \
  $(OPENAL_DIR)/OpenAL32/alSource.c        \
  $(OPENAL_DIR)/OpenAL32/alState.c         \
  $(OPENAL_DIR)/OpenAL32/alThunk.c         \
  $(OPENAL_DIR)/Alc/ALc.c                  \
  $(OPENAL_DIR)/Alc/alcConfig.c            \
  $(OPENAL_DIR)/Alc/alcEcho.c              \
  $(OPENAL_DIR)/Alc/alcModulator.c         \
  $(OPENAL_DIR)/Alc/alcReverb.c            \
  $(OPENAL_DIR)/Alc/alcRing.c              \
  $(OPENAL_DIR)/Alc/alcThread.c            \
  $(OPENAL_DIR)/Alc/ALu.c                  \
  $(OPENAL_DIR)/Alc/bs2b.c                 \
  $(OPENAL_DIR)/Alc/null.c                 \
  $(OPENAL_DIR)/Alc/panning.c              \
  $(OPENAL_DIR)/Alc/mixer.c                \
  $(OPENAL_DIR)/Alc/audiotrack.c           \
  $(OPENAL_DIR)/Alc/opensles.c


LOCAL_MODULE    := openal
LOCAL_SRC_FILES := $(AL_SOURCES)

LOCAL_C_INCLUDES := \
  $(HOME)/src/openal-soft/jni/OpenAL \
  $(HOME)/src/openal-soft/jni/OpenAL/include \
  $(HOME)/src/openal-soft/jni/OpenAL/OpenAL32/Include \


LOCAL_CFLAGS += \
  -DAL_ALEXT_PROTOTYPES \

MAX_SOURCES_LOW ?= 4
MAX_SOURCES_START ?= 8
MAX_SOURCES_HIGH ?= 64

LOCAL_CFLAGS += -DMAX_SOURCES_LOW=$(MAX_SOURCES_LOW) -DMAX_SOURCES_START=$(MAX_SOURCES_START) -DMAX_SOURCES_HIGH=$(MAX_SOURCES_HIGH)
LOCAL_CFLAGS += -DPOST_FROYO

include $(BUILD_STATIC_LIBRARY)

Step 3

Get an Android port of freealut.

Step 4

Add an Android.mk file to src/ directory containing:

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE := alut

LOCAL_SRC_FILES := \
  alutBufferData.c \
  alutCodec.c \
  alutError.c \
  alutInit.c \
  alutInputStream.c \
  alutLoader.c \
  alutOutputStream.c \
  alutUtil.c \
  alutVersion.c \
  alutWaveform.c

LOCAL_C_INCLUDES := \
  $(HOME)/src/freealut/include \
  $(HOME)/src/openal-soft/jni/OpenAL/include \

#LOCAL_CFLAGS +=

include $(BUILD_STATIC_LIBRARY)

Step 5

Some config.h stuff does not properly make it into the source. Not sure why, so I manually edited alutInputStream.c alutInternal.h and alutUtil.c to overcome this.

Step 6

Now you can pull these static libraries into your build by adding to your project's Android.mk file something like this:

LOCAL_STATIC_LIBRARIES += alut openal
...
$(call import-module,openal-soft/jni)
$(call import-module,freealut/src)

UPDATE #1

When porting to MSWindows, I found that OpenAL is pretty tricky there as well. Some info for future reference:

• You need to link against libOpenAL32.dll.a
• You need to supply soft_oal.dll with your game, renamed as OpenAL32.dll
• As with Android and OSX, the CMake stuff of freealut is horribly broken, so you need to pull in the sources to your project and hack around the platform specific stuff.

UPDATE #2

When building for 64bit arm, or arm64-v8a as it is known, I get crashes (SIGSEGV) in alSourcePlay. Maybe openal-soft is not 64 bit safe? The android port is pretty old, as the last change is from 2012.

Flight Simulation on touch screens.

My current project is biplane combat simulator. I have been developing it under GNU/Linux, and porting it to Android. It needs a commercially viable platform to run on, and I am pretty sure GNU/Linux games are not making money at the moment. This may change with the Steam box, but for now, I have to look at iOS, Android or maybe a console platform.

After porting my biplane game to Android, I learned a surprising fact: When it comes to flight simulators, there is absolutely no substitute whatsoever for a physical analog joystick. You really need the self-centering stick with plenty of instant feedback if you want to control an airplane.

The first substitute I tried was touch control, with a virtual joystick on the screen. I added snapping-to-center when you release your touch, so that there is some sort of self-centering going on. But this was not enough to make the plane controllable. The lack of tactile feedback means that you have little awareness of the absolute position of the virtual joystick without looking at the control graphics.

I may need to redo this touch control experiment with iOS though. Android is notoriously bad when it comes to responsive touch control. Try this for fun: swipe some content (web page, e.g.) up and down at roughly 5Hz or so. You will see that the motion of the content is opposite of the motion of your finger. If your finger goes up, the content still goes down from your swipe 100ms ago.

I thought that tilt-control (changing pitch and roll of the phone to directly control the virtual flight stick) would fix the awareness issue. But it comes with a loss of snap-to-center functionality. To center the control, you actually have to carefully move the device orientation back to its neutral position. Android now has sensor fusion and can use both gyroscope and accelerometer to accurately determine the gravity vector. But even using this (via Android sensor type 'Gravity') was not enough to constitute a usable control system.

So for now I will make a bold statement: Android and flight simulators do not mix. I guess it means I need to use another platform to release this for. Maybe PC, but then, so far I've not been able to get a project greenlit for Steam, and PC gaming seems to be synonymous to Steam gaming these days.

Recording Game Play Videos

This is mainly a note to future self, and also others that need to record a video of game play under Ubuntu. First off, it is of no use to use non-standard resolutions when uploading to YouTube. I recorded some videos with a vertical resolution of 608 lines, which ended up as 480p encoded on YouTube. So it's best to use a window size of 1280x720 to get a quality YouTube video.

My tool of choice for recording a video under Ubuntu is the tool called 'RecordMyDesktop'. I thought I had a very fast (Haswell) CPU, but when using 'encode on the fly' the output video will be full of faults. (Typically frozen video, or image lagging behind the sounds.) So under the Advanced tab, it is essential to set the 'encode on the fly' to disabled. It turns out that my SSD drive is fast enough to store this raw 1280x720 stream, so that's fine. You do need to wait for encoding after you stop recording the video though. Also in the advanced settings make sure you don't have 'Record Window Decorations' set, and that the sound device is 'Pulse'.

There is an annoying bug in RecordMyDesktop that will save the video under the wrong file name. When you select 'Save As', and choose a file name, it will actually save the video under the previously used file name. And next time you save a recording, and pick another name, it will use the old name again.