Input of a Rhythm Game

Question

So, Unity does not do a lot of rhythm games on android. I decided to find out why, and program one as assignment (the basics of it anyway). My most important hurdle is user input. As we know that input is based on frame rate in unity, and a music game (i assume) would prefer the smallest possible delay between button press and action.

If we look at music, at around 15 to 20ms of delay, the human ear hears something is "off beat".

I heard Android Unity games run at 30FPS (since 60FPS sucks the battery dry), simple math indicates: 1000/30 = 33ms per frame. Calculating in the 15ms we can probably not notice, we are at 18ms of possible disaster. assuming we always reach this 30FPS at any given moment.

When i get input from a user, i can play a sound on that input on the exact same frame. However, we could be 18ms off.

Now there is a way to get DIRECT controls from mouse and keyboard, which uses OnGui() instead of Update(), to get events of the keyboard or mouse clicks on the spot. The problem is, android probably doesn't work with this, (this doesn't work for gamepads either) and the methods sounds downright strange, especially when we try to play sounds from the OnGui() method.

My question: What would you do, and why? Should we just accept the possible 18ms off, and assume we reach the 30FPS, or should we look for a reliable way to get input directly, instead of waiting for an update to come by?

Thanks for any insight you can give me, i have not found any articles on this that are useful just yet. -Smiley

EDIT I just did some basic testing with a metronome, running at 100FPS in the editor (which should be 10ms per frame) tapping my spacebar on a metronome inside unity. The results i got were just horrible. Tapping rapidly: I get as close as 20ms to my metronome tick, but nothing closer. Tapping on the beat: I got at least 200ms off my target tick. Unless i am confused with this rhythm, this is just wrong.

Currently i use Debug.Log to get my test data to the log. Can anyone please confirm for me if this may be the cause (causes some long delay? i know debug isn't that optimized), or is the timing actually that bad on it?

Thanks in advance, -Smiley

Answer 1

First, I'd like to add a few things to your comments and analysis:

There is a hell of a lot more to measuring the latency between the tactile input and what your eyes perceive.

Probably the biggest one I'm seeing missing in your tests is the latency between the graphics card and the PC monitor you're testing on. Many common LCD monitors these days have a latency of between 15-30ms in processing lag. I don't know how much this relates to mobile screens and hardware, but I would suggest you take the time to perform additional tests on your target hardware before drawing further conclusions.

To more directly answer your question:

I would continue to use Unity however I would continue to research the best methods of taking the input and feeding it back to the player as fast as possible. In the above comments @rutter has pointed you to what appears to be a pretty good thread on the issue.

Of specific note I would look at using FixedUpdate() to decouple the framerate of the game from the input processing speed.

I think it is also worth putting time into researching the psychology of the perception of latency. For example, if your game is a sort of Guitar Hero game of matching the playing song, you could simply take into account the lag you know is there and in your game logic take that into account when checking input.

Answer 2

I think you are over-complicating this, and that the accuracy issue is no where near as bad as you think.

People usually hit the buttons a little early in order to sync what they are seeing and hearing.

It also depends alot on if you have some kind of scrolling display that they are trying to match up to... if the display is scrolling smoothly at 30fps (without big jumps) they they are still able to make their timing presses fairly accurate.

I would surmise that although people can hear when their timing is off, their actual timing of hitting the buttons at exactly the right time is not that accurate anyway.

Here is one other simple solution... which I think is what rock band and guitar hero often do... You start playing the note/sound at the correct time anyway.... then change it to a broken sound if you detect they missed it or goofed up.