You can use a flat slice if you like - then you just have null/0 for entries that don't have a entry. That approach would be bad if your non-consecutive values can range to very high values.
Using this code: http://play.golang.org/p/gLni-BzMKy
I got these results after running 100,000,000 indexes on a slice and map: Map: 3062 ms Slice: 56 ms
That being said - the speed difference shouldn't matter in almost any real world use case. I would just use a map personally.
Edit: And I agree with the other comment that says to initialize the map outside the function so it only has to be constructed once and not on every call.