The most common technique is to use an epsilon (the second thing you described). It can be extremely difficult/impossible to make a generic epsilon that works for all input numbers nicely though. If you're dealing with numbers around 0.00001 or numbers around 1000000000 an epsilon of 0.01 would likely be horrible for you. Read this for a really comprehensive analysis of epsilon techniques.
The first solution you described is very common in time math. Everything is represented in an integer number of ticks. The ticks can represent 1 second, or 1 millisecond, or whatever you want. You can then convert them into a decimal of another unit if you like, or compare them. The only thing is that you do need to choose a tick size, and nothing can represent smaller than 1 tick unit.