Question

I have written a method to take an integer and convert it to either decimal, hex, or binary depending on what you choose using commenting. I will build a choice in later. The problem I am having is that the wrong values are being chosen and I think it has to do with the modulus operator returning the wrong value. It may be a rounding issue because I always seem to be off by 1 digit, but not for every char in the string. Any help would be appreciated. For example, right now it is set up give the decimal value. So if you run this and enter 1705, it should return 1705, but instead it returns 2705. Hex and Binary are not much better.

Do While val >= 1    'finds the remainder and assigns it array

    r = val Mod base 
    buf(i) = arr(r)

    'Possible rounding issue:
    val = val / base     'need whole number division to find val. 

    i = i - 1 'decrement the buf array    

Loop
Was it helpful?

Solution

Mod() operates on integer values. If the value you're passing isn't an integer value, VBScript will convert it to one by rounding it to the nearest integer. That's what's happening with your value of 1705. After three iterations, you're left with 1.705, which Mod() rounds to 2.

So now the question becomes, why are you passing floating-point values to Mod() in the first place? The reason is because VBScript is doing floating point division in the following statement:

val = val / base

Since you don't care about the remainder (you've already captured it), this statement needs to be performing integer division, not floating-point division. Fortunately, VBScript makes integer division easy. Just use a backslash instead of a forward slash:

val = val \ base
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top