Depending on how you define larger values, your concerns are valid.
Let does "fixed with" integer aritmethics with no overflow checking whatsoever, the "fixed width" is implementation/system dependent. I do not know if posix defines the a minimum "width". If I recall correctly, bash does the calculations using longs.
Note on protability; Korn, bash and POSIX compatible shells have let, older bourne shells do not.