There's nothing "natural" about linear color values. Your own eyes respond to brightness logarithmically, and the phosphors in televisions and monitors respond exponentially.
GIF and JPEG files do not specify a gamma value (JPEG can with extra EXIF data), so it's anybody's guess what their color values really mean. PNG does, but most people ignore it or get it wrong, so they can't be trusted either. If you're displaying images from a file, you'll just have to guess or experiment. A reasonable guess is to use 2.2 unless you know the file came from an Apple device, in which case use 1.0 (Apple devices are generally linear).
If you need to store images accurately, Use JPEG from a camera with good EXIF data embedded for photos, and maybe something like TIFF for uncompressed images. And use high-quality software that understands these things. Unfortunately, that's pretty rare and/or expensive.