Store Handle in custom data field?

Question 1

I'm not completely sure what you want. If you don't understand type errors involving IO type mismatches, you should probably read an introduction to IO in Haskell first. Anyway, this is code that works:

import System.IO

data CustomType = CustomType {
    file :: Handle
  }

fileName :: FilePath
fileName = "foo"

process :: IO ()
process = do
  someFile <- openFile fileName AppendMode
  let object = CustomType { file = someFile }
  hPutStrLn (file object) "abc"
  hClose (file object)

If you want to type the commands in GHCi instead, you can enter every line of the do-block in the process action individually, like this:

GHCi> someFile <- openFile fileName AppendMode
GHCi> let object = CustomType { file = someFile }
GHCi> hPutStrLn (file object) "abc"
GHCi> hClose (file object)

Question 2

So you have created your type like this:

data CustomType = CustomType {
    file::Handle
}

Now try this in ghci:

ghci> let a = openFile "someFile.txt" AppendMode
ghci> :t a
a :: IO Handle

So Handle is wrapped with IO type. You can use the Monad bind operator to extract the Handle out of it.

ghci> let b = a >>= \handle -> return $ CustomType handle

The return function will wrap the CustomType again in the IO monad. You can verify this again in the ghci:

ghci> :t b
b :: IO CustomType

The type of bind or >>= is like this:

ghic> :t (>>=)
(>>=) :: Monad m => m a -> (a -> m b) -> m b

Try to replace m with IO and you get:

(>>=) :: IO a -> (a -> IO b) -> IO b

And that's why you have to use return function with the bind operator, so that it will typecheck.

Question 3

So your confusion seems to be with how to use values inside a IO monad. The thing about the IO monad is that its infectious, so whenever you want to do something with a IO value you will get a IO result. This might sound like a pain in the ass, but its actually pretty nice in reality as it keeps the parts of your program pure and gives you complete control over when actions are performed. Instead of trying to get out of the IO monad you have to learn to embrace the haskell way of applying functions to monadic values. Every monad is a functor so you can apply a pure function to a monadic value with fmap. The extra power a monad gives is that it allows you to join contexts together. So if you have a IO a and a IO b you can join the IO's together to get IO (a, b). We can use this knowledge to solve your problem:

openFile has the signature:

openFile :: FilePath -> IOMode -> IO Handle

As mentioned above there is no way* to remove the IO from Handle, so the only thing you can do is to put your type inside the IO monad too. You can for example make this function that uses fmap to apply your object to the IO Handle:

createMyObject :: FilePath -> IO CustomType
createMyObject fp = CustomType `fmap` openFile fp AppendMode

Now you have your object but its in a IO monad so how do you use it? The outermost layer of your application is always in the IO monad. So your main function should have a signature like IO (). Inside the main function you can use other IO values like they are pure by using do notation. The (<-) keyword is kind of like join we talked about above. It draws the value from another IO into the current IO:

main :: IO ()
main = do
    myObjectPure <- createMyObject "someFilePath.txt"

    let myHandle :: Handle -- No IO!
        myHandle = file myObjectPure

    -- Now you can use the functions that takes a pure handle:
    hPutStrLn myHandler "Yay"

By the way you probably shouldn't use a Handle directly this way because it will be really easy for you to forget to close it. Its better to use something like withFile which will close the handle for you when its done.

*Actually there is a way, but you don't need to know about it yet because its very unlikely that you are solving a problem that actually needs it, and its too easy to abuse for someone new.