There are a number of libraries in development to do what you suggest. I believe all of them are discussed on this forum. https://groups.google.com/forum/#!forum/mechanical-sympathy which may also have many topics which may interest you.
In short you can do it using Unsafe, or a library which uses it. In fact I have one of my own, but again it is in development.
For the effort involved this will only make much of a difference if you have many GB of data. At this point the reduce GC times and reduced size of the heap are the main advantages, on saving a single de-reference.