Sounds like you have a buggy {top|shell|grep} in that order of likelyhood ;) Lemme guess it's a busybox-based system?
Perhaps a better alternative is to write a simple script to monitor the process' direct stats through the /proc file system. Some entries of interest to you might be
/proc//maps - info about currently mapped memory regions
/proc//smaps - memory consumption for each of the process's mappings
/proc//stat - some useful stats including resident set size (total current memory usage)
The man page proc(5) has much more information about what information is contained within these /proc/ entries.
Happy hunting!