In the default configuration which you had part of there is an acl called localnet. You could modify the localnet acl to include only your source address from where you are connecting instead of the defaults, or you could make your own acl for your source address. You need to use it in the http_access commands which grant access.
I see all the following http_access commands used in your configuration:
http_access allow manager localhost #1
http_access deny manager #2
http_access allow all #3
http_access allow localnet #4
http_access deny !Safe_ports #5
http_access deny CONNECT !SSL_ports #6
http_access allow localnet #7
http_access allow localhost #8
http_access deny all #9
4 and 7 are redundant. 3 could be removed if your source address matched localnet. 3 also covers up some of the security features given in 5 and 6. I propose the following where localnet acl has been modified for only your source address:
acl localnet src <source_ip>
http_access allow manager localhost #1
http_access deny manager #2
http_access deny !Safe_ports #3
http_access deny CONNECT !SSL_ports #4
http_access allow localnet #5
http_access allow localhost #6
http_access deny all #7
I think that would do the trick.