Question 1: Is there a way to install an SMTP server to work with Apache? If so, how?
As Devon mentioned in your comments, SMTP and Apache are different servers/services and use different protocols all together. From what I can understand from your question, Yes you can setup the built in SMTP server on your 2008 R2 server and use it to send mail as long as you have a static IP from your ISP where the server resides (if you have a dynamic IP where your server is your IP may be permanently blocked from being able to send to anybody but a local mail server which in your case would be either a exchange server or a 3rd party email application in which would only reside locally for testing purposes). If you really want to know what is happening to your email I would follow this link on getting your SMTP logs setup and viewing what is really happening to the mail that is sent through PHP.
http://www.vsysad.com/2012/04/setup-and-configure-smtp-server-on-windows-server-2008-r2/
Also on a side note, if you are using the PHP mail function or any type of mail function (not just PHP) typically you would not use SSL on port 25 unless you configured SMTP to use SMTP over SSL in that manner (The settings you have setup on the plugin does not really make sense).
Question 2: Can I configure php mail to work without using SMTP? If so, how?
The plugin you are using has the ability for you to not send to a localhost (Local SMTP server) but rather another address like gmail, yahoo, secureserver (godaddy) or any other mail service provider which would be SMTP for your particular situation. Depending on the provider the SMTP configuration will change based on the setup that you have for the account.
Here is a brief list of servers that you could use instead of localhost and you would require SMTP authentication (which requires an account with one of them).
http://www.arclab.com/en/amlc/list-of-smtp-and-pop3-servers-mailserver-list.html