There are some popular ways to define the public API. Which one you choose is mostly a matter of taste.
One way is documentation. You simply state in the documentation, which protocols are part of the public API, and what the contract of those protocols is. YARD even has predefined tags for this.
Another way is testing. I think Merb did this. The public API was described in its RSpec tests. The private parts were obviously also tested, but those tests lived in a different directory.
This is actually pretty cool, because it allows you to tie together code changes and semantic version changes: everytime you add a test to the public directory you need to bump the minor version. Everytime you delete or modify a test in the public directory you need to bump the major version.
Or the other way round: you are not allowed to change or delete tests during a minor revision.