Although section 2.5 does introduce the concept of type hierarchies, the "inheritance" mentioned is not as general as OOP's inheritance. In the context of a numeric tower, a type inherits all operations defined on a supertype, that's all.
In fact, in the second edition of SICP OOP is not treated formally - although I think I saw some SICP course notes dealing with OOP once, but they were not part of the book. The closest thing you get to OOP in SICP is the technique of message-passing, but without inheritance.
Check your Scheme interpreter's documentation, it's possible that some full-fledged object system is included within it - but it will have nothing to do with SICP.