Scrum roles mixed up

Question

Here at the office our Scrum roles are a bit mixed up, so I would appreciate any advice on how to improve it.

We are a small team of 4 developers (one of which is a de facto scrum master) and we also have a line manager and a Product Owner. The line manager simultaneously embraces Scrum and has a hard time "letting go"; he has the Product Backlog in his head and also prioritizes it.

A previous attempt to take it out of his head only lead to a short lived up-to-date Product Backlog in Pivotaltracker.com – politically speaking he was the programmer who built the current system 7 years ago. The Product Owner has been in the company for about 2 years and seems to have a clear vision of the product, but he is underpowered by our line manager (who in fact is held responsible for this and other projects). Other than this, our implementation of Scrum is “by the book”.

What do you think are disadvantages of this structure we find ourselves into? What could be improved? How would you improve?