No, you can't drill into related tables in this way, because the data of an author is simply not in the data source you are displaying with the data list. But you don't have to create function for each property of an author you want to display. You may just create a function which will return whole author object, which is in your case CustomTableItem. The function may look like this.
public CustomTableItem GetAuthor(object id)
{
int authorId = ValidationHelper.GetInteger(id, 0);
var pr = new CustomTableItemProvider();
var item = pr.GetItem(authorId, "customtable.author");
return item;
}
Then in a transformation you will use GetValue() method to get the value.
Author: <%# GetAuthor(Eval("AuthorID").GetValue("FirstName")) %>
Be aware of each call of the function will issue a database request, so i would suggest to use some kind of caching. Either output cache for whole page or you may implement some caching mechanism directly inside the function.
The other option you also have is to use CustomQueryRepeater/DataSource and write your own SQL query where you join book data with author data. Then you could use simply <%# Eval("FirstName") %>
directly in yout transformation.