You're using the wrong syntax, you need to add the filegroup first, marking it as containing filestream data and then add a file to that filestream filegroup, like this:
ALTER DATABASE MyDB
ADD FILEGROUP MyDBFiles CONTAINS FILESTREAM
GO
ALTER DATABASE MyDB
ADD FILE
(
NAME = 'SomeCoolName',
FILENAME= 'C:\FileTable\Data'
)
TO FILEGROUP MyDBFiles
GO