Your question about correct quoting is hard to answer, because you don't create something that you can easily debug. Consider the following code instead:
$pattern = 'sed -i \'s/max-players=4/max-players=8/g\' %s\'/server.properties';
$command = sprintf($pattern, $filedir);
$result = ssh2_exec($con, $command);
Then you can more easily validate your actual command, e.g. by inspecting / outputting / logging: $command
.
Also you can more easily compare what the difference between the two variants is.
Also enable error reporting and logging to the highest level, you might just have the problem that the variable $DIR
is not defined:
Notice: Undefined variable: DIR in ...
This will make your first command to fail because there is no such path/file then. That is highly likely your issue.
Take care.