Filemaker has no SumIf() function. You need to create a calculation field in the LineItems table, along the lines of:
If ( Paid = "N" ; Cost )
then sum this field at the invoice level (and/or summarize it at the LineItems table itself), instead of the Cost field.
--
BTW, it is much more convenient to define Boolean (yes/no) fields as Number and use the values of 1 for True, 0 (or empty) for False. Then the calculation can be simply:
If ( not Paid ; Cost )