Why does this Prolog Fibonacci function cause a "instantiation_error"?

Question 1

In the second clause:

fib(N,A,B,F) :-
    N1 is N-1, Sum is A+B, fib(N1, B, Sum, F).

N is a variable to be decremented, and in your call to:

fib(What, 6765).

The variable is not yet defined, so you get the instantiation error on N1 is N - 1.

In swipl I do even get the error:

?- fib(W, 6765).
ERROR: fib/4: Arguments are not sufficiently instantiated

Question 2

Now that you know it's an error, do you mind to know if it's actually possible to answer your query?

How do you would approach the problem? Your function it's ok, isn't it? Exactly, because it's a function, and not a relation, you get the error.

It's a bit complicate to solve it, but CLP can do !

See this fascinating example from CLP(FD) documentation (cited here)

:- use_module(library(clpfd)).

n_factorial(0, 1).
n_factorial(N, F) :-
        N #> 0, N1 #= N - 1, F #= N * F1,
        n_factorial(N1, F1).

We need something like this, but for fibonacci. See how easy it is:

:- [library(clpfd)].

fib(0,A,_,A).
fib(N,A,B,F) :-
    N #> 0,
    N1 #= N-1,
    Sum #= A+B,
    fib(N1, B, Sum, F).
fib(N, F) :- fib(N, 0, 1, F).

i.e. replace is/2 by #=/2 and we get

?- fib(20,Result).
Result = 6765 .

?- fib(X,6765).
X = 20 ;
^C

note, after the first response the program loops! Do you a see a way to correct it? Or another question could be worth...

Question 3

A more clear and more natural predicate definition may be:

//The two base steps
fib1(0,0). 
fib1(1,1).
//the recursive step
fib1(N,F) :- 
        N >= 0, M is N-2, O is N-1, fib1(M,A), fib1(O,B), F is A+B.

It is also a definition with only one predicate: fib/2