I found the answer myself, after a good hard look.
This line should be at the beginning of your file, right under reset or somewhere around there
rgb(r,g,b) = int(r)*65536 + int(g)*256 + int(b)
And this is your plotting command
plot "output.txt" using 4:5:(rgb($9,$10,$11)) \
with points pt 7 ps 2 lc rgb variable notitle
So basically, you make a new variable rgb with 3 variables, the red, the green and the blue component. For my problem, i had to add 3 columns to my data files that described my color. So if my color was #5535cf (i calculate the colors randomly) my rgb value would be (85, 53, 207) I would then add 3 columns to my output.txt data file with 85, 53 and 207.
These numbers are in column 9, 10 and 11 from my output file, which explains the $9 $10 and $11. There should be the columnnumbers where your r g and b values are.
If you add lc rgb variable notitle to your plot command, the points should take the color you assigned them in your data-file.
Hope this helps a lot of people!