Indy TIdHTTP URI with login/password not working

Question

I'm writing an HTTP API wrapper to integrate with particular IP Cameras. Part of the process requires login credentials in the URI, for example:

http://user:pass_with_#_sign@address:port/somespecificpath/

I'm using Indy's TIdHTTP.Get to send the request to the device. However, the password consists of a '#' character. If this URI is placed in any browser with the plain password, the # character throws it off. Therefore, I need to encode the password's # to %23...

http://user:pass_with_%23_sign@address:port/somespecificpath/

When I paste this URI into any browser, it successfully logs in and does what it needs. However, when I pass the exact same URI into TIdHTTP.Get, it does not successfully log in, and therefore I cannot do anything as long as the password contains # (or %23). Changing the password to not include a # is far too sloppy of a solution. Indy must be messing something up with this URI/password.

Is this a bug in Indy, or is there something else I need to do to make Indy accept such an encoded password?

---

UPDATE

I added a new account on one of the cameras with username and password without any special characters which need encoding, and authentication still does not work. It seems as if Indy is stripping out the login credentials completely from the URI, and doesn't even send these credentials. Next thing I need to do is monitor the URI which is actually sent.

---

UPDATE 2

I did a packet capture via WireShark and have verified that the credentials ...user:pass@... are not even sent - they're stripped from the URI that Indy actually sends.

---

UPDATE 3

TLama suggested that I get a capture of what's sent when using a browser. I tried this, and sure enough even when using a browser the capture doesn't show these login credentials either... even though it works from a browser. So I have to figure out another way to identify whether or not these credentials are sent.

---

UPDATE 4

I tried (before seeing Remy's answer) to provide these credentials in Request.Username and Request.Password instead of in the URI, and I still have no success. I keep getting "Unauthorized" back from the device.

---

UPDATE 5

The documentation for this API mentions nothing relevant to how users are authenticated other than this paragraph:

Grandstream Video Surveillance API (Application Programming Interface) supports HTTP 1.0 protocol (RFC1945). This document explains in detail the parameter of functions in client side, via the supported GET/POST method. Users will require administrator privilege to retrieve or set the parameters.

And on that note, I did switch the TIdHTTP protocol version to 1.0.

---

UPDATE 6

Hopefully the last update needed... I did another comparison with the packet captures between a browser (Chrome) and TIdHTTP. Chrome actually sends two requests, the first one does not have any credentials, but in the second request there's a node in the header Authorization: Basic... and Credentials: User:Pass, whereas using TIdHTTP only sends 1 single request without these credentials.

---

UPDATE 7

7 is a lucky number :-) I just realized, the very first request I make to the device returns "Unauthorized", but all following requests I make (using the same TIdHTTP instance) are successful! So going back to my prior update, just like I see in the browser capture, it takes that second repetitive request for it to work.

Answer 1

I have solved the issue by sending two sequential Get requests. After observing the packet captures between a browser and Indy, I noticed browsers would always send one request without credentials, and then another identical request with credentials. So it only sends the credentials when it needs to. Indy was only sending one request, but if I send another request right afterward, I have success.

So, the request now looks like...

FWeb.Get(U);
FWeb.Get(U, R);

Of course it really should be in the order of "If the first request is unauthorized, then send another request with credentials".

Answer 2

# is an illegal character in a URL prior to the fragment portion of the URL, that is why it has to be encoded as %23 when used in other areas of the URL.

A username/password is not actually part of a real URL, that is why they get stripped off when TIdHTTP sends the URL to a server (monitor the traffic of any web browser and you will see the same thing happen).

To use HTTP authentication with TIdHTTP, you need to use the TIdHTTP.Request.Username and TIdHTTP.Request.Password properties instead (and you do not need to URL encode the values), eg:

IdHTTP1.Request.Username := 'user';
IdHTTP1.Request.Password := 'pass_with_#_sign';
IdHTTP1.Get('http://address:port/somespecificpath/');

If you pass a URL that has an encoded username/password in it, TIdHTTP will strip off the values and move them to the Request.Username and Request.Password properties for you, but they will remain in their original encoded format, eg:

IdHTTP1.Get('http://user:pass_with_%23_sign@address:port/somespecificpath/')
// this will set Request.Username to 'user',
// Request.Password to 'pass_with_%23_sign', and
// send a request for 'http://address:port/somespecificpath/'

If you are being given an encoded URL to start with, you can use the TIdURI class to manually decode it prior to then calling TIdHTTP.Get(), eg:

var
  RequestUrl: string;
  Uri: TIdURI;
begin
  RequestUrl := 'http://user:pass_with_%23_sign@address:port/somespecificpath/';
  ...
  Uri := TIdURI.Create(RequestURL);
  try
    IdHTTP1.Request.Username := TIdURI.URLDecode(Uri.UserName);
    IdHTTP1.Request.Password := TIdURI.URLDecode(Uri.Password);
    RequestURL := Uri.URI;
  finally
    Uri.Free;
  end;
  IdHTTP1.Get(RequestUrl);
  ...
end;

Update: either way, make sure you have appropriate IdAuthentication... units, or the IdAllAuthentications unit, in your uses clause to enable Indy's HTTP authentication classes for TIdHTTP to use.