How to clean up this code that fetches images from the Flickr database?

Question

Using the GWT API, I was able to pull images off of the Flickr database and insert them into a spread sheet but the process I used seemed to be unnecessarily painstaking.

Here is the function I used to pull the image URL off of the database:

function kwURL(keyword) {
  var response = UrlFetchApp.fetch("https://www.flickr.com/search/?q=" + keyword)
  var content = response.getContentText();
  var length = content.length;
  var count = 0;

  for (var i = 0; i <= length - 3; i++) {
    if (content.substring(i, 5 + i).equals("src=\"")) {
      var start = i + 5;
      
      count += 1
      
      if (count == 4) {
        for (var e = start; e < length - 1; e++) {
          if (content.substring(e, e + 1).equals("\"")) {
            var end = e
            
            break;
          }
        }
      }
      
      if (count == 4) {
        break;
      }
    }
  }
  
  Logger.log(content.substring(start, end));
  
  return content.substring(start, end):
}

The count variable just chooses the fourth image on the HTML page since the first three are logos and stuff like that.

Here is the code that pulls words from the first column and searches the keyword in the Flickr database and inserts the found image into the second column:

function onChange() {
  var sheet = SpreadsheetApp.getActiveSheet();
  var rows = sheet.getDataRange();
  var numRows = rows.getNumRows();
  
  for (var i = 2; i <= numRows + 2; i++) {
    var keyword = sheet.getRange(i, 1).getValue();
    
    if (keyword != "") {
      sheet.insertImage(kwURL(keyword), 2, i);
    }
  }
}

I could not find any other program that was trying to do the same thing in the Google Apps Script documentation or programs that I searched on the internet, so hopefully someone came clean up this messy code I made form scratch.

Answer 1

Screen-scraping (which is essentially what you're doing here) should only be used as a last resort. Any change to the Web page you're fetching could potentially break your code. A much better way to solve your problem is to use a publicly-available API provided for third-party developers to request some data in a standard format. And it looks like Flickr provides just that: https://www.flickr.com/services/api/flickr.photos.search.html (note: I have no experience with this API. I simply found out that it exists after a quick google search).

If you fetch the proper API resource, you should get back a JSON object containing a list of URLs pointing directly to images. No more weird for() loops, no more ugly (if magicNumber == 4) or if (magicString == '\"src\"') code. It should be pretty straightforward at that point to insert these images into your spreadsheet.