How to retrieve a large number of pom.xml files from github?

Question

We are working on a project where we need a large number of Maven projects and collect dependency information from the POM files.

Can anyone suggest, how can I retrieve github project information according to this use case?

Answer 1

There are two approaches, using the github search api or a few calls to their other apis (explained below).

Either way, just look through their docs. I looked through their docs for a few minutes and found all this stuff. It'll probably help you a lot if you look too.

---

First get the name of all the public repositories (docs):

GET /repositories

And then for each repository you can get the tree (docs):

GET /repos/:owner/:repo/git/trees/:sha

Which will return something like this:

{
  "sha": "9fb037999f264ba9a7fc6274d15fa3ae2ab98312",
  "url": "https://api.github.com/repos/octocat/Hello-World/trees/9fb037999f264ba9a7fc6274d15fa3ae2ab98312",
  "tree": [
    {
      "path": "file.rb",
      "mode": "100644",
      "type": "blob",
      "size": 30,
      "sha": "44b4fc6d56897b048c772eb4087f854f46256132",
      "url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/44b4fc6d56897b048c772eb4087f854f46256132"
    },
    {
      "path": "subdir",
      "mode": "040000",
      "type": "tree",
      "sha": "f484d249c660418515fb01c2b9662073663c242e",
      "url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/f484d249c660418515fb01c2b9662073663c242e"
    },
    {
      "path": "exec_file",
      "mode": "100755",
      "type": "blob",
      "size": 75,
      "sha": "45b983be36b73c0788dc9cbcb76cbb80fc7bb057",
      "url": "https://api.github.com/repos/octocat/Hello-World/git/blobs/45b983be36b73c0788dc9cbcb76cbb80fc7bb057"
    }
  ]
}

And then you can check each path to see if it contains pom.xml