Download all messages from a Google group

Question

I'm in the process of developing a Google apps migration/archive system and at this point in development I'm trying to come up with a way to download all messages in all the groups that my domain users have created. I know that I can set up forwarding filters and have all messages archived to an email, but this doesn't help with older messages.

Is there a way to download these messages from a Google group and if so, is there away in the admin API to get a list of all groups that users have created?

Answer 1

If you don't mind using #bash, you may try a tool I wrote

https://github.com/icy/google-group-crawler

It can download all mbox files from Google Group. If you have a cookie file, you can even download all files from a private Google Group, and/or to see all original emails. It can also read rss feeds and fetch the latest posts ; and this is useful for daily mirror.

An example result is here http://l.archlinuxvn.org/archlinuxvn/. MHonArch is used to convert mbox files into HTML format.

Answer 2

Ultimately I ended up using the gdata python library to get a list of all groups along with their respective URLs. From there I used selenium to scrape the groups for messages and all replies. Probably not the best solution but it works for what I need.

Answer 3

I made a simple scrap utility by using selenium and htmlunit.. you can use it.. it is not very optimized and can help you download messages of small groups only(up-to 7000 msgs)

https://github.com/himukr/google-grp-scraper

Answer 4

A work around could be where we try something like the below

1. Forward google groups messages to an email box
2. Use the IMAP Protocol & Gmail's app passwords to download all emails with the tag
3. Sort and filter through the resulting emails

Note: Not sure if this will work for historical messages, only the newer ones