What to do with old access logs? [closed]

Question

I have 51 GB of HTTP (Apache) access logs for my website going back five years. I'm sort of a hoarder, as far as my personality, which is why I am reluctant to delete them, even though it's taking up more disk space than I would like. Should I bite the bullet and delete? Or anonymize?

What do people do with their old log files?

Answer 1

The answer really depends on what you usually need to do with logs, what is your business and your (company?) security policy.

On my side, I usually never delete logs. They are really useful to spot security holes, behaviors, trends.

If storage is a problem, consider to use a storage service such as Amazon S3 and don't forget to compress them! I usually move there all the gzipped logs older than one month.

Answer 2

Most log files that I've "hoarded" I will either zip up based on month/quarter/year then you'll have them, but log files zip nicely. If space is an issue, burn to DVDs (make more than one copy too).

Answer 3

That's a lot of logs. Are you using logrotate? Just zipping them up may give you the room you need. All my server logs are rolled into gzipped files. One years worth of logs from a small to medium corporate intranet is 174M.

Answer 4

I agree with others:

If it makes you feel better, compress and burn to disc.

If you need extra comfort, store offsite.

If you just need something to distract you from the decision of letting go, you can write a parser that strips your logs down to the bare essentials, finds the diffs, or otherwise mangles them so you have some boiled down, insightful, juicy data set. Then you can have two things to keep. :)

Answer 5

We have over 75 Gigs of logs a day zipped. We currently don't keep more than a months worth of logs for obvious reasons but we're looking into solutions including a DFS so we can easily add more. Storage is cheap but you always have to think about the "what ifs". At $0.15/month/gig you're only talking about $8/month to store on S3 and then you don't have to worry about loosing anything. I suggest you look into that.

Drives are cheap too, if you're that concerned pick up some cheap $50 drives and copy the logs to that and store them at a friends/relatives house. :-)