Renaming HTML files using <title> tags

Question

I'm a relatively new to programming. I have a folder, with subfolders, which contain several thousand html files that are generically named, i.e. 1006.htm, 1007.htm, that I would like to rename using the tag from within the file.

For example, if file 1006.htm contains Page Title , I would like to rename it Page Title.htm. Ideally spaces are replaced with dashes.

I've been working in the shell with a bash script with no luck. How do I do this, with either bash or python?

this is what I have so far..

#!/usr/bin/env bashFILES=/Users/Ben/unzipped/*
for f in $FILES
do
   if [ ${FILES: -4} == ".htm" ]
      then
    awk 'BEGIN{IGNORECASE=1;FS="<title>|</title>";RS=EOF} {print $2}' $FILES
   fi
done

I've also tried

#!/usr/bin/env bash
for f in *.html;
   do
   title=$( grep -oP '(?<=<title>).*(?=<\/title>)' "$f" )
   mv -i "$f" "${title//[^a-zA-Z0-9\._\- ]}".html   
done

But I get an error from the terminal exlaing how to use grep...

Answer 1

use awk instead of grep in your bash script and it should work:

#!/bin/bash   
for f in *.html;
   do
   title=$( awk 'BEGIN{IGNORECASE=1;FS="<title>|</title>";RS=EOF} {print $2}' "$f" )
   mv -i "$f" "${title//[^a-zA-Z0-9\._\- ]}".html   
done

don't forget to change your bash env on the first line ;)

EDIT full answer with all the modifications

#!/bin/bash
for f in `find . -type f | grep \.html`
   do
   title=$( awk 'BEGIN{IGNORECASE=1;FS="<title>|</title>";RS=EOF} {print $2}' "$f" )
   mv -i "$f" "${title//[ ]/-}".html
done

Answer 2

Here is a python script I just wrote:

import os
import re

from lxml import etree


class MyClass(object):
    def __init__(self, dirname=''):
        self.dirname   = dirname
        self.exp_title = "<title>(.*)</title>"
        self.re_title  = re.compile(self.exp_title)

    def rename(self):
        for afile in os.listdir(self.dirname):
            if os.path.isfile(afile):
                originfile = os.path.join(self.dirname, afile)
                with open(originfile, 'rb') as fp:
                    contents = fp.read()
                try:
                    html  = etree.HTML(contents)
                    title = html.xpath("//title")[0].text
                except Exception as e:
                    try:
                        title = self.re_title.findall(contents)[0]
                    except Exception:
                        title = ''

                if title:
                    newfile = os.path.join(self.dirname, title)
                    os.rename(originfile, newfile)


>>> test = MyClass('/path/to/your/dir')
>>> test.rename()

Answer 3

You want to use a HTML parser (likelxml.html) to parse your HTML files. Once you've got that, retrieving the title tag is one line (probably page.get_element_by_id("title").text_content()).

Translating that to a file name and renaming the document should be trivial.

Answer 4

A python3 recursive globbing version that does a bit of title sanitising before renaming.

import re
from pathlib import Path
import lxml.html


root = Path('.')
for path in root.rglob("*.html"):

    soup = lxml.html.parse(path)
    title_els = soup.xpath('/html/head/title')

    if len(title_els):
        title = title_els[0].text

        if title:
            print(f'Original title {title}')
            name = re.sub(r'[^\w\s-]', '', title.lower())
            name = re.sub(r'[\s]+', '-', name)
            new_path = (path.parent/name).with_suffix(path.suffix)

            if not Path(new_path).exists():
                print(f'Renaming [{path.absolute()}] to [{new_path}]')
                path.rename(new_path)
            else:
                print(f'{new_path.name} already exists!')