CeWL v2.2 (Custom Word List generator) - released

Version 2.2

Added grabbing words from the meta keywords and description tags, from HTML comments and from select HTML attribute tags, currently alt and title. If you want to add more attributes just edit the attribute_names array to add new ones.

CeWL will spider a target site and generate up to three lists:

• A word list of all unique words found on the target site
• A list of all email addresses found in mailto links
• A list of usernames/author details from meta data found in any documents on the site

CeWL needs the rubygems package to be installed along with the following gems:

• http_configuration
• mime-types
• mini_exiftool
• rubyzip
• spider

Usage

cewl [OPTION] ... URL

• —help, -h: Show help
• —depth x, -d x: The depth to spider to, default 2
• —min_word_length, -m: The minimum word length, this strips out all words under the specified length, default 3
• —offsite, -o: By default, the spider will only visit the site specified. With this option it will also visit external sites
• —write, -w file: Write the ouput to the file rather than to stdout
• —ua, -u user-agent: Change the user agent
• -v: Verbose, show debug and extra output
• —no-words, -n: Don’t output the wordlist
• —meta, -a: Include meta data
• —email, -e: Include email adtresses
• —meta-temp-dir directory: The directory used used by exiftool when parsing files, the default is /tmp
• URL: The site to spider.

If you need to use a proxy server, you will need to uncomment three lines in CeWL, just look through for the comments.

More information about CeWL