Methanol is a modular, customizable Web crawling system with crawlers optimized for speed. It is designed to allow the administrator to set up any kind of filetype handling, parsing, and indexing rules.
|Tags||spider web crawling web crawler|
|Operating Systems||Linux FreeBSD|
|Implementation||C MySQL PHP|
Release Notes: This releases features new server-side programs as well as lots of improvements to the Web crawler client. There is now full support for HTML entities, conversion between character encodings, and many new features.