OpenWebSpider is a small web crawler and search indexer written in C. It
supports loadable modules for content filters and media/PDF extraction.
