摘要
主题网络爬虫采用集中式体系结构,具有对单台服务器性能要求高、可扩展性差等缺点。提出了一种基于Hadoop的分布式主题网络爬虫架构,通过将主题网络爬虫部署在分布式集群中的不同机器,运用MapReduce编程模型对数据进行抓取分析,使不同机器共同完成对指定任务的抓取工作。实验证明,采用分布式架构,通过动态调节分布式集群中的节点个数,能够明显改善主题网络爬虫的抓取效果。
Topic Web crawler uses a centralized architecture for a single server have high performance requirements, seal- ability poor shortcomings,this paper presents a distributed topic crawler Hadoop--based architecture. Topic by different machines in a distributed Web crawler deployment cluster, using the MapReduce programming model for data analysis crawl, crawl all the different machines together to complete work on a given task. Experiments show that the use of a dis tributed architecture, distributed by dynamically adjusting the number of nodes in the cluster, can significantly improve the topic crawler to crawl effect.
出处
《软件导刊》
2016年第3期24-26,共3页
Software Guide
