Common Crawl Web Corpus
An open web-crawl corpus for researchers building text datasets and analysing the public web.
About this listing
Common Crawl maintains an open repository of web-crawl data for research and analysis. The corpus spans multiple years and receives new pages each month, making it useful for teams assembling language datasets or studying changes across the web.
The foundation provides discovery surfaces including a URL index, a CDXJ index and web graphs. These help researchers locate relevant crawl material before planning their own extraction and analysis.
Access to the corpus is free. Compute, storage and any separately selected model services are not included. Users should review the foundation's terms and applicable rights in underlying web content before reuse; open access is not a blanket licence to republish every crawled page.