A whole site — into a text base for a neural net

The scenario crawls the site, gathers links, grabs page text, tidies it up and lays it out in a sheet.

  • Crawls the site by its links
  • Separates images from pages
  • Grabs and tidies the text
  • Lays it all out in a sheet