[NTLK] Requiring registration to download?

Grant Hutchinson grant at splorp.com
Thu Apr 13 16:57:09 EDT 2017


>> If it was way back machine why would they download the same
>> files over and ove again, and ignore robots.txt?
> 
> Can't be the way back machine then.  I was going to ask about your
> robots.txt but that answered it.  Also it wouldn't be every day.
> Changes perhaps, every few days or weeks.  But not every day.

I agree with Dan. It isn't the Internet Archive doing the bulk downloading.

However, you might consider removing the following lines from your robots.txt file, because the Wayback Machine isn't currently archiving all of your .css files.

Disallow: /components/
Disallow: /modules/

g.

----------------------------------------------------------------------

Grant Hutchinson
NewtonTalk List Dad & Web Custodian





More information about the NewtonTalk mailing list