Another interesting day at work today. I spent another day with Dave N, just looking at the code really. Driven by error messages and exceptions in the log files, tracking them down, fixing them where we can (exceptions on the mainframe are soooo expensive). We found one error caused by a robot, TurnItIn, hitting the site, failing to correctly interpret or ignore some links written to the page with javascript (don’t ask 😉 ), and then making invalid requests as it tried to follow those ‘links’. Hopefully the robots.txt fix for that one will work. Quite why turnitin is trying to crawl a clothing catalogue site I have no idea. No academic papers here!
Short link to this post: https://z1.tl/3q
