What are the pages would you recommend disallowing in robots.txt file?

Guest · Post by **Guest** » Thu Aug 25, 2022 8:55 am

Hey guys, what are the pages would you recommend disallowing in robots.txt file?

Thank you!

Adam · Post by **Adam** » Thu Aug 25, 2022 9:22 am

images is a common thing to ignore if you don't have unlimited hosting bandwidth

Johan · Post by **Johan** » Thu Aug 25, 2022 9:23 am

In short; nothing

Because if something shouldn't be indexed, it shouldn't be accessable.

Although it can help to disallow admin panels (like /wp-admin/), but other than that keep everything allowed for all bots.

Not to mention that some bots even ignore robots.txt or the robots meta, so ensuring your application doesn't give access is more important anyway.