I have to disagree with you for several reasons.
1. Google is now displaying breadcrumbs in SERP and they are more likely to do so based on hierarchies.
2. Hierarchically designed IA makes it MUCH easier to analyze indexation and web traffic.
3. While internal linking structures are clearly driving SERP, it is much easier to understand and visualize your internal linking structures if you have a hierarchical structure.
4. Someday, you may need to move some content around or do other tricky and inconvenient things. You will thank yourself if you took the time to build out a really solid IA first.
5. Believe it or not, some users do see the URL and it helps establish information scent.

Having everything “flat”, that is the URI’s path identifies the resource without using slashes, is an option, and just that. You can do it, combine it with slash delimited paths, query strings, … whatever. Always do what’s best for the actual site, and don’t listen to crappy advice that tells you otherwise, even when it comes from a major search engine and is beginner level stuff trying to bring a first understanding of what information architecture is to the noobs.

Just bear in mind that neither search engines nor human vistors care much about your URIs - they follow links, and only links. So if it’s difficult to create a hierarchical structure (usually it’s raping common sense at some point), then don’t do it, or do it just for parts of your content where it makes sense.

are you advocating everything being flat 1 tier?

whats the benefits of this?

Theres no doubt that sites like the bbc make good use of folder like structures??

Thanks a lot for sharing man.

In the first case Chrom really doesn’t send referer, but in second - does. And this behavior doesn’t depend on source referer. I’ve checked twice

[No performance issues.]

That said, I do think the solution itself is pretty slick.

[You didn’t get it. A smart robots.txt is just one weapon in the arsenal. And yes, some rogue bots request it seeking site areas that aren’t fully indexed caused by Disallow: statements in robots.txt, usually to avoid duplicate content issues with reprinted content. Once they request robots.txt, they’ll never see any content again. As said above, rogue bot detection has to be done before any piece of content is served to any user agent. A smart robots.txt is just one source that detects and persists sneaky requests, so that the offender can be properly blocked from everything.]