diff options
| author | kballou <kballou@devnulllabs.io> | 2015-09-03 18:00:02 -0600 |
|---|---|---|
| committer | kballou <kballou@devnulllabs.io> | 2017-09-02 19:31:44 -0600 |
| commit | 3c38a412fd3bcb24c817f79983fbbb28254e4493 (patch) | |
| tree | 67e49fad5cb3756ce7f31a70d63e101ded87654b | |
| parent | ce4efe31c0f949e567dc88f44ec7d03d48a47c16 (diff) | |
| download | blog.kennyballou.com-3c38a412fd3bcb24c817f79983fbbb28254e4493.tar.gz blog.kennyballou.com-3c38a412fd3bcb24c817f79983fbbb28254e4493.tar.xz | |
Adjust robot rules
| -rw-r--r-- | static/robots.txt | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/static/robots.txt b/static/robots.txt index 058446d..41b7ac2 100644 --- a/static/robots.txt +++ b/static/robots.txt @@ -1,5 +1,8 @@ -User-agent: * -Disallow: /about-me/ -Disallow: /css/ -Disallow: /js/ +User-agent: Googlebot +Disallow: /media/ + +User-agent: archive.org_bot Disallow: /media/ + +User-agent: * +Disallow: / |