diff options
| author | kballou <kballou@devnulllabs.io> | 2015-09-03 18:00:02 -0600 |
|---|---|---|
| committer | kballou <kballou@devnulllabs.io> | 2015-09-03 18:00:02 -0600 |
| commit | aa9d8394d847e4822e3e14a80a04e852bade4d6c (patch) | |
| tree | e63bfcaab008588f91e0a98584051925a8b03bda | |
| parent | cab7abff941cefbce24dfebd05d002d09b5f56d0 (diff) | |
| download | kennyballou.com-aa9d8394d847e4822e3e14a80a04e852bade4d6c.tar.gz kennyballou.com-aa9d8394d847e4822e3e14a80a04e852bade4d6c.tar.xz | |
Adjust robot rules
| -rw-r--r-- | blag/static/robots.txt | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/blag/static/robots.txt b/blag/static/robots.txt index 058446d..41b7ac2 100644 --- a/blag/static/robots.txt +++ b/blag/static/robots.txt @@ -1,5 +1,8 @@ -User-agent: * -Disallow: /about-me/ -Disallow: /css/ -Disallow: /js/ +User-agent: Googlebot +Disallow: /media/ + +User-agent: archive.org_bot Disallow: /media/ + +User-agent: * +Disallow: / |