robotstxt.org valuation and analysis
Robots.txt Information
| Robot | Path | Permission |
| GoogleBot | / | ✔ |
| BingBot | / | ✔ |
| BaiduSpider | / | ✔ |
| YandexBot | / | ✔ |
User-agent: * | ||
Meta Tags
| Title | The Web Robots |
| Description | skip to content Advertisement Navigation The /robots.txt <META> tags Frequently Asked Questions Mailing list Other Sites About robotstxt.org Tools / |
| Keywords | N/A |
Server Information
| WebSite | |
| Host IP | 93.93.131.3 |
| Location | United Kingdom |
Related Websites
| Site | Rank |
More to Explore
robotstxt.org Valuation
|
US$20,545,463
Last updated: 2023-04-26 17:09:32
robotstxt.org has Semrush global rank of 515,165. robotstxt.org has an estimated worth of US$ 20,545,463, based on its estimated Ads revenue. robotstxt.org receives approximately 2,370,631 unique visitors each day. Its web server is located in United Kingdom, with IP address 93.93.131.3. According to SiteAdvisor, robotstxt.org is safe to visit. |
Traffic & Worth Estimates
| Purchase/Sale Value | US$20,545,463 |
| Daily Ads Revenue | US$18,966 |
| Monthly Ads Revenue | US$568,952 |
| Yearly Ads Revenue | US$6,827,416 |
| Daily Unique Visitors | 158,043 |
| Note: All traffic and earnings values are estimates. | |
DNS Records
| Host | Type | TTL | Data |
| robotstxt.org. | A | 3599 | IP: 93.93.131.3 |
| robotstxt.org. | AAAA | 3598 | IPV6: 2a00:1098:0:86:1000::33 |
| robotstxt.org. | NS | 86400 | NS Record: ns2.mythic-beasts.com. |
| robotstxt.org. | NS | 86400 | NS Record: ns1.mythic-beasts.com. |
| robotstxt.org. | MX | 3600 | MX Record: 10 mx.mythic-beasts.com. |
| robotstxt.org. | TXT | 300 | TXT Record: google-site-verification=FuTkdKut4zpT5f61O29qfjnWKg3u5iA292tOglYwZuo |
| robotstxt.org. | TXT | 300 | TXT Record: v=spf1 include:_spf.mythic-beasts.com ~all |
HtmlToTextCheckTime:2023-04-26 17:09:32
| skip to content Advertisement Navigation The /robots.txt <META> tags Frequently Asked Questions Mailing list Other Sites About robotstxt.org Tools /robots.txt checker Robots Database IP lookup Advertisement The Web Robots Pages Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. On this site you can learn more about web robots. About /robots.txt explains what /robots.txt is, and how to use it. The FAQ answers many frequently asked questions, such as How do I stop robots visiting my site? and How can I get the best listing in search engines?" The Other Sites page links to external resources for robot writers and webmasters. The Robots Database has a list of robots. The /robots.txt checker can check your site’s /robots.txt file and meta tags. The IP Lookup can help find out more |
Ads.txtCheckTime:2023-04-26 17:09:32
google.com, pub-9311532361854131, DIRECT, |
HTTP Headers
HTTP/1.1 301 Moved Permanently Date: Wed, 20 Oct 2021 05:14:20 GMT Server: Apache/2.4.38 (Debian) Location: http://www.robotstxt.org/ Cache-Control: max-age=3600 Expires: Wed, 20 Oct 2021 06:14:20 GMT Content-Type: text/html; charset=iso-8859-1 HTTP/1.1 200 OK Date: Wed, 20 Oct 2021 05:14:20 GMT Server: Apache/2.4.38 (Debian) Vary: User-Agent,Accept-Encoding Last-Modified: Sun, 03 May 2020 13:29:25 GMT ETag: "110b-5a4be6736ce42" Accept-Ranges: bytes Content-Length: 4363 Content-Type: text/html |
robotstxt.org Whois Information
Domain Name: ROBOTSTXT.ORG Registry Domain ID: D34003338-LROR Registrar WHOIS Server: whois.tucows.com Registrar URL: http://www.tucows.com Updated Date: 2021-08-25T20:29:44Z Creation Date: 2000-09-04T14:43:31Z Registry Expiry Date: 2022-09-04T14:43:31Z Registrar: Tucows Domains Inc. Registrar IANA ID: 69 Registrar Abuse Contact Email: domainabuse@tucows.com Registrar Abuse Contact Phone: +1.4165350123 Domain Status: ok https://icann.org/epp#ok Registrant Organization: none Registrant State/Province: Norfolk Registrant Country: GB Name Server: NS1.MYTHIC-BEASTS.COM Name Server: NS2.MYTHIC-BEASTS.COM DNSSEC: signedDelegation URL of the ICANN Whois Inaccuracy Complaint Form https://www.icann.org/wicf/) >>> Last update of WHOIS database: 2021-09-08T13:23:38Z <<< |