Sign Up for Free

RunKit +

Try any Node.js package right in your browser

This is a playground to test code. It runs a full Node.js environment and already has all of npm’s 400,000 packages pre-installed, including robotize with all npm packages installed. Try it out:

var robotize = require("robotize")

This service is provided by RunKit and is not affiliated with npm, Inc or the package authors.

robotize v1.1.0

Generates a robots.txt

robotize

npm version build status downloads

Generates a robots.txt

This module generates a robots.txt. The generated robots.txt conforms to the standards set by Google. Use it to programmatically generate a robots.txt file for your site.

Installation

$ npm install robotize

Example

const robotize = require("robotize");
const opts = {
  useragent: "googlebot",
  allow: ["index.html", "about.html"],
  disallow: ["404.html"],
  sitemap: "https://www.site.com/sitemap.xml"
};

robotize(opts, (err, robots) => {
  if (err) {
    throw new Error(err);
  } else {
    console.log(robots);
  }
});

Will log:

User-agent: googlebot
Allow: index.html
Allow: about.html
Disallow: 404.html
Sitemap: https://www.site.com/sitemap.xml

Options

Robotize accepts an object with options. The options are:

  • useragent: the useragent - String, default: *
  • allow: an array of the url(s) to allow - Array of Strings
  • disallow: an array of the url(s) to disallow - Array of Strings
  • sitemap: the sitemap url - String

Robotize expects at least one of the last three options. So either allow, disallow or sitemap must be passed.

Credits

Forked from robots-generator.

License

MIT

Metadata

RunKit is a free, in-browser JavaScript dev environment for prototyping Node.js code, with every npm package installed. Sign up to share your code.
Sign Up for Free