Sign Up for Free

RunKit +

Try any Node.js package right in your browser

This is a playground to test code. It runs a full Node.js environment and already has all of npm’s 400,000 packages pre-installed, including crawlx with all npm packages installed. Try it out:

var crawlx = require("crawlx")

This service is provided by RunKit and is not affiliated with npm, Inc or the package authors.

crawlx v1.5.2

crawlx is a Lightweight web crawler with powerful plugins!


Build Status docs npm NPM Coverage Status

⚡Lightweight web crawler with powerful plugins!

const x = require("crawlx").default;

  url: "",
  parse: [
      author: ".author",
      authorUrl: ".author+a@href",
      text: ".text",
      tags: "[a.tag]",
      type: () => "quote"
    s => ((s["crawled"] = new Date()), s)
  follow: ["[.author+a@href]", followAuthorRule]
}).then(task => {

function followAuthorRule(url) {
  return {
    parse: {
      name: ["h3 | reverse", v => v.toUpperCase()],
      born: ".author-born-date | date"
    callback(task) {


  • Make http request with got
  • Priority queue of requests
  • Simple plugin system
  • Promise support
  • Flexible schema with powerful parse plugin, using only one rule object
  • Easily paginate and follow links with builtin follow plugin
  • Spawner mode: add a url directly


npm install crawlx



See more examples: crawlx/examples

RunKit is a free, in-browser JavaScript dev environment for prototyping Node.js code, with every npm package installed. Sign up to share your code.
Sign Up for Free