1. About puppeter

Puppeter is a node library. It provides a set of API s for manipulating chrome. Generally speaking, it is a headless chrome browser (of course, you can also configure it to have a UI, which is not available by default). Since it is a browser, puppeter can do all the things we can do manually on the browser. In addition, puppeter means "puppet" in Chinese, so you can easily manipulate it to achieve:

1) Generate web page screenshots or PDF
2) Advanced crawler, which can crawl web pages with a large amount of asynchronous rendering content
3) Simulate keyboard input, automatic form submission, web page login, etc., and realize UI automatic test
4) Capture the timeline of your site to track your site and help analyze site performance issues

If you have used PhantomJS, you will find that they are somewhat similar, but the puppeter is maintained by the official Chrome team. As the saying goes, "people with parents" has a better prospect.

2. Operating environment

Check the official API of puppeter and you will find the full screen async, await, etc. These are ES7 specifications, so you need to:

1. The version of Nodejs cannot be lower than v7.6.0. It needs to support async and await
2. You need the latest chrome driver, which will be downloaded automatically when you install the puppeter through npm

npm install puppeteer --save

3. Basic usage

First open the official DEMO

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

The above code implements the screenshot of the web page. First, briefly interpret the above lines of code:

1. First, use the puppeter Launch() creates a Browser instance Browser object
2. Then create a Page page object through the Browser object
3. Then page Goto() jump to the specified page
4. Call page Screenshot() takes a screenshot of the page
5. Close browser

Does it feel so simple? Anyway, I think it is simpler than PhantomJS, not to mention selenium webdriver. Here are some common API s for puppeter.

3.1 puppeteer.launch(options)

Use puppeter Launch () runs the puppeter, which return s a promise and uses the then method to obtain the browser instance. Of course, the higher version of nodejs already supports the await feature, so the above example uses the await keyword. This needs special explanation. Almost all puppeter operations are asynchronous. In order to use a large number of then to reduce the readability of the code, all demo codes in this article use async, await mode. This is also the writing method officially recommended by puppeter. Hard on the students of async/await Poke here

Detailed explanation of options parameter

Parameter name	Parameter type	Parameter description
ignoreHTTPSErrors	boolean	Whether to ignore the Https error message during the request process. The default value is false
headless	boolean	Whether to run chrome in "headless" mode, that is, do not display UI. The default value is true
executablePath	string	By default, puppeter uses its own chrome webdriver. If you want to specify your own webdriver path, you can set this parameter
slowMo	number	Slows the puppeter operation in milliseconds. This parameter is very useful if you want to see the whole working process of the puppeter.
args	Array(String)	Other parameters passed to the chrome instance. For example, you can use "– ash host window boundaries=1024x768" to set the browser window size. For more parameter lists, please refer to here
handleSIGINT	boolean	Whether to allow the process signal to control the chrome process, that is, whether to use CTRL+C to close and exit the browser
timeout	number	The maximum time to wait for the Chrome instance to start. The default is 30000 (30 seconds). No time limit if 0 is passed in
dumpio	boolean	Whether to import browser processes stdout and stderr into process Stdout and process In stderr. The default is false.
userDataDir	string	Set the USER data directory. By default, linux is in ~/ config directory. By default, the window is in C:\Users{USER}\AppData\Local\Google\Chrome\User Data, where {USER} represents the currently logged in USER name
env	Object	Specifies the environment variables visible to Chromium. The default is process env.
devtools	boolean	Whether to automatically open the DevTools panel for each tab. This option is valid only when headless is set to false

3.2 Browser object

When the puppeter is connected to a Chrome instance, a Browser object will be created. There are two ways:

Puppeteer.launch and puppeter connect.

The DEMO implementation below reconnects the browser instance after disconnecting

const puppeteer = require('puppeteer');

puppeteer.launch().then(async browser => {
  // preservation Endpoint，So you can reconnect  Chromium
  const browserWSEndpoint = browser.wsEndpoint();
  // from Chromium Disconnect
  browser.disconnect();

  // use endpoint Re and Chromiunm Establish connection
  const browser2 = await puppeteer.connect({browserWSEndpoint});
  // Close Chromium
  await browser2.close();
});

Browser object API

Method name	Return value	explain
browser.close()	Promise	Close browser
browser.disconnect()	void	Disconnect browser
browser.newPage()	Promise(Page)	Create a Page instance
browser.pages()	Promise(Array(Page))	Get all open Page instances
browser.targets()	Array(Target)	Get all active targets
browser.version()	Promise(String)	Get browser version
browser.wsEndpoint()	String	Returns the socket connection URL of the browser instance. You can reconnect the chrome instance through this URL

Well, I won't introduce the API of puppeter one by one. The detailed API provided by the official Poke here

4. Puppeter practice

After understanding the API, we can come to some actual combat. Before that, let's first understand the design principle of puppeter. In short, the biggest difference between puppeter and webdriver and PhantomJS is that it is designed from the perspective of user browsing. Webdriver and PhantomJS were originally designed to do automated testing, so it is designed from the perspective of machine browsing, So they use different design philosophies. For example, to join me, I need to open the home page of jd.com and conduct a product search to see the implementation process of using puppeter and webdriver:

Implementation process of puppeter:

1. Open JD Homepage
2. Place the cursor focus in the search input box
3. Keyboard click to input text
4. Click the search button

Implementation process of webdriver:

1. Open JD Homepage
2. Locate the input element of the input box
3. Set the value of input to the text to search
4. Standalone event that triggers the search button

Personally, I feel that the design philosophy of puppeter is more in line with any operation habit and more natural.

Let's use a simple requirement implementation to learn about the introduction of puppeter. This simple requirement is:

Grab 10 mobile phone products in Jingdong Mall and take screenshots of the product details page.

First, let's sort out the operation process

1. Open JD Homepage
2. Enter "mobile" keyword and search
3. Get the A tag of the top 10 products, get the href attribute value, and get the product details link
4. Open the detail pages of 10 commodities respectively, and intercept the web page pictures

To realize the above functions, you need to find elements, obtain attributes, keyboard events, etc. next, we will explain them one by one.

4.1 getting elements

The Page object provides two API s to get Page elements

(1). Page.$(selector) get a single element. The bottom layer is to call document Queryselector(), so the selector format of the selector follows css selector specification

let inputElement = await page.$("#search", input => input);
//The following is equivalent
let inputElement = await page.$('#search');

(2). Page.$$(selector) get a group of elements, and the underlying call is document querySelectorAll(). Returns an array of Promise(Array(ElemetHandle)) elements

const links = await page.$$("a");
//The following is equivalent
const links = await page.$$("a", links => links);

All the returned objects are ElemetHandle objects

4.2 getting element attributes

The logic for puppeter to obtain element attributes is a little different from that of js in the previous paragraph. According to the usual logic, it should obtain the element now and then obtain the element attributes. However, as we know above, the API for obtaining elements ultimately returns ElemetHandle objects. If you check the API for ElemetHandle, you will find that it does not have an API for obtaining element attributes

In fact, puppeter specifically provides a set of API s for obtaining attributes, page$ Eval() and page$$ eval()

(1). Page.$$eval(selector, pageFunction[,... args]) to obtain the attributes of a single element. The selector here is the same as page$ (selector) is the same.

const value = await page.$eval('input[name=search]', input => input.value);
const href = await page.$eval('#a", ele => ele.href);
const content = await page.$eval('.content', ele => ele.outerHTML);

4.3 execute customized JS script

The Page object of the puppeter provides a series of evaluate methods through which you can execute some custom js code. The following three API s are provided

(1). page.evaluate(pageFunction,... args) returns a serializable ordinary object. pageFunction represents the function to be executed on the page. Args represents the parameters passed to pageFunction. The following pageFunction and args represent the same meaning.

const result = await page.evaluate(() => {
    return Promise.resolve(8 * 7);
});
console.log(result); // prints "56"

This method is very useful. For example, when we get a screenshot of a page, the default is to only take a screenshot of the size of the current browser window. The default value is 800x600. If we need to get a complete screenshot of the whole web page, we can't do it. Page. The screenshot () method provides parameters that can set the size of the screenshot area, so we can solve this problem as long as we get the width and height of the page after the page is loaded.

(async () => {
    const browser = await puppeteer.launch({headless:true});
    const page = await browser.newPage();
    await page.goto('https://jr.dayi35.com');
    await page.setViewport({width:1920, height:1080});
    const documentSize = await page.evaluate(() => {
        return {
            width: document.documentElement.clientWidth,
            height : document.body.clientHeight,
        }
    })
    await page.screenshot({path:"example.png", clip : {x:0, y:0, width:1920, height:documentSize.height}});

    await browser.close();
})();

(2). Page.evaluateHandle(pageFunction,... args) executes a pageFunction in the page context and returns the JSHandle entity

const aWindowHandle = await page.evaluateHandle(() => Promise.resolve(window));
aWindowHandle; // Handle for the window object. 

const aHandle = await page.evaluateHandle('document'); // Handle for the 'document'.

As you can see from the above code, page The evaluatehandle () method also passes Promise The resolve method directly returns the final processing result of Promise, but encapsulates the last returned object into a JSHandle object. It is essentially no different from evaluate.

The following code is used to obtain the dynamic (including js dynamically inserted elements) HTML code of the page

const aHandle = await page.evaluateHandle(() => document.body);
const resultHandle = await page.evaluateHandle(body => body.innerHTML, aHandle);
console.log(await resultHandle.jsonValue());
await resultHandle.dispose();

(3). page.evaluateOnNewDocument(pageFunction,... args) calls pageFunction before the document page is loaded. If there is an iframe or frame in the page, the context of the function call will become a sub page, that is, iframe or frame. Since it is called before the page is loaded, this function is generally used to initialize the javascript environment, such as resetting or initializing some global variables.

4.4 Page.exposeFunction

In addition to the above three APIs, there is a similar and very useful API, that is, page Exposefunction, an API used to register global functions on a page, is very useful:

Because sometimes some functions are needed to process some operations on the page, although you can use page The evaluate() API defines functions on the page, such as:

const docSize = await page.evaluate(()=> {
    function getPageSize() {
        return {
            width: document.documentElement.clientWidth,
            height : document.body.clientHeight,
        }
    }

    return getPageSize();
});

However, such functions are not global and need to be redefined in each evaluate. Code reuse cannot be achieved. For example, there are many toolkits in nodejs that can easily implement complex functions, such as md5 encryption function. It is not convenient to use pure js to implement this function, but nodejs is a matter of several lines of code.

The following code implements adding an md5 function to the window object of the Page context:

const puppeteer = require('puppeteer');
const crypto = require('crypto');

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg.text));
  await page.exposeFunction('md5', text =>
    crypto.createHash('md5').update(text).digest('hex')
  );
  await page.evaluate(async () => {
    // use window.md5 to compute hashes
    const myString = 'PUPPETEER';
    const myHash = await window.md5(myString);
    console.log(`md5 of ${myString} is ${myHash}`);
  });
  await browser.close();
});

As you can see, page The exposefunction API is very convenient and useful to use. For example, register the readfile global function for the window object:

const puppeteer = require('puppeteer');
const fs = require('fs');

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  page.on('console', msg => console.log(msg.text));
  await page.exposeFunction('readfile', async filePath => {
    return new Promise((resolve, reject) => {
      fs.readFile(filePath, 'utf8', (err, text) => {
        if (err)
          reject(err);
        else
          resolve(text);
      });
    });
  });
  await page.evaluate(async () => {
    // use window.readfile to read contents of a file
    const content = await window.readfile('/etc/hosts');
    console.log(content);
  });
  await browser.close();
});

5,Page. Simulate modifying the simulator (client) running configuration

Puppeter provides some API s for us to modify the configuration of browser terminals

1. Page.setViewport() modify browser window size
2. Page.setUserAgent() sets the UserAgent information of the browser
3. Page.emulateMedia() changes the CSS media type of the page for simulated media emulation. The optional values are "screen", "print" and "null". If it is set to null, media emulation is disabled.
4. Page. Simulate() simulates devices and parameter device objects, such as iPhone, Mac, Android, etc

page.setViewport({width:1920, height:1080}); //Set the window size to 1920 x1080
page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36');
page.emulateMedia('print'); //Set printer media style

In addition, we can also simulate non PC devices. For example, the following code simulates iPhone 6 accessing google

const puppeteer = require('puppeteer');
const devices = require('puppeteer/DeviceDescriptors');
const iPhone = devices['iPhone 6'];

puppeteer.launch().then(async browser => {
  const page = await browser.newPage();
  await page.emulate(iPhone);
  await page.goto('https://www.google.com');
  // other actions...
  await browser.close();
});

Puppeter supports simulation of many devices, such as Galaxy, iPhone, IPad, etc. for detailed device support, please click here DeviceDescriptors.js.

6. Keyboard and mouse

The API of keyboard and mouse are relatively simple. Several APIs of keyboard are as follows:

• keyboard.down(key[, options]) triggers the keydown event
• keyboard.press(key[, options]) press a key. The key indicates the name of the key, such as the arrow left key. For detailed key name mapping, see Poke here
• keyboard.sendCharacter(char) enter a character
• keyboard.type(text, options) enter a string
• keyboard.up(key) triggers the keyup event

page.keyboard.press("Shift"); //Press Shift key
page.keyboard.sendCharacter('hi');
page.keyboard.type('Hello'); // One time input completion
page.keyboard.type('World', {delay: 100}); // Input slowly like a user

Mouse operation:

• mouse.click(x, y, [options]) to move the mouse pointer to the specified position, and then press the mouse. This is actually mouse Move and mouse Down or mouse Up shortcut
• mouse.down([options]) triggers the mousedown event. Options are configurable:
  • options.button. The optional values are [left, right, middle]. The default value is left, indicating the left mouse button
  • options.clickCount the number of presses, clicks, double clicks, or other times
  • Delay key delay time
• mouse.move(x, y, [options]) move the mouse to the specified position, options Steps indicates the step size of the movement
• mouse.up([options]) triggers the mouseup event

7. Several other useful API s

Puppeter also provides several very useful API s, such as:

7.1 Page.waitFor series API

• page.waitFor(selectorOrFunctionOrTimeout[, options[,... args]]) integrated API s of the following three
• page.waitForFunction(pageFunction[, options[,... args]]) waits for the completion of pageFunction execution
• page.waitForNavigation(options) waits for the basic elements of the page to be loaded, such as synchronized HTML, CSS, JS and other codes
• page.waitForSelector(selector[, options]) waits for the element of a selector to be loaded asynchronously. This API is very useful, you know.

For example, if I want to get an element loaded asynchronously through js, I can't get it directly. You can use page Waitforselector to solve:

await page.waitForSelector('.gl-item'); //Wait for the element to load, otherwise the asynchronously loaded element cannot be obtained
const links = await page.$$eval('.gl-item > .gl-i-wrap > .p-img > a', links => {
    return links.map(a => {
        return {
            href: a.href.trim(),
            name: a.title
        }
    });
});

In fact, the above code can solve our top requirements and capture JD products. Because they are loaded asynchronously, this method is used.

7.2 page.getMetrics()

Via page Getmetrics () can get some page performance data and capture the timeline trace of the website to help diagnose performance problems.

• Timestamp timestamp of the metric sample
• Documents page documents
• Frames page frame s
• Number of event listeners in JSEventListeners page
• Nodes page DOM nodes
• LayoutCount total page layouts
• RecalcStyleCount style recalculation
• LayoutDuration merge duration of all page layouts
• RecalcStyleDuration the combined duration of all page style recalculations.
• ScriptDuration duration of all script executions
• TaskDuration all browser task duration
• Jsheapsusedsize the heap size occupied by JavaScript
• JSHeapTotalSize JavaScript heap total

8. Summary and source code

This article has learned some basic and common APIs of puppeter through a practical requirement. The API version is v0.13.0-alpha Please refer to the latest state-of-the-art API Official puppeter API.

In general, puppeter is really a good headless tool with simple operation and powerful functions. It is good for UI automation testing and some small tools.

The requirements implementation source code we started is posted below for reference only:

//Delay Functions 
function sleep(delay) {
    return new Promise((resolve, reject) => {
        setTimeout(() => {
            try {
                resolve(1)
            } catch (e) {
                reject(0)
            }
        }, delay)
    })
}

const puppeteer = require('puppeteer');
puppeteer.launch({
    ignoreHTTPSErrors:true, 
    headless:false,slowMo:250, 
    timeout:0}).then(async browser => {

    let page = await browser.newPage();
    await page.setJavaScriptEnabled(true);
    await page.goto("https://www.jd.com/");
    const searchInput = await page.$("#key");
    await searchInput.focus(); //Navigate to the search box
    await page.keyboard.type("mobile phone");
    const searchBtn = await page.$(".button");
    await searchBtn.click();
    await page.waitForSelector('.gl-item'); //Wait for the element to load, otherwise get the element that is not loaded asynchronously
    const links = await page.$$eval('.gl-item > .gl-i-wrap > .p-img > a', links => {
        return links.map(a => {
            return {
                href: a.href.trim(),
                title: a.title
            }
        });
    });
    page.close();

    const aTags = links.splice(0, 10);
    for (var i = 1; i < aTags.length; i++) {
        page = await browser.newPage()
        page.setJavaScriptEnabled(true);
        await page.setViewport({width:1920, height:1080});
        var a = aTags[i];
        await page.goto(a.href, {timeout:0}); //Prevent the page from being too long and loading timeout

        //Inject code and slowly slide the scroll bar to the bottom to ensure that all elements are loaded
        let scrollEnable = true;
        let scrollStep = 500; //Steps per scroll
        while (scrollEnable) {
            scrollEnable = await page.evaluate((scrollStep) => {
                let scrollTop = document.scrollingElement.scrollTop;
                document.scrollingElement.scrollTop = scrollTop + scrollStep;
                return document.body.clientHeight > scrollTop + 1080 ? true : false
            }, scrollStep);
            await sleep(100);
        }
        await page.waitForSelector("#footer-2014", {timeout:0}); //Determine if the bottom is reached
        let filename = "images/items-"+i+".png";
        //Here's a Puppeteer of bug It hasn't been resolved. I found that the maximum height of the screenshot can only be 16384 px， The excess was cut off.
        await page.screenshot({path:filename, fullPage:true});
        page.close();
    }

    browser.close();
});

Reprint: http://www.r9it.com/20171106/puppeteer.html