如何在多个线程中运行`selenium-chromedriver`

How to run `selenium-chromedriver` in multiple threads(如何在多个线程中运行`selenium-chromedriver`)

问题描述

我正在使用 selenium 和 chrome-driver 从某些页面 scrape 数据，然后使用该信息运行一些额外的任务(例如，在某些页面上输入一些评论)

I am using selenium and chrome-driver to scrape data from some pages and then run some additional tasks with that information (for example, type some comments on some pages)

我的程序有一个按钮.每次按下它都会调用 thread_(self)(如下)，开始一个新线程.目标函数 self.main 具有在 chrome-driver 上运行所有 selenium 工作的代码.

My program has a button. Every time it's pressed it calls the thread_(self) (bellow), starting a new thread. The target function self.main has the code to run all the selenium work on a chrome-driver.

def thread_(self):
    th = threading.Thread(target=self.main)
    th.start()

我的问题是用户第一次按下后.这个 th 线程将打开浏览器 A 并做一些事情.当浏览器 A 正在做一些事情时，用户将再次按下按钮并打开运行相同 self.main 的浏览器 B.我希望每个打开的浏览器同时运行.我遇到的问题是，当我运行那个线程函数时，第一个浏览器停止并且第二个浏览器打开.

My problem is that after the user press the first time. This th thread will open browser A and do some stuff. While browser A is doing some stuff, the user will press the button again and open browser B that runs the same self.main. I want each browser opened to run simultaneously. The problem I faced is that when I run that thread function, the first browser stops and the second browser is opened.

我知道我的代码可以无限创建线程.我知道这会影响电脑性能，但我可以接受.我想加快 self.main 完成的工作！

I know my code can create threads infinitely. And I know that this will affect the pc performance but I am ok with that. I want to speed up the work done by self.main!

推荐答案

Threading for selenium 加速

考虑以下函数来举例说明与单一驱动程序方法相比，使用 selenium 的线程如何提供一些加速.下面的代码 scraps 来自 selenium 使用 BeautifulSoup 打开的页面的 html 标题.页面列表是links.

Threading for selenium speed up

Consider the following functions to exemplify how threads with selenium give some speed-up compared to a single driver approach. The code bellow scraps the html title from a page opened by selenium using BeautifulSoup. The list of pages is links.

import time
from bs4 import BeautifulSoup
from selenium import webdriver
import threading

def create_driver():
   """returns a new chrome webdriver"""
   chromeOptions = webdriver.ChromeOptions()
   chromeOptions.add_argument("--headless") # make it not visible, just comment if you like seeing opened browsers
   return webdriver.Chrome(options=chromeOptions)  

def get_title(url, webdriver=None):  
   """get the url html title using BeautifulSoup 
   if driver is None uses a new chrome-driver and quit() after
   otherwise uses the driver provided and don't quit() after"""
   def print_title(driver):
      driver.get(url)
      soup = BeautifulSoup(driver.page_source,"lxml")
      item = soup.find('title')
      print(item.string.strip())

   if webdriver:
      print_title(webdriver)  
   else: 
      webdriver = create_driver()
      print_title(webdriver)   
      webdriver.quit()

links = ["https://www.amazon.com", "https://www.google.com", "https://www.youtube.com/", "https://www.facebook.com/", "https://www.wikipedia.org/", 
"https://us.yahoo.com/?p=us", "https://www.instagram.com/", "https://www.globo.com/", "https://outlook.live.com/owa/"]

现在在上面的 links 上调用 get_tile.

Calling now get_tile on the links above.

顺序方法

单个 chrome 驱动程序并按顺序传递所有链接.我的机器需要 22.3 秒(注意:windows).

A single chrome driver and passing all links sequentially. Takes 22.3 s my machine (note:windows).

start_time = time.time()
driver = create_driver()

for link in links: # could be 'like' clicks 
  get_title(link, driver)  

driver.quit()
print("sequential took ", (time.time() - start_time), " seconds")

多线程方法

为每个链接使用一个线程.结果在 10.5 秒内 >快 2 倍.

Using a thread for each link. Results in 10.5 s > 2x faster.

start_time = time.time()    
threads = [] 
for link in links: # each thread could be like a new 'click' 
    th = threading.Thread(target=get_title, args=(link,))    
    th.start() # could `time.sleep` between 'clicks' to see whats'up without headless option
    threads.append(th)        
for th in threads:
    th.join() # Main thread wait for threads finish
print("multiple threads took ", (time.time() - start_time), " seconds")

这里和这个更好是其他一些工作示例.第二个在 ThreadPool 上使用固定数量的线程.并建议存储在每个线程上初始化的 chrome-driver 实例比每次都创建-启动它更快.

This here and this better are some other working examples. The second uses a fixed number of threads on a ThreadPool. And suggests that storing the chrome-driver instance initialized on each thread is faster than creating-starting it every time.

我仍然不确定这是否是 selenium 的最佳方法有相当大的加速. 因为 threadingin-python?rq=1">无 IO 绑定代码 将结束顺序执行(一个线程一个接一个).由于 Python GIL(全局解释器锁)，Python 进程无法并行运行线程(利用多个 cpu 核).

Still I was not sure this was the optimal approach for selenium to have considerable speed-ups. Since threading on no IO bound code will end-up executed sequentially (one thread after another). Due the Python GIL (Global Interpreter Lock) a Python process cannot run threads in parallel (utilize multiple cpu-cores).

使用包multiprocessing

start_time = time.time() 

processes = [] 
for link in links: # each thread a new 'click' 
    ps = multiprocessing.Process(target=get_title, args=(link,))    
    ps.start() # could sleep 1 between 'clicks' with `time.sleep(1)``
    processes.append(ps)        
for ps in processes:
    ps.join() # Main wait for processes finish

return (time.time() - start_time)