使用 Xpath 处理较大的 XML 文件的最佳方法是什么

What is the best way to use Xpath for processing larger XML files?(使用 Xpath 处理较大的 XML 文件的最佳方法是什么?)

问题描述

我有一个要求，我必须使用大型 XML(4 GB 文件)来查找其中的值.基本上我必须编写大约 30 个不同的 Xpath 并将值存储在一个列表中.当我尝试解析 XML 时，它会引发内存错误.我已经尝试使用 lxml 和 ElementTree 来处理开始和结束事件，但处理时间太长还是不行，而且我的 Pycharm/Jupyter 笔记本会引发内存错误.

I have a requirement where I have to use an large XML (4 GB file) for finding values in it. Basically I have to write around 30 different Xpath and store the values in a list. When I try to parse an XML, it throws memory error. I have tried using lxml and ElementTree with start and end events, still no luck the processing time is too high and my Pycharm/Jupyter notebook throws me memory error.

有没有更好的方法呢?尽管此实现不限于任何编程语言，但我更喜欢 Python，因为它是我的右手.提前致谢.

Is there a better way to do it? Even though this implementation is not restricted to any programming language, I prefer Python because its my right hand. Thanks in advance.

例如搜索:如果我想要类别正在烹饪的年份的值.然后我使用 ./bookstore[@category=cooking]/book/year.所以值为 2005

Eg of a search: If I want the value of year where category is cooking. Then I use ./bookstore[@category=cooking]/book/year. So the value is 2005

同样，我必须根据我的业务需求找到我的标签的值.在我的要求中，XML 并不像下面的示例那样简单.

Similarly I have to find the values of my tags based on my business requirements. In my requirement the XML is not simple as the below example.

<?xml version="1.0" encoding="UTF-8"?>

<bookstore>

<book category="cooking">
  <title lang="en">Everyday Italian</title>
  <author>Giada De Laurentiis</author>
  <year>2005</year>
  <price>30.00</price>
</book>

<book category="children">
  <title lang="en">Harry Potter</title>
  <author>J K. Rowling</author>
  <year>2005</year>
  <price>29.99</price>
</book>

<book category="web">
  <title lang="en">XQuery Kick Start</title>
  <author>James McGovern</author>
  <author>Per Bothner</author>
  <author>Kurt Cagle</author>
  <author>James Linn</author>
  <author>Vaidyanathan Nagarajan</author>
  <year>2003</year>
  <price>49.99</price>
</book>

</bookstore>

推荐答案

不知道这样能不能满足你的需求.先试试看.如果有问题，我们可以继续沟通.

I don't know if this will meet your needs. Try it first. If there is a problem, we can continue to communicate.

from simplified_scrapy import SimplifiedDoc,req,utils
def getBook(lines,category,year):
  html = "".join(lines)
  book = SimplifiedDoc(html).book
  if book.category==category and book.year.text==year:
    return book.category,book.title.text,book.authors.text,book.year.text,book.price.text
lst = []
with open('bookstore.xml', 'r') as file: # bookstore.xml is your xml file path
  lines = []
  flag = False
  for line in file:
    if flag or line.find('<book ')>=0:
      flag = True
      lines.append(line)
    if line.find('</book>')>=0:
      b = getBook(lines,'web','2003')
      if b: 
        lst.append(b)
        break # If you only want the first one, add a break
      flag = False
      lines = []
print (lst)

结果:

[('web', 'XQuery Kick Start', ['James McGovern', 'Per Bothner', 'Kurt Cagle', 'James Linn', 'Vaidyanathan Nagarajan'], '2003', '49.99')]

这篇关于使用 Xpath 处理较大的 XML 文件的最佳方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！