在线程内调用时，Python Queues 内存泄漏

Python Queues memory leaks when called inside thread(在线程内调用时，Python Queues 内存泄漏)

问题描述

我有 python TCP 客户端，需要循环发送媒体(.mpg)文件到C"TCP 服务器.

I have python TCP client and need to send media(.mpg) file in a loop to a 'C' TCP server.

我有以下代码，在单独的线程中，我正在读取 10K 文件块并将其发送并在循环中重新执行，我认为这是因为我实现了线程模块或 tcp send. 我正在使用 Queues 在我的 GUI (Tkinter) 上打印日志，但过了一段时间它内存不足..

I have following code, where in separate thread I am reading the 10K blocks of file and sending it and doing it all over again in loop, I think it is because of my implementation of thread module, or tcp send. I am using Queues to print the logs on my GUI ( Tkinter ) but after some times it goes out of memory..

更新 1 - 根据要求添加了更多代码

线程类Sendmpgthread"用于创建线程发送数据

Thread class "Sendmpgthread" used to create thread to send data

.
. 
def __init__ ( self, otherparams,MainGUI):
    .
    .
    self.MainGUI = MainGUI
    self.lock = threading.Lock()
    Thread.__init__(self)

#This is the one causing leak, this is called inside loop
def pushlog(self,msg):
    self.MainGUI.queuelog.put(msg)

def send(self, mysocket, block):
    size = len(block)
    pos = 0;
    while size > 0:
        try:
            curpos = mysocket.send(block[pos:])
        except socket.timeout, msg:
            if self.over:
                 self.pushlog(Exit Send)
                return False
        except socket.error, msg:
            print 'Exception'     
            return False  
        pos = pos + curpos
        size = size - curpos
    return True

def run(self):
    media_file = None
    mysocket = None 

    try:
        mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        mysocket.connect((self.ip, string.atoi(self.port)))
        media_file = open(self.file, 'rb') 

        while not self.over:
            chunk = media_file.read(10000)
            if not chunk:   # EOF Reset it
                print 'resetting stream'
                media_file.seek(0, 0)
                continue
            if not self.send(mysocket, chunk): # If some error or thread is killed 
                break;

            #disabling this solves the issue
            self.pushlog('print how much data sent')       

    except socket.error, msg:
        print 'print exception'
    except Exception, msg:
        print 'print exception'

    try:
        if media_file is not None:
            media_file.close()
            media_file = None            
        if mysocket is not None:
            mysocket.close()
            mysocket = None
    finally:
            print 'some cleaning'   

def kill(self):
    self.over = True

我发现这是因为 Queue 的错误实现，因为评论该部分解决了问题

更新 2 - 从 Thread 类调用的 MainGUI 类

class MainGUI(Frame):
    def __init__(self, other args):
       #some code
       .
       .
        #from the above thread class used to send data
        self.send_mpg_status = Sendmpgthread(params)
        self.send_mpg_status.start()     
        self.after(100, self.updatelog)
        self.queuelog = Queue.Queue()

    def updatelog(self):
       try:
           msg = self.queuelog.get_nowait() 

           while msg is not None:
               self.printlog(msg)
               msg = self.queuelog.get_nowait() 
        except Queue.Empty:
           pass

        if self.send_mpg_status: # only continue when sending   
            self.after(100, self.updatelog)

    def printlog(self,msg):
        #print in GUI

推荐答案

由于 printlog 正在添加到 tkinter 文本控件，该控件占用的内存将随着每条消息而增长(它必须存储所有日志消息才能显示它们).

Since printlog is adding to a tkinter text control, the memory occupied by that control will grow with each message (it has to store all the log messages in order to display them).

除非存储所有日志至关重要，否则一个常见的解决方案是限制显示的最大日志行数.

Unless storing all the logs is critical, a common solution is to limit the maximum number of log lines displayed.

一个简单的实现是在控件达到最大消息数后从一开始就消除多余的行.添加一个函数获取控件的行数 然后，在 printlog 中类似于:

A naive implementation is to eliminate extra lines from the begining after the control reaches a maximum number of messages. Add a function to get the number of lines in the control and then, in printlog something similar to:

while getnumlines(self.edit) > self.maxloglines:
    self.edit.delete('1.0', '1.end')

(以上代码未测试)

更新:一些一般准则

请记住，看起来像内存泄漏的情况并不总是意味着函数错误，或者内存不再可访问.很多时候，正在积累元素的容器缺少清理代码.

Keep in mind that what might look like a memory leak does not always mean that a function is wrong, or that the memory is no longer accessible. Many times there is missing cleanup code for a container that is accumulating elements.

此类问题的基本通用方法:

A basic general approach for this kind of problems:

• 就代码的哪一部分可能导致问题形成意见
• 通过注释掉该代码来检查它(或继续注释代码直到找到候选人)
• 在负责的代码中查找容器，添加代码以打印其大小
• 决定哪些元素可以安全地从容器中移除，以及何时移除
• 测试结果

这篇关于在线程内调用时，Python Queues 内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持编程学习网！