Process Memory grows huge

I am using Tornado CurlAsyncHTTPClient. My process memory keeps growing for both blocking and non blocking requests when I instantiate corresponding httpclients for each request. This memory usage growth does not happen if I just have one instance of the httpclients(tornado.httpclient.HTTPClient/tornado.httpclient.AsyncHTTPClient) and reuse them.

Also If I use SimpleAsyncHTTPClient instead of CurlAsyncHTTPClient this memory growth doesnot happen irrespective of how I instantiate.

Here is a sample code that reproduces this,

import tornado.httpclient
import json
import functools

instantiate_once = False
tornado.httpclient.AsyncHTTPClient.configure('tornado.curl_httpclient.CurlAsyncHTTPClient')

hc, io_loop, async_hc = None, None, None
if instantiate_once:
    hc = tornado.httpclient.HTTPClient()
    io_loop = tornado.ioloop.IOLoop()
    async_hc = tornado.httpclient.AsyncHTTPClient(io_loop=io_loop)

def fire_sync_request():
    global count
    if instantiate_once:
        global hc
    if not instantiate_once:
        hc = tornado.httpclient.HTTPClient()
    url = '<Please try with a url>'
    try:
        resp = hc.fetch(url)
    except (Exception,tornado.httpclient.HTTPError) as e:
        print str(e)
    if not instantiate_once:
        hc.close()

def fire_async_requests():
    #generic response callback fn
    def response_callback(response):
        response_callback_info['response_count'] += 1
        if response_callback_info['response_count'] >= request_count:
            io_loop.stop()
    if instantiate_once:
        global io_loop, async_hc
    if not instantiate_once:
        io_loop = tornado.ioloop.IOLoop()
    requests = ['<Please add ur url to try>']*5
    response_callback_info = {'response_count': 0}
    request_count = len(requests)
    global count
    count +=request_count
    hcs=[]
    for url in requests:
        kwargs ={}
        kwargs['method'] = 'GET'
       if not instantiate_once:
            async_hc = tornado.httpclient.AsyncHTTPClient(io_loop=io_loop)
        async_hc.fetch(url, callback=functools.partial(response_callback), **kwargs)
        if not instantiate_once:
        hcs.append(async_hc)

    io_loop.start()
    for hc in hcs:
        hc.close()
    if not instantiate_once:
        io_loop.close()

if __name__ == '__main__':
    import sys
    if sys.argv[1] == 'sync':
        while True:
            output = fire_sync_request()
    elif sys.argv[1] == 'async':
        while True:
            output = fire_async_requests()

Here set instantiate_once variable to True, and execute python check.py sync or python check.py async. The process memory increases continuously

With instantiate_once=False, this doesnot happen.

Also If I use SimpleAsyncHTTPClient instead of CurlAsyncHTTPClient this memory growth doesnot happen.

I have python 2.7/ tornado 2.3.2/ pycurl(libcurl/7.26.0 GnuTLS/2.12.20 zlib/1.2.7 libidn/1.25 libssh2/1.4.2 librtmp/2.3)

I could reproduce the same issue with latest tornado 3.2

Please help me to understand this behaviour and figure out the right way of using tornado as http library.

---

HTTPClient and AsyncHTTPClient are designed to be reused, so it will always be more efficient not to recreate them all the time. In fact, AsyncHTTPClient will try to magically detect if there is an existing AsyncHTTPClient on the same IOLoop and use that instead of creating a new one.

But even though it's better to reuse one http client object, it shouldn't leak to create many of them as you're doing here (as long as you're closing them). This looks like a bug in pycurl: https://github.com/pycurl/pycurl/issues/182

---

Use pycurl 7.19.5 and this hack to avoid memory leaks:

Your Tornado main file:

tornado.httpclient.AsyncHTTPClient.configure("curl_httpclient_leaks_patched.CurlAsyncHTTPClientEx")

curl_httpclient_leaks_patched.py

from tornado import curl_httpclient

class CurlAsyncHTTPClientEx(curl_httpclient.CurlAsyncHTTPClient):

    def close(self):
        super(CurlAsyncHTTPClientEx, self).close()
        del self._multi

链接地址: http://www.djcxy.com/p/66926.html

上一篇: 节点流导致大量内存占用或泄漏

下一篇: 进程内存变得巨大