Why doesn't BeautifulSoup want to run under a flask?

Question

Flask

There is a chat bot for VK. While it has no special functions, it displays the same line for any message. There is also an Avito parser that collects several ads and glues information about them into a string. This parsing module itself is called from the engine.py file. And when you run the engine.py itself, everything is perfectly parsed and the desired line is displayed. When importing the engine as a module in another file and then launching it, everything is also successful. But as soon as I insert the same function call into the main file of my chatbot, which describes the Flask handlers for web paths, everything collapses and the following heroku log appears:

2018-05-12T21:35:59.266624+00:00 app[web.1]: [2018-05-12 21:35:59,265] ERROR in app: Exception on / [POST]
2018-05-12T21:35:59.266660+00:00 app[web.1]: Traceback (most recent call last):
2018-05-12T21:35:59.266663+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1982, inwsgi_app
2018-05-12T21:35:59.266665+00:00 app[web.1]:     response = self.full_dispatch_request()
2018-05-12T21:35:59.266667+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1614, infull_dispatch_request
2018-05-12T21:35:59.266669+00:00 app[web.1]:     rv = self.handle_user_exception(e)
2018-05-12T21:35:59.266671+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1517, inhandle_user_exception
2018-05-12T21:35:59.266672+00:00 app[web.1]:     reraise(exc_type, exc_value, tb)
2018-05-12T21:35:59.266674+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
2018-05-12T21:35:59.266677+00:00 app[web.1]:     raise value
2018-05-12T21:35:59.266679+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1612, infull_dispatch_request
2018-05-12T21:35:59.266680+00:00 app[web.1]:     rv = self.dispatch_request()
2018-05-12T21:35:59.266682+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 1598, indispatch_request
2018-05-12T21:35:59.266684+00:00 app[web.1]:     return self.view_functions[rule.endpoint](**req.view_args)
2018-05-12T21:35:59.266685+00:00 app[web.1]:   File "/app/online_rieltor.py", line 27, in processing
2018-05-12T21:35:59.266688+00:00 app[web.1]:     ads_list = engine.get_ads()
2018-05-12T21:35:59.266689+00:00 app[web.1]:   File "/app/engine.py", line 12, in get_ads
2018-05-12T21:35:59.266692+00:00 app[web.1]:     return parser.get_all_ads('Санкт-Петербург', 'Фрунзенская', ['3', '6'], ['1', '4'], ['22', '37'], ['2000000', '5000000'])
2018-05-12T21:35:59.266694+00:00 app[web.1]:   File "/app/Parser/parser.py", line 10, in get_all_ads
2018-05-12T21:35:59.266696+00:00 app[web.1]:     avito_list = Avito.get_avito_ads(city, subway, floor, rooms, area, price)
2018-05-12T21:35:59.266698+00:00 app[web.1]:   File "/app/Parser/avito/Avito.py", line 26, in get_avito_ads
2018-05-12T21:35:59.266700+00:00 app[web.1]:     ads_list = soup.find('div', class_='catalog-list').find_all('div', class_='item_table')
2018-05-12T21:35:59.268657+00:00 app[web.1]: AttributeError: 'NoneType' object has no attribute 'find_all'
2018-05-12T21:35:59.271337+00:00 app[web.1]: 87.240.160.178 - - [12/May/2018:21:35:59 +0000] "POST / HTTP/1.1" 500 291 "-" "curl/7.26.0"

And the bot, in turn, is silent and a record appears in VK that the server responded with an error. Based on the log, I realized that the error is in the div tag with the catalog-list class, which cannot be found, so it turns out to be NoneType. But after all, in other cases everything worked, I did not change anything, help me figure it out.
online_realtor.py:

from flask import Flask, request, json
from werkzeug.contrib.fixers import ProxyFix
import vk
import engine
from vk_config import vk_config

vk_conf = vk_config()
app = Flask(__name__)

@app.route('/test')
def test_method():
    return engine.get_ads()

@app.route('/', methods=['POST'])
def processing():
    data = json.loads(request.data)
    
    if data['type'] == 'confirmation':
        return vk_conf['confirmation']
    elif data['type'] == 'message_new':
        session = vk.Session()
        api = vk.API(session, v=5.0)
        user_id = data['object']['user_id']
        ads_list = engine.get_ads()
        string = ''
        for i in range(5):
            string += 'city: ' + ads_list[i].city + '\n'
            string += 'subway: ' + ads_list[i].subway + '\n'
            string += 'address: ' + ads_list[i].address + '\n'
            string += 'price: ' + ads_list[i].price + '\n'
            string += 'link: ' + ads_list[i].link + '\n'
            string += '\n\n'
        api.messages.send(access_token=vk_conf['access_key'], user_id=str(user_id), message=string)
    
    return 'ok'


app.wsgi_app = ProxyFix(app.wsgi_app)
if __name__ == '__main__':
    app.run()

Answer 1

Everything is extremely simple, Avito does not respond to requests from Heroku.