Why can't I get some blocks from a page using Python?

Question

Python

Good afternoon, faced such problem.
I am writing a parser for a site and I need to display all the latest news that appears on the page, everything is fine with the first block, but when I try to get news from the lower block (less interesting news), the parser simply does not see it. Can you help?

import urllib.request
import bs4
from bs4 import BeautifulSoup
import requests
from selenium import webdriver
import time
import datetime
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities


hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
       'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
       'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
       'Accept-Encoding': 'none',
       'Accept-Language': 'en-US,en;q=0.8',
       'Connection': 'keep-alive'}


now = datetime.datetime.today()
i = 1
prev = "11111111111"
prev2 ="11111111111"

ua = dict(DesiredCapabilities.PHANTOMJS)
ua["phantomjs.page.settings.userAgent"] = hdr
browser = webdriver.PhantomJS(desired_capabilities=ua)
browser.get("https://blog.coinbase.com/")

soup = BeautifulSoup(browser.page_source)
# a = soup.find("a", class_="u-block u-width140 u-height120 u-floatRight u-marginLeft30 u-xs-size80x80 u-backgroundCover u-borderLighter")
div2 = soup.find("div", class_="u-fontSize24 u-xs-fontSize18")
print(div2)

Website: https://blog.coinbase.com/
Link to news that can't be displayed: https://blog.coinbase.com/upcoming-story-about-coi...
Link to news that can't be displayed: https:/ /blog.coinbase.com/coinbase-pro-disables-ma...

Answer 1

Because they are reloaded via AJAX.