How to redirect to another page using jsoup and keep displaying content content in ListView?

Question

Java

There is a site where the main content is a list of posts with text. A block of HTML code is responsible for each post.

<div class="col-xs-12" style="margin:0.5em 0;line-height:1.785em">Some text</div>

To parse posts and display them in a ListView, I implemented such an AsyncTask.

class NewPostsAsyncTask extends AsyncTask<String, Void, String> {

    @Override
    protected void onPreExecute() {
        super.onPreExecute();

        progressDialog = new ProgressDialog(MainActivity.this);
        progressDialog.setTitle("Новые");
        progressDialog.setMessage("Загрузка...");
        progressDialog.setIndeterminate(false);
        progressDialog.show();
    }

    @Override
    protected String doInBackground(String... params) {
        Document doc;

        try {
            doc = Jsoup.connect(URL).get(); 

            content = doc.select("[style=margin:0.5em 0;line-height:1.785em]");
            titleList.clear();

            for (Element contents : content) {
                if (!contents.text().contains("18+")) {
                    titleList.add(contents.text());
                }
            }

        } catch (IOException e) {
            e.printStackTrace(); 
        }

        return null;
    }

    @Override
    protected void onPostExecute(String s) {
        super.onPostExecute(s);
        listView.setAdapter(adapter);
        progressDialog.dismiss();
    }
}

But I have one problem. All messages are not stored on the same page. At the bottom of the page after all the posts there is a block with page numbers and links to them.

And this HTML code is responsible for this block.

<div class="row"><div class="col-xs-12">
        <div class="paginator">

                <span class="pagina">1683</span> " | " 

                <span class="pagina"><a href="/page/1682">1682</a></span> " | " 

                <span class="pagina"><a href="/page/1681">1681</a></span> " | " 

                <span class="pagina"><a href="/page/1680">1680</a></span> " | " 

                <span class="pagina"><a href="/page/1679">1679</a></span> " | " 

                <span class="pagina"><a href="/page/3">3</a></span> " | "

                <span class="pagina"><a href="/page/2">2</a></span> " | " 

                <span class="pagina"><a href="/page/1">1</a></span>

        </div>
    </div>
</div>

How do I navigate to another page, parse other posts and display them in a ListView after the previous posts? As a result, I want to have all the posts of this site in one ListView. Can you show me how to do this?

Answer 1

It is not very clear why you needed to display them all on one page, but for this you just need to parse all the pages indicated in the pagination one by one. The simplest is to do something like this:

titleList.clear();
for (int i = 1; i <= 1683; i++) {
    doc = Jsoup.connect("/page/" + i).get();
    content = doc.select("[style=margin:0.5em 0;line-height:1.785em]");

    for (Element contents : content) {
        if (!contents.text().contains("18+")) {
            titleList.add(contents.text());
        }
    }
}

For dynamic loading you can use AbsListView . Firstly, we can catch the scroll event there, as well as get the number of visible elements.

public abstract class InfiniteScrollListener implements AbsListView.OnScrollListener {
    private int bufferItemCount = 10;
    private int currentPage = 0;
    private int itemCount = 0;
    private boolean isLoading = true;

    public InfiniteScrollListener(int bufferItemCount) {
        this.bufferItemCount = bufferItemCount;
    }
   
    public abstract void loadMore(int page, int totalItemsCount);

    @Override
    public void onScrollStateChanged(AbsListView view, int scrollState) {
        
    }

    @Override
    public void onScroll(AbsListView view, int firstVisibleItem, int visibleItemCount, int totalItemCount)
    {
        if (totalItemCount < itemCount) {
            this.itemCount = totalItemCount;
            if (totalItemCount == 0) {
                this.isLoading = true; }
        }

        if (isLoading && (totalItemCount > itemCount)) {
            isLoading = false;
            itemCount = totalItemCount;
            currentPage++;
        }

        if (!isLoading && (totalItemCount - visibleItemCount)<=(firstVisibleItem + bufferItemCount)) {
            loadMore(currentPage + 1, totalItemCount);
            isLoading = true;
        }
    }
}

Well, to hook this to your list:

myListView.setOnScrollListener(new InfiniteScrollListener(5) {
    @Override
    public void loadMore(int page, int totalItemsCount) {
        doc = Jsoup.connect("/page/" + page).get();
        content = doc.select("[style=margin:0.5em 0;line-height:1.785em]");

        for (Element contents : content) {
            if (!contents.text().contains("18+")) {
                titleList.add(contents.text());
            }
         }
        //Обновляем список
        adapter.notifyDataSetChanged();
    }
});

Everything should work. There are also ready-made solutions like this one .

Answer 2

Better at the bottom of the list, put a button like "Load more". and through the variable with each click increase the counter accordingly and changing the address of the new page. So it will be easier for the user. If you have, for example, 1000 pages, then the user will have to wait