How to sort queryset by related OneToOne fields on the same table?

Question

Django

I'm adding something like an embedded trello to my project. You need to link the positions of the elements on the front with the positions of the elements on the back.
The models look like this:

class Board(models.Model):
    name = models.CharField(verbose_name="Название", max_length=200)
    user = models.ForeignKey(AUTH_USER_MODEL, verbose_name="Пользователь", on_delete=models.CASCADE)

    class Meta:
        verbose_name = "Доска"
        verbose_name_plural = "Доски"

    def __str__(self):
        return self.name


class Column(models.Model):
    name = models.CharField(verbose_name="Название", max_length=200)
    board = models.ForeignKey("canban.Board", verbose_name="Доска", on_delete=models.CASCADE)
    next = models.OneToOneField("canban.Column", on_delete=models.SET_NULL, null=True, default=None, related_name="previous")

    class Meta:
        verbose_name = "Колонка"
        verbose_name_plural = "Колонки"

    def __str__(self):
        return self.name


class Task(models.Model):
    title = models.CharField(verbose_name="Название", max_length=200)
    column = models.ManyToManyField("canban.Column")
    next = models.OneToOneField("canban.Task", on_delete=models.SET_NULL, null=True, default=None, related_name="previous")

    class Meta:
        verbose_name = "Задача"
        verbose_name_plural = "Задачи"

    def __str__(self):
        return self.title

You need to sort the columns and tasks in the columns respectively their nextand previous.
It is clear that the first element will be the one that does not have previous, and the last one will be the one that does not have next. It is also clear that the position of each element is the number of calls to it previousuntil previousit becomes null.
The essence of the task is to know nextor previousany element at any time to get its position as an integer in order to annotate it and sort the queryset by it.
The problem is that with a large size of the task table, the calculation of the position will take a very long time.
If you have any ideas on how to quickly sort such a list (preferably with a single query to the database), please help with advice or an idea.