E
E
ehles2014-01-26 22:24:06
MySQL
ehles, 2014-01-26 22:24:06

How to implement fetch from mysql tables related as one to many?

There are signs in the database:
BlogRecord - the number of entries is more than 30,000.
Comment - the number of entries is more than 250,000.
Everything is implemented on Django, the models look something like this:

class BlogRecord(models.Model):
    text = models.TextField(null=True, blank=True)
    # прочие поля

class Comment(models.Model):
    blog_record = models.ForeignKey(BlogRecord, db_index=True)
    # прочие поля

First, a selection is made from the comments table, more than 150k entries can fall into the selection, then you need to get a list of blog entries from this selection. The problem is that such a request takes an unacceptably long time - more than 30 seconds, I would like to reduce this time to 1-2 seconds.
Initially, the sample was done like this:
# сложная выборка по разным полям из таблицы комментариев
comments = models.Comment.objects.filter() 
blog_record = models.BlogRecord.objects.filter(
        id__in=comments.values_list('blog_record')
    )

Then I tried to transfer part of the calculations to the python code, but it didn’t get any better.
comment_ids = # Большой список ID комментариев полученный в результате выборки.
models.BlogRecord.objects.filter(id__in=comment_ids)

Maybe there are some other methods that can change something in the database structure?
PS I'm not very strong in django, maybe I'm doing something wrong.

Answer the question

In order to leave comments, you need to log in

3 answer(s)
Y
yamel, 2014-01-27
@ehles

It is possible to try to transfer conditions of selection of comments to the second request.
For example, like this:

blog_record = models.BlogRecord.objects.filter(comment__author=103, comment__status='publish')

M
maxfox, 2014-01-27
@maxfox

I didn’t quite understand what you want to do, but it definitely needs to be done in one request.
"By transferring some of the calculations to Python code" you only slow down the process. The fastest selection is made on the side of the DBMS - it is written in C and optimized as much as possible. Therefore, whatever you do there, try to put it in one request.
Describe what result you want to get, then it will be easier to suggest.

E
ehles, 2014-01-27
@ehles

Did as suggested by @yamel :

comment_params = {"comment__author":103}
blog_records = models.BlogRecord.objects.filter(
     **comment_params
     ).annotate(Count('comment__blogrecord'))

Thanks to all!!!
Yes, here is the result, I tested it on a small request:
it was 50 seconds, it became 0.9 seconds.
PS I lied a little with the number of records, there are an order of magnitude more of them.)))
PPS along the way found another bug that slowed down the execution of queries by getting into what SQL queries django generates.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question