I
I
Ilya2015-02-24 15:14:24
PostgreSQL
Ilya, 2015-02-24 15:14:24

PostgreSQL, how to speed up checking if a column has a unique value?

There is a simple table:

CREATE TABLE public.notices (
  id TEXT NOT NULL,
  json TEXT,
  CONSTRAINT notices_idx PRIMARY KEY(id)
) 
WITH (oids = false);

json may contain long text.
Before adding records, I check their presence in the table, this stage has gone through three stages of development:
  1. One by one check:select * from notices where id = '123' limit 1);
  2. One by one check:
    select exists(select 1 from notices where id = '123');

  3. Checking the whole crowd:
    (SELECT id FROM notices WHERE id='123' LIMIT 1) UNION ALL (SELECT id FROM notices WHERE id='234' LIMIT 1) UNION ALL ...


The last method gave a significant speedup, but the number of records grows, the number of id in the request is large, so the speed of checking a pack of 50 thousand id for a database of ~ 1 million records is from 20 to 60 seconds on the current machine.
I would like to speed up the check to a minimum. How can you optimize?
Since the json field can be very long, I blame him too. If I allocate id to a separate table, is there any speedup?

Answer the question

In order to leave comments, you need to log in

3 answer(s)
V
Viktor Koltcov, 2015-02-24
@Gorily

There is an index for the id field, is it the primary key?
3 check - in my horror. Better like this:
SELECT id FROM notices WHERE id='123' OR id='234'...

I
Ivan Velichko, 2015-02-24
@Ostrovski

You actually have access by key, store in key-value storage simply, for example in Redis.

A
Aristarkh Zagorodnikov, 2015-02-28
@onyxmaster

Probably in most cases the objects are already there, so what's the best way to get the missing ones?

select "id" from "notices" where "id" not in ('1', '2', ...)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question