PostgreSQL: query slow, planner estimates 0.01-0.1 of actual results

Question

I will preface this by saying I'm not well-versed in SQL. I mainly work with ORMs, and this recent headache has brought me to dive into the world of queries, planners, etc..

A very common query is behaving weirdly on my website. I have tried various techniques to solve it but nothing is really helping, except narrowing down the released_date field from 30 days to 7 days. However, from my understanding the tables we're talking about aren't very big and PostgreSQL should satisfy my query in acceptable times.

(Notice: the query below actually queries for a 60 day range - this is because I need 30 days forward and 30 days back, for our purposes, this is a 60 day query.)

Some statistics:

core_releasegroup row count: 3,240,568

core_artist row count: 287,699

core_subscription row count: 1,803,960

Relationships:

Each ReleaseGroup has M2M with Artist, each Artist has M2M with UserProfile through Subscription. I'm using Django which auto-creates indices on foreign-keys, etc..

Unique characteristics of database:

Most music releases have a release_date that corresponds to a Friday, because that's the day most artists prefer to release their music. (Sidenote: could any benefits be made from converting the date field to a timestamp and just generating a random hour for each release?)

Here's the query:

SELECT "core_releasegroup"."id", "core_releasegroup"."title", "core_releasegroup"."type", "core_releasegroup"."release_date", "core_releasegroup"."applemusic_id", "core_releasegroup"."applemusic_image", "core_releasegroup"."geo_apple_music_link", "core_releasegroup"."amazon_aff_link", "core_releasegroup"."is_explicit", "core_releasegroup"."spotify_id", "core_releasegroup"."spotify_link" 
FROM "core_releasegroup" 
INNER JOIN "core_artist_release_groups" 
ON ("core_releasegroup"."id" = "core_artist_release_groups"."releasegroup_id") 
WHERE ("core_artist_release_groups"."artist_id" 
IN 
(SELECT U0."artist_id" FROM "core_subscription" U0 WHERE U0."profile_id" = 1) 
AND "core_releasegroup"."type" IN ('Album', 'Single', 'EP', 'Live', 'Compilation', 'Remix', 'Other') 
AND "core_releasegroup"."release_date" BETWEEN '2020-08-20'::date AND '2020-10-20'::date);

Here's the initial table schema:

CREATE TABLE public.core_releasegroup (
    id integer NOT NULL,
    created_date timestamp with time zone NOT NULL,
    modified_date timestamp with time zone NOT NULL,
    title character varying(560) NOT NULL,
    type character varying(30) NOT NULL,
    release_date date,
    applemusic_id character varying(512),
    applemusic_image character varying(512),
    applemusic_link character varying(512),
    spotify_id character varying(512),
    spotify_image character varying(512),
    spotify_link character varying(512),
    is_explicit boolean NOT NULL,
    spotify_last_refresh timestamp with time zone,
    spotify_next_refresh timestamp with time zone,
    geo_apple_music_link character varying(512),
    amazon_aff_link character varying(620)
);

indices:

mb_12=# SELECT * FROM pg_indexes WHERE tablename = 'core_releasegroup';

Artist table schema:

CREATE TABLE public.core_artist (
    id integer NOT NULL,
    created_date timestamp with time zone NOT NULL,
    modified_date timestamp with time zone NOT NULL,
    name character varying(560) NOT NULL,
    applemusic_id character varying(512) NOT NULL,
    itunes_link character varying(512),
    spotify_id character varying(512),
    spotify_image character varying(512),
    spotify_link character varying(512),
    last_refreshed timestamp with time zone,
    applemusic_image character varying(512),
    spotify_last_refresh timestamp with time zone,
    spotify_resolve_manually boolean NOT NULL,
    last_itunes_refresh timestamp with time zone,
    next_itunes_refresh timestamp with time zone
);

indices:

 schemaname |  tablename  |                indexname                | tablespace |                                                          indexdef
------------+-------------+-----------------------------------------+------------+----------------------------------------------------------------------------------------------------------------------------
 public     | core_artist | core_artist_applemusic_id_009c5120_uniq |            | CREATE UNIQUE INDEX core_artist_applemusic_id_009c5120_uniq ON public.core_artist USING btree (applemusic_id)
 public     | core_artist | core_artist_pkey                        |            | CREATE UNIQUE INDEX core_artist_pkey ON public.core_artist USING btree (id)
 public     | core_artist | core_artist_applemusic_id_009c5120_like |            | CREATE INDEX core_artist_applemusic_id_009c5120_like ON public.core_artist USING btree (applemusic_id varchar_pattern_ops)
 public     | core_artist | core_artist_last_refreshed_2480a006     |            | CREATE INDEX core_artist_last_refreshed_2480a006 ON public.core_artist USING btree (last_refreshed)

Artist-ReleaseGroup:

     Column      |  Type   | Collation | Nullable |                        Default                         | Storage | Stats target | Description
-----------------+---------+-----------+----------+--------------------------------------------------------+---------+--------------+-------------
 id              | integer |           | not null | nextval('core_artist_release_groups_id_seq'::regclass) | plain   |              |
 artist_id       | integer |           | not null |                                                        | plain   |              |
 releasegroup_id | integer |           | not null |                                                        | plain   |              |
Indexes:
    "core_artist_release_groups_pkey" PRIMARY KEY, btree (id)
    "core_artist_release_grou_artist_id_releasegroup_i_c5191f18_uniq" UNIQUE CONSTRAINT, btree (artist_id, releasegroup_id)
    "core_artist_release_groups_artist_id_4dc95871" btree (artist_id)
    "core_artist_release_groups_releasegroup_id_cea5da71" btree (releasegroup_id)
Foreign-key constraints:
    "core_artist_release__releasegroup_id_cea5da71_fk_core_rele" FOREIGN KEY (releasegroup_id) REFERENCES core_releasegroup(id) DEFERRABLE INITIALLY DEFERRED
    "core_artist_release_groups_artist_id_4dc95871_fk_core_artist_id" FOREIGN KEY (artist_id) REFERENCES core_artist(id) DEFERRABLE INITIALLY DEFERRED

Subscription schema:

CREATE TABLE public.core_subscription (
    id integer NOT NULL,
    created_date timestamp with time zone NOT NULL,
    modified_date timestamp with time zone NOT NULL,
    last_notification_time timestamp with time zone NOT NULL,
    artist_id integer NOT NULL,
    profile_id integer NOT NULL,
    library_id character varying(512)
);

Indices:

 schemaname |     tablename     |                      indexname                       | tablespace |                                                                 indexdef
------------+-------------------+------------------------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------------------------
 public     | core_subscription | core_subscription_pkey                               |            | CREATE UNIQUE INDEX core_subscription_pkey ON public.core_subscription USING btree (id)
 public     | core_subscription | core_subscription_profile_id_artist_id_a4d3d29b_uniq |            | CREATE UNIQUE INDEX core_subscription_profile_id_artist_id_a4d3d29b_uniq ON public.core_subscription USING btree (profile_id, artist_id)
 public     | core_subscription | core_subscription_artist_id_0b3ffc23                 |            | CREATE INDEX core_subscription_artist_id_0b3ffc23 ON public.core_subscription USING btree (artist_id)
 public     | core_subscription | core_subscription_profile_id_ee2db298                |            | CREATE INDEX core_subscription_profile_id_ee2db298 ON public.core_subscription USING btree (profile_id)

 schemaname |     tablename     |                   indexname                   | tablespace |                                                                indexdef
------------+-------------------+-----------------------------------------------+------------+----------------------------------------------------------------------------------------------------------------------------------------
 public     | core_releasegroup | core_releasegroup_applemusic_id_aec4a5cb_uniq |            | CREATE UNIQUE INDEX core_releasegroup_applemusic_id_aec4a5cb_uniq ON public.core_releasegroup USING btree (applemusic_id)
 public     | core_releasegroup | core_releasegroup_pkey                        |            | CREATE UNIQUE INDEX core_releasegroup_pkey ON public.core_releasegroup USING btree (id)
 public     | core_releasegroup | core_releasegroup_applemusic_id_aec4a5cb_like |            | CREATE INDEX core_releasegroup_applemusic_id_aec4a5cb_like ON public.core_releasegroup USING btree (applemusic_id varchar_pattern_ops)
 public     | core_releasegroup | core_releasegroup_release_date_03a267f7       |            | CREATE INDEX core_releasegroup_release_date_03a267f7 ON public.core_releasegroup USING btree (release_date)
 public     | core_releasegroup | core_releasegroup_type_58b6243d               |            | CREATE INDEX core_releasegroup_type_58b6243d ON public.core_releasegroup USING btree (type)
 public     | core_releasegroup | core_releasegroup_type_58b6243d_like          |            | CREATE INDEX core_releasegroup_type_58b6243d_like ON public.core_releasegroup USING btree (type varchar_pattern_ops)

Here's the PostgreSQL execution plan: (notice the estimates)

See updated plan below, after running ANALYZE; on database.

 Nested Loop  (cost=2437.52..10850.51 rows=4 width=495) (actual time=411.911..8619.311 rows=362 loops=1)
   Buffers: shared hit=252537 read=29104
   ->  Nested Loop  (cost=2437.09..10578.84 rows=569 width=499) (actual time=372.265..8446.324 rows=36314 loops=1)
         Buffers: shared hit=143252 read=29085
         ->  Bitmap Heap Scan on core_releasegroup  (cost=2436.66..4636.70 rows=567 width=495) (actual time=372.241..7707.466 rows=32679 loops=1)
               Recheck Cond: ((release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date) AND ((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[])))
               Heap Blocks: exact=29127
               Buffers: shared hit=10222 read=27872
               ->  BitmapAnd  (cost=2436.66..2436.66 rows=567 width=0) (actual time=366.750..366.750 rows=0 loops=1)
                     Buffers: shared hit=15 read=8952
                     ->  Bitmap Index Scan on core_releasegroup_release_date_03a267f7  (cost=0.00..342.46 rows=16203 width=0) (actual time=8.834..8.834 rows=32679 loops=1)
                           Index Cond: ((release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date))
                           Buffers: shared read=92
                     ->  Bitmap Index Scan on core_releasegroup_type_58b6243d_like  (cost=0.00..2093.67 rows=113420 width=0) (actual time=355.071..355.071 rows=3240568 loops=1)
                           Index Cond: ((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[]))
                           Buffers: shared hit=15 read=8860
         ->  Index Scan using core_artist_release_groups_releasegroup_id_cea5da71 on core_artist_release_groups  (cost=0.43..10.46 rows=2 width=8) (actual time=0.018..0.020 rows=1 loops=32679)
               Index Cond: (releasegroup_id = core_releasegroup.id)
               Buffers: shared hit=133030 read=1213
   ->  Index Only Scan using core_subscription_profile_id_artist_id_a4d3d29b_uniq on core_subscription u0  (cost=0.43..0.48 rows=1 width=4) (actual time=0.004..0.004 rows=0 loops=36314)
         Index Cond: ((profile_id = 1) AND (artist_id = core_artist_release_groups.artist_id))
         Heap Fetches: 362
         Buffers: shared hit=109285 read=19
 Planning Time: 10.951 ms
 Execution Time: 8619.564 ms

Please note that the above is a stripped down version of the actual query I need. Because of its unbearable slowness, I've stripped down this query to a bare-minimum and reverted to doing some filtering and ordering of the returned objects in Python (which I know is usually slower). As you can see, it's still very slow.

After a while, probably because the memory/cache are populated, this query becomes much faster: (see updated after running ANALYZE)

 Nested Loop  (cost=2437.52..10850.51 rows=4 width=495) (actual time=306.337..612.232 rows=362 loops=1)
   Buffers: shared hit=241776 read=39865 written=4
   ->  Nested Loop  (cost=2437.09..10578.84 rows=569 width=499) (actual time=305.216..546.749 rows=36314 loops=1)
         Buffers: shared hit=132503 read=39834 written=4
         ->  Bitmap Heap Scan on core_releasegroup  (cost=2436.66..4636.70 rows=567 width=495) (actual time=305.195..437.375 rows=32679 loops=1)
               Recheck Cond: ((release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date) AND ((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[])))
               Heap Blocks: exact=29127
               Buffers: shared hit=16 read=38078 written=4
               ->  BitmapAnd  (cost=2436.66..2436.66 rows=567 width=0) (actual time=298.382..298.382 rows=0 loops=1)
                     Buffers: shared hit=16 read=8951
                     ->  Bitmap Index Scan on core_releasegroup_release_date_03a267f7  (cost=0.00..342.46 rows=16203 width=0) (actual time=5.619..5.619 rows=32679 loops=1)
                           Index Cond: ((release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date))
                           Buffers: shared read=92
                     ->  Bitmap Index Scan on core_releasegroup_type_58b6243d_like  (cost=0.00..2093.67 rows=113420 width=0) (actual time=289.917..289.917 rows=3240568 loops=1)
                           Index Cond: ((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[]))
                           Buffers: shared hit=16 read=8859
         ->  Index Scan using core_artist_release_groups_releasegroup_id_cea5da71 on core_artist_release_groups  (cost=0.43..10.46 rows=2 width=8) (actual time=0.003..0.003 rows=1 loops=32679)
               Index Cond: (releasegroup_id = core_releasegroup.id)
               Buffers: shared hit=132487 read=1756
   ->  Index Only Scan using core_subscription_profile_id_artist_id_a4d3d29b_uniq on core_subscription u0  (cost=0.43..0.48 rows=1 width=4) (actual time=0.002..0.002 rows=0 loops=36314)
         Index Cond: ((profile_id = 1) AND (artist_id = core_artist_release_groups.artist_id))
         Heap Fetches: 362
         Buffers: shared hit=109273 read=31
 Planning Time: 1.088 ms
 Execution Time: 612.360 ms

This is still slow in SQL terms (I guess?), but much more acceptable. The problem is, even though this is a very common view in my web-app (an often executed query), it is still not kept around in RAM/cache, and so I see these huge response-time spikes too often.

I've tried every combination of constructing those queries. Some remedy attempts:

1. Remove the core_releasegroup.type filter altogether to inspect whether it's the culprit
2. Create a DESC index on core_releasegroup.release_date, because it's more common than just an ASC index.
3. A DISTINCT ON(id) clause.

The only thing that helps consistently is decreasing the timeframe from 1 month to 1 week.

My last attempt is to see whether the planner estimations are to blame here, and if they're fixable. If not, I'll start considering de-normalization.

Or is there something else I'm missing?

UPDATE

Ok, after a comment from Gerard H. Pille, It appears that both my local copy of the database & my production database have never been analyzed. Here's the execution plan (still off) after running ANALYZE;: (notice the row estimate is still off, and query time is still pretty long)

 Gather  (cost=1697.69..43260.89 rows=69 width=424) (actual time=522.372..5147.785 rows=362 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   Buffers: shared hit=218811 read=81318
   ->  Nested Loop  (cost=697.69..42253.99 rows=29 width=424) (actual time=993.252..5122.081 rows=121 loops=3)
         Buffers: shared hit=218811 read=81318
         ->  Hash Join  (cost=697.26..35407.52 rows=8819 width=4) (actual time=3.201..507.423 rows=23496 loops=3)
               Hash Cond: (core_artist_release_groups.artist_id = u0.artist_id)
               Buffers: shared hit=416 read=17749
               ->  Parallel Seq Scan on core_artist_release_groups  (cost=0.00..31150.65 rows=1355965 width=8) (actual time=0.153..223.448 rows=1084772 loops=3)
                     Buffers: shared hit=2 read=17589
               ->  Hash  (cost=690.92..690.92 rows=507 width=4) (actual time=2.636..2.637 rows=1689 loops=3)
                     Buckets: 2048 (originally 1024)  Batches: 1 (originally 1)  Memory Usage: 76kB
                     Buffers: shared hit=315 read=155
                     ->  Index Scan using core_subscription_profile_id_ee2db298 on core_subscription u0  (cost=0.43..690.92 rows=507 width=4) (actual time=0.067..1.966 rows=1689 loops=3)
                           Index Cond: (profile_id = 1)
                           Buffers: shared hit=315 read=155
         ->  Index Scan using core_releasegroup_pkey on core_releasegroup  (cost=0.43..0.78 rows=1 width=424) (actual time=0.195..0.195 rows=0 loops=70489)
               Index Cond: (id = core_artist_release_groups.releasegroup_id)
               Filter: ((release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date) AND ((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[])))
               Rows Removed by Filter: 1
               Buffers: shared hit=218395 read=63569
 Planning Time: 4.796 ms
 Execution Time: 5147.966 ms

And after RAM/cache has been populated:

 Gather  (cost=1697.69..43260.89 rows=69 width=424) (actual time=103.916..745.476 rows=362 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   Buffers: shared hit=218972 read=81137
   ->  Nested Loop  (cost=697.69..42253.99 rows=29 width=424) (actual time=134.936..730.317 rows=121 loops=3)
         Buffers: shared hit=218972 read=81137
         ->  Hash Join  (cost=697.26..35407.52 rows=8819 width=4) (actual time=2.027..441.156 rows=23496 loops=3)
               Hash Cond: (core_artist_release_groups.artist_id = u0.artist_id)
               Buffers: shared hit=556 read=17589
               ->  Parallel Seq Scan on core_artist_release_groups  (cost=0.00..31150.65 rows=1355965 width=8) (actual time=0.042..195.811 rows=1084772 loops=3)
                     Buffers: shared hit=3 read=17588
               ->  Hash  (cost=690.92..690.92 rows=507 width=4) (actual time=1.823..1.823 rows=1689 loops=3)
                     Buckets: 2048 (originally 1024)  Batches: 1 (originally 1)  Memory Usage: 76kB
                     Buffers: shared hit=469 read=1
                     ->  Index Scan using core_subscription_profile_id_ee2db298 on core_subscription u0  (cost=0.43..690.92 rows=507 width=4) (actual time=0.058..1.283 rows=1689 loops=3)
                           Index Cond: (profile_id = 1)
                           Buffers: shared hit=469 read=1
         ->  Index Scan using core_releasegroup_pkey on core_releasegroup  (cost=0.43..0.78 rows=1 width=424) (actual time=0.012..0.012 rows=0 loops=70489)
               Index Cond: (id = core_artist_release_groups.releasegroup_id)
               Filter: ((release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date) AND ((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[])))
               Rows Removed by Filter: 1
               Buffers: shared hit=218416 read=63548
 Planning Time: 1.077 ms
 Execution Time: 745.604 ms

Still quite slow, and it seems like the indices aren't being used?

Explain after adding composite_index release_date DESC, type, and running VACUUM(ANALYZE):

 Nested Loop  (cost=1.29..18720.56 rows=46 width=425) (actual time=0.990..423.496 rows=362 loops=1)
   Buffers: shared hit=223769 read=63437 written=4
   ->  Nested Loop  (cost=0.86..2732.78 rows=20594 width=4) (actual time=0.048..30.681 rows=70489 loops=1)
         Buffers: shared hit=4245 read=1005
         ->  Index Only Scan using core_subscription_profile_id_artist_id_a4d3d29b_uniq on core_subscription u0  (cost=0.43..17.14 rows=498 width=4) (actual time=0.017..0.549 rows=1689 loops=1)
               Index Cond: (profile_id = 1)
               Heap Fetches: 0
               Buffers: shared hit=5 read=3
         ->  Index Only Scan using core_artist_release_grou_artist_id_releasegroup_i_c5191f18_uniq on core_artist_release_groups  (cost=0.43..5.04 rows=41 width=8) (actual time=0.004..0.011 rows=42 loops=1689)
               Index Cond: (artist_id = u0.artist_id)
               Heap Fetches: 0
               Buffers: shared hit=4240 read=1002
   ->  Index Scan using core_releasegroup_pkey on core_releasegroup  (cost=0.43..0.78 rows=1 width=425) (actual time=0.005..0.005 rows=0 loops=70489)
         Index Cond: (id = core_artist_release_groups.releasegroup_id)
         Filter: ((release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date) AND ((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[])))
         Rows Removed by Filter: 1
         Buffers: shared hit=219524 read=62432 written=4
 Planning Time: 0.999 ms
 Execution Time: 423.600 ms

Should I be satisfied with this result? or there room for more improvement? I've applied VACUUM(ANALYZE) to the production database as well and there's some improvement but this is still slower than I'd like.

Answer 1

Can you add an index on core_releasegroup(type,release_date) and try this:

SELECT
   "core_releasegroup"."id",
   "core_releasegroup"."title",
   "core_releasegroup"."type",
   "core_releasegroup"."release_date",
   "core_releasegroup"."applemusic_id",
   "core_releasegroup"."applemusic_image",
   "core_releasegroup"."geo_apple_music_link",
   "core_releasegroup"."amazon_aff_link",
   "core_releasegroup"."is_explicit",
   "core_releasegroup"."spotify_id",
   "core_releasegroup"."spotify_link"
  FROM "core_releasegroup"
  where exists (select null from
                 "core_artist_release_groups" arg,
                 "core_subscription" U0
                 WHERE U0."profile_id" + 0 = 1
                   and U0."artist_id" = arg."artist_id" + 0
                   and arg."releasegroup_id" = "core_releasegroup"."id" + 0
               )
    AND "core_releasegroup"."type" IN
          ('Album', 'Single', 'EP', 'Live', 'Compilation', 'Remix', 'Other') 
    AND "core_releasegroup"."release_date"
           BETWEEN '2020-08-20'::date AND '2020-10-20'::date;

?

Explain I'd like him to use (I don't have all the indices in the schema I built, moreover: all tables are empty).

 Nested Loop Semi Join  (cost=11.90..59.08 rows=1 width=3699)
   Join Filter: (core_releasegroup.id = arg.releasegroup_id)
   ->  Index Scan using rg_rgrd on core_releasegroup  (cost=0.14..9.04 rows=1 width=3699)
         Index Cond: (((type)::text = ANY ('{Album,Single,EP,Live,Compilation,Remix,Other}'::text[])) AND (release_date >= '2020-08-20'::date) AND (release_date <= '2020-10-20'::date))
   ->  Hash Join  (cost=11.76..49.91 rows=10 width=4)
         Hash Cond: (arg.artist_id = u0.artist_id)
         ->  Seq Scan on core_artist_release_groups arg  (cost=0.00..30.40 rows=2040 width=8)
         ->  Hash  (cost=11.75..11.75 rows=1 width=4)
               ->  Seq Scan on core_subscription u0  (cost=0.00..11.75 rows=1 width=4)
                     Filter: (profile_id = 1)
(10 rows)