{"id":22663,"date":"2026-07-24T09:58:29","date_gmt":"2026-07-24T09:58:29","guid":{"rendered":"https:\/\/scannn.com\/hatchet-%c2%b7-the-startups-postgres-survival-guide\/"},"modified":"2026-07-24T09:58:29","modified_gmt":"2026-07-24T09:58:29","slug":"hatchet-%c2%b7-the-startups-postgres-survival-guide","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/hatchet-%c2%b7-the-startups-postgres-survival-guide\/","title":{"rendered":"Hatchet \u00b7 The startup's Postgres survival guide"},"content":{"rendered":"\n<div id=\"\">\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Over the past half year or\u00a0so, I\u2019ve\u00a0been writing an\u00a0internal doc for our engineers trying to\u00a0distill two years of\u00a0Postgres battles into a\u00a0somewhat cohesive document. While I\u00a0love <a class=\"_primary_jxyxs_66\" href=\"https:\/\/www.postgresql.org\/docs\/18\/index.html\">the Postgres manual<\/a>, I\u00a0find it\u2019s\u00a0hard to\u00a0turn to\u00a0when shit hits the fan because it\u2019s\u00a0just so\u00a0darn comprehensive. I\u00a0thought this might be\u00a0useful for others and would appreciate feedback (or\u00a0other tidbits that you\u2019ve\u00a0learned running Postgres in\u00a0production).<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Before starting Hatchet, while I\u00a0was familiar with SQL, the extent of\u00a0my\u00a0knowledge was basically: if\u00a0a\u00a0query is\u00a0slow, you need an\u00a0index. That\u2019s\u00a0the starting point for this doc; I\u2019m\u00a0going to\u00a0assume you\u2019re\u00a0familiar with SQL basics, rows, tables, and know roughly what an\u00a0index is.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\"><em>And if\u00a0Claude is\u00a0writing all of\u00a0your queries, this might be\u00a0a\u00a0waste of\u00a0time! I\u00a0recommend <a class=\"_primary_jxyxs_66\" href=\"https:\/\/github.com\/supabase\/agent-skills\">supabase\/agent-skills<\/a><\/em><\/p>\n<hr class=\"_hr_1fiiu_193\"\/>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"a-quick-note-on-orms\"><a class=\"_primary_jxyxs_66\" href=\"#a-quick-note-on-orms\">A\u00a0quick note on\u00a0ORMs<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">This guide should still be\u00a0useful, but you might need to\u00a0translate some of\u00a0these tips into your ORM of\u00a0choice. Lots of\u00a0optimizations as\u00a0you scale just aren\u2019t\u00a0possible with ORMs unless you can break past the abstraction layer and write SQL.\u00a0You can do\u00a0this gracefully or\u00a0non-gracefully; <a class=\"_primary_jxyxs_66\" href=\"https:\/\/www.prisma.io\/docs\/orm\/prisma-client\/using-raw-sql\/typedsql\">Prisma TypedSQL<\/a> or\u00a0equivalents look interesting for this. We\u00a0use <code class=\"_code_1fiiu_64 \">sqlc<\/code> at\u00a0Hatchet which gets us\u00a0very similar behavior; highly recommend if\u00a0you\u2019re\u00a0a\u00a0Go\u00a0stack.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"table-of-contents\"><a class=\"_primary_jxyxs_66\" href=\"#table-of-contents\">Table of\u00a0contents<\/a><\/h4>\n<h3 class=\"_h3_jxyxs_9 h3 _balance_jxyxs_54\" id=\"the-simple-stuff-good-reads-writes-and-schemas\"><a class=\"_primary_jxyxs_66\" href=\"#the-simple-stuff-good-reads-writes-and-schemas\">The simple stuff: good reads, writes and\u00a0schemas<\/a><\/h3>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Let\u2019s\u00a0start with the basics: queries and schemas at\u00a0low volume.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"writing-a-good-schema\"><a class=\"_primary_jxyxs_66\" href=\"#writing-a-good-schema\">Writing a\u00a0good schema<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">After you\u2019re\u00a0deployed, schemas are by\u00a0far the hardest to\u00a0change moving forward, so\u00a0it\u2019s\u00a0worth spending some time on\u00a0them. I\u2019d\u00a0recommend building your schema iteratively: start with a\u00a0rough approximation for your tables and primary keys, then write some queries on\u00a0those tables based on\u00a0your application needs. You can approximate this with some questions: <em>Is this a\u00a0high-read and\/or high-write table? What are the most common filters on\u00a0reads? Which columns am\u00a0I\u00a0updating the\u00a0most?<\/em><\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">If\u00a0you want to\u00a0be\u00a0more formal about it, you can look into <a class=\"_primary_jxyxs_66\" href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/database-normalization\">database normalization<\/a> into 1NF\/2NF\/3NF, but I\u2019ve\u00a0found normal forms to\u00a0sometimes be\u00a0at\u00a0odds with query efficiency and ease of\u00a0use, which is\u00a0critical when you\u2019re\u00a0moving fast\u2014sometimes it\u2019s\u00a0just easier to\u00a0dump data into a\u00a0<code class=\"_code_1fiiu_64 \">jsonb<\/code>\u00a0column.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">My\u00a0rules of\u00a0thumb for schemas\u00a0are:<\/p>\n<ul>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Use identity columns (auto-incrementing integers, slightly more performant than <code class=\"_code_1fiiu_64 \">bigserial<\/code>) or\u00a0built-in UUIDs for primary\u00a0keys<\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Always use <code class=\"_code_1fiiu_64 \">timestamptz<\/code><\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Always use primary\u00a0keys<\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Use foreign keys with cascading deletes for low-volume tables, particularly where database consistency and correctness are important. Careful at\u00a0higher volume.<\/li>\n<\/ul>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"writing-good-read-queries\"><a class=\"_primary_jxyxs_66\" href=\"#writing-good-read-queries\">Writing good read\u00a0queries<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Let\u2019s\u00a0start with <code class=\"_code_1fiiu_64 \">SELECT<\/code> queries. A\u00a0useful\u2014albeit slightly inaccurate\u2014mental model for fast selects is: under the hood, Postgres is\u00a0either going to\u00a0find a\u00a0single row in\u00a0a\u00a0table very quickly, or\u00a0it\u2019s\u00a0going to\u00a0read every single row in\u00a0your table using something called a\u00a0<em>sequential scan<\/em>\u00a0<span class=\"emoji\"><\/span>.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">It\u2019s\u00a0going to\u00a0find a\u00a0single row very quickly when you filter\u00a0by:<\/p>\n<ul>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">An\u00a0explicit index<\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">A\u00a0unique constraint (just a\u00a0special case of\u00a0index)<\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">A\u00a0primary key (these are automatically indexed in\u00a0Postgres)<\/li>\n<\/ul>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Indexes by\u00a0default use a\u00a0<code class=\"_code_1fiiu_64 \">btree<\/code> implementation. It\u2019s\u00a0most helpful to\u00a0think of\u00a0indexes as\u00a0just another table in\u00a0Postgres, with data stored in\u00a0a\u00a0specific format which is\u00a0optimized for lookups (more on\u00a0this later). These trees are great because finding a\u00a0single row happens in\u00a0approximately <code class=\"_code_1fiiu_64 \">log(n)<\/code> time, where <code class=\"_code_1fiiu_64 \">n<\/code> is\u00a0the number of\u00a0rows in\u00a0the table\u2014in other words, really\u00a0fast.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">When Postgres can\u2019t\u00a0use an\u00a0index, it\u2019ll\u00a0use something called a\u00a0sequential scan, or\u00a0<code class=\"_code_1fiiu_64 \">seq scan<\/code>. Seq scans are much slower than index lookups, but modern databases are so\u00a0fast at\u00a0loading rows into memory that you probably won\u2019t\u00a0even notice at\u00a0first: seq scans on\u00a0tables with less than 20k rows are pretty much\u00a0instant.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"writing-performant-joins\"><a class=\"_primary_jxyxs_66\" href=\"#writing-performant-joins\">Writing performant\u00a0joins<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">For inner joins, there\u2019s\u00a0rarely an\u00a0argument for not using primary keys as\u00a0the inner join; it\u00a0usually speaks to\u00a0a\u00a0schema design or\u00a0normalization problem. Treat <code class=\"_code_1fiiu_64 \">ON<\/code> clauses with the same respect as\u00a0a\u00a0<code class=\"_code_1fiiu_64 \">WHERE<\/code> clause\u2014the same principles apply. Use an\u00a0index.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"compound-indexes-and-aligning-order-by-to-your-indexes\"><a class=\"_primary_jxyxs_66\" href=\"#compound-indexes-and-aligning-order-by-to-your-indexes\">Compound indexes and aligning ORDER BY\u00a0to\u00a0your indexes<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Often the first slow query in\u00a0your application will be\u00a0a\u00a0list query across a\u00a0large table. Something\u00a0like:<\/p>\n<div data-codeblock=\"\" class=\"_codeBlock_150cy_1 \">\n<p>Loading syntax highlighting&#8230;<\/p>\n<\/div>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">In\u00a0this case, you can use a\u00a0compound index\u2014a sensible one might\u00a0be:<\/p>\n<div data-codeblock=\"\" class=\"_codeBlock_150cy_1 \">\n<p>Loading syntax highlighting&#8230;<\/p>\n<\/div>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">In\u00a0more complex cases, a\u00a0good rule of\u00a0thumb is: the <code class=\"_code_1fiiu_64 \">ORDER BY<\/code> columns should be\u00a0the last columns in\u00a0the index, and you should align columns to\u00a0the ordering in\u00a0the <code class=\"_code_1fiiu_64 \">ORDER BY<\/code>. Note that Postgres can scan btrees in\u00a0both directions, so\u00a0sometimes the <code class=\"_code_1fiiu_64 \">DESC<\/code> is\u00a0irrelevant\u2014but for compound indexes it\u2019s\u00a0good practice. More information <a class=\"_primary_jxyxs_66\" href=\"https:\/\/www.cybertec-postgresql.com\/en\/benefits-of-a-descending-index\">here<\/a>.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"writing-good-write-queries\"><a class=\"_primary_jxyxs_66\" href=\"#writing-good-write-queries\">Writing good write\u00a0queries<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">The premise of\u00a0successful writes\u00a0is:<\/p>\n<ol>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\"><strong class=\"_primary_jxyxs_66\">Keep transactions short.<\/strong> Don\u2019t\u00a0go\u00a0querying an\u00a0external service in\u00a0the middle of\u00a0a\u00a0transaction unless you have a\u00a0really good reason\u00a0to.<\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\"><strong class=\"_primary_jxyxs_66\">Be\u00a0careful of\u00a0the rows you\u2019re\u00a0locking<\/strong> for writing; in\u00a0other words, only lock what you need. Every time you update a\u00a0row, you\u2019re\u00a0taking out a\u00a0lock on\u00a0that row for a\u00a0short period of\u00a0time until the transaction\u00a0commits.<\/li>\n<\/ol>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">As\u00a0your system gets busier, you\u2019re\u00a0going to\u00a0start noticing the impact of\u00a0locks more. In\u00a0particular, you might try to\u00a0create an\u00a0index at\u00a0some point in\u00a0the future with a\u00a0simple <code class=\"_code_1fiiu_64 \">CREATE INDEX<\/code> command: turns out this locks your table and prevents inserts and updates! When creating an\u00a0index on\u00a0an\u00a0existing large table, always use <code class=\"_code_1fiiu_64 \">CREATE INDEX CONCURRENTLY<\/code>.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"migrations\"><a class=\"_primary_jxyxs_66\" href=\"#migrations\">Migrations<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Getting really good at\u00a0writing migrations is\u00a0an\u00a0important technical advantage: it\u00a0helps you iterate much faster and increases your uptime. As\u00a0a\u00a0starting point, try to\u00a0keep migrations additive (in\u00a0other words, don\u2019t\u00a0delete or\u00a0remove columns) and run them in\u00a0a\u00a0transaction wherever possible; this will make rollbacks and partial migrations much easier to\u00a0deal with. As\u00a0you get more advanced, you can start looking into <a class=\"_primary_jxyxs_66\" href=\"https:\/\/martinfowler.com\/bliki\/ParallelChange.html\">expand and contract<\/a> migrations.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">The simplest mental model for good migrations is: does this block all of\u00a0my\u00a0writes, or\u00a0does it\u00a0not? Creating an\u00a0index without <code class=\"_code_1fiiu_64 \">CONCURRENTLY<\/code> blocks all your writes, so\u00a0you might see downtime. Generally, operations which call <code class=\"_code_1fiiu_64 \">ALTER TABLE<\/code> should be\u00a0worth a\u00a0second look; for example, adding a\u00a0new check constraint to\u00a0a\u00a0very large table can block your writes as\u00a0well (unless you add it\u00a0with the <code class=\"_code_1fiiu_64 \">NOT VALID<\/code>\u00a0keyword).<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"connection-management\"><a class=\"_primary_jxyxs_66\" href=\"#connection-management\">Connection\u00a0management<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Every time you execute a\u00a0transaction or\u00a0query against your database, you\u2019re\u00a0utilizing a\u00a0connection. Connections are expensive in\u00a0a\u00a0number of\u00a0dimensions (cpu and memory), and high connection churn can lead to\u00a0a\u00a0lot of\u00a0unnecessary resource waste, so\u00a0connections should be\u00a0long-lived. Connection storms (when you start using up\u00a0a\u00a0ton of\u00a0new connections at\u00a0the same time) can also lead to\u00a0very hard to\u00a0debug edge cases related to\u00a0internal Postgres\u00a0locks.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Because of\u00a0all these connection footguns, external connection poolers like pgbouncer are great! If\u00a0you can\u2019t\u00a0add this for whatever reason, in-memory connection poolers are a\u00a0great second option. For example, because Hatchet is\u00a0open-source, we\u00a0don\u2019t\u00a0assume that all user databases use connection poolers, so\u00a0we\u00a0use <a class=\"_primary_jxyxs_66\" href=\"https:\/\/pkg.go.dev\/github.com\/jackc\/pgx\/v5\/pgxpool\">pgxpool<\/a> (an\u00a0in-memory connection pool for Go) for this\u00a0purpose.<\/p>\n<h3 class=\"_h3_jxyxs_9 h3 _balance_jxyxs_54\" id=\"intermediate-the-query-planner-bulk-updates-and-autovacuum\"><a class=\"_primary_jxyxs_66\" href=\"#intermediate-the-query-planner-bulk-updates-and-autovacuum\">Intermediate: the query planner, bulk updates, and\u00a0autovacuum<\/a><\/h3>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"introducing-the-leakiest-of-abstractions-the-query-planner\"><a class=\"_primary_jxyxs_66\" href=\"#introducing-the-leakiest-of-abstractions-the-query-planner\">Introducing the leakiest of\u00a0abstractions, the query\u00a0planner<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">At\u00a0a\u00a0certain point, your queries might become complex enough that a\u00a0simple index won\u2019t\u00a0cut it\u00a0(and you shouldn\u2019t\u00a0endlessly add indexes to\u00a0your tables\u2014they come with overhead). The queries might involve many <code class=\"_code_1fiiu_64 \">JOIN<\/code> statements or\u00a0different types of\u00a0joins where the correct path for querying the data isn\u2019t\u00a0clear.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">At\u00a0this point, you will need to\u00a0concern yourself with the <em>query planner<\/em>. At\u00a0best, the query planner is\u00a0a\u00a0<a class=\"_primary_jxyxs_66\" href=\"https:\/\/www.joelonsoftware.com\/2002\/11\/11\/the-law-of-leaky-abstractions\/\">leaky abstraction<\/a>. It\u2019s\u00a0an\u00a0internal implementation, and you have virtually no\u00a0control over it, but you have to\u00a0know its spontaneous and sometimes irrational behavior. It\u2019s\u00a0like working with an\u00a0LLM!<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">The query planner looks at\u00a0the query you pass in, and it\u00a0figures out how it\u00a0should translate your query into a\u00a0set of\u00a0internal operations in\u00a0the database. For example, it\u00a0might look at\u00a0your query, and realize that it\u00a0needs to\u00a0use an\u00a0index. In\u00a0an\u00a0ideal world, the query planner would know, for every query and set of\u00a0parameters, the perfect plan to\u00a0use. But the query planner is\u00a0operating on\u00a0limited information, and sometimes it\u00a0doesn\u2019t\u00a0pick the best\u00a0option.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">This limited information is\u00a0the table statistics. You can actually query it\u00a0directly in\u00a0Postgres:<\/p>\n<div data-codeblock=\"\" class=\"_codeBlock_150cy_1 \">\n<p>Loading syntax highlighting&#8230;<\/p>\n<\/div>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">These statistics are collected for every <code class=\"_code_1fiiu_64 \">ANALYZE<\/code>. This also happens when autovacuum is\u00a0run (see below), so\u00a0more frequent autovacuums also mean that your query statistics will be\u00a0more up\u00a0to\u00a0date. A\u00a0common reason why your query is\u00a0behaving improperly is\u00a0not analyzing frequently\u00a0enough.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">The reason I\u00a0think it\u2019s\u00a0useful to\u00a0view queries as\u00a0binary\u2014they either seq scan or\u00a0they don\u2019t\u00a0seq scan\u2014is: the more you micro-optimize a\u00a0query, the more of\u00a0a\u00a0risk you take that the query planner goes rogue. If\u00a0you stick to\u00a0querying by\u00a0primary keys and indexes, the query planner will have a\u00a0much easier\u00a0time.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Let\u2019s\u00a0say that there\u2019s\u00a0nothing obviously wrong in\u00a0your query, but it\u2019s\u00a0still slow\u2014how do\u00a0you go\u00a0about debugging this? Some Postgres database providers (like Google CloudSQL) will sample your queries and save slow ones\u2014but many don\u2019t. This is\u00a0where <code class=\"_code_1fiiu_64 \">EXPLAIN ANALYZE<\/code> is\u00a0your friend. This outputs the query plan for the query and executes the query (careful running this in\u00a0production\u2014you can use <code class=\"_code_1fiiu_64 \">EXPLAIN<\/code> without <code class=\"_code_1fiiu_64 \">ANALYZE<\/code> to\u00a0get a\u00a0query plan), and then compares its estimates based on\u00a0the table statistics to\u00a0the actual number of\u00a0rows scanned. I\u00a0usually place my\u00a0sql query in\u00a0a\u00a0file, prefix it\u00a0with <code class=\"_code_1fiiu_64 \">EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS, FORMAT JSON)<\/code> and\u00a0run:<\/p>\n<div data-codeblock=\"\" class=\"_codeBlock_150cy_1 \">\n<p>Loading syntax highlighting&#8230;<\/p>\n<\/div>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">And then use <a class=\"_primary_jxyxs_66\" href=\"https:\/\/explain.dalibo.com\">explain.dalibo.com<\/a> to\u00a0visualize the execution\u00a0plan.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"sometimes-it-just-makes-sense-to-seq-scan\"><a class=\"_primary_jxyxs_66\" href=\"#sometimes-it-just-makes-sense-to-seq-scan\">Sometimes it\u00a0just makes sense to\u00a0seq scan<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">There are cases where you think an\u00a0index should be\u00a0used, but the query planner is\u00a0still seq scanning anyway, despite table statistics being up\u00a0to\u00a0date and the index being valid. In\u00a0these cases, Postgres is\u00a0usually estimating that the cost of\u00a0the seq scan will be\u00a0smaller than the cost of\u00a0the index scan. Index scans do\u00a0come with some overhead; indexes are stored separately from the actual data in\u00a0the table (called the heap)\u2014finding all of\u00a0the rows in\u00a0the heap can be\u00a0expensive!<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Unless you can dramatically restructure your query, you might have to\u00a0accept that it\u2019s\u00a0going to\u00a0seq scan, or\u00a0think about something like partitioning (more on\u00a0that below).<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"writing-lots-of-data\"><a class=\"_primary_jxyxs_66\" href=\"#writing-lots-of-data\">Writing lots of\u00a0data<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Let\u2019s\u00a0say your application is\u00a0scaling and you need to\u00a0write a\u00a0lot of\u00a0data fast. Each query has some overhead associated with it\u00a0(separate from the connection overhead we\u00a0talked about before): this includes the round-trip time to\u00a0the database, the time it\u00a0takes the internal application connection pool to\u00a0acquire a\u00a0connection, and the time it\u00a0takes Postgres to\u00a0process the query (including a\u00a0set of\u00a0<a class=\"_primary_jxyxs_66\" href=\"https:\/\/github.com\/postgres\/postgres\/blob\/master\/src\/backend\/storage\/lmgr\/README\">internal Postgres locks<\/a> which can be\u00a0bottlenecks in\u00a0high-throughput scenarios).<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">To\u00a0reduce this overhead, we\u00a0can pack a\u00a0batch of\u00a0rows into each query. The simplest way to\u00a0do\u00a0this is\u00a0to\u00a0send all queries to\u00a0the Postgres server at\u00a0once in\u00a0an\u00a0implicit transaction (in\u00a0Go, we\u00a0can use <code class=\"_code_1fiiu_64 \">pgx<\/code> to\u00a0execute a\u00a0<code class=\"_code_1fiiu_64 \">SendBatch<\/code>). Batching is\u00a0very powerful: we\u00a0found that it\u00a0can ~10\u00d7 your throughput. I\u00a0wrote more about this plus some other tips for writing data quickly <a class=\"_primary_jxyxs_66\" href=\"https:\/\/hatchet.run\/blog\/fastest-postgres-inserts\">here<\/a>.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"default-autovacuum-settings-can-kill-your-database\"><a class=\"_primary_jxyxs_66\" href=\"#default-autovacuum-settings-can-kill-your-database\">Default autovacuum settings can kill your\u00a0database<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Autovacuum is\u00a0a\u00a0critical operation in\u00a0Postgres databases that sometimes needs to\u00a0be\u00a0tuned, especially in\u00a0high-write scenarios. The autovacuum daemon is\u00a0responsible for a\u00a0number of\u00a0things, including cleaning up\u00a0dead tuples and managing transaction\u00a0ids.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">What\u2019s\u00a0a\u00a0<em>dead tuple<\/em>? A\u00a0tuple is\u00a0an\u00a0instance of\u00a0a\u00a0row on\u00a0the filesystem. Every time you update or\u00a0delete a\u00a0row, a\u00a0version of\u00a0that row is\u00a0left in\u00a0Postgres until all transactions which started before that row was updated or\u00a0deleted have committed or\u00a0rolled back. These rows which can no\u00a0longer be\u00a0read by\u00a0any transactions are <em>dead tuples<\/em>.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">If\u00a0you\u2019re\u00a0writing data quickly enough, sometimes autovacuum can\u2019t\u00a0keep up, which will get you into a\u00a0very unhealthy state, very quickly. You\u2019ll\u00a0see this when you query for active processes on\u00a0the database:<\/p>\n<div data-codeblock=\"\" class=\"_codeBlock_150cy_1 \">\n<p>Loading syntax highlighting&#8230;<\/p>\n<\/div>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">If\u00a0you see an\u00a0autovacuum query running for more than ~1 hour, you might want to\u00a0consider changing your autovacuum settings! See <a class=\"_primary_jxyxs_66\" href=\"https:\/\/www.cybertec-postgresql.com\/en\/tuning-autovacuum-postgresql\/\">this article<\/a> for more information.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">It\u2019s\u00a0worth monitoring this: if\u00a0you use up\u00a0all transaction ids in\u00a0the system before they can be\u00a0reclaimed by\u00a0autovacuum, you\u2019ll\u00a0reach a\u00a0dreaded state called <strong class=\"_primary_jxyxs_66\">transaction id\u00a0wraparound<\/strong>. This will mean a\u00a0big chunk of\u00a0downtime.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"other-types-of-bloat\"><a class=\"_primary_jxyxs_66\" href=\"#other-types-of-bloat\">Other types of\u00a0bloat<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Besides dead tuples, there are two other kinds of\u00a0bloat you\u2019ll\u00a0often encounter in\u00a0a\u00a0busy Postgres\u00a0system:<\/p>\n<ol>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Table bloat caused by\u00a0partially filled pages. Postgres stores rows on\u00a0pages on\u00a0disk, each of\u00a0which are 8kb in\u00a0size. When Postgres can\u2019t\u00a0fit new rows onto an\u00a0existing page, it\u00a0creates a\u00a0new one. But when dead tuples are reclaimed, this can lead to\u00a0pages not being entirely filled, which can increase the disk usage of\u00a0Postgres, sometimes significantly. The best way to\u00a0avoid table bloat is\u00a0by\u00a0tuning autovacuum before you\u2019re\u00a0bloated. But there are some extensions to\u00a0help with bloated tables, like <code class=\"_code_1fiiu_64 \">pg_repack<\/code>, because the built-in Postgres <code class=\"_code_1fiiu_64 \">VACUUM FULL<\/code> is\u00a0rarely a\u00a0good idea. Note that Postgres 19 is\u00a0getting <code class=\"_code_1fiiu_64 \">REPACK...CONCURRENTLY<\/code>, which I\u00a0haven\u2019t\u00a0tested, but seems like potentially a\u00a0good solution for concurrent table\u00a0repacking.<\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Index bloat is\u00a0a\u00a0special case of\u00a0table bloat, and is\u00a0similarly solved by\u00a0good autovacuum settings. But Postgres has a\u00a0built-in command for dealing with this, which is\u00a0<code class=\"_code_1fiiu_64 \">REINDEX INDEX CONCURRENTLY<\/code>.<\/li>\n<\/ol>\n<h3 class=\"_h3_jxyxs_9 h3 _balance_jxyxs_54\" id=\"some-advanced-stuff\"><a class=\"_primary_jxyxs_66\" href=\"#some-advanced-stuff\">Some advanced\u00a0stuff<\/a><\/h3>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">I\u00a0wanted to\u00a0end with a\u00a0set of\u00a0advanced Postgres features which have been particularly useful for us\u00a0at\u00a0Hatchet.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"for-update-skip-locked\"><a class=\"_primary_jxyxs_66\" href=\"#for-update-skip-locked\">FOR UPDATE SKIP\u00a0LOCKED<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">The best way to\u00a0think about this Postgres feature is\u00a0that it\u00a0reserves the rows that you\u2019re\u00a0selecting for use in\u00a0your transaction without interfering with other queries. We\u00a0use it\u00a0primarily for implementing our <a class=\"_primary_jxyxs_66\" href=\"https:\/\/hatchet.run\/blog\/multi-tenant-queues\">job queue<\/a>; a\u00a0single-query queue in\u00a0Postgres can be\u00a0implemented like\u00a0this:<\/p>\n<div data-codeblock=\"\" class=\"_codeBlock_150cy_1 \">\n<p>Loading syntax highlighting&#8230;<\/p>\n<\/div>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">It\u2019s\u00a0also very useful in\u00a0cases where you\u2019re\u00a0doing many independent updates of\u00a0rows, or\u00a0you\u2019re\u00a0managing leases on\u00a0objects in\u00a0your system across many instances of\u00a0your application (for example, we\u00a0use this to\u00a0distribute tenant leases across Hatchet\u00a0engines).<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"partitioning\"><a class=\"_primary_jxyxs_66\" href=\"#partitioning\">Partitioning<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Postgres has built-in partitioning which allows you to\u00a0subdivide your tables based on\u00a0row values, like timestamps or\u00a0hashes. This can be\u00a0incredibly useful for time-series data (in\u00a0our case, historical task data) because:<\/p>\n<ol>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Each partition can be\u00a0autovacuumed independently, allowing you to\u00a0scale up\u00a0autovacs on\u00a0your table<\/li>\n<li class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Deleting old data is\u00a0near-instant\u2014you simply drop the table partition, rather than iterating through\u00a0rows<\/li>\n<\/ol>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">Partitioning does come with drawbacks: there can be\u00a0overhead on\u00a0read queries if\u00a0Postgres doesn\u2019t\u00a0prune partitions during the planning phase (something that Postgres has gotten much better at\u00a0in\u00a0recent releases).<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">I\u2019ve\u00a0written more about our experience with partitioning <a class=\"_primary_jxyxs_66\" href=\"https:\/\/hatchet.run\/blog\/postgres-partitioning\">here<\/a>.<\/p>\n<h4 class=\"_h4_jxyxs_13 h4\" id=\"tricks-for-large-table-migrations\"><a class=\"_primary_jxyxs_66\" href=\"#tricks-for-large-table-migrations\">Tricks for large table\u00a0migrations<\/a><\/h4>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">(note that this isn\u2019t\u00a0referring to\u00a0a\u00a0database migration, but rather moving large amounts of\u00a0data from one table to\u00a0another, which we\u00a0find ourselves doing a\u00a0few times a\u00a0year)<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">If\u00a0you try to\u00a0migrate really large tables, it\u00a0can take many hours to\u00a0copy data in\u00a0a\u00a0single transaction. This isn\u2019t\u00a0good\u2014long-running transactions prevent autovacuum from doing its job properly, and therefore make the entire system bloated with dead tuples. Also if\u00a0you want to\u00a0keep writing to\u00a0the old tables, the new table won\u2019t\u00a0see the new\u00a0data.<\/p>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">So\u00a0we\u00a0need to\u00a0figure out how to\u00a0migrate data safely without a\u00a0transaction, with new writes (after the migration has started) being copied to\u00a0the new table. One of\u00a0the tricks we\u2019ve\u00a0learned is\u00a0to\u00a0use Postgres triggers and run a\u00a0large batched backfill outside of\u00a0a\u00a0transaction, using unique constraints on\u00a0the primary key to\u00a0prevent duplicate\u00a0writes.<\/p>\n<hr class=\"_hr_1fiiu_193\"\/>\n<p class=\"_body_jxyxs_22 paragraph _secondary_jxyxs_58\">That\u2019s\u00a0it! If\u00a0you have any other tidbits about Postgres scaling\u2014or questions about Postgres scaling in\u00a0general\u2014feel free to\u00a0reach out.<\/p>\n<\/div>\n<p><a href=\"https:\/\/hatchet.run\/blog\/postgres-survival-guide?utm_source=tldrdata\">Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Over the past half year or\u00a0so, I\u2019ve\u00a0been writing an\u00a0internal doc for our engineers trying to\u00a0distill two years of\u00a0Postgres battles into a\u00a0somewhat cohesive document. While I\u00a0love the Postgres manual, I\u00a0find it\u2019s\u00a0hard to\u00a0turn to\u00a0when shit hits the fan because it\u2019s\u00a0just so\u00a0darn comprehensive. I\u00a0thought this might be\u00a0useful for others and would appreciate feedback (or\u00a0other tidbits that you\u2019ve\u00a0learned running [&hellip;]<\/p>\n","protected":false},"author":16,"featured_media":22664,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[143],"tags":[],"class_list":["post-22663","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22663","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/comments?post=22663"}],"version-history":[{"count":0,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/posts\/22663\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media\/22664"}],"wp:attachment":[{"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/media?parent=22663"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/categories?post=22663"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scannn.com\/lv\/wp-json\/wp\/v2\/tags?post=22663"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}