profile

+++ title = "Gonzalo Tudela Chavero" summary = "Backend & platform engineer. Performance, scale and long-running jobs that can be stopped, resumed and trusted, on a network of 170+ sites with 680k+ posts." fields = [ ["Name", "Gonzalo Tudela Chavero"], ["Role", "Backend & Platform Engineer"], ["Focus", "performance, scale, reliability"], ["Stack", "PHP, MySQL, Varnish, Redis, Rust"], ["Code", "github.com/GonzaloTudela"], ["Contact", "linkedin.com/in/gonzalotch"], ] +++

Overview

I work on the backend of a publishing group's network of 170+ WordPress sites. About 80 of them are translated into ~45 languages and together hold 680k+ posts. It all runs on shared PHP-FPM, MySQL, Varnish and Redis that I do not configure, so most of my work is making code behave well on a platform it cannot change: bounded memory, bounded time, no duplicate work, and safe to kill at any moment.

Long-running jobs that can be stopped, resumed and trusted

• Multi-day migration, resumable by design. A CLI migrator moves ~145,000 translation slots per site. The cursor is written atomically (temp file + rename) after every item; SIGINT/SIGTERM let the item in flight finish; SIGUSR1 dumps the process stack to the log. "Success" is printed only after database-wide assertions pass. Pilot: 24,360 of 24,360 indexed URLs still resolving, zero 404s.
• Reversible bulk deletion at 80k-post scale. Each run persists its state, cursor, snapshot and log. CLI and web cron hand the same run over through a row lock with TTL, owner and heartbeat. Backups are verified before anything is deleted; restore streams gzip with CRC checks and resumes from a (file, line) cursor. 80,893 posts pruned on one site, 41,062 on another.
• A cron that never got past selecting. It ran out of its 256 MB loading 99,465 rows and processed none. Rebuilt around batches of 50, a keyset cursor that only advances over finished work and a lock released only by its owner. Memory per post: 2.61 → 1.33 KB.
• Retries that degrade instead of amplifying. Per-backend circuit breaker: opens after 5 failures, 60 s doubling up to 900 s, with ±20% jitter so hundreds of rows never retry in lockstep.

Concurrency and consistency on shared infrastructure

• Queue claims without double delivery. SELECT ... FOR UPDATE SKIP LOCKED inside a transaction, rollback on partial claims, a generation counter, and reclaiming of rows stuck in processing after a crash. The receiver upserts on (original, language): a resend never duplicates a post.
• Locks matched to the failure they prevent. MySQL advisory locks for migrations and purges, with fail-open or fail-closed argued in the code; row locks with heartbeat for long jobs; flock single-flight so two identical expensive queries never run at once.
• Idempotent down to the schema. A restore had dropped a primary key, so every upsert silently inserted: 37,984 rows for 5,972 posts. The code now detects the missing key instead of trusting the migration tool.

Making slow things fast

• Sitemap worker: ~16,700 → 54-331 queries per pass. N+1 removed in three layers by batch-priming; one file went from 758 queries to 2, peak memory from ~330 MB to ~150 MB, output byte-identical.
• 1.2 GB → 1.2 MB. A dry run indexed 5.4 M keys to look up 761 URLs. It now indexes only what it needs, and still finds all 761.
• Microseconds instead of a request. Deleted URLs live in a sorted binary file of 8-byte keys, searched with binary search over fseek: 23 us per lookup with 10 M keys, no database, no Redis.
• A cache index eating Redis. A third-party plugin grew a 370-480 MB key index and 22-second requests; declaring those groups non-persistent fixed it.

SQL that holds up at millions of rows

• Query plans read, not guessed. EXPLAIN showed a filesort that was harmless at 14k rows and fatal at 42 languages x 150k posts; ordering by the driving table's index let LIMIT stop the scan early.
• Keyset pagination and snapshots everywhere: ID > n ORDER BY ID LIMIT k, each run pinned to its initial MAX(ID).
• Search that cannot take the database down. LIKE '%...%' replaced by boolean FULLTEXT with a self-healing index and a fallback; paths that are not slugs never cost a query, which mattered when bots generated thousands of 404s per hour.
• Destructive SQL with a safety net. Fleet-wide templates must batch by primary key, count before and after, and back up first; a linter and a self-test on real MySQL caught a collation error in 3 of 16 templates before production.

Cache correctness and invalidation

• Purges that never purged. Varnish answered 200 to every PURGE whether it evicted anything or not. Using Age as the only honest signal I found four chained defects, one of them affecting any non-ASCII slug purged from WordPress. After the fixes, 42 translations purge in 1.67 s for 504 requests, so I retired the deferred queue I had built for the old cost.
• Breaking a purge storm. Sequential purges held workers for 5 s x N nodes, callers retried, retries multiplied purges. Fix: parallel curl_multi with hard timeouts, respond before purging, a 60 s throttle and deduplication.
• The layer nobody checked. Code still ran in its old version 70 minutes after a deploy: OPcache is now flushed as a third cache layer and verified.

Integrations and failure modes

• A proxy with no timeout was filling the PHP pool. On Linux max_execution_time counts CPU, not wall time. My self-reapplying patch answers 503 + Retry-After instead of an empty 200, with timeouts chosen from 64,225 measured responses: 2.5x fewer workers held during an outage.
• Workers waiting on themselves. Image sizes read over HTTP from the same site made each worker wait for another worker in its own pool. Reading them from the database fixed 1,622 failed self-requests in a single log.
• Answer first, work after. Webhooks acknowledge in 37 ms and do the slow part after the response; a production-safe crawler caps itself at 1 req/s and stops on 429s.

Incidents and root cause

• A bistable deadlock, mitigated the next day. After a large prune, bots and a proxy sent ~138,000 404s in 90 minutes and PHP workers ended up waiting on a proxy waiting on the same pool. Within 48 h an early 410 handler was in production: p50 11.2 ms against 156-320 ms for the theme's 404.
• Not silencing a noisy alert. A nightly 5xx spike was traced across sites, databases and storage to backup jobs; after the provider changed them it stayed at <=0.4%.

Measurement and verification

• Alerting as code with backtests: a redesigned alert fires 11 times where the old one fired 29. Log-query cost measured before designing (~0.65 s per hour of data), which led to 6-hour slices and caching of closed slices.
• Mutation testing applied by hand and logged ("25 applied, 25 caught"), shared contract vectors across repositories, CI that rejects a tag that does not match the version it ships. One tool alone carries 2,416 tests.
• I build with AI coding agents inside verification gates: separate implementer and adversarial reviewer, tests seen failing first, nothing accepted on the agent's word.

Systems, Rust and my own infrastructure

• gondor, my own server: HAProxy terminating TLS and fingerprinting clients (JA4), Traefik, CrowdSec, Docker, and Prometheus, Loki and Grafana watching it all.
• This site is Rust end to end: actix-web serving precompressed, content-hashed assets under a strict CSP, and a ratatui terminal compiled to WebAssembly, drawn with WebGL2 through a CRT shader.
• Input measurement on Windows in Rust (Win32): showed that Windows 11 throttles background mouse input to ~125 Hz, against 1,000 Hz with a low-level hook.

Toolbox

 Keyset pagination ......... FOR UPDATE SKIP LOCKED .... transactional outbox
 Advisory locks ............ TTL + heartbeat locks ..... atomic checkpoints
 Signal-safe CLI stop ...... idempotent upserts ........ circuit breakers
 Adaptive batch sizing ..... memory guards ............. streaming I/O
 EXPLAIN-driven indexing ... FULLTEXT search ........... curl_multi fan-out
 Respond-then-work ......... single-flight caching ..... multi-layer purges
 Alerting as code .......... mutation testing .......... contract vectors