Ahmad Leo Yudanto

The catalog consumers kept rebalancing. Every rebalance paused the group, lag grew, and the answer had been to add pods. More pods meant more members joining and leaving, which meant more rebalancing. The cluster had grown to 200 pods and the lag was still there.

The root cause was consumers taking too long between polls and getting kicked out of the group, plus an out-of-memory crash that restarted pods mid-batch. I fixed the consumer, not the cluster:

# consumer, per topic
partition.assignment.strategy = StickyAssignor   # keep partitions on rebalance
max.poll.interval.ms          = ↑ raised         # slow batches no longer evicted
max.poll.records              = ↓ lowered        # smaller, predictable batches
ackMode                       = RECORD
concurrency                   = 1 thread per partition

• Fixed the OOM in the consumer service so pods stopped restarting mid-batch.
• Made consumption idempotent: a message already logged as processed is skipped, so retries are safe.
• Added a dead-letter topic and moved heavy scheduled topics into their own application mode, so they stopped loading the main consumer group.

Same traffic, 30 pods, lag back under control, including campaign days that jump from about 90,000 to 500,000 orders. Nobody gets paged for it any more.

When I joined, most services had between 0 and 15% unit test coverage and no quality gate. I added JaCoCo and SonarQube to the existing CI/CD pipeline so every merge reports coverage, then brought the services in my scope above 90%.

The part I'm proudest of is how it stays there. I wrote our team's engineering rules as Markdown, then packaged them as AI skills: one writes tests to our conventions, one reviews pull requests and posts comments, one resolves test-file merge conflicts, and one debugs by reproducing a production issue in a unit test with real data first.

• Everyone on my team now uses AI to write tests and debug, under the same written rules, so the output looks like ours.
• Production issues on our services have dropped noticeably since tests became the default.
• My SDET background is why this worked: I design the tests first and let the tooling fill them in.

Promotions are where e-commerce money goes wrong: one rule off and the cart gives away margin, or a customer is charged more than the shelf label says. I lead the squad that builds them.

• Free Gift (Hadiah Gratis): a new promo type. Meet the condition and a free item joins the cart. New APIs for eligible gifts and add-to-cart, with stock handled so a gift never oversells. In its first year: 2.5 million orders with a gift and 6.5 million free items handed out.
• Non-base to base migration: moved existing promotions onto the base promo model without breaking live campaigns.
• Paired-product exclusion: removed paired products from specific promos, which touched pricing, strike-through labels and upsell logic across cart and catalog.

Every launch shipped on schedule, and nobody on the team worked a weekend for it. I plan for that on purpose: a team that isn't exhausted makes fewer mistakes in money logic.