Odiya Location Ingestion Server

01The problem

If every location upload is written straight to the DB, the server collapses along with the write load when it spikes. So the design buffers records in a Redis queue and saves them in chunks, but that comes with two traps. If dequeuing is not atomic, two processes can split the same data between them and lose it, and if a single corrupted record slips into a batch, the whole batch rolls back and no user's locations get stored.

02What I decided

I unified the dequeue path into a single atomic bulk pop. I tried both the range-read-then-trim approach and a script-based approach, but load tests measured actual data loss with them. If a save fails, the popped records are reversed and pushed back to the front of the queue, and duplicates are filtered out on re-save. Parsing and validation happen before the save, so only broken records get quarantined and automatically reprocessed.

• Atomic bulk pop only, saving in chunks of up to 5,000 records every 60s
• At-least-once storage guaranteed via reverse-order restore and deduplication on failure
• Corrupted records moved to a quarantine table and periodically re-injected, with unrecoverable ones excluded
• Server commands designed as a one-way channel, piggybacked on upload responses
• Single-execution locks on all schedulers to block double runs during zero-downtime deploy overlap windows

03From legacy Tomcat to zero-downtime deploys

The deployment I inherited put a war file onto a standalone Tomcat. Deploying meant taking the server down, swapping the file, and bringing it back up. Ingestion stopped entirely during that window, and whatever child devices uploaded then either piled into retry queues or was dropped. On a location service, deploy time is time a parent cannot see their child. Builds also ran on the server itself, so a failed build left it half dead.

• Artifacts were consolidated into a single executable jar while the old war path kept being produced for a while. Rewriting the deployment method and the application structure at the same time was not worth the risk
• Never build on the server became the rule. Images are built locally, compressed, and streamed over SSH so the server only receives and starts them. A failed build can no longer reach production
• Two slots run side by side: the new version starts on the idle one, a readiness check that reaches DB and Redis has to pass before the front proxy hands traffic over, and only then does the old slot stop. From a user's perspective nothing is interrupted
• Production carries an extra gate that checks the login page answers correctly and that the identity-verification redirect leaves over https. An earlier incident had that redirect going out over http and getting blocked inside the app's web view
• Images carry rollback tags, so a bad release goes back to the previous version immediately, with no rebuild required
• Because the two slots briefly run together, every scheduled job takes a single-execution lock so batches never double-run. The new risk that zero-downtime deployment introduced got closed at the same time

04Outcome

The data loss we used to measure in load tests is gone, and when corrupted data arrives, only that record is quarantined while the overall ingestion keeps running. Even during a DB outage, connection waits are cut short so the failure does not spread to the whole server, and all three environments deploy via traffic switching, so collection never stops.