Postmortem:

Yesterday, we made some code changes to adapt to an issue we saw when processing tracking from a particular carrier.

Unfortunately, the effect of that code changes caused the background process that consumes the carrier tracking events to slow down by several orders of magnitude. This morning, once we realised that tracking was not being processed fast enough, we made the changes required to fix that issue, and upon releasing that change, the system sped back up and got through to roughly 500,000 tracking events that were sitting in the queue.

Our intention going forward is to set up alerts against that queue such that if we see the queue growing beyond what we expect, we will be immediately alerted so that we can look into the fix proactively rather than reactively as we have in this situation.

MachShip can see status events are now being correctly processed

The fix has been deployed, and the tracking statuses are being consumed and updated.

The MachShip team will continue to monitor and advise when all events are up to date

The tracking issue has been identified, and a fix is being pushed into MachShip now

This fix will result in any statuses that have yet to update will be processed, and there should be no loss of data

The tracking of consignments in MachShip isn't updating.

The MachShip team is now investigating this as a high priority and will advise further updates once they are available.