First Deploy First Rollback

In 2026-05-27 I did my real deployment to production. In 2026-05-28, I did my first rollback.

It was a bittersweet experience because of course I’d rather deploy perfect code instead of having to revert to a previous working version, but truth is I learned a lot more because of the rollback.

Let me try and explain what happened in this wannabe post-mortem blogpost.

The requested feature

My task was simple: to adapt a system to be able to deal with a new kind of identification the brazilian government will start to issue in July 2026: the alphanumeric CNPJ (Cadastro Nacional de Pessoas Jurídicas).

CNPJ, as of this post (June 2026), is a 14 digit number that is used to identify companies. For example, the following is a valid CNPJ number: 29.507.878/0001-08 (dots, slashes and dashes are just for visualization). Beginning July, the government will begin to issue CNPJ whose digits may be, besises numbers, any letter of the alphabet. In short, alphanumeric digits. The following is a valid alphanumeric CNPJ: M8.BHG.844/0001-19

In terms of the application, basically I had to check if CNPJ were properly stored as VARCHARs in the databases and check if the application itself did some sort of processing that could break if CNPJ characters turned out not to be numbers.

The application was mostly ready to deal with this new scenario. I had to change very little things. So, along with this change, my manager also told me: check all the used library and bump their versions to the most recent ones. That is where our problem came from.

Breaking Changes

The application is a C#/.NET Web API, created using the ASP.NET framework. It mostly listens to requests from a frontend, does some processing, queries and store data in a SQL Server Database and, in two situations, it sends requests to a third-party system.

To send requests to the outside world, the application uses the Refit Nuget Package. This package had a breaking change that hit us because we bumped versions. We updated from Refit 5.x to Refit 10.x. In version 6, the developers introduced a breaking change.

In Version 6, Refit changed the default serializer it uses to serialize/deserialize JSON payloads from and to memory objects. It used Newtonsoft.Json and it changed to System.Text.Json.

This turned out to be a problem because a part of the code used Newtonsoft.Json tags to change the way some in-memory objects were serialized. Specifically, a tag was used to rename a JSON field when serializing an in-memory object.

Since the serializer changed, this renaming did not occur, thus breaking the contract with the third-party system.

The Fix

The fix was pretty simple: I just had to specify Newtonsoft.Json as the desired serializer during the Refit service injection.

Some nice extras

Man, was I glad that the CI/CD pipeline was working properly. Just had to press a button and the rollback was done pretty quickly. This situation would be pretty bad if I have to manually rollback the version. Keep in mind this application had it last deployment 2 years before mine. Phew!

Also, I am glad that the observability plataform allowed be to quickly query logs and identify root cause. I don’t love Dynatrace, but it worked.

Now I will write a better post-mortem than this one and share with the team. I think it might do some good for a future developer (maybe it will be me, haha).

Thank you for your interest and time!