What to do when production goes down: Two weeks ago in my new job one of the applications that the company uses to serve a large part of its internal customers stopped working, this is an application that uses GIS services to represent, consult and print georeferential information corresponding to electrical entities. When the affected departments reported the incident, I realized that within the server that managed these applications, the folders with the necessary files were empty.
I immediately set about trying to recover as much information as I needed, in order to put back the necessary applications and bash scripts that should be running in the background.
One of the applications consisted of a Java Glassfish application server and within that server two more applications were deployed which had the following task:
- One consisted of the application, that is, the web application itself developed in Java with the standard of JavaServer Faces (JSF).
- The other is an API to serve information corresponding to maps and render said maps. The application use this API within its modules.
- And another that consisted of a GIS server for its in a module of the application (JSF) where the users have the possibility of interacting online with the geographic entities.
Scrutinizing the server I found the source code of the three applications, I immediately set me to work to deploy them and put them into production, but here things got complicated.
The first two (The Web Application and the API) are both applications deployed by the Glassfish server that we mentioned earlier, and the files I found were apparently corrupted or missing information so I asked a member from the development team the source code of the Web application (JSF) in the company's repositories, as well as the API and eureka but there was another problem, when testing the application in a local environment it worked correctly, but when I tested it in production trying to move between the modules (links or menus of the Web Application) the server threw error 500. So I came to the conclusion that the files corresponding to the glassfish server that I found could be equally damaged so download a new one, configure bash files to change environment variables such as JAVA_HOME, etc to deploy the server.
Done, glassfish server installed, apps and APIs deployed, but happiness was short lived 😔. When trying to search for geographic entities in the application, the application did not return anything, so I had to learn the basics of the JSF application workflow and reading the source code I realized that the SQL queries that the application made to the DB had a parameter that was no longer being used, so I had to decompile that package from source, change the SQL queries, compile the app, package it, deploy it with Glassfish, and that's it.
This whole process of testing, error, reading code, consulting my colleagues and so on took me almost two weeks, thank heavens there were alternative methods by which other internal clients could continue with their operations, although not at the same speed and fluidity as before but now everything is back to normal and what I have learned in these two weeks is priceless, being able to recover applications from a production server, configure them from scratch without having the slightest idea was an amazing trip.
With this I want to invite you to lose your fear of breaking things while doing tests, but always do them locally😅 production is not touched. 🚫
If you liked this article and want to continue learning from my experiences in the world of technology, please subscribe to my blog. Also, if you think this article may be useful to someone you know, feel free to share it.
Top comments (0)