Got some reports that the data 1 server was having issues. The two reports were that it was not finishing the invoicing process on any corp and that a person could not edit a user account. I logged in and took a look at what was going on, sure enough, the invoice process was stopping part way through. It took awhile, but we found that the problem was the invoice history table. That table was tied in to both the add and edit invoice processes. Because we had ordered a full server restore, it was replacing tables with older data from a back-up. We also had a couple of people still on the server trying to put in new data. Just the way the tables worked, it created a conflict and was giving us a duplicate record error.

The other problem was the main user table. Originally, it looked like it had been hit as part of the server hack. We later found out that it too was trying to sync old data and new data and just needed a little bit of loving. In the meantime, we posted a temporary login page and added notes to it throughout the day as we found out new pieces and made small break throughs. It was pretty stressful on our side of the fence. I know that some of the clients, on their side, were stressing it as well.

Throughout the day we 7 or 8 of us here at adilas helping out. We had Calvin helping me on the servers and checking database stuff. Wayne popped in and helped us reset a number of things and do some looking around to see if we could see any other harmful activity. Alan popped in and helped us with fixing the database problems towards the end of the day. None of them were very straight forward. Steve was helping to coordinate who was doing what and testing with fake data to make sure things were flowing through. Shari O. and Drea were doing emails, phone calls, and text messages. I was digging through log files, working with the other guys, and back and forth with the server team in Arizona. Very busy day and it took the whole team to figure out different parts of the puzzle. I'm super grateful for their help.

Only the one server was down, but that was pretty crazy. Lots of lessons learned and it was re-restored, released, and pretty heavily tested by around 6 pm. We added a link to the main login page on that server to show a link of what happened and some notes from throughout the day. One of the most stressful things was wondering if the problems were part of the hack job or just circumstances of mixing old and new data from the restore from back-ups.

On a positive note, I would say that the biggest lesson for the day was - enjoy the learning process. Sometimes that is fun and sometimes not, but the cool thing is, if you enjoy learning, it will always make you better. Pretty cool. Long day.