A Freakin’ Nightmare
In my last entry I suggested that there may be a little down time associated with a server/data center move that my hosting service would be doing. The expectation was that the move would occur on Friday evening and that by Sunday I would be back on-line. I felt like an expectant father pacing the floor worrying about my sites and my data. IX Web Hosting was very good at notifying their customers of what to expect and keeping us informed about how things were going. Early Sunday morning I received notification that the server move was completed and was even ahead of schedule. This was awesome! I went out and checked each of my sites and sure enough they seemed to be up and functioning. I left feeling that the worst was behind me. Little did I know that this was just the eye of a very big storm.
After spending some time with my family on Sunday (something fairly rare unless you count the Arizona Diamondbacks games we go to and Trina refuses to count those as family activities), I came home and decided to update my Diary of a Diehard blog as well as this one and let everyone know we were back up and running. I opened a browser and went to update the blog. Instead of the normal login page I got a screen filled with error messages. Wow, I must have typed that in wrong I thought. I attributed it to just being out of the habit for a few days. I re-entered the address and again I got a page filled with error messages. I immediately thought perhaps something did not get moved properly so I tried the other blog to see if I could get in there. No, that was the same problem. I quickly went through each site and found that they all had the same error messages. I went out to the IX Web Hosting site and opened a problem ticket and then started in depth troubleshooting of the problem. Over the next two hours I had identified that the server was missing three key components. I updated the problem ticket with the information I found and I wanted for a response. Minutes turned to hours and hours turned to days. The only update to my trouble ticket were the ones that I had entered. This is unusual since IX Web Hosting has been quite responsive in times past when I needed assistance. I turned to their Live Chat to get an update. The chat took a long time to load and even then I waited 20 minutes before a representative came online. I explained the situation and gave him the ticket number. After a 15 minute silence I got a response that the admins would look at it shortly and I should be back up and running in a couple of hours. Although longer than I wanted, I accepted that. Four hours later I was in the same place and the ticket had not been updated. I again went to the Live Chat and again had to endure the delay for a representative to respond. When they did they apologized and said that things would be fixed in four hours. The time had doubled but I was still not panicked. I decided to give them 6 hours just in case since I know how it can be dealing with a large migration such as they just went through. Feeling sorry for them I gave then the night to recoup thinking that by Monday everything would be back to normal. No, Monday came and the server was still mis-configured. I went back to Live Chat and asked for an update. This time they said it would be an additional 12 hours before they would get to my problem. This was starting to really irritate me. I had basically been down since Thursday and now it would be Monday night before I got back online. Still what could I do but wait. By late Monday night the server was still broken. I again went to Live Chat and waited. When a representative finally arrived I asked for status and was told it would be fixed within the hour. As you might have guessed the time came and went without an update or a resolution. By this time I was ticked off and shot off an email to the owner of the company who offered to listen if anyone had a problem. By the next morning a senior customer service manager looked at the site and declared it working. Obviously he didn’t even read the problem ticket or else he would have understood that it was not that the modules were not loaded it was that they were not configured. He closed the problem ticket as resolved. I then had to reopen it and explain how testing should be conducted. I even went as far as describing in detail what needed to be fixed and provided web links describing each step of the process. Again this problem ticket sat without an update. I determined that this needed to be fixed so I went to the Live Chat area after each deadline came and went to remind them that they had promised to fix this problem by a certain time. This went on through Tuesday and Wednesday. Last evening I was still dealing with the problem and went back to Live Chat. This time the representative was rather short and somewhat rude. I didn’t take it personally but chalked it up to cultural differences since IX Web Hosting has outsourced much of their technical support to the Ukraine. This representative said he could not help me as he “doesn’t do problem tickets”. Yeah that was about the last thing I wanted to hear at that moment so I questioned him as to what his purpose was on this earth then. He stated that he was there to answer simple questions. Oh this should be fun. I began peppering him with “simple questions”. As he began answering these it became clear that there were modules missing from my server. Wow, what an epiphany that was! He suggested that I update the problem ticket to state that I needed to have these modules installed. At this moment I began banging my head against the desk before taking a deep breath and explaining that the problem ticket did contain this information. Oh, he said, I will forward this to the admins for them to install this for you. I told him I would wait for him to do that. The ticket was forwarded to the server administrators at 4:12 PM on Wednesday. I sat in the chat and waited patiently. The installation and configuration of these modules takes about 10-15 minutes tops so I was starting to feel better knowing I would soon be back online. The Admin team got to my problem ticket and installed the three missing modules at 1:53 AM this morning. That is nearly 10 hours after they were assigned the work. I wanted to tear my hair out. I decided that I would step away from the keyboard and take some deep cleansing breaths before I posted anything I would later regret. So after some sleep and a bowl of fruit I am a little calmer. The sites are now working as they were before the migration. Now I have a full day of trying to recover and post what I could not for the past 6 days. Maybe I need to start looking for a new hosting provider. IX Web Hosting has been great in the past. I just wonder if this is an indication of what I can expect in the future or if this was some sort of an anomaly in service breakdown. It’s something I plan on monitoring very closely.