Last Saturday morning my blog was down for the count. I had received an email from Are My Sites Up stating that Creeva.com was down. It told me that my blog went down late night on Friday. I verified on Saturday that it was still down, and I had wanted to do some writing on it. I waited. I attempted to call tech support but after being the first caller inteh queu for over an hour I surrendered. Saturday came and went and no blog. Sunday morning site was still down.
I called into tech support again Sunday afternoon. I was fifth in the queu and I was moving forward about 15 minutes at a time. Either people ahead of me were getting help, or they were giving up like I did the day before. I held on and finally managed to get through to a person. Upon talking to the tech support I learned that my hosting server had moved. Not only had it been moved, it was moved back in October.
I had known that they were going to move servers, but the e-mail announcing this stated that most people would not notice an changes. What this says to me is that a new server will be put up with the same IP address with very little down time in the migration. This however was not done, so all of my data was moved to a new server. Since October I had been updating my information and blog on the old server. When I managed to correct my DNS entries at my registrar, I discovered my blog was out of date since early November. I was missing literally hundreds of articles once you take into account my lifestreaming posts that don’t make the front page or RSS feeds. I was livid.
I called back and they stated they would get it synced back up off of the old server. Ten hours later there was no change. I waited until Monday when I knew there would be more then just support in the office. Since I’m a reader of the Consumerist I followed on of their recommended ways of dealing with this is, I sent an executive e-mail carpret bomb here is what I wrote:
To Whom Can Help Me,
I have been a customer with AN Hosting (anhosting.com) a division of Midphase Hosting (midphase.com) since 12/24/07. In October you were planning a server consolidation/migration plan (notice sent out is given below my e-mail (removed from blog article)) – which stated that some of our accounts would be migrated to new servers. At that time I created an offline backup of my domain. Things went on working as they normally did until Friday night 1/16/09 7:15 PM. At this time my site went offline. This was verified by the service Are My Sites Up? (aremysitesup.com). I was not aware and didn’t check my email to be informed about this until Saturday 1/17/09 early evening. I called the support line and though I was the first customer in the queue I waited over an hour before I gave up.
On Sunday 1/18/09 I called in with my problem. It seems my “free” domain was working properly – I don’t really use this site so I hadn’t checked it. My add-on domains however were not working at all. My main site which is an add-on site (creeva.com) was not working. The technician stated that I had been moved to a new server and the old server was turned off, but my data should still be available. My old server IP address was [REDACTED], the new server you had migrated me to was [REDACTED]. The technician on the phone stated that if anything was wrong with the site to call back and they could pull the back-ups off the old server. I thanked the technician and got off the phone to start relocating my DNS and testing the site.
Before I migrated my DNS I adjusted the local host file on my computer to make sure everything was working on my main site. Upon loading it and checking the entries it seems at the closest date I could pin down my latest blog entry was written at 11/10/07. It’s hard to pin down since in October due to other concerns I had written and scheduled blog posts for most of the month of November and some going forward to early December. The data may even be from October because I use these functions of wordpress. I called back in to the support line and explained the issue and asked if I could get the backup loaded that he previously mentioned.
He opened a ticket for me and stated he would get this assigned. In the ticket language he was vague – here is was he entered on my behalf:
Hi, I am having issues with my addon domain(creeva.com). My account was migrated
to another box. I am not able to view the latest version of my website, its
pointing back to November 10. I’ve been posting updates to my blog. Can you
please look into my account?
I’m not sure why it had to be opened under my name using the “I” reference – since the technician opened the ticket (ticket number [REDACTED]). I would think this is misleading to the technicians working on the case. I checked out the language after I got the auto-email generated when the ticket is assigned to another technician. I added the following entries after the auto reply to clarify the situation:
None of my add-on domains to my account are up to date – these have been
constantly updated through the last few months – yet under the current server
migration I was told I was migrated about 3 weeks ago. So there would still
have been a discrepancy of a month during hte migration.
The migration letter stated that it would be seamless to the end user –
normally in most scenarios this would include making sure that the data was at
the latest possible and maintaining the same IP address – since I’m sure I am
not hte only customer that use another registrar. If monitoring this was made
more apparent in the letter going out about the migration I’m sure this would
have saved the technical support some headaches.
I was finished some major changes on my add-on site creeva.com at about 4PM
1/16/09 – if a backup of the previous server ( [REDACTED]) can be found after
that time that would be terrific. Is it Midphase’s policy to make backups
before taking a server offline?
Also wanted to ad – my blog at creeva.com may seem up to date more Nov. 10 – but
everything going through a publishing right now was written to publish at a
future (now past date) and the blog is catching up with it’s scheduled posts.
The current date of the latest post is not an accurate reflection of the latest
data from a back up – the bet I can ascertain the date was at least Nov. 10th
if not earlier.
A while later one of your technicians had checked on the issue and issued this reply:
We have investigated your issue. We could find that your account was migrated
from our old server [REDACTED] to our brand new server [REDACTED]. Please let me know if
you are using any ftp clients to upload the files. If so please use the details
that we have mentioned below to upload the files and update your website. Please
check the settings of your FTP client.
Server name : [REDACTED]
Server IP : [REDACTED]
Username : [REDACTED]
We have also checked the blog site creeva.com and entered a post and the post
was successfully updated. We have attached 2 screenshots along with this mail
for your reference. Please check it.
Please let me know the exact issue regarding your subdomains. It seems that you
are uploading the new updates to our old server. So please use the above details
in your ftp client so as to upload the files to the new server.
Technical Support Representative.
Hosting Services Inc.
Since he didn’t seem to understand what I was asking I reiterated my request – also notice in [REDACTED]’s reply that he gave me a third IP address that I should be using:
I need the data off of [REDACTED] – which is where my add-on host was before
yesterday – I wasn’t using an FTP client – I was directly entering the data via
the wordpress interface – I checked my e-mail logs and at no time was I alerted
there was going to be a server move where I would have had to adjusted the IP
address records of my add-on domains, if I had received that I would have been
more pro-active and migrated the DNS earlier.
The problem is I need the lastest information from Friday off of the
[REDACTED] – one of the advantages on your home page is the fact that you
do nightly back-ups – so the backups should still be available.
I’m missing approxiamately 200 posts on my blog – affiliates update
information, theme updates – and that’s just for one add-on domain. I also
find it odd that you give me the IP address [REDACTED] – earlier tech
support gave me [REDACTED] – which restored my access to the site – I added
the IP address you gave me and added it to my hosts file locally before I
updated my registrar – I was hoping it would give me a more up-to date version
of my blog – it didn’t and just delivered me to a CGI page and the site doesn’t
load at all.
The problem is not access – and I can re-adjust my own FTP software – my
problem is data -freshness
Can I please get my data restored from the old server?
While you stated that during the migration – “In most cases, the migration is completely seamless. We have brought in a team of experienced migration experts to ensure this effort is handled with the best of care. If you should have a problem relating to your site, please do not hesitate to contact us: email@example.com. We have a team standing by 24/7 to help out!” I question the migration experts that performed this migration. Currently at my place of employment we are performing similar actions with our infrastructure – and there was more customer outreach then you have given. Since you know what are add-on domains are are your servers – any customer with a third party registrar should have been alerted by e-mail that their DNS records would have had to be adjusted.
Further action should have been done to sites on the old servers that were still getting traffic. Since my site was receiving regular and steady traffic that should have been a tip off to action by your IT personnel that the customer should be alerted to adjust their DNS records. This is all a failing on your team of migration “experts” by things that could easy have been prevented by a little due diligence on their part.
The biggest coincidence is that over hte last week I’ve been in discussion with someone to move over to your service. I also had planned on migrating one of the non-profit organizations ove to your service next year to a VPS plan when their contract runs up. Currently in good faith I can’t advise this to either of those accounts. This disruption and loss of data is just not acceptable when it could have been easily prevented by checking what was going on the old server before you took it offline.
I am merely asking to have my backups from [REDACTED] restored to my account. Is this possible or do I have to make some other plans? I am also sending the Consumer Advocacy blog “The Consumerist” (consumerist.com) a copy of this e-mail. While the migration diligence seems to be in error you can still correct this problem and an unhappy customer by performing adequate customer service. One of your features of An Hosting that is touted on your site is ” Backup Storage -Nothing is more shocking and heartbreaking than losing ones data and hard work. We back up our servers nightly to ensure you always have ANHosting to depend on!” I am solely asking for my data to be restored as close to 4PM 1/16/09 as possible and my faith that I can “always have ANHosting to depend on!”
I sent this e-mail out to the American general manager of the company, the consumerist, the European Executives, and support. I received an e-mail from the general manager and he got a technician working on the issue right away. The issue was resolved in a couple hours. They pulled my data off the old server and synced it to the new one. There was still some minor issues that I had to correct – wordpress needed to be re-upgraded, but all my SQL databases were up to date. They also kept me informed during the whole migration.
I really wish I could have had that level of care taken of me on Sunday, or at least a time frame when the data could have been restored. When they try to equate a blog with a normal static website, it just shows ignorance on their part. It does show the problem I have with doing regular back-ups – which is in the architecture stages right now. The only thing I ask of my hosting provider is to live up to the promises they have given me. I don’t open tickets – really I’ve talked to them only about 3 times in the last year. The first being a simple problem I had while waiting for my account to activate, the second inquiring about the share SSL certs, and the third this issue. I’m glad even though I “yelled” across the Internet more then I should have, that they came through and everything is now working.