If you were reading my Twitter stream about 11 days ago you will have seen me tweet the following message.
That feeling when your
#AWS reserved instance ran out and you didn’t realise. 🙁
The story was I logged into AWS and noticed the reserved instance for my website had expired about a month and a half before. For those that don’t know, buying reserved instances reduces your compute costs because you are committing to some resource usage over an extended period of time. There are various combinations of time, up-front payment, instance type and region that save you different amounts of cash… Once I noticed it had expired I bought a new reserved instance and thought no more of it.
Fast forward to this weekend and I checked my “EC2 Reserved Instance Utilization Report”, to make sure my actual instance was using my reserved instance allocation and I was not getting overcharged. The default screen suggested that I wasn’t using my reserved instance, which left me kind-of vexed. I checked the settings and noticed the reserved instance had been bought at “Region” level, rather than for my “Availability Zone”, which is “EU (Ireland)”. I reset it to my availability zone and decided to investigate…
What I noticed was visiting the EC2 reports was flipping my region settings from “EU (Ireland)” to “Global”. When I next visited any EC2 screen my region was flipped from “Global” to “US East (N. Virginia)”.
I reported this to AWS and got a message back suggesting this was impossible. I recorded a video of it and sent that. I also reached out on Twitter and it seems I’m not the only person able to reproduce this bug. After several more interactions with support, it seemed this was the only “Global” service that was having this issue. When I switched to IAM or S3 and back to EC2 I remained in the proper region…
At this point I was feeling a little miffed that I had lost out on 9 days of reserved instance pricing, so I asked for the difference to be refunded. I accepted the 40+ days where I had no reserved instance were my fault… I also made some suggestions about reserved instance notifications and proper buy-the-same-again functionality, that would really help in this situation.
After a few minutes I got a little paranoid and though I ought to double-check that my reserved instance wasn’t being used, so I had a little play around with reporting and noticed the default report (a summary) was not showing it, but if I flipped to a report with finer granularity I could see the reserved instance was being used, which in itself seemed a little odd. I wrote back to support and asked them to ignore my request for the 9 day refund as the reserved instance was being used. As far as I was concerned, the situation was over now as I had not lost out on any money due to this interface bug.
A little while later I got a message from support confirming my reserved instance was indeed being used, so I had not lost out over those 9 days, but they also refunded the difference between the reserved instance pricing and the full price I paid for the month and a half where I had no reserved instance! Wow! I really didn’t expect that!
In financial terms, the refund is nothing to Amazon, or me for that matter, but that sort of gesture is fantastic from a customer service perspective. I instantly switched from being a little annoyed by the whole situation to wanting Amazon to marry my daughter (Heli). 🙂
Before last year’s Oracle OpenWorld I wrote a post called “Oracle : Tech Company or Service Company?“, where I discussed what I think Oracle need to do to compete with the other cloud providers. I can’t imagine Oracle resolving an incident like this so quickly, at the weekend, which resulted in giving me some money I probably didn’t deserve. I hope someone from Oracle reads this post and registers the fact this is what you are competing with!
Cheers
Tim…
I’ve found the same thing – you have to keep a close eye on your Reserved Instances, and I found early on that you need to check, double-check and triple-check your options when choosing an R.I. because if it doesn’t match your instance, it will simply sit there unused.
I’ve made a few mistakes over time but Amazon have always been quick to refund my losses due to my mistakes. They also relaxed their rules last year so that a RI is no longer tied to a particular availability zone; and you can even choose to buy a “convertible” RI which can be switched to a different instance type, which is nice:
https://aws.amazon.com/blogs/aws/ec2-reserved-instance-update-convertible-ris-and-regional-benefit/
It’s kind of annoying that RIs are not “linked” in any way to the instance they are covering – which means more flexibility I guess; but it means that they won’t warn you if your RI is not being used. S.O.P. is to run the RI report a day after buying or renewing an RI. It would be nice if you could say “this RI is intended for this instance” when purchasing it, and the site could then warn me if the RI is unsuitable, and when it’s soon to expire.