As most of the Roblox community knows, we recently experienced an extended downtime on our platform. We’re sorry it took so long to restore service. An important value at Roblox is “Respect for the community” and in this case, we apologize for the inconvenience to our community.
On the afternoon of Thursday, October 28, users started experiencing problems connecting to our platform. This immediately became our highest priority. Teams have begun working around the clock to pinpoint the source of the problem and get things back to normal.
This is a particularly difficult incident because it involves a combination of factors. A core system in our infrastructure became overloaded, fueled by a small error in our backend service communication under heavy load. This is not due to any peak in external traffic or any particular experience. Instead, the failure was due to an increase in the number of servers in our data center. As a result, most of the services at Roblox cannot communicate and deploy effectively.
Due to the difficulty of diagnosing the actual error, the recovery process took longer than any of us would have liked. After successfully identifying this root cause, we were able to resolve the issue through performance tuning, reconfiguration, and scaling back some of the loads. We were able to fully restore service as of this afternoon.
We will release an autopsy with more details once we complete our analysis, along with actions we will take to avoid such problems in the future. Additionally, we will be implementing a policy to make our community of creators economically inclusive as a result of this shutdown. More details on this to come. As part of our value “Respect to the community”, we will continue to be transparent during the autopsy.
To the best of our knowledge, there is no loss of player endurance dataand your Roblox experience should now be completely back to normal. You can always contact us support team if you have any problems using Roblox now or in the future.
We appreciate the patience and support of our players, developers, and partners during this time.
