Following on from our earlier communication, we would like to share our full recovery plan.
Please note that this plan will require some planned and scheduled downtime.
Following last Tuesday's service outage, we took steps to rapidly implement a long term solution, unfortunately, those measures led to the short unexpected outage which took place this morning.
We have reviewed the impact, and as a result we will need to implement a more staggered approach to manage a phased change.
Recovery Plan Overview:
Phase 1:
Date: Tuesday, 1st October
Action: We plan to restore RICOH Spaces services to the state they were in as of Friday, 27th September, to remove changes that were applied this morning.
Impact: There will be planned downtime across all RICOH Spaces services for approximately 30 minutes.
Time: Downtime will begin at 4 PM GMT, which was chosen to minimise disruption across various time zones while also allowing us to monitor the reversion under sufficient traffic load into the early evening.
Phase 2:
Date: Wednesday, 2nd October
Action: We will implement VPC platform changes into the Next (pre-prod) environment. These will be tested and monitored for 24 hours. The aim of these changes are to provide a long term resolution for the incident that occurred on 24th Sept.
Impact: This phase will have no effect on the production platform and therefore should not affect any customers using the live environment.
Phase 3:
Date: Thursday, 3rd October
Action: Once we are confident we have had sufficient time to monitor the VPC platform changes, we will progress these changes to the Production environment.
Impact: This will result in planned downtime on all RICOH Spaces services, estimated to take place from 4 PM GMT for approximately 2 hours. Our team will be on standby into the evening to ensure full services are resumed and that any application disruptions or disconnections are resolved overnight.
Additional Information:
We understand that this plan may not be ideal for some of our users across various regions, but we believe it will have minimal impact on the majority of users. It is important that we monitor these changes against real-time load to ensure these issues do not arise again in the future.
As of now, all feature and bug releases are on hold until the above changes are complete.
We truly appreciate your patience and understanding during this process.
Thank you for your continued support.