Background:
RICOH Spaces v2 has been experiencing some intermittent degradation in performance for some customers in specific areas of the application.
The platform consistently operates within agreed thresholds for the majority of the time. Nevertheless, we are aware that there are occasional bottlenecks that occur during peak times and in specific areas of the platform. As a result, a brief loading message may appear for a few seconds while data is being retrieved, such as "Loading the floorplan."
To create a global, multi-regional and highly scalable solution RICOH Spaces underwent a significant platform and re-architecture change to deliver v2. During the project delivery, the platform underwent load testing within the API architecture and the results were all positive. To further mitigate any risk customer migrations were managed in a phased approach and no issues with performance were reported during the first three months of production usage, even after 50% of our customers were migrated. After the last phase of migrations, in April, the DevOps team captured some degradation in performance which also started to be reported by some customers.
Devices affected:
Web
Mobile
Resolution:
Following detailed analysis, it has been identified that enhancements are required across the architecture to fine-tune the application and platform interaction. A change plan has been created to address this.
We have several changes to implement, and due to their impact, they need to be applied over a period of time, this allows for monitoring and review between each change. This program of change has already started and includes;
Application integration enhancements
Implementation of Redis, an industry-leading memory cache solution
Cloud Nat enhancements to support the distribution of services
GCP Serverless VPC Connector fine-tuning for load management
Pre-calculated Data loading process changes
Timeline:
The first set of changes was applied in the April release, and the second set of changes (which included the first phase of Redis) was applied in the mid-May release on 15th May.
Following the changes in April and May we have seen some significant improvements in performance with latency reduced, some requests are now 3 x faster, even during peak periods. We are continuing to monitor our performance against the 95th percentile of requests.
We have changes planned over the next 3 weeks after which we will review the level of service in relation to performance to determine whether any further action is required. We fully expect all bottlenecks to be resolved following the programme of changes.
Next steps:
Further updates will be provided during the next 3 weeks.