The Secret Behind Non-disruptive Cloud Infrastructure Upgrade

[Samsung Newsroom]

Samsung Account is a global account service that brings the Samsung universe together, from all Samsung services to online and offline stores. It handles large-scale traffic with security and reliability. As a core Samsung service, all tasks on Samsung Account, from general service deployments to cloud infrastructure upgrades, must be carried out without interruption to the service. This blog introduces the architecture designed for an Elastic Kubernetes Service upgrade and shares our experience with upgrading the cloud infrastructure without interruptions to the high-traffic Samsung Account service.

What is Samsung Account?

Samsung Account is an account service that brings together more than 60 services and applications in 256 countries with over 1.7 billion user accounts. It is used for Samsung Electronics services including Samsung Pay, SmartThings, and Samsung Health, as well as for authentication on various devices such as mobile, wearable, TV, PC, etc. Samsung Account helps deliver a secure and reliable customer experience with one account on a variety of contact points from online stores (such as samsung.com) and offline stores to our customer services.

Evolution of Current Samsung Account Architecture

As the number of user accounts and connected services has grown, the infrastructure and service of Samsung Account has also evolved. It switched to the AWS-based cloud for service stability and efficiency in 2019, and is currently servicing 4 regions: 3 global regions (EU, US, AP) and China.

Currently, Samsung Account consists of more than 70 microservices. In 2022, Samsung Account switched to the Kubernetes base in order to reliably support Microservices Architecture (MSA). Kubernetes is an open-source orchestration platform that supports the easy deployment, scaling, and management of containerized applications. In 2023, Samsung Account reinforced disaster recovery (DR) to be able to provide failover across global regions, and expanded the AP region to improve user experience.

In other words, Samsung Account has repeatedly evolved its infrastructure and services, and is currently running stably with traffic over 2.7 million requests per second (RPS) and over 200K DB transactions per second (TPS).

Each AWS-based Samsung Account region, with its own virtual private cloud, (VPC) is accessible through user devices, server-to-server, or the web. In particular, the web access provides a variety of features such as samsung.com and TV QR login on AWS CloudFront, a Content Delivery Network (CDN).

Samsung Account microservices are being serviced on containers within Elastic Kubernetes Service (EKS) clusters, and internal communication between regions uses VPC peering.

Samsung Account is using several managed services from AWS to deliver various features. It is using Aurora, DynamoDB, and Managed Streaming for Apache Kafka (MSK) as storage to build data sync between regions, and it provides account services based on different managed services including ElastiCache, Pinpoint, and Simple Queue Service (SQS).

Let's elaborate on the AWS Managed Services that Samsung Account uses. The first is EKS, which is a Kubernetes service for running over 70 microservices on MSA. Next, Aurora is used to save and query data as an RDB and DynamoDB does the same but as a NoSQL database. Along with them, ElastiCache (Redis OSS) is used to manage cache and sessions and MSK handles delivering events from integrated services and data sync. If you’re building an AWS-based service yourself, you would probably use these managed services as well.

Frustrating Upgrades Contrasting the Convenience of Managed Services

There is a major challenge to consider when you use these managed services, though. End of support comes, on average, after 1.5 years for EKS and 2 years for Aurora. Various other services like ElastiCache and MSK face the same problem. Such service support termination is natural for AWS, but upgrading these services when support ceases is often a painful task for those running them. Because operation resources are often reduced upon switching to the cloud, large-scale upgrades that come around every 1 or 2 years have to be performed without enough resources for emergency response.

These managed service upgrades put a major burden on Samsung Account. More than 60 integrated services have to be upgraded without causing interruptions, and the upgrades must be rolled out across a total of 4 regions. On top of that, Samsung Account is developing and running more than 70 microservices, so a significant amount of support and cooperation from development teams is required. The most challenging of all is that the upgrades need to be performed while dealing with traffic of over 2.7M RPS and DB traffic of 200K TPS.

EKS Upgrade Sequence and Restrictions

You might think upgrading EKS on AWS is easy. In general, when upgrading EKS, you start with the control plane including etcd and the APIs that manage EKS. Afterwards, you move to the data plane where the actual service pods are on, and finally to EKS add-ons. In theory, it is possible to upgrade EKS following this sequence without any impact to the service operation.

However, there are restrictions to general EKS upgrades. If an upgrade fails in any of the 3 steps above due to missing EKS API specs or incompatibility issues, a rollback is not available at all. In addition, it is difficult to do a compatibility check for the services and add-ons in advance.

Multi-cluster Architecture for Non-disruptive EKS Upgrades

After much thought, Samsung Account decided to go with a simple but reliable option to perform EKS upgrades. It's possible that many other services are using a similar way to upgrade EKS or run actual services.

Samsung Account chose to upgrade EKS based on a multi-cluster architecture with 2 EKS clusters. The architecture is built to enable an existing EKS version to continue providing the service, while a new EKS version on a separate cluster performs a compatibility validation with various microservices and add-ons before receiving traffic.

The advantage of this method is that you can implement a rollback plan where the old EKS version takes over the traffic if any issues occur when switching to the new EKS version. A lesson we have learned from providing the Samsung Account service under high traffic is that there will be issues when you actually start processing traffic, no matter how perfectly you've built your infrastructure or service. For these reasons, it is essential to have a rollback plan in place whenever you deploy a service or upgrade your infrastructure.

When you perform a multi-cluster upgrade, traffic must be switched between the old and new EKS clusters. Simply put, there are 2 main approaches. One approach is to switch traffic by placing a proxy server between the 2 clusters. The other approach is to switch the target IP using DNS. Needless to say, there may be a variety of other ways to accomplish this.

In the first option, using a proxy server, you may encounter overload issues when handling high-volume traffic, such as with Samsung Account. Additionally, there are too many Application Load Balancers (ALBs) used for approximately 70 microservices, making it impractical to create a proxy server for each ALB.

In the second option, using DNS, the actual user, client, and server replace the service IP of the old EKS with that of the new EKS during a DNS lookup, redirecting requests to a different target at the user level. The DNS option does not require a proxy server, and switching traffic is easy by simply editing the DNS record. However, there is a risk that the traffic switch might not happen immediately due to propagation-related delays with DNS.

The DNS-based traffic switch architecture was applied to achieve a non-disruptive EKS upgrade for Samsung Account.

Let us describe the DNS layers of Samsung Account with a hypothetical example. The top domain is account.samsung.com, and there are 3 global region domains under it, classified based on latency or geolocation. For us.account.samsung.com, the layers are split into service.us-old-eks.a.s.com and service.us-new-eks.a.s.com, representing the old and new domains. This is a simple, hypothetical example. In reality, Samsung Account uses more DNS layers. During the recent EKS upgrade, we switched traffic between the internal domains of the 2 EKS clusters based on weighted records while adjusting the ratio, rather than switching all at once. For instance, when a user sends a request to account.samsung.com, it goes through us.account.samsung.com, and the actual EKS service IP is applied at the end based on the specified weight.

Retrospective of the Non-disruptive EKS Upgrade

In summary, I would say "it's a successful upgrade if the connected services haven't noticed." With this EKS upgrade, we deployed and switched traffic for a total of 3 regions, 6 EKS clusters, and more than 210 microservices over the course of one month. The traffic switch was conducted with ratios set based on each service's load and characteristics, and no issues with connected services were reported during this one month EKS upgrade.

Of course, as they say, "it's not over until it's over." We did have a minor incident where there were insufficient internal IPs in the internal subnet due to many EKS nodes and service pods becoming active simultaneously, which scared us for a moment. We secured the IP resources by reducing the number of pods for Kubelet and add-ons by about a thousand and quickly scaling up the EKS nodes. One thing we realized while switching traffic with DNS is that 99.9% of the entire traffic can be switched within 5 minutes when the DNS weight is adjusted.

Closing Note

Richard Branson, co-founder of Virgin Group, once said, "You don't learn to walk by following rules. You learn by doing, and by falling over." Samsung Account has been growing and evolving, addressing many bumps along the way. We continue to resolve various challenges with the stability of our service as the priority, keeping this "learning while falling over" spirit in mind. Thank you.

View the full blog at its source