Disclaimer : This lab is available to learn the MASC and the lab is a shared resource used by many people. DON'T ADD,CHANGE or DELETE the configuration. Please use the lab responsibly.
About MASC - Multi-cloud Application Security with Tanzu Service Mesh, Avi AKO/GSLB and Antrea-NSX Lab
Introduction to Multi-cloud Application Security Lab running on multiple cloud providers including Tanzu Kubernetes Grid for on-premise SDDC to Public clouds (EKS,AKS,OKE,GKE). In this lab, you can hands-on Tanzu Service Mesh, Avi AKO/GSLB and Antrea-NSX security features by yourself running on multiple Kubernetes clusters on different Public cloud operations. You will experience how we can easily solve North-South and East-West Security challenges on customer multi-cloud environment and more.
Tanzu Service Mesh (TSM) provides end-to-end connectivity, resiliency, security, and insights for modern applications running in single and multi-cloud environments. Tanzu Service Mesh is a leader in service mesh innovation – providing policy control and visibility across end-to-end communications from application end-users, to services and APIs, and data – enabling compliance with service level objectives (SLOs) and data protection and privacy regulations.
Tanzu Service Mesh (TSM) provides strong, multi-cloud, easy-to-operationalize network defenses that secure application traffic within and across clouds. TSM makes it easier for you to enable Zero Trust application access across multi-cloud environments—so you can secure traffic across applications and individual workloads with security controls that are consistent, automated, attached to the workload, and elastic in scale.
This lab is intended for intermediate to advanced-level users exploring VMware Tanzu Service, Antrea CNI and Avi AKO/GSLB use cases, helping you to explore security concepts and plan with TSM.
How Tanzu Service Mesh works
This lab will use a scenario involving TSM which provides advanced, end-to-end connectivity, security, and insights for modern applications — across application end-users, microservices, APIs, and data. TSM provides service mesh capabilities across single and multiple clusters, clouds, and data centers. Teams can build application resiliency and data security policies and bake security testing into their existing DevOps toolchain, ensuring application high availability, performance and secure transactions. In addition to rich application performance metrics and security visibility, it offers application and data-level security policies — for example, attribute-based access control (ABAC) policies, end-to-end encryption policies, and API segmentation, parameter validation and threat protection policies.
Enterprise-grade, integrated load balancing, ingress and container networking. With TSM, platform and IT operators can easily implement container ingress services, including L4-L7 local and global server load balancing (GSLB), web application firewall (WAF), DNS, and IPAM in a single platform across any cloud. In addition, it has built-in container networking leveraging Antrea, making it easy for platform operators to apply and change network policies to run applications without the risk of disruptions and with guaranteed security enforcement into each cluster. Application moder
Let's now see how Tanzu Service Mesh (TSM) and Modern App Security (Antrea/Avi AKO/GSLB) Lab Guide can help prevent and protect against these attacks.
Section 1: Before You Begin
To login to the environment, perform the following steps:
- First, open a web browser of your choice (Incognito recommended) and navigate to vmtestdrive.com. Select LOG IN.
If you do not already have an account please reference the instructions found here.
Note: we only support customer sign up with their corporate email - do not to use personal email like Gmail (if doing so, no email activation will be sending out). - If you are signing in for the first time and don’t have a TestDrive account, click GET STARTED and follow the instructions for creating your TestDrive portal account.
- Enter your TestDrive Username and Password and select ENTER.
- Locate the Multi-Cloud Security product under the Intrinsic Security tab and click Launch. Make sure that you open Multi-cloud Application Security Lab and refer it on a separate tab.
In case of long idle or got disconnected, please log-out from the upper-right corner and re-login to Launch a new Horizon desktop or switch to Incognito browser instead of Chrome/Firefox.
- A new tab will open with Workspace ONE. Enter your TestDrive Username and Password, then click Sign in.
Note: Please provide the short username (not your email ID) and password to login.
- Go to Apps tab.
- Search for the NSXSM desktop and launch it.
- Now you'll be on the Multi-Cloud Security desktop. At this point you can begin the walkthrough steps listed in the next section.
Section 2: Modern App Security with Tanzu Service Mesh (TSM)
In this section, we will walk you through the core component of how easily TSM can secure the communications between services and the automation framework (APIs). TSM plays a vital role in distributed modern applications for their service communication. Ops teams need to be able to monitor and remediate security threats automatically. Existing workloads should easily be protected without interruption. In this section, you will learn:
-The key security concerns and challenges in a modern apps, multi-cloud environment
-How VMware Service Mesh can address the modern app security challenges by securing communications as well as services and data while it’s being processed.
The technologies covered by our VMware Service Mesh TSM include end-to-end encryption, mTLS, API security, runtime protection, container isolation, how to monitor and remediate to security threats.
Global namespace, a unique concept in Tanzu Service Mesh, defines an application boundary. A global namespace connects the resources and workloads that make up the application into one virtual unit to provide consistent traffic routing, connectivity, resiliency, and security for applications across multiple clusters and clouds
Let's start reviewing the ACME Global Namespace in the TSM console.
1. Click on Home -> Under GNS Overview ,
- Click on three dots and select Edit "Global Namespace"
2. Review the ‘GNS Name’ and Domain name for GNS. Domain provides automatic service discovery and manages service identities within that Global namespace. Click on Next
3. Observe Namespace Mapping Rules - It defines the services included in Global namespace.
- prod-tanzu-tkg-dc01 is the first On-Prem Kubernetes cluster and Name space mapped is acme-app. The front-end of the application with shopping micro service is deployed on this cluster.
- prod-tanzu-tkg-dc02 is the second On-Prem Kubernetes cluster and Name space mapped is acme-app.The backend application Database with catalog micro service is deployed on this cluster
4. Click on Next to navigate to Auto Discovery.
- API Discovery - Tanzu Service Mesh API discovery is the capability that allows auto-discovery of APIs signatures between microservices running inside or outside the mesh
- PII Data Discovery - Tanzu Service Mesh learns the APIs with PII between the microservices deployed in the application environment.
5. Public Services It is a way to expose a service outside its Global Namespace to enable external users to access the services.
- Service Name It is the service we want to expose it to the external users for accessing the application
- Service Port Port number where application service uses for connection
- Public URLs Url available for external users to make a request to the application
6. GSLB & Resiliency Defines Global Load Balancing Scheme, Health checks & High availability for public services in the GNS.
1. Click on Home. Under GNS Overview click on acmegns
2. Observe the properties of Global Namespace like health status and security type. After that scroll down to GNS Topology
3. TSM generates a topology graph dynamically by observing the traffic flow between services in the GNS. The topology shows three key metrics of services.
- The Services incoming requests per second(rps)
- The Error rate, that is percentage of failed requests to the service
- The 99th percentile latency of requests processed by the service.
Prod-tanzu-tkg-dc01 On-Prem Kubernetes cluster with frontend services of ACME Shopping Application.
Prod-tanzu-tkg-dc02 - On-Prem Kubernetes cluster with backend catalog services of ACME Shopping Application.
The connection between the shopping service in cluster01(Prod-tanzu-tkg-dc01) and the catalog service (Prod-tanzu-tkg-dc02) shows the traffic flows between them.
A service group is a collection of services in the Global Namespace. We can observe the aggregated metrics for the services in the group or consistently enforce policies across the service group.
Service groups serves two main purposes:
- To monitor relevant metrics such as ‘requests per second’, ‘latency’ & ‘error rate’ across the services in the group.
- We can also define & apply consistent Access control polices to the entire service group.
- Click on Security under GNS Topology.
- An access control policy is applied between shopping service group and catalog service group.
2. Click on frontend-catalog to review the policy.
- Shopping service added to the frontend group and Catalog service added to the catalog group.
- Click on edit policy to view the detailed Access Control Policy configuration
3. Review the Access Rules of Access Control Policy between the service groups.
- Policy Name Name of the policy for Access rule.
- GNS Scope Global namespace scope, under which service groups are created
- Source and Destination Services Drop down will list all the services groups created under specified GNS scope, Service group can be selected as per the requirement.
- Policy Intent Define the type (Allow/Deny) of traffic using drop down All Traffic, Specific TCP Connections, Specific HTTP Requests. Click on cancel after reviewing.
Note: For creating service group navigate to Inventory -> Service Groups
1. Click on APIs, select shopping service and open it to view all the APIs.
2. Observe the shopping service topology under service dependencies.
- Service Topology depicts the traffic flow between the Shopping service to all other connected services via APIs.
- Under APIs & Connections, All the incoming API methods and Path are listed.
Click on any API(/products). All the API data like Schema and logs are available
3. You can use the topology to understand about all the APIs involved and to determine the key security events.
- API Overview Contains Key security events, Top errors, latency information.
- API Schema consists of Request information, API Response code and JSON data of Schema.
- All the API traffic logs data is recorded between source and destination services.
Click on back button twice to navigate GNS Topology under acmegns namespace
Tanzu Service Mesh discovers the components of the application, APIs, PII data and application users. It then learns the behavior of application and creates a baseline of normal behavior.
1. Click on PIIs, we can see GNS topology with no PII data detected. Let’s place an order in ACME shopping application to visualize how PII data is automatically captured and presented within the topology.
2. Open the ACME fitness shopping application in chrome.
- Open google chrome and access the application and place the order of an item of your choice.
- http://shopping.vmware-demo.net
- Select any item from catalog and add to cart
- Click on Item s IN CART on top right corner of the window.
- In shopping cart, click on Proceed to Checkout.
- Under Checkout-Address, Provide the required information as shown below and click on Continue to Delivery Method
- Continue to Payment Method and populate the payment information as shown below and click on continue to order review
- Card Type - Mastercard
- Credit Card Number - 5105105105105100
- CCV - 123
- Expiration Month - 01
- Expiration Year -2023
- Under order review, proceed to place an order as a final step to complete the order placement
3. Navigate to TSM console to review PII data under GNS Topology.
- In the GNS topology the data flow between the services is highlighted
- The PII data traffic in each microservice from beginning to the end is captured.
- How the application handles sensitive PII data, which services are involved in total transaction is shown.
- Click on Attacks under GNS Topology.
2. Click on shopping service to view the detailed overview of security events and APIs involved in the attack.
- Under service dependencies select Attacks.
- Under Incoming APIs click on /Products API to navigate to security events.
- Under API Overview security analytics provides total security events with PII and Attacks detected data.
- Scroll down to security events. Observe the Timestamp, severity, Event title, Destination Service and Attack information for each attack.
- Post reviewing the events, Click on Home.
3. API security policies are to secure and segment the applications against API and deep payload layer threats. TSM’s Global Namespace construct, to offer discovery, detection and behavioral security and observability capabilities across multi-cloud environments.
- Under Home Expand Policies > Click on API security under Policies
- In API Security, under acmegns, Click on three dots next to shopping and select Edit Configuration
- API security policies offers to enforce consistent policies with Access control, PII Data Security, Attack defense and schema validation.
- Observe the type of actions allowed for each security policy.
2.7 TSM Security Threat Remediation (tbd)
Tanzu Service Mesh helps teams overcome the performance and security visibility gaps resulting from distributed microservices architectures and adoption of multiple platforms and clouds. Operations teams have access to rich troubleshooting tools, including multi-cloud topology maps and traffic flows, performance and health metrics, and application-to-infrastructure correlation.
Section 3: Security with NSX Advanced Load Balancer (Avi)
SQL injection attacks of data-driven web apps, also simply called SQLi attacks, have been a serious problem of late. A SQLi attack happens when an attacker exploits a vulnerability in the web app’s SQL implementation by submitting a malicious SQL statement via a fillable field. In other words, the attacker will add code to a field to dump or alter data or access the backend. A successful malicious SQL statement could give an attacker administrator access to a database, allowing them to select data such as employee ID/password combinations or customer records, and delete, modify, or data dump anything in the database they choose. The right SQL injection attack can allow access to a hosting machine’s operating system and other network resources, depending on the nature of the SQL database.
For this demo, the real question is, how do we prevent it to the best of our ability?
First, we must understand how to prevent SQL Injection Attacks.
The Open Web Application Security Project OWASP provides an overview of how to avoid SQL injection attacks. While there are various SQL injection attack tools on the market, there is no substitute for implementing best practices for preventing these attacks. Here are some of the OWASP top strategies for preventing SQLi attacks.
- Prepared statements/parameterization
- Stored procedures
- Input validation
- Escape user-supplied input
- Limit privileges
- Update and patch regularly
- Web application firewall (WAF): is an important part of a larger security solution that detects SQLi along with other threats. WAFs typically do this in part by relying on detailed lists of signatures that are constantly updated, so they can surgically excise threats, including malicious SQL queries.
So, let us focus on WAF and how Avi WAF can help.
AviWAFprotects web applications from OWASP Top 10 threats such as SQL Injection Attacks and Cross-site Scripting (XSS) and other common security vulnerabilities while offering customizable rule sets for each application.
The architectural advantages of the Avi platform power Avi’s WAF, gaining real-time application security insights thanks to the platform’s strategic location in theapplication trafficpath. This architectural advantage and the platform’smulti-cloudcapabilities extend to WAF network security
Majority of the SQLi exploits are successful when CRS rules and signatures are not updated.
The biggest advantage of Avi WAF is that it can subscribe to the latest updates on WAF signatures and CRS rules.
Now let us see how WAF protects the application from an SQLi attack
https://avinetworks.com/docs/21.1/waf-monitoring/
Access to Avi Controller
The console is accessed through a supported supported web browser Chrome. Login to Avi Controller:
- Click on Avi-Controller Auto Logon shortcut on the Desktop. Shortcut will open the URL to NSX: https://dal-avictrl01.nsx-sdfw.local/
- Username/Password: Sign in is automated, so please don't touch the keyboard/mouse during 15s of process. If you do so, please close all Chrome windows and reopen the link again.
Avi WAF protects web applications from OWASP Top 10 threats such as SQL Injection Attacks and Cross-site Scripting (XSS) and other common security vulnerabilities while offering customization rule sets for each application.
- Click on cloud services as an admin to subscribe to the latest updates on WAF signatures and CRS rules
- Click on Edit
- Click to enable the updates
- Click Save
- Let us now change the tenant to "Demo"
- Click on the Demo tenant
- Next up, we will see how Avi WAF automatically protects the application from SQL Injection threats.
- For this demo, we will consider the WAF_Demo_VS virtual service. The shield icon around the virtual service indicates the application is protected by the Avi WAF. Let us click on it to application details and other details including security logs and WAF policies.
- Notice the application details including the end to end timings. It is a great way to understand the application behavior and instantly detect if the application behavior is sub-optimal or if it not responsive. For this demo, let us look at the logs.
- Click on logs
- Notice certain requests are rejected by the WAF. One particular request that is of interest to us for this demo is the SQL Injection.
- We notice the request ended abnormally because of a WAF policy match. To learn more abou the response code 4xx, let us scroll down a bit more.
- Notice the WAF is suggesting the match with a signature called CRS_942_Application_Attack_SQLi
- Notice the CRS rule 942100
- We can further learn about this signature by editing the signatures settings in the WAF policy section.
- Click on signatures
- Click to expand
- Click to show the rule
- Notice the CRS version CRS-2021-4 which is the latest release from OWASP that is protecting the application. We can scroll down further to notice the exact policy match and understand why the request was rejected in the first place.
- Notice the exact reason for the request being rejected. Avi WAF proactively provided application security by syncing with the latest releases from OWASP.
- We can confirm the latest AVI release including the Avi CRS which is the default signature based protection for Avi WAF is consistent and up to date with the OWASP releases.
- Finally we can confirm the same from the OWASP website as well.
Section 4: Secure Modern Apps with Antrea CNI and NSX
Introduction
NSX becomes the single pane of glass for policy management when connected to Antrea clusters. The Antrea clusters could be running on VMware Tanzu platform, RedHat OpenShift or any upstream Kubernetes cluster. Inventory management, tagging, dynamic grouping and troubleshooting can be extended to Antrea clusters along with native Kubernetes network policies and Antrea network policies to be centrally managed by NSX.
Antrea NSX Adapter is a new component introduced to the standard Antrea cluster to make the integration possible. This component communicates with K8s API and Antrea Controller and connects to the NSX APIs. When a NSX admin defines a new policy via NSX APIs or UI, the policies are replicated to all the clusters as applicable. These policies will be received by the adapter which in turn will create appropriate CRDs using K8s APIs. The Antrea Controller which is watching these policies run the relevant computation and sends the results to the individual Antrea Agents for enforcement. As these policies are run, statistics are reported back to the Antrea Controller for aggregation. Because the cluster is integrated with NSX, these aggregated values are reported to NSX adapter, which in turn reports back to NSX and made available to the admin.
Access NSX 3.2 Manager.
The console is accessed through a supported supported web browser Chrome. Login to NSX Manager:
- Click on NSX-Mgr AutoLogon shortcut on the Desktop. Shortcut will open the URL to NSX: https://ns-nsxmgr01.nsx-sdfw.local/
- Username/Password: Signin is automated, so please don't touch the keyboard/mouse during 15s of process. If you do so, please close all Chrome windows and reopen the link again.
- In case you prefer using manual login, please click on NSX icon inside Lab Access folder and enter your Testdrive credential@vmtestdrive.com / password.
Multi-cloud Networking with Antrea Demo Steps
In this demo we will demonstrate to manage network policies across multiple K8s cluster and NSX also provides central policy management, visibility, and network troubleshooting for these multi-cloud K8s clusters.
We have two K8s clusters onprem and aws all are running on Antrea CNI and clusters are registered with NSX .
- Click on Inventory Overview and Validate the Containers section.
- Click on Containers and select clusters , Review the K8s-Cluster01 & 02.
- Click on Namespaces to see all the deployed namespaces.
- Click on Security, Under Policy Management Select Distributed Firewall.
- The DFW rules are applied to two container clusters deployed on prem and aws cloud.
- These are application DFW rules configured as per the application services.
- Select the rule frontend to appsvc and click on Action select Drop
- Access the acme fitness application from URL http://shopping.vmware-demo.local
- You can observe the application is accessible but all the catalog items are not available in the website
- Navigate to NSX manger
- Select Plan & Troubleshoot
- Click on Traceflow , Select Antrea Traceflow
- Fill in the below details to start Traceflow and Click on Trace
Cluster - k8s-cluster01
Protocol Type - TCP
Source Port - 3000 Destination Port - 8082
Source Pod - shopping Destination Pod - cart
14. Observe the Traffic dropped by Cluster Network Policy.
15. Click on Security, Under Policy Management Select Distributed Firewall.
16. Select the rule frontend to appsvc and click on Action select Allow
17. You can observe the application accessibility is restored completely.
Antrea with NSX provides Multi cloud , multi-cluster Kubernetes cluster central policy management, visibility, and network troubleshooting
1. Click on Inventory Overview from the NSX Manage and Navigate to Container Section. Click on Clusters as shown in the below image.
2. Observe the k8s-cluster information from the NSX UI. Expand the K8s-Cluster5 and click on each section "Nodes", "Pods", Kubernetes Services and view the detailed information.
3. Click on Namespaces and scroll down to find the Namespace "yelb" running on cluster k8s-cluster5. Click on pods to see the running pods in Yelb namespace.
We have familiarized with the Containers inventory section. We can get the detailed information about the K8s clusters, Pods and Kubernetes services running in the cluster.
You can create Antrea groups when the NSX has one or more Antrea container clusters registered to it. An Antrea can include static IP address, membership criteria, or both. IP address can be Pod or Service IP addresses.
Currently supported member types for creating Antrea group are Name space , Service & Pod.
1. In Inventory section, Click on Groups . List of existing groups are shown under groups section. Check the groups starting with name yelb. For demo we will edit one group called yelb_frontend to view the membership criteria. Click on 3 dots next to yelb_frontend and click on edit.
2. Click on criteria to view the Antrea Group Membership Criteria. Click on Cancel
You can create Distributed Firewall policies (security policies) in NSX and apply them to registered Antrea container clusters to secure traffic between Pods within a container cluster.
An NSX security policy can be applied to multiple Antrea container clusters. However, the policy can secure traffic between Pods within a single Antrea container cluster. The following traffic is not protected:
- Pod-to-Pod traffic between Antrea container clusters.
- Traffic between Pods in an Antrea container cluster and VMs on hosts in the NSX environment.
1. Click on Security and Distributed Firewall under Policy Management. DFW rules has been configured under the Yelb_App Policy. These rules establish the connectivity between all the pods that makes the yelb application.
2. Access the yelb application. Open a new tab in the chrome browser and type the url http://yelb-ui.vmware-demo.local/ to view the application. You can interact with application by clicking on vote button.
4.5 Network Troubleshooting with NSX Antrea Traceflow
NSX can be leveraged to trace the packet flow between two Pods. NSX-T can perform a packet trace between two Pods based on source and destination Namespaces, Nodes or Pods by leveraging Antrea Traceflow.Once the trace is done, each hop the packet takes is visible along with any Network Policy it hits along the way. This provides invaluable insight into troubleshooting and debugging applications inside an Antrea cluster
1. Select the DFW Rule frontend to app form the policy Yelb_App and change the action from Allow to Reject. Click on publish.
2. Access the yelb application. Open a new tab in the chrome browser and type the url http://yelb-ui.vmware-demo.local/ to view the application. You can observe the Vote button doesn't work anymore indicating that traffic between frontend to application is being blocked.
3. We will do a Antrea Traceflow to troubleshoot the packet drop issue. Click on Plan & Troubleshoot , Select Traffic Analysis under Troubleshooting tools and Select Antrea Traceflow. Update the below details to run the trace.
Cluster - K8s-cluster5
Protocol Type - TCP
Destination Port - 4567
Source Pod - yelb-ui
Destination Pod - yelb-appserver
Click on Trace.
4. Traceflow results are shown as Traffic is Dropped by Cluster Network Policy with Network Policy ID for further analysis. Click on EgressMetric to see the policy ID
5. Update the previously changed DFW rule to Allow. Click on Security , Distributed Firewall under Policy Management. Select the rule frontend to app and click on action change it to Allow. Click on Publish.
6. Once the DFW rule is updated the Yelb application works fine.
Summary
Tanzu Service Mesh is an enterprise-class service mesh that helps solve the challenges associated with deploying a distributed microservices application by providing service-mesh functions across multiple clusters and clouds.
Tanzu Service Mesh provides service mesh capabilities for resources in a distributed application by arranging these objects in a logical group called global namespace. A global namespace is not tied to a single cluster and connects resources between two or more clusters. Each global namespace manages service discovery, observability, encryption, policies, and service-level objectives (SLOs) for its objects regardless of where they reside - in multiple clusters, sites, or clouds.
By abstracting the service mesh from the physical boundaries of a single Kubernetes cluster and a single cloud, and by extending the scope from service-to-service communication to users-to-service-to data communication, Tanzu Service Mesh is able to control, secure, and operate applications, no matter where their components are deployed.
Additional Resources
VMware Security on Tech Zone and Hands-on Lab
- Pitch: Light-board NSX and Container Networking for Tanzu user - https://www.youtube.com/watch?v=HxwmBV3eGfE
- Storytelling: Modern App high level story - Combat Service Latency with Tanzu Service Mesh and Load Balancing - https://www.youtube.com/watch?v=2RmBfMWJ9E8
- Storytelling Demo 1 - Modern App connectivity demo with TSM- Avi GSLB integration - https://www.youtube.com/watch?v=LPLuVDaARGs
- Storytelling Demo 2 - NSX and Antrea integration - https://www.youtube.com/watch?v=barGggRV3vA
- Storytelling Demo 3 - Antrea - NSX NCP and NSX-Antrea when and why - https://www.youtube.com/watch?v=7Xpq-GVp2Qo
- Try: TSM Tanzu Service Mesh Hands-on Lab: http://docs.hol.vmware.com/hol-isim/hol-2021/hol-isim-player.htm?isim=HOL-2132-92-ISM_SM.json
- Try: TKO Tanzu Kubernetes Operation Hands-on Lab: https://pathfinder.vmware.com/v3/path/tko_path/section/step1/activity/tanzu_kubernetes_operation
- Blog NSX-Antrea: Connect and Secure your Apps with Antrea and VMware NSX 3.2 - Network and Security Virtualization - VMware
- Learn more: VMware Modern App Connectivity Solution- https://www.vmware.com/products/modern-app-connectivity.html
Contact us: [email protected]