Protect Ingress Traffic with AWS Guard Duty and Aviatrix IPS Security Appliance

AWS VPC Ingress Routing allows customers to insert (or service chain) a security appliance/gateway or firewall for the traffic flows coming from the Internet and going towards the public-facing applications such as a web server. With Amazon VPC Ingress Routing, customers can define routing rules at the Internet Gateway (IGW) to redirect ingress traffic to third-party appliances, before it reaches the final destination.

Aviatrix takes full advantage of the Amazon VPC Ingress Routing Enhancement by combining it with

  • Aviatrix Security Gateway’s policy-based FQDN Filtering capabilities and
  • Amazon GuardDuty’s continuous threat intelligence feed

What is AWS GuardDuty?

AWS GuardDuty is a threat and intrusion detection (IDS) service but it does not provide intrusion prevention (IPS) capabilities. The Aviatrix Controller programs an inline Aviatrix Security Gateway (called Public Subnet Filtering Gateway) to dynamically filter traffic to prevent unauthorized and malicious traffic.

You can find more information on this topic here

LAB6 – GCP Remote User VPN / Client VPN

Business Objective

An important security requirement for GCP VPCs is to effectively control remote user access in a policy based manner. The cloud and the COVID-19 pandemic makes most users “remote.” Not only for employees who are out of the office, the “remote” label can be applied to developers, contractors, and partners whether they’re in the office or around the globe.

Note: User VPN, Client VPN or OpenVPN are interchangeable terms.

Remote User VPN / Client VPN Overview

While a bastion host using an SSH tunnel is an easy way to encrypt network traffic and provide direct access, most companies looking for more robust networking will want to invest in a SMAL based VPN solution. Because …

  • Single instance VPN servers in each VPC results in tedious certificate management
    • No centralized enforcement give rise to questions like “who can access what VPC?”
    • With more than dozen users and more than a few VPCs, management and auditing of the user access can become a major challenge

What’s needed is an easily managed, secure, cost-effective solution. Aviatrix provides a cloud-native and feature-rich client VPN solution.

  • The solution is based on OpenVPN® and is compatible with all OpenVPN® clients
  • In addition, Aviatrix provides its own client that supports SAML authentication directly from the client
  • Each VPN user can be assigned to a profile with access privileges – down to hosts, protocols and ports
  • Any Identity provider auth for LDAP/AD, Duo, Okta, Centrify, MFA, Client SAML and other integrations
  • Centralized visibility of all users, connection history and all certificates across your network.

LAB Topology and Objective

  • This LAB is not dependent on any previous labs.
    • This LAB will build on the topology we have deployed already in the previous LABs. Following is what we have deployed already.
  • A GCP Spoke gateway (gcp-spoke-vpn) is already deployed in the gcp-spoke-vpn-vpc.
    • This is needed to make sure remote users, employees, developers or partners have a clear demarcation point (called Cloud Access layer in MCNA architecture) before they access the enterprise or corporate resources/workloads/VMs/etc.

Objective

  • Students will use their laptops to connect to this lap topology using an Aviatrix SAML client VPN and will become part of this topology. This will allow them to access any resources using the private IP address

Deploy Smart SAML Remote User VPN Solution

Deploy User VPN

Controller –> Gateway –> Create New (with the following information)

While creating this gateway, you must select “VPN Access” checkbox. This will make this gateway as OPENVPN Gateway for Aviatrix User VPN Solution

Common Mistake

The process could take upto ~10 minute to complete. It is hard to predict the deployment time even when you are deploying it in the same region and same cloud all the time. 

Once the gateway is deployed, you can see the status and the GCP LB address that was created as part of the automation.

After the gateway deployment, the topology looks like following

GCP TCP LB Configuration (Reference Only)

Following screen shots show the TCP LB details in GCP that was created by Aviatrix automation. LB helps in scaling out the solution without any disruption to the user profile or certificates.

Notice: Students do not have access to this details. It is shared here fro reference purposes only.

Profile Based Zero-Trust Access Control

Each VPN user can be assigned to a profile that is defined by access privileges to network, host, protocol and ports. The access control is dynamically enforced when a VPN user connects to the public cloud via an Aviatrix VPN gateway.

Create a new profile: Controller –> OpenVPN –> Profile

Create a policy to allow users access to only VMs in gcp-spoke-vpc1.

Now add a remote uswr and assign profile to it. Make sure to provide correct email address here.

Add a New VPN User

Controller –> OPENVPN –> Add a New VPN User


Download the .ovpn profile file from the Aviatrix Controller

Now download the Aviatrix OpenVPN Client: https://docs.aviatrix.com/Downloads/samlclient.html

MAC: https://s3-us-west-2.amazonaws.com/aviatrix-download/AviatrixVPNClient/AVPNC_mac.pkg
Windows: https://s3-us-west-2.amazonaws.com/aviatrix-download/AviatrixVPNClient/AVPNC_win_x64.exe
Linux: Check the Download link here

Now Open VPN client is connected to your network.

Testing and Verification

  • Ping VM in the gcp-spoke-vpc-2
    • This should not ping because as per the zero-trust profile, the remote users are not allowed to access any resources expect in gcp-spoke-vpc-1
  • Ping the VM in the gcp-spoke-vpc-1
    • Most likely it will not ping because the “gcp-spoke-vpn” VPC is not assigned to any MCNS Domain yet
shahzadali@shahzad-ali ~ % ping 10.20.11.130
PING 10.20.11.130 (10.20.11.130): 56 data bytes
92 bytes from gcp-spoke-vpn.c.cne-pod24.internal (10.20.19.2): Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 7c18   0 0000  01  01 544d 192.168.19.6  10.20.11.130 

Request timeout for icmp_seq 0
92 bytes from gcp-spoke-vpn.c.cne-pod24.internal (10.20.19.2): Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 b6da   0 0000  01  01 198b 192.168.19.6  10.20.11.130 

Request timeout for icmp_seq 1
92 bytes from gcp-spoke-vpn.c.cne-pod24.internal (10.20.19.2): Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 fcc1   0 0000  01  01 d3a3 192.168.19.6  10.20.11.130 

^C
--- 10.20.11.130 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
shahzadali@shahzad-ali ~ % 

Now change MCNS setting and assign gcp-spoke-vpn to Green domain.

Now the new topology will look like following

Now connectivity is established and ping will start working as you can see in the following output as well.

Request timeout for icmp_seq 7
92 bytes from gcp-spoke-vpn.c.cne-pod24.internal (10.20.19.2): Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 5400 2b8e   0 0000  01  01 a4d7 192.168.19.6  10.20.11.130 

Request timeout for icmp_seq 8
Request timeout for icmp_seq 9
Request timeout for icmp_seq 10
64 bytes from 10.20.11.130: icmp_seq=11 ttl=60 time=70.931 ms
64 bytes from 10.20.11.130: icmp_seq=12 ttl=60 time=63.498 ms
64 bytes from 10.20.11.130: icmp_seq=13 ttl=60 time=62.943 ms
64 bytes from 10.20.11.130: icmp_seq=14 ttl=60 time=69.129 ms
64 bytes from 10.20.11.130: icmp_seq=15 ttl=60 time=62.002 ms
64 bytes from 10.20.11.130: icmp_seq=16 ttl=60 time=68.655 ms
^C
--- 10.20.11.130 ping statistics ---
17 packets transmitted, 6 packets received, 64.7% packet loss
round-trip min/avg/max/stddev = 62.002/66.193/70.931/3.477 ms
shahzadali@shahzad-ali ~ % 

Conclusion

  • Aviatrix User VPN is a powerful solution
  • MCNS provides additional security beyond the profile based user-vpn

LAB5 – Overlapping Subnet / IP (Duplicate IP) Solution in GCP

Objective

ACE Enterprise in GCP wants to connect to different partners to consume SaaS services. These partners could be present in physical DC or Branches, or in VPC/VNET in the cloud such as GCP/AWS/Azure/etc. ACE cannot dictate or control the IPs/Subnets/CIDR those partners have configured and must support “Bring Your own IP” which might overlap with what is already configured in GCP.

In our topology GCP Spoke3 VPC subnet is overlapping with AWS Spoke2 VPC subnet (10.42.0.0). We need to make sure that VM in GCP Spoke3 VPC is able to communicate with EC2 in AWS Spoke2 VPC.

Topology Modifications

In order for this lab to work, we are simulating AWS Spoke2 VPC as a remote site/branch

  • Step#1: Detach the aws-spoke2 gateway from transit
  • Step#2: Delete the aws-spoke2-gw
  • Step#3: Deploy a standard Aviatrix Gateway (s2c-gw) in aws-spoke2-vpc

Step#1

Controller –> Multi-Cloud Transit –> Setup –> Detach AWS VPC2

Step#2

Controller –> Gateway –> Highlight AWS VPC2 Spoke GW –> Delete

Step#3

Controller –> Gateway –> Add New (use the following screen)

This getaway could be any router or firewall or vpn device in the on-prem or cloud location.

Topology

  • Following diagram shows the topology after the modifications are done
  • In the topology notice that there is no path between gcp-spoke3-vpc and s2c-gw in aws-spoke2-vpc
  • Also notice the local and remote virtual IPs allocated for the VPCs/Sites that are overlapping. This is needed so that these overlapping VPCs/Sites can communicate with each other using those virtual IPs
    • This is possible using Aviatrix patented technology where as a an enterprise you do not need to worry about programming advanced NAT rather Aviatrix intent based “Mapped NAT” policy will automatically take care of all the background VPC/VNET route programming, Gateway Routing, Secure Tunnel Creation, Certificate Exchange, SNAT, DNAT, etc.

Configuration

Bi-directional setup is needed for this to work. We will create two connections in coming configuration steps.

Connection From GCP to Partner Site (AWS) – Leg1

Controller –> SITE2CLOUD –> Add a new connection (with following details)

VPC ID / VNet Name: vpc-gcp-spoke-3
Connection Type: Mapped
Connection Name: partner1-s2c-leg1
Remote Gateway Type: Aviatrix
Tunnel Type: Policy-based
Primary Cloud Gateway: gcp-spoke-3
Remote Gateway IP Address: Check Controller’s Gateway section to find the public ip address
Remote Subnet (Real): 10.42.0.0/24
Remote Subnet (Virtual): 192.168.10.0/24
Local Subnet (Real): 10.42.0.0/24
Local Subnet (Virtual): 192.168.20.0/24

This will create the first leg of the connection from Cloud (GCP) to Partner Site (AWS). Tunnel will stay down until the other end is configured.

Download the Configuration

Aviatrix Controller provides a template that can be used to configure the remote router/firewall. Click on the Site-to-Cloud connection that you have just created. Click “EDIT” and then download the configuration for Aviatrix.

Downloaded file (vpc-vpc-gcp-spoke-3~-~cne-pod24-partner1-s2c-leg1.txt) contents just for reference purposes. We will import this file in next step.

{
  "ike_ver": "1",
  "name": "partner1-s2c-leg1",
  "type": "mapped",
  "tunnel_type": "policy",
  "peer_type": "avx",
  "ha_status": "disabled",
  "null_enc": "no",
  "private_route_enc": "no",
  "PSK": "Password123!",
  "ph1_enc": "AES-256-CBC",
  "ph2_enc": "AES-256-CBC",
  "ph1_auth": "SHA-256",
  "ph2_auth": "HMAC-SHA-256",
  "ph1_dh_group": "14",
  "ph2_dh_group": "14",
  "ph2_lifetime": "3600",
  "remote_peer": "34.86.77.74",
  "remote_peer_private_ip": "10.42.0.2",
  "local_peer": "54.71.151.196",
  "remote_subnet_real": "10.42.0.0/24",
  "local_subnet_real": "10.42.0.0/24",
  "remote_subnet": "192.168.20.0/24",
  "local_subnet": "192.168.10.0/24",

  "tun_name": "",
  "highperf": "false",
  "ovpn": "",
  "enable_bgp": "false",
  "bgp_local_ip" : "",
  "bgp_remote_ip" : "",
  "bgp_local_asn_number": "0",
  "bgp_remote_as_num": "0",
  "bgp_neighbor_ip_addr": "",
  "bgp_neighbor_as_num": "0",
  "tunnel_addr_local": "",
  "tunnel_addr_remote": "",
  "activemesh": "yes"
}

Close the dialogue box now.

Connection Partner Site (AWS) to GCP AWS – Leg2

Now we need to create a new connection and import the file we just downloaded

Controller –> SITE2CLOUD –> Add a new connection –> Select “aws-west-2-spoke-2” VPC –> Import

VPC ID / VNet Name: aws-west-2-spoke-2
Connection Type: auto-populated (Mapped)
Connection Name: partner1-s2c-leg2
Remote Gateway Type: Aviatrix
Tunnel Type: auto-populated (Policy-based)
Algorithms: auto-populated (do not change these settings)
Primary Cloud Gateway: auto-populated (aws-spoke2-vpc-s2c-gw)
Remote Gateway IP Address: auto-populated
Remote Subnet (Real): auto-populated (10.42.0.0/24)
Remote Subnet (Virtual): auto-populated (192.168.20.0/24)
Local Subnet (Real): auto-populated (10.42.0.0/24)
Local Subnet (Virtual): auto-populated (192.168.10.0/24)

Now it will take about a minute for both tunnels to come up

This makes is look like following

Verification

Now you can ping the overlapping subnet VM on the AWS side from GCP. Make sure to use the virtual subnet for AWS with the last octet of the VM being the same.

ubuntu@vm-gcp-spoke-3:~$ ifconfig
ens4 Link encap:Ethernet HWaddr 42:01:0a:2a:00:82
inet addr:10.42.0.130 Bcast:10.42.0.130 Mask:255.255.255.255
inet6 addr: fe80::4001:aff:fe2a:82/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1460 Metric:1
RX packets:1492666 errors:0 dropped:0 overruns:0 frame:0
TX packets:871540 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:518583431 (518.5 MB) TX bytes:111566034 (111.5 MB)

ubuntu@vm-gcp-spoke-3:~$ ping 192.168.10.84
PING 192.168.10.84 (192.168.10.84) 56(84) bytes of data.
64 bytes from 192.168.10.84: icmp_seq=1 ttl=62 time=83.8 ms
64 bytes from 192.168.10.84: icmp_seq=2 ttl=62 time=83.5 ms
64 bytes from 192.168.10.84: icmp_seq=3 ttl=62 time=83.6 ms
64 bytes from 192.168.10.84: icmp_seq=4 ttl=62 time=83.6 ms
64 bytes from 192.168.10.84: icmp_seq=5 ttl=62 time=84.2 ms
^C
--- 192.168.10.84 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 83.554/83.784/84.203/0.391 ms
ubuntu@vm-gcp-spoke-3:~$

This concludes the lab. Final topology looks like following

Note: This lab is not depend on previous labs.

LAB4 – GCP FQDN Based Egress Security

This lab will demonstrate how to provide Fully Qualified Domain Name (FQDN) based Egress Filtering security using Aviatrix. Only those FQDNs will be allowed which are permitted in the configured policy.

Egress FQDN Filtering Overview

Aviatrix FQDN Egress is a highly available security service specifically designed for workloads or applications in
the public clouds.

Aviatrix Egress FQDN Filtering is Centrally managed by the Aviatrix Controller and executed on Aviatrix FQDN gateway
in the VNET/VPC/VCN in the distributed or centralized architecture. All the internet bound traffic (TCP/UDP
including HTTP/HTTPS/SFTP) is first discovered and based on the results admin can create egress filters using
whitelist or blacklist model.

Egress FQDN filtering allows organizations to achieve PCI compliance by limiting application’s access to
approved FQDNs. This is a common replacement for SQUID proxy type of manual solutions. There are several
ways to deploy Egress FQDN filtering depending on requirements.

This lab will use existing GCP Spoke GWs to provide filtering to protect instances that are on private subnet but require Egress Security. For more scalable solution, enterprises opt for a dedicated Egresss FQDN GW rather than using the existing Spoke GW for this function.

Topology

  • The workload in gcp-spoke2-vpc will follow zero trust security model
    • Workload/VM in gcp-spoke2-vpc will only have access to https://*.ubuntu.com and https://*.google.com FQDN.
    • Rest of the traffic will be blocked with the base zero trust policy
  • We will configure the gcp-spoke2-gw as Egress FQDN GW as well to enforce this security policy
  • We will use VM in gcp-spoke3-vpc as “Jump Host” for this testing

Enable Egress FQDN Filtering

Controller –> Security –> Egress Control –> New TAG

Controller –> Security –> Egress Control –> Egress FQDN Filter –> Edit “Allow-List” TAG –> Add New

Now click “SAVE”, then “UPDATE” and CLOSE.

Now make sure that the base policy is “White” which stands for “Zero Trust”. This will make sure only the FQDNs in the “Allowed List” are accessible and rest of the FQDNs are blocked.

Now we will attach this filter policy to gcp-spoke-2-gw and then enable it.

It will look like following

The status is still disabled and now we need to enable it.

Testing and Verification

We have completed the following topology

  • ssh into the gcp-spoke3 VM using its public ip address (vm_gcp_public_ip_spoke3)
    • User: ubuntu / pass: Password123!
  • Then from there ssh into the gcp-spoke2 VM using its private ip address (vm_gcp_private_ip_spoke2)
    • User: ubuntu / pass: Password123!
  • gcp-spoke2 is where we have enforced the Egress FQDN policy
    • Since both spoke2 and spoke3 are in Blue segment, they can communicate to each others. If you would try to ssh into gcp-spoke2-vm from gcp-spoke1-vm, it will not work
shahzadali@shahzad-ali ~ % ssh ubuntu@34.86.180.56

ubuntu@34.86.180.56's password:
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.15.0-1087-gcp x86_64)
Documentation: https://help.ubuntu.com
Management: https://landscape.canonical.com
Support: https://ubuntu.com/advantage
3 packages can be updated.
0 updates are security updates.
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
*** System restart required ***
Last login: Sat Jan 2 16:41:56 2021 from 172.124.233.126
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@vm-gcp-spoke-3:~$ ifconfig

ens4 Link encap:Ethernet HWaddr 42:01:0a:2a:00:82
inet addr:10.42.0.130 Bcast:10.42.0.130 Mask:255.255.255.255
inet6 addr: fe80::4001:aff:fe2a:82/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1460 Metric:1
RX packets:1461966 errors:0 dropped:0 overruns:0 frame:0
TX packets:846760 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:510003616 (510.0 MB) TX bytes:107570824 (107.5 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:126 errors:0 dropped:0 overruns:0 frame:0
TX packets:126 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14552 (14.5 KB) TX bytes:14552 (14.5 KB)
ubuntu@vm-gcp-spoke-3:~$


ubuntu@vm-gcp-spoke-3:~$ ssh ubuntu@10.20.12.130

ubuntu@10.20.12.130's password:
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.15.0-1087-gcp x86_64)
Documentation: https://help.ubuntu.com
Management: https://landscape.canonical.com
Support: https://ubuntu.com/advantage
3 packages can be updated.
0 updates are security updates.
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
*** System restart required ***
Last login: Sat Jan 2 16:42:30 2021 from 10.20.12.2
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.
ubuntu@vm-gcp-spoke-2:~$
ubuntu@vm-gcp-spoke-2:~$

ubuntu@vm-gcp-spoke-2:~$ wget https://www.google.com
--2021-01-02 17:46:12-- https://www.google.com/
Resolving www.google.com (www.google.com)… 74.125.197.147, 74.125.197.103, 74.125.197.104, …
Connecting to www.google.com (www.google.com)|74.125.197.147|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.21’
index.html.21 [ <=> ] 12.54K --.-KB/s in 0s
2021-01-02 17:46:12 (29.4 MB/s) - ‘index.html.21’ saved [12844]

ubuntu@vm-gcp-spoke-2:~$ wget https://cloud.google.com
--2021-01-02 17:46:59-- https://cloud.google.com/
Resolving cloud.google.com (cloud.google.com)… 74.125.20.113, 74.125.20.102, 74.125.20.100, …
Connecting to cloud.google.com (cloud.google.com)|74.125.20.113|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 706920 (690K) [text/html]
Saving to: ‘index.html.22’
index.html.22 100%[==============================================================>] 690.35K 3.44MB/s in 0.2s
2021-01-02 17:47:00 (3.44 MB/s) - ‘index.html.22’ saved [706920/706920]
ubuntu@vm-gcp-spoke-2:~$

ubuntu@vm-gcp-spoke-2:~$ wget https://www.ubuntu.com
--2021-01-02 17:48:34-- https://www.ubuntu.com/
Resolving www.ubuntu.com (www.ubuntu.com)… 91.189.88.180, 91.189.88.181, 91.189.91.45, …
Connecting to www.ubuntu.com (www.ubuntu.com)|91.189.88.180|:443… connected.
HTTP request sent, awaiting response… 301 Moved Permanently
Location: https://ubuntu.com/ [following]
--2021-01-02 17:48:35-- https://ubuntu.com/
Resolving ubuntu.com (ubuntu.com)… 91.189.88.180, 91.189.91.44, 91.189.91.45, …
Connecting to ubuntu.com (ubuntu.com)|91.189.88.180|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 121017 (118K) [text/html]
Saving to: ‘index.html.23’
index.html.23 100%[==============================================================>] 118.18K 319KB/s in 0.4s
2021-01-02 17:48:36 (319 KB/s) - ‘index.html.23’ saved [121017/121017]
ubuntu@vm-gcp-spoke-2:~$

Now if we try to access any other FQDN, that should fail

ubuntu@vm-gcp-spoke-2:~$ wget https://www.espn.com
--2021-01-02 17:49:27-- https://www.espn.com/
Resolving www.espn.com (www.espn.com)… 13.224.10.82, 13.224.10.114, 13.224.10.88, …
Connecting to www.espn.com (www.espn.com)|13.224.10.82|:443… connected.


ubuntu@vm-gcp-spoke-2:~$ wget https://www.cnn.com
--2021-01-02 17:51:00-- https://www.cnn.com/
Resolving www.cnn.com (www.cnn.com)… 151.101.1.67, 151.101.65.67, 151.101.129.67, …
Connecting to www.cnn.com (www.cnn.com)|151.101.1.67|:443… connected.

Egress FQDN Stats on Controller

Controller –> Security –> FQDN Stats

Per Gateway Stats

Egress FQDN Search

==============================
Search results on Gateway gcp-spoke-2
==============================
2021-01-02T16:42:54.990606+00:00 GW-gcp-spoke-2-34.83.204.207 avx-nfq: AviatrixFQDNRule3[CRIT]nfq_ssl_handle_client_hello() L#291  Gateway=gcp-spoke-2 S_IP=10.20.12.130 D_IP=13.224.10.88 hostname=www.espn.com state=MATCHED drop_reason=BLACKLISTED Rule=*.espn.com,SourceIP:IGNORE;0;0;443
2021-01-02T16:43:12.620897+00:00 GW-gcp-spoke-2-34.83.204.207 avx-nfq: AviatrixFQDNRule0[CRIT]nfq_ssl_handle_client_hello() L#291  Gateway=gcp-spoke-2 S_IP=10.20.12.130 D_IP=13.224.10.82 hostname=www.espn.com state=MATCHED drop_reason=BLACKLISTED Rule=*.espn.com,SourceIP:IGNORE;0;0;443
2021-01-02T17:49:27.437085+00:00 GW-gcp-spoke-2-34.83.204.207 avx-nfq: AviatrixFQDNRule0[CRIT]nfq_ssl_handle_client_hello() L#291  Gateway=gcp-spoke-2 S_IP=10.20.12.130 D_IP=13.224.10.82 hostname=www.espn.com state=NO_MATCH drop_reason=NOT_WHITELISTED

2021-01-02T17:49:41.679243+00:00 GW-gcp-spoke-2-34.83.204.207 avx-nfq: message repeated 7 times: [ AviatrixFQDNRule0[CRIT]nfq_ssl_handle_client_hello() L#291  Gateway=gcp-spoke-2 S_IP=10.20.12.130 D_IP=13.224.10.82 hostname=www.espn.com state=NO_MATCH drop_reason=NOT_WHITELISTED]
2021-01-02T17:49:55.759092+00:00 GW-gcp-spoke-2-34.83.204.207 avx-nfq: AviatrixFQDNRule0[CRIT]nfq_ssl_handle_client_hello() L#291  Gateway=gcp-spoke-2 S_IP=10.20.12.130 D_IP=13.224.10.82 hostname=www.espn.com state=NO_MATCH drop_reason=NOT_WHITELISTED
2021-01-02T17:50:02.669462+00:00 GW-gcp-spoke-2-34.83.204.207 avx-nfq: message repeated 6 times: [ AviatrixFQDNRule0[CRIT]nfq_ssl_handle_client_hello() L#291  Gateway=gcp-spoke-2 S_IP=10.20.12.130 D_IP=13.224.10.82 hostname=www.espn.com state=NO_MATCH drop_reason=NOT_WHITELISTED]
2021-01-02T17:50:05.926066+00:00 GW-gcp-spoke-2-34.83.204.207 avx-nfq: AviatrixFQDNRule0[CRIT]nfq_ssl_handle_client_hello() L#291  Gateway=gcp-spoke-2 S_IP=10.20.12.130 D_IP=13.224.10.82 hostname=www.espn.com state=NO_MATCH drop_reason=NOT_WHITELISTED

CoPilot Egress FQDN Stats

https://copilot-pod24.mcna.cc/#/login   user: copilot / pass: Copilot123!  (Read-Only on Controller)

CoPilot –> Security –> Egress

CoPilot Live Status

CoPilot Search

LAB3 – GCP Multi-Cloud Network Segmentation (MCNS)

It is important to provide security compliance and fulfill audit requirements by using various methods and network segmentation is one of them. Providing Network Security segmentation is a critical business requirement. Aviatrix MCNS is helping many customers who achieved this requirement.

So far we have built following topology

Our objective in this lab to segment VPCs in GCP based on the workload. Here are the business requirements

  • There are two types of workload present in GCP called Green and Blue
  • The workload in Blue and Green must not be allowed to communicate to each other
  • Workloads within Blue and Green segments must be allowed to communicate with each other
  • These segments must also extend to AWS as well.
  • These segments should also extend to the on-prem data centers to provide segmentation for hybrid connectivity and their respective workload deployed in the on-premise DC locations.

Following is how the final topology will look like after all the business objectives are met

Enable MCNS on Transit gateways

Controller –> Multi-Cloud Transit –> Segmentation –> Plan –> Enable for “gcp-transit”

Controller –> Multi-Cloud Transit –> Segmentation –> Plan –> Enable for “aws-transit-gw-us-west-2”

Create Multi-Cloud Security Domains (MCSD)

Create two MCSD Green and Blue. These two domains are not connected to each other by default.

Controller –> Multi-Cloud Transit –> Segmentation –> Plan –> Create MCSD

Repeat it again for Blue

Controller –> Multi-Cloud Transit –> Segmentation –> Plan –> Create MCSD

MCNS Connection Policy

The following screen shows that Green and Blue are not connected as per their Security or “Connection Policy”.

Assign GCP and AWS VPC to MCSD

In order to enforce the intent/policy we just created, we need to assign VPCs to their respective Security Domain based on the business policy.

  • gcp-spoke-1 :: Green
  • gcp-spoke-2 :: Blue
  • gcp-spoke-3 :: Blue
  • gcp-to-dc-route-1 :: Green
  • aws-spoke-1 :: Green
  • aws-spoke-2 :: Blue
  • aws-to-dc-router-2 :: Green

Controller –> Multi-Cloud Transit –> Segmentation –> Build –> Associate Aviatrix Spoke to MCSD

Repeat this step as per the business requirement

Controller –> Multi-Cloud Transit –> Segmentation –> List –> Domains to verify the configuration

Verify the Connectivity

Now Ping from vm_gcp_private_ip_spoke1 (Green Segment) to other test machines (as listed below) and check the connectivity

  • vm_gcp_private_ip_spoke2 Blue Segment (10.20.12.130) – should not work
  • vm_gcp_private_ip_spoke3 Blue Segment (10.42.0.130) – should not work
  • vm_aws_private_ip_spoke1 Green Segment (10.101.0.84) – should work
ubuntu@vm-gcp-spoke-1:~$ ping 10.20.12.130
PING 10.20.12.130 (10.20.12.130) 56(84) bytes of data.
From 10.20.11.2 icmp_seq=1 Time to live exceeded
From 10.20.11.2 icmp_seq=2 Time to live exceeded
From 10.20.11.2 icmp_seq=3 Time to live exceeded
^C
--- 10.20.12.130 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2003ms

ubuntu@vm-gcp-spoke-1:~$ ping 10.42.0.130
PING 10.42.0.130 (10.42.0.130) 56(84) bytes of data.
From 10.20.11.2 icmp_seq=1 Time to live exceeded
From 10.20.11.2 icmp_seq=2 Time to live exceeded
From 10.20.11.2 icmp_seq=3 Time to live exceeded
From 10.20.11.2 icmp_seq=4 Time to live exceeded
^C
--- 10.42.0.130 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3003ms

ubuntu@vm-gcp-spoke-1:~$ ping 10.101.0.84
PING 10.101.0.84 (10.101.0.84) 56(84) bytes of data.
64 bytes from 10.101.0.84: icmp_seq=1 ttl=60 time=63.3 ms
64 bytes from 10.101.0.84: icmp_seq=2 ttl=60 time=61.3 ms
64 bytes from 10.101.0.84: icmp_seq=3 ttl=60 time=61.5 ms
64 bytes from 10.101.0.84: icmp_seq=4 ttl=60 time=61.3 ms
^C
--- 10.101.0.84 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 61.376/61.914/63.303/0.806 ms
ubuntu@vm-gcp-spoke-1:~$ 

Now keep the ping running from gcp-spoke-1 VM to 10.20.12.130 and change the policy to connect Green and Blue. Notice that ping starts working.

Now change the policy as it was before so that Blue is not allowed to Green

LAB2 – GCP Multi-Cloud Network Transit / Hub-Spoke

In this lab, we will build the hub and spoke network in GCP. All the GCP VPCs and their respective subnets are already created for you to save time.

GCP Spoke-2 GW VPC Network/Subnet Creation

This step is already done for you so please do not attempt.

Controller can create those subnets using API/Terraform as well. It makes it easy to make it part of your CI/CD pipleline.

Controller –> Useful Tools –> Create A VPC –> Add New
Controller –> Useful Tools –> Create A VPC –> Add New Row

You should view the VPCs created for you already.

Controller –> Useful Tools –> VPC Tracker

Now click “VIEW GCLOUD SUBNET” to see further details

Deploy GCP Spoke-2 Gateway

Since we have all the subnets created properly, it is time to deploy the GCP Spoke-2 GW using the Controller.

Controller –> Multi-Cloud Transit –> Setup

Connect GCP Spokes to GCP Transit

At this stage we have 4 spokes in GCP

  • gcp-spoke1 –> connected to gcp transit gw
  • gpc-spoke2 –> not connected to gcp transit gw
  • gcp-spoke3 –> connected to gpc transit gw (this will be use for user-vpn later in the lab)
  • gcp-spoke4 –> not connected to gcp transit gw

Login to CoPilot (user: Copilot / pass: Copilot123!) now to see the connectivity in real-time

In this part of the lab, we will connect the remaining spoke gateway to aviatrix transit-gw

Controller –> Multi-Cloud Transit –> Setup –> Attach Spoke Gateway to Transit Network

Spoke#2

Spoke#3

Now notice the topology changed and all GCP spokes are connected now

Veryify GCP Spoke Connectivity

Now Ping from vm_gcp_private_ip_spoke1 (user: ubuntu / pass: Password123!) to other test machines (as listed below) and check the connectivity

  • vm_gcp_private_ip_spoke2 (10.20.12.130) – should work
    • using Aviatrix hub/spoke
  • vm_gcp_private_ip_spoke3 (10.42.0.130) – should work
    • using Aviatrix hub/spoke

Connect AWS Spoke to AWS-Transit

If we look at the AWS side, we have aws-spoke1 connected to aws-transit but the spoke2 is not yet connected as shown in the topology below

Now we will use Aviatrix Controller to establishe this connection. The aws-spoke2 gateway is already deployed for you

Controller –> Multi-Cloud Transit –> Setup –> Attach Spoke Gateway to Transit Network

After this setup, following is the topology

Veryify AWS Spoke Connectivity

Now Ping from vm_gcp_private_ip_spoke1 (user: ubuntu / pass: Password123!) to AWS test EC2s (as listed below) and check the connectivity

  • vm_aws_private_ip_spoke1 (10.101.0.84) – should work
    • This works because the traffic is routed via the dc-router-1. There is a private link between dc-router-1 and dc-router-2 connecting GCP and AWS using services like Equinix/Megaport/Pureport/Epsilon/etc.
    • Use tracepath or traceroute ($sudo apt install traceroute) to confirm
  • vm_aws_private_ip_spoke2 (10.42.0.84) – should not work
    • Because this subnet overlaps with the gcp spoke-vpc3 subnet
    • We will fix this issue later in the lab

Following output shows a traffic route from gcp-spoke-1 test VM to AWS test EC2 in aws-spoke1

ubuntu@vm-gcp-spoke-1:~$ tracepath 10.101.0.84
 1?: [LOCALHOST]                                         pmtu 1460
 1:  gcp-spoke-1.c.cne-pod24.internal                      1.563ms 
 1:  gcp-spoke-1.c.cne-pod24.internal                      0.260ms 
 2:  gcp-spoke-1.c.cne-pod24.internal                      0.256ms pmtu 1396
 2:  10.20.3.2                                             1.846ms 
 3:  10.20.3.2                                             0.775ms pmtu 1390
 3:  169.254.100.2                                        51.410ms 
 4:  172.16.0.2                                           55.184ms 
 5:  10.10.0.93                                           75.906ms 
 6:  10.101.0.70                                          77.206ms 

Multi-Cloud Transit Peering (Connection GCP and AWS)

Now we have setup all the gateway and build the hub-spoke (transit) in GCP, it is time to connect GCP with AWS now. This peering is secure and encrypted and takes care of number of configuration options and best practices that are needed for such a complex connectivity. You will appreciate the simplicity of doing that.

Select Multi-Cloud Transit –> Transit Peering –> ADD NEW

Note that the order of cloud selection does not matter here. After few seconds, the status will be green as shown in the screen shot below.

Verify Transit Connectivity

Ping from vm_gcp_private_ip_spoke1 (user: ubuntu / pass: Password123!) to GCP and AWS VMs (as listed below) and make sure connectivity works

  • vm_gcp_private_ip_spoke2 (10.20.12.130) – should work
    • using Aviatrix hub/spoke
  • vm_gcp_private_ip_spoke3 (10.42.0.130) – should work
    • using Aviatrix hub/spoke
  • vm_aws_private_ip_spoke2 (10.42.0.84) – should not work
    • Because this subnet overlaps with the gcp spoke-vpc3 subnet
    • We will fix this issue later in the lab
  • vm_aws_private_ip_spoke1 (10.101.0.84) – should work
    • This time the packet will use gcp to aws aviatrix transit gateway route for connectivity
    • Use tracepath or traceroute ($sudo apt install traceroute) to confirm
    • In case this link goes down, the connectivity will be provided using the DC router DCI link as backup. This setup is important for Business Continuity and Disaster Recover
ubuntu@vm-gcp-spoke-1:~$ traceroute 10.101.0.84
traceroute to 10.101.0.84 (10.101.0.84), 30 hops max, 60 byte packets
1 gcp-spoke-1.c.cne-pod24.internal (10.20.11.2) 1.322 ms 1.354 ms 1.413 ms
2 10.20.3.2 (10.20.3.2) 2.684 ms 2.708 ms 2.710 ms
3 10.10.0.93 (10.10.0.93) 62.072 ms 62.032 ms 62.032 ms
4 10.101.0.70 (10.101.0.70) 61.975 ms 61.981 ms 61.895 ms

Controller –> Multi-Cloud Transit –> List –> Transit Gateway –> gcp-transit –> Show Details

Following screen shows the best path available to reach 10.101.0.84

Controller –> Multi-Cloud Transit –> List –> Transit Gateway –> gcp-transit –> Show Details –> Route Info DB Details

Following screen shows that 10.101.0.0/24 was received from two paths and transit peering was selected as the best path

This completes the transit topology and testing. You can verify it in CoPilot as well.

LAB1 – Google Cloud and Aviatrix Testing/POC LAB Guide

Introduction

This document is the lab guide for GCP Test Flight Project. Anyone with basic GCP knowledge is the audience. It is GCO focused with connection to AWS as optional component of the cloud.

Topology

Following is the starting topology. Some components are pre-built to save time.

Once you finish all lab steps, following is what it will look like

Main Use-Cases Covered

  • Cloud and Multi-Cloud Transit (Hub and Spoke) connectivity
  • Multi-Cloud Network Segmentation (MCNS)
  • Egress FQDN
  • User-VPN
  • Multi-Cloud Transit with AWS
  • Policy Based Remote User SAML/SSL VPN
  • Hybrid / On-Premise Connectivity (S2C)
  • Traffic Engineering with SD and BGP advance knobs
  • Day2 Operations, troubleshooting and monitoring (Aviatrix CoPilot)

Warning / Pre-Requisite / Notes

  • Do not change the password of any device or server in the lab pod
  • Do not change controller password
  • In most of the places:
    • The Aviatrix Controller is referred to as “Controller”
    • The Aviatrix Gateway is referred to as “Gateway”

LAB1 – Verify Connectivity

Make sure the lab is in good standing. Verify the following tasks by logging into the Aviatrix Controller UI. Make sure you log in to your own pod. Pod name is displayed on top.

Make sure you have resources deployed and matches to following


The GCP Project is already on-boarded to save time in the Aviatrix Controller under Accounts Access Account


Now change the email address to your corporate email address under Accounts –> Account Users –> admin (do not change the Controller password)

Aviatrix gateway are pre-deployed to save time. Make sure all gateways are up and running.

Check the transit gateway under Multi-Cloud Transit –> List –> Transit

Check the spoke gateway under Multi-Cloud Transit –> List –> Spoke

Verify GCP VM SSH Connectivity

GCP VMs only requires a password. Password is Password123!
There is no .pem file needed to login to them. Login to vm_gcp_public_ip_spoke1. This IP address is provided in the LAB POD file

shahzadali@shahzad-ali Pem Files % ssh ubuntu@35.224.13.215
ubuntu@35.224.13.215's password:
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.15.0-1087-gcp x86_64)
 
 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
 
11 packages can be updated.
0 updates are security updates.
 
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
 
 
Last login: Sat Nov 28 16:42:18 2020 from 172.124.233.126
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
 
ubuntu@vm-gcp-test-spoke-1:~$

ubuntu@vm-gcp-spoke-1:~$ ifconfig

ens4 
Link encap:Ethernet HWaddr 42:01:0a:14:0b:82
inet addr:10.20.11.130 Bcast:10.20.11.130 Mask:255.255.255.255
inet6 addr: fe80::4001:aff:fe14:b82/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1460 Metric:1
RX packets:1025064 errors:0 dropped:0 overruns:0 frame:0
TX packets:663466 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:511233248 (511.2 MB) TX bytes:81897766 (81.8 MB)

lo 
Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
ubuntu@vm-gcp-spoke-1:~$

Now Ping from vm_gcp_private_ip_spoke1 to other test machines (as listed below) and check the connectivity

  • vm_gcp_private_ip_spoke2 (10.20.12.130) – should not work
  • vm_gcp_private_ip_spoke3 (10.42.0.130) – should not work
  • vm_aws_private_ip_spoke1 (10.101.0.84) – should work
  • vm_aws_private_ip_spoke2 (10.42.0.84) – should not work

Can you guess why it worked or did not work?

Verify AWS EC2 SSH Connectivity

ssh into AWS VM in Spoke1using its public ip address and .pem file (the address is provided in the lab pod file you received). If you get the following error, then please fix your .pem file permission first.

shahzadali@shahzad-ali Pem Files % ssh ubuntu@35.163.104.122 - instance_priv_key.pem
ubuntu@35.163.104.122: Permission denied (publickey).
 
shahzadali@shahzad-ali Pem Files % chmod 400 instance_priv_key.pem

ssh using the user-name and .pem file again and ping the second AWS instance

shahzadali@shahzad-ali Desktop % ssh ubuntu@34.217.68.104 -i instance_priv_key_gcp_pod24.pem
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-1072-aws x86_64)
Documentation: https://help.ubuntu.com
Management: https://landscape.canonical.com
Support: https://ubuntu.com/advantage
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
79 packages can be updated.
0 updates are security updates.
New release '18.04.5 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
*** System restart required ***
Last login: Thu Dec 31 20:36:52 2020 from 172.124.233.126
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.
ubuntu@ip-10-101-0-84:~$

Ping from vm_aws_private_ip_spoke1 to …

  • vm_gcp_private_ip_spoke1 (10.20.11.130) – should work
  • vm_gcp_private_ip_spoke2 (10.20.12.130) – should not work
  • vm_gcp_private_ip_spoke3 (10.42.0.130) – should not work
  • vm_aws_private_ip_spoke2 (10.42.0.84) – should not work

Verify On-Prem Router SSH Connectivity

ssh into the on-prem dc-router-1 (we are using Cisco CSR to simulate it). User is admin and Password is “Password123”

shahzadali@shahzad-ali Desktop % ssh admin@54.219.225.218
Password: 


dc-router-1#show ip int brief
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.20.162  YES DHCP   up                    up      
Loopback0              10.20.11.254    YES TFTP   up                    up      
Tunnel1                169.254.100.2   YES TFTP   up                    up      
Tunnel42               172.16.0.1      YES TFTP   up                    up      
VirtualPortGroup0      192.168.35.101  YES TFTP   up                    up      
dc-router-1#

From dc-router-1 ping gcp and aws instances private ip addresses

  • vm_gcp_private_ip_spoke1 (10.20.11.130) – should no work
  • vm_gcp_private_ip_spoke2 (10.20.12.130) – should not work
  • vm_gcp_private_ip_spoke3 (10.42.0.130) – should not work
  • vm_aws_private_ip_spoke1 (10.101.0.84) – should not work
  • vm_aws_private_ip_spoke2 (10.42.0.84) – should not work

ssh into the on-prem dc-router-1 (we are using Cisco CSR to simulate it). User is admin and Password is “Password123”

shahzadali@shahzad-ali Desktop % ssh admin@54.193.196.247
The authenticity of host '54.193.196.247 (54.193.196.247)' can't be established.
RSA key fingerprint is SHA256:fi8bbpJc8LCE32dn9RL1EIDzznl+mgQ5V5u5vR/hxFo.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '54.193.196.247' (RSA) to the list of known hosts.
Password: 
dc-router-2#
dc-router-2#show ip int brief 
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.10.120  YES DHCP   up                    up      
Tunnel1                169.254.101.2   YES TFTP   up                    up      
Tunnel42               172.16.0.2      YES TFTP   up                    up      
VirtualPortGroup0      192.168.35.101  YES TFTP   up                    up      
dc-router-2#

From dc-router-2 ping gcp and aws instances private ip addresses

  • vm_gcp_private_ip_spoke1 (10.20.11.130) – should no work
  • vm_gcp_private_ip_spoke2 (10.20.12.130) – should not work
  • vm_gcp_private_ip_spoke3 (10.42.0.130) – should not work
  • vm_aws_private_ip_spoke1 (10.101.0.84) – should not work
  • vm_aws_private_ip_spoke2 (10.42.0.84) – should not work

This completes the verification lab1. We will now move to other use-cases.

Secure S3 Bucket Access Over Direct Connect Private VIF

Problem Statement

  • The AWS recommended and native solution to access S3 buckets (via Direct Connect (DX) from on-prem location) is to use Public VIF
  • Accessing S3 bucket over DX Public VIF can pose serious security threats to enterprises
  • AWS advertises the entire S3 Public subnet range (one or all the regions) to on-prem which implies that …
    • All on-prem users can upload to any S3 bucket, including to their personal S3 buckets on their own personal accounts, leading to confidential data leakage
    • Potentially higher utilization of DX circuit (non-compliant data) that could choke the DX and may incur higher charges ($$$)

Solution

The solution proposed here not only works for the traffic coming from on-prem location but also allows secure connectivity to S3 traffic coming from other AWS VPCs or other public clouds as well (Multi-Cloud scenario)

Following are the high level steps to implement this solution

  • Create a dedicated S3 VPC (as spoke)
  • Create S3 end-point in the S3 VPC
  • Deploy Aviatrix Global Transit and attach the S3 VPC (as spoke) to it
  • Deploy Aviatrix S3 Gateway in dedicated S3 VPC
  • Enable private S3 feature in Aviatrix Controller
    • The Controller automatically configures the AWS NLB and load balances multiple AVX S3 GW for high availability, redundancy and performance
  • Apply the security policy to allow S3 access from specific CIDRS
    • Controller enforce zero-trust S3 security policy
    • The CIDRs specified in the policy are allowed to access S3 (rest are blocked by default)
  • Create on-prem DNS private zone to point the S3 bucket FQDN to private IP

Production Topology

Following is the enterprise topology to solve the this business challenge

Traffic Flow

  • Business requirement is that on-prem corporate resources, such as laptop/desktop, can access the S3 Bucket in AWS
    • This access must be encrypted over DX link at line rate
    • The encryption should allow maximum utilization of 10G DX cuircuit
  • There is an optional Aviatrix CloudN hardware appliance in the topology
    • CloudN HPE (High Performance Encryption appliance) provide end-to-end and line-rate encryption of all the traffic crossing over the AWS DX link
    • The traffic over DX link is not encrypted by default device that is why it is important to use CloudN appliance without compromising the throughput (Cloud Native IPSec encyrption is limited to 1.25 Gbps only)
  • S3 bucket traffic goes from on-prem DX link to Aviatrix Transit GW (AVX-TR-GW)
    • This is possible because the on-prem DNS server is configured to send S3 bucket traffic towards S3 VPC
  • The S3 VPC (AVX-SPK-GW) is attached to AVX-TR-GW as a spoke
  • S3 bucket traffic goes to AVX-S3-GW (AVX-SPK-GW)
  • AVX-S3-GW inspect the traffic and if the policy allows it, then it forwards the traffic to S3 end-point configured inside the S3 VPC

Deployment Details

In this deployment

  • Aviatrix Transit and Spoke gateways are already deployed in their respective VPCs
  • Aviatrix Transit and Spoke peering is already configured
  • On-Prem to direct connect connectivity is already established

Tip: You can test this solution even if you do not have a DX circuit. What you can do is to deploy another Aviatrix Transit GW in another Cloud or region. This second Transit could be treated as on-prem location. Now peer both Aviatrix Transit gateways together to establish connectivity.

1 – Create AWS S3 Endpoint

S3 endpoint created in AWS

AWS S3 Endpoint details

2- Deploy Aviatrix S3 Gateway

Deploy two Aviatrix generic or standard gateway from the “Gateway” left navigation tab in the S3-Spoke-VPC.

S3 Gateways
Aviatrix GWs designated for s3 traffic

3- Configure S3 Bucket Access Parameter

Under Security –> Private S3 configure the S3 bucket access parameters.

  • Select the gateway name first
  • The select the source CIDR
    • This creates access policy such that only CIDRs mentioned here are allowed to access certain S3 buckets
  • Now specify the bucket name. In my case I am using “shahzad-public-s3.s3.amazonaws.com as my bucket name
    • You can find this name by loggin into your AWS console
Aviatrix S3 private access parameters

You have to repeat the same steps for second S3 gateway (this second gateway is optional and for high availability). You can specify more than two gateway as well, depending on your load requirement


aws console to find out s3 bucket name
Tip: The S3 object URL is only visible after upload a file to S3 bucket

4- AWS ELB Created

In the background, an AWS ELB –> NLB is created and two S3 gateways are being load balanced. Following screens show the NLB configuration in AWS console done by Aviatrix Controller

NLB Created by aviatrix controller

The listener is listening on https:443 for s3 bucket

NLB is load balancing between two s3 gateway

Testing and Validation

Following screen is taken from the on-prem laptop. Notice that a ping to my S3 bucket is returning the private IP address of AWS NLB IP address. Which basically takes the request to one of the Aviatrix S3 gateways.

C:\>ping shahzad-public-s3.s3.amazonaws.com
Pinging shahzad-public-s3.s3.amazonaws.com [10.11.144.163] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.


Ping to show public s3 fqdn returns the private nlb ip address

The "S3 Bucket FQDN Name Resolution IP: 10.11.144.163" is actually the IP address of the private interface of AWS NLB. This is the IP address that should be resolved to in your on-prem DNS for S3 bucket FQDN (i.e shahzad-public-s3.s3.amazonaws.com")

Now type the S3 Bucket FQDN in the browser on your on-prem laptop/desktop/server

https://shahzad-public-s3.s3.amazonaws.com/S3_Bucket_Over_Private_VIF.txt

Following screen shows that we can successfully access the S3 public FQDN using the private VIF


S3 Bucket FQDN Access Visibility

You can also check the stats of the FQDN access because in the background the S3 private access feature is using the Egress FQDN URL.

s3 fqdn stats for entire deployment

s3 fqdn stats for one specific gateway

Design Notes

  • Aviatrix Controller will automatically deploy the AWS-NLB per region
  • There is no limit to the number of Aviatrix-S3-GW you can deploy. These gateways are Active/Active and being load balanced by the AWS-NLB
  • Even if you deploy one S3-GW, you will notice an AWS-NLB deployed. This is default behavior and cannot be changed. Because for production, our recommendation is to deploy at least two S3-GWs in two different AZs within the same region
  • It is customer’s responsibility to pick the right size of S3-GW depending on the traffic needs
    • Aviatrix Controller does not create or manage auto-scaling group
    • Adding new gateway of different size is just a matter of one click in the Controller
  • Using the source CIDR security policy model, the rest of the access is zero-trust for S3 FQDN.
    • Different on-prem teams could segment the traffic based on VLAN or Account they own in the AWS
  • Aviatrix strongly recommends to use additional S3 security provided by AWS
    • Remember it is a shared security model

Multi-Cloud Network and Security (MCN) Architecture: Ownership vs SaaS Approach

When it comes to building and running the Cloud Networks, predominantly there are two distinct approaches available to Network/Security Engineers, Architects, and decision-makers.

  1. Owning the Network and Security Architecture Approach
  2. As a Service (SaaS) Approach

Let’s take a look at the following point and understand both approaches. In the end, an enterprise must pick the approach that would solve their business challenges and will satisfy the compliance, governance, and audit requirement.

Owning the Architecture Approach

  • Enterprises should own the architecture end-to-end. Do not fall into the traps of the early days of Cloud adoption where shadow IT and DevOp guys took control and started building networking on their own
  • Almost all the Enterprises I talk to, want to own, control, and management control, data and operations plane. Similar to the way they were owning the on on-prem networking and security
  • How would you get the deep level of monitoring, logs and visibility from the SaaS platform? What I have seen that if enterprise do not own the platform, then they are at the mercy (SLA) of SaaS provider

Trust Factors

  • How much do you trust a SaaS based Multi-Cloud Networking and Security provider?
  • You have to trust your CSP (AWS/Azure/GCP/etc) I get that. Buy should you add an extra layer of trust as an enterprise?
    • It is trust (..cloud hardware) over trust (cloud hperplane) over trust (cloud provider security model) over trust (multi-cloud provider SaaS platform).

Competition Factor

  • Are you ok sitting next to multiple tenants on the same SaaS platform? One of them might be your competitors
    • Again, this is something you have to decide as an enterprise.
    • There is a reason that some retail customers are not hosting their applications on AWS and going to Azure. You could apply the same logic here as well.
    • If this SaaS goes down, you and your competitor both goes down. Not good because where is your competitive advantage then?

Pace of Innovation

  • Pace of innovation might be slow
    • If there is a feature an enterprise need, then in the SaaS model, it will be hard for enterprise to ask to add that feature into the product.
    • Typically SaaS providers need to support and enable a good number of tenants and it is not easy for them to quickly build and release new features

Compliance/Audit/Governance/GDPR

  • In the SaaS offering, some one else dictates its own terms and conditions. It is hard for you as an enterprise to dicated and create your own policies, governance and operational model.

Credits

Following people helped me review and write this blog
Hammad Alam

GCP Networking Best Practices

Delete Default VPC and Subnets

You should delete the deafult VPC and Subnets created by GCP automatically. Once you have the defult subnet and VPC deleted, following is how it looks like. In the screen shot below notice that all the default subnets are gone and you can only see the ones I created manually with “Mode = Custom”

The reason to delete the default is that most likely they could overlap either with on-prem deployment or other Clouds (such as AWS/Azure). As an architect you should maintain your own good IP address assignment hygiene.

For example if you look at my Multi-Cloud Networking IP scheme you will notice there is a theme there. Pretty much same as we used to do in On-prem networking world. This help us troubleshoot and manage easily.

The lab I have built is at a very small scale. You might want to adjust these ranges based on your future growth plans and strategy.

Private Google Access

When you create a VPC, you should enable your subnets to access Google Services using the Private IP addressing as well. To access services like storage bucket or big-query services.

GCP Shared VPC Concepts

Warning: Setting up GCP Shared VPC is not easy. It requires various API rights and roles before you are allowed to create it from the UI. I had to do lots of hit and trials. The GCP documentation is not clear and not straightforward. I am the super admin for my GCP organization called netjoints.com and still I had to enable many roles in different places to create a shared service VPC

There are two types (host and service) of projects in GCP

  • Host Project(s)
    • Host project are created using Shared VPC option
    • Simply put, host project shares subnets with the service projects
    • In majority of the cases, one host project is enough
  • Service Projects
    • This is where actual VMs are being deployed
    • The VM is being deployed in a subnet that is shared by “Shared VPC Host Project”

Following diagram illustrate the concept and relation between Shared VPC host project and Service Prpject

Service project to multiple VPCs

Setting up Shared VPC Host Project

First step to create a standard GCP Project that will be treated as Host Project. It is good practice to name it Host Project.

Then enable GCP Shared VPC

Select “Setup Shared VPC”

Save & Continue will take you to following screen.

In the above screen shot, I am sharing all my subnets. Depending on your org. policy, you might want to share few but not all.

Following are some requirements where a shared service VPC (aka Host Project) is needed

  • Use Shared VPC for administration of multiple working groups
  • Use multiple host projects if resource requirements exceed the quota of a single project
  • Use multiple host projects if you need separate administration policies for each VPC network
  • Create a shared services VPC if multiple VPC networks need access to common resources but not each other

Now create VPC in shared services

Now that the Host Project owner has created the subnets, he/she can share them with the Service Project as necessary. Following screen shows members being added to the Shared VPC and their roles assigned according to your org. policy.

Service Projects need to have Compute Engine API enabled to be configured as service projects.

Now in the next step, we will attach the service project called “Service Project A” to the shared VPC.

Select the name of the service project from the shared VPC UI as shown below

Finally you can see in the following screen that Service Project is successfully attached with the Shared VPC and all subnets are being shared

Following screen shot shows the view from the Service Project-A side of the house. You can see the networks shared to this service project. Now the compute resources can consume these subnets.

AWS Direct Connect and Direct Connect Gateway Scale Limits

Direct Connect (DX)

  • DX is region specific offering
    • It allows On-Prem physical locations to connect to a specific AWS region/location
  • DX supports max of 50 VIFs (including Private and Public) per physical connection
  • DX does not support Transit VIF for AWS-TGW connectivity

Direct Connect Gateway (DXGW)

  • Only supports Private and Transit VIFs
    • DXGW mainly used to access private resources in VPCs
  • Does not support public VIF
    • DXGW does not provide any benefit of Public Internet Connectivity
  • VGW associated with a DXGW must be “attached” to a VPC
  • Does not support transitive routing or transit connectivity
    • VPC in Region-1 cannot directly communicate with VPC in Region-2
    • DX Location-1 cannot directly communicate with DX Location-2
  • Up to 30 DX physical connections can connect to one single DXGW for physical link redundancy purposes
    • In another words 30 DX locations/regions
  • DX supports max of 50 VIFs (for DXGW only Private and Transit VIFs are applicable)
    • It means one can have Max of 50 DXGW per physical DX link
    • But one DXGW can connect to max of 10 VPCs
    • It means Max of 500 VPCs (50 x 10 VPC) per physical DX link across accounts and regions

DXGW with AWS-TGW Limitations

  • Transit VIF can only be attached to a DXGW
  • Only one Transit VIF for any AWS Direct Connect 1/2/5/10 Gbps connection
    • Less than 1G connections does not support Transit VIF
    • Max of 3 AWS-TGW can connect to one DXGW behind one Transit VIF
  • A single DXGW cannot attach with both Private and Transit VIF
    • This could be a serious limitation for some customers
    • I think the underline assumption is that if a customer is alreadt using AWS-TGW then why would he want to use a private VIF attached to the same DXGW?

DXGW without and with AWS-TGW Comparision

DXGW without AWS-TGWDXGW with AWS-TGW
10 VPCs per DXGW3 TGWs per DXGW
50 DXGW max (b/c of 50 Private VIF)With Transit  VIF only one DXGW is possible
500 VPCs total5,000 VPCs per TGW
15,000 VPC per DX physical link
Private VIF supported on all Direct Connect connection typesTransit VIF supported only on dedicated or hosted connections of speed 1Gbps and above
No additional chargesAdditional charge for TGW data processing

References

https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-limits.html
https://docs.aws.amazon.com/directconnect/latest/UserGuide/limits.html

Credits

Abdul Rahim
Kamran Habib
Saad Mirza
Hammad Alam

Importance of Right Network Architecture vs Cost

An Architect, CTO or any technical decision maker has a huge responsibility to approve and adopt the “right network architecture” that is aligned with the business requirement.

We have seen enterprises who picked the wrong or compromised network architecture and then paid the price, way more than the initial cost to build and run the network, in the long run.

Here we are sharing some nuggets for the technical decision makers

  • A bad architecture can cost you a lot in the long run. A lot more than what you have spent on building and running it
  • Don’t make long term architecture decisions based on short-term problems
  • Right architecture is more important than feature set
  • Simple architecture is the best architecture

Building an architecture and putting a design is one time deal, but you end up running that design for years to come. As an architect if you don’t make smart choices to build the correct architecture, your enterprise will be paying a lot more.

Also think about support. You need a trusted and seasoned support partner working side by side, when problem arises.

Real World Customer Example

Let me give you an example. Here is an architecture a customer wanted to go with.

This customer wanted to …

  • Build the Aviatrix transit peering within an AWS region
  • Build the Aviatrix transit peering across multiple AWS regions and clouds (GCP)
  • Deploy AWS-TGW using Aviatrix Controller but just to attach the AWS-TGW with the AVX-Transit-GW

Essentially all the red-lines in the topology above were to be controlled and managed by Aviatrix.
For VPC1, VPC2 and so on, this customer wanted to do it manually. Thinking that it was just one time job.

In order to save few $$$, they wanted to make just one comprises in the architecture and I will explain how costly that one compromise could be in the long run

Customer did not want to use Aviatrix’s AWS-TGW Orchestration/Control to attach the Spoke VPCs with AWS-TGW

Ripple Effect of a Single Compromise

I will break it down into different section and the ripple effect a single compromise can make

  • Aviatrix Controller won’t be able to monitor and propagate existing and new routes
    • Application VPC routes must be updated manually
    • AWS-TGW route tables must be updates manually
    • Transit VPC route table must be updated manually
  • Customer will loose the Aviatrix Controller’s TGW Audit functionality
    • Aviatrix has a feature called “TGW Audit” that monitors all the routing tables and updates in the Spoke, Transit and from On-Prem to make sure the intent matches with realized state (for example if someone by mistake add or remove a route manually from the VPCs or no overlapping IP issues etc.)
    • This could be huge operational burden on the team
  • Customer will not be providing proper alerts about the route updates and warnings about incorrect or duplicate routes
    • No network correctness
  • Customer will not be able to use new route approval feature (Aviatrix has this unique feature where any new route learned from on-prem requires admin approval before it can be propagated into the Cloud Network)
  • Beside that there are other functionalities that Aviatrix is planning to build for AWS-TGW and Aviatrix-TGW that probably won’t work in such a network design
  • No way to do network segmentation for workloads in different VPCs
    • No Security Domain functionality available
  • Potential of AWS-TGW sprawl
    • Multiple AWS-TGW might be needed for traffic separation
    • Huge management overhead
  • Some of the Aviatrix Flight-Path functionality might be broken in future
  • In future if Aviatrix releases, capacity planning and monitoring tools, that might not work in this type of architecture
  • Adding the Firewall in the architecture will not be possible. This could be a huge compliance and security risk a customer would be taking for security sensitive data
  • For User-VPN use-case, customer must accommodate VPN subnets manually on TGW and Aviatrix Transit
  • Aviatrix support won’t be able to solve and troubleshoot end-to-end because the VPCs were not attached by Aviatrix Controller
  • Customer is taking the risk of not having end-to-end Encryption
    • AWS-TGW does not provide encryption for Spoke VPCs
    • This could be moot point in this architecture, because customer decided to use AWS-TGW as attachment but it is important to call out for compliance, audit, GDPR and security reasons

Credits

Wanted to say thanks to the following people for providing input to this post

Tomasz Klimczyk
Don Leone
Mark Cunningham
Hammad Alam
Nauman Mustafa
Saad Mirza

Aviatrix ACE Professional Bootcamp Prerequisite

Technical Prerequisite

Familiarity and basic know-how of at least one Cloud Provider is must to attend the bootcamp. For instance attendees should know concepts of

  • VPC/ VNet
  • Account/ Subscription
  • AMI / VM / EC2
  • VPN GW / VGW / Internet GW (IGW)
  • CIDR / Subnet / Routes
  • etc

List of items participants need to bring

  • Laptop needed with SSH/RDP tools installed
  • For Windows, make sure to have software like puttygen to generate cert based password
  • The underlying Cloud for labs is AWS
    • The same labs are applicable to other Cloud such as Azure, GCP and OCI.
    • The beauty of Aviatrix is that it will hide all the complexities of Clouds and provides a unified/normalized management, control, data and operations plane
  • All users must have an account with admin. privileges in AWS. It could be a personal account that could be deleted after the bootcamp. For Azure and GCP instructor will use their account to showcase multi-cloud use-cases

List of items needed in the training room

  • Projector with HDMI or USB-C cable
  • White-Board (it should not be in-front of Projector. Ideally it could be on the side)
    • Dry-erase markers
  • Easel Board with markers
    • Will be used by attendees to draw their design diagrams
  • The WIFI provided should allow ssh/rdp out

Misc.

  • Attendees are responsible for their Flight/Transportation/Lodging

Azure Transit Network Deployment with Native VNet Peering

Unless you were living under a rock :-), everyone knows that Microsoft Azure is picking really fast in the Enterprise market. Understanding the Multi-Cloud Network (MCN) architecture is a must for Network/Cloud architects and Transit Networking is one of the Cloud Core element of MCN architecture.

This blog will discuss the deployment of an Azure Transit Network design pattern called “Aviatrix Transit with VNet Peering”.

Refer to my previous blog for different Azure Transit Network design options and pros/cons

Topology

We will be using following topology to deploy the Azure Transit Networking with native VNet peering with Aviatrix Transit GW in the Transit/Hub VNet.

Simple and Quick Deployment

The process of deploying the Azure Transit Network is extremely simple by using the Aviatrix Controller UI. You need to perform 3 simple steps and the entire setup can be up and running in about 30 min.

IP Addressing

I will be using following IP addressing scheme for this deployment (also shown in the topology diagram above)

  • Aviatrix-Transit-VNet-Central  10.160.0.0/16
    • Aviatrix-Transit-GW-Central  10.160.0.4/22
  • Aviatrix-Spoke-VNet-Central-1  10.161.0.0/16
  • Aviatrix-Spoke-VNet-Central-2  10.162.0.0/16

Region Selection

I am using US-Central region for this deployment but it is not really mandatory to deploy all the hub and spokes in the same region. They all can be spread across multiple regions as well.


Step#1: Aviatrix Controller Creates Azure VNets and Resource Group (RG)

Use Aviatrix Controller UI to creates the Azure VNets. The process allows to pick the VNet region and CIDR range. A corresponding and unique Azure Resource Group (RG) is also created at this step.

Behind the Scene

Here is what happens in the background in Azure (you can verify it from the Azure portal itself)

  • Aviatrix first creates a new Azure Resource Group
  • Then Aviatrix Controller creates VNet in that RG
  • Aviatrix Controller also creates four /20 subnets from /16 CIDR range
    • Controller makes it easy and select the subnet range automatically. For example for /24 CIDR, controller will create /28 subnets
  • Controller then creates a User Route-Table and associates subnets in the newly created User Route-Table

Les us take a look at the screen-shots from Azure portal for the above mentioned bullet points

Aviatrix first creates a new Azure Resource Group

Aviatrix Controller creates VNet in the newly created RG

Azure Virtual Network (VNet) Properties

Aviatrix Creates four /20 subnets. 2 public and 2 private
User Route Table created without any routes. Only “user” subnets associated with the user-route table. Public subnets are not associated with any route table at this stage


Step#2: Aviatrix Controller Deploys Transit GW VM in Transit VNet:RG

Now deploy Aviatrix Transit GW VM in Azure using the Aviatrix Controller UI. Make sure to deploy this VM in the Azure Public subnet that was created in Step#1.

Aviatrix Controller deploys the AVX-Transit GW in the Hub/Transit VNet

The controller UI shows the progress of this deployment as shown below

[03:47:10] Starting to create ARM GW Aviatrix-Transit-GW-Central.
[03:47:11] Connected to Azure ARM.
[03:47:22] Deploying virtual machine...
[03:50:32] Deploy virtual machine done.
[03:50:33] License check is complete.
[03:50:33] Added GW info to Database.
[03:50:34] Aviatrix-Transit-GW-Central AVX SQS Queue created. 
[03:50:34] Create message queue done.
[03:50:34] Initializing GW.....
[03:50:34] Copy configuration to GW Aviatrix-Transit-GW-Central done.
[03:50:35] Copy new software to GW Aviatrix-Transit-GW-Central done.
[03:50:35] Copy /etc/cloudx/cloudx_code_file.json.enc to GW Aviatrix-Transit-GW-Central done.
[03:50:35] Copy /etc/cloudx/cloudx_code_key_file.txt to GW Aviatrix-Transit-GW-Central done.
[03:50:35] Copy scripts to GW Aviatrix-Transit-GW-Central done.
[03:50:35] Copy sdk to GW Aviatrix-Transit-GW-Central done.
[03:50:39] Copy libraries to GW Aviatrix-Transit-GW-Central done.
[03:50:39] Installing software ....
[03:50:41] Issuing certificates....
[03:50:41] Issue certificates done
[03:51:14] GW software started.
[03:51:38] Software Installation done.
[03:51:40] Run self diagnostics done. 

Behind the Scene

At this stage the Aviatrix Transit VM is deployed. Let me show you what happens behind the scene by logging into the Azure Portal

Aviatrix Transit Resource Group now has the AVX-Transit VM/GW

Pay attention to the above screen shot. Following are the resources that Aviatrix Controller orchestrate behind the scene

  • Creates new VNet
  • Creates VM in the newly created VNet (see screen-shot below)
  • Creates network interface for the VM
  • Allocated Public IP address to the VM
  • Creates Availability set and assign it to VM
  • Creates NSG (Azure Network Security Group) and assign it to the VM
  • Creates storage account
  • Assign the user-route-table to VM subnet

Following screen shows the Aviatrix Transit GW VM details

Aviatrix Transit GW VM details

Inbound Rules Added by Aviatrix Controller for Transit-GW at the NIC Level
Outbound Rule Added by Aviatrix Controller for Transit-GW
NSG Created by Aviatrix (all rules on one screen)


Step#3: Aviatrix Controller Orchestrate Azure Native Transitive Peering

Now attach Azure ARM Spoke through Native Peering using Aviatrix Controller

Native Peering between Spoke and Transit VNets

Repeat the above step for the second Spoke VNet as well.

Behind the Scene

  • Aviatrix Controller creates native peering
  • Creates Route Table I
  • nstall RFC1918 routes in the spoke VNets and points it to Transit VNet
Native Peering Created by Aviatrix Controller

Following two screen shot show that Aviatrix controller automatically creates a bi-directional peering relationship between Transit GW and Spoke VNet

Peering Details from Aviatrix Transit to Spoke-1 VNet

Peering Details from Spoke-1 VNet to Aviatrix Transit GW VM
Aviatrix Manages Route Table Creation and Life-Cycle
“Aviatrix-Spoke-VNet-Central-1 public” Route Table points to Aviatrix Transit GW IP as the Next Hope
Similarly, Aviatrix-Spoke-VNet-Central-2 public” Route Table points to Aviatrix Transit GW IP as next hops

No need for route(s) is needed in transit VNet routing table because the routing in the Transit VNet is handled by the Aviatrix GW itself
Aviatrix Controller UI shows Azure peering information
Aviatrix Transit GW Routing Table
You can also verify Azure Spoke VNet Routing Table from the Aviatrix Controller UI

Transit Network Validation/Testing

Now we will deploy two test VMs to validate the deployment. The VMs will be deployed using CentOS OS and will get a public IP address so that we can ssh into them for testing purposes.

  • Azure-Test-VM-Spoke1 (Public: 13.67.225.200, Private: 10.161.0.4)
  • Azure-TestVM-Spoke2 (Public: 40.78.147.153, Private: 10.162.0.4)
Azure-Test-VM-Spoke1

Similarly a second Azure Test VM was created as shown in the screen-shot below

[shahzad@Azure-Test-VM-Spoke1 ~]$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.161.0.4  netmask 255.255.240.0  broadcast 10.161.15.255
        inet6 fe80::20d:3aff:fe9f:8c29  prefixlen 64  scopeid 0x20<link>
        ether 00:0d:3a:9f:8c:29  txqueuelen 1000  (Ethernet)
        RX packets 69315  bytes 39635034 (37.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 70959  bytes 14573682 (13.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 186  bytes 15872 (15.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 186  bytes 15872 (15.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[shahzad@Azure-Test-VM-Spoke1 ~]$ ping 10.162.0.4
PING 10.162.0.4 (10.162.0.4) 56(84) bytes of data.
64 bytes from 10.162.0.4: icmp_seq=1 ttl=63 time=1.95 ms
64 bytes from 10.162.0.4: icmp_seq=2 ttl=63 time=1.95 ms
64 bytes from 10.162.0.4: icmp_seq=3 ttl=63 time=2.24 ms
64 bytes from 10.162.0.4: icmp_seq=4 ttl=63 time=1.67 ms
64 bytes from 10.162.0.4: icmp_seq=5 ttl=63 time=2.19 ms
64 bytes from 10.162.0.4: icmp_seq=6 ttl=63 time=2.30 ms

Conclusion

Aviatrix makes it extremely simple and easy to deploy Azure Transit Network with native VNet peering option. The strength of the solution is that enterprises can build a common and unified transit solution in other cloud such as AWS and GCP and create a true multi-cloud network architecture with consistent operations and management options.

For more details, refer to Aviatrix documentation here.

Aviatrix User-VPN Deployment with AWS UDP Based NLB

The steps mentioned here are not supported yet. It should be treated as a workaround only.

Introduction

AWS LBN Supports UDP

AWS recently started supporting UDP protocol for its NLB (Network Load Balancer). Customers are looking to add support for UDP based NLB now. While this support will shortly be available in the product, there is a workaround to deploy such a topology.

Note: Aviatrix User-VPN GW uses TCP:443 for incoming heath-check probes

Deployment Overview

  • Create an Aviatrix GW (AGW) with VPN Access option but without enabling cloud-native ELB integration
  • This will create the AGW and by default it listens on UDP 1194 port
  • Manually create AWS NLB in the AWS console with the UDP option and port 1194
  • Manually create the target group with user-VPN AGW(s) in it
  • Make sure to override the health-check port and use TCP 443 for it

Deployment Details

Following screen shots shows a working deployment

Topology

Deploy Aviatrix User-VPN GW

Deploy an Aviatrix User-VPN GW with “VPN Access” checked and without enabling ELB using Aviatrix Controller.

Gateway config shows following in the Aviatrix diagnostics section. Notice the port 1194 here.

"VPN Service": {
"port": {
"1194": [
"up",
"reachable"
]
},

Create a new user and assign this user to the Aviatrix User-VPN GW

Create NLB in AWS Console

Create a UDP based NLB using the AWS console. Once the NLB is created, you will notice following config in the AWS console. Notice the DNS name for this NLB. This is the name we will use later in the config.

Name: shahzad-udp-nlb
arn:aws:elasticloadbalancing:ap-southeast-1:481151252831:loadbalancer/net/shahzad-udp-nlb/a2e01e8690702d00
DNS name: shahzad-udp-nlb-a2e01e8690702d00.elb.ap-southeast-1.amazonaws.com
(A Record)

AWS Network Load Balancer

Following screen also shows the name of the NLB and the DNS name associated with it.

NLB Listner

By default the AWS UDP based NLB listen at UDP port 1194 which is the port Aviatrix GW also listen at. You can observe it in the following screen

NLB Listener Details

Now we nee to create target group that will point to the Aviatrix User-VPN GW.

Health Check Configuration for Aviatrix GW

Make sure to modify the health-check port to 443 (by default it will be configured with 1194)

Modify User-VPN Certificate File

Download the User-VPN certificate file and replace the IP address with the DNS name of the AWS NLB.

client
comp-lzo
nobind
persist-key
persist-tun
auth-nocache
tun-mtu 1500
remote shahzad-udp-nlb-a2e01e8690702d00.elb.ap-southeast-1.amazonaws.com 1194
proto udp
mssfix
route-method exe
verb 3
route-delay 2
mute 20
reneg-sec 0
cipher AES-256-CBC
auth SHA512
key-direction 1
explicit-exit-notify
dev-type tun
dev tun

Connect VPN User

Now we connect using this profile. I am using OpenVPN connect client version 2.7.1.100.

User will be connected and will show in the Aviatrix Controller UI as well

Credits

Thank you Liming Xiang and Felipe Vasconcellos for reviewing and making adjustments to this post.

Azure Transit Network Design Patterns

Microsoft Azure is getting lot of Cloud business in the Enterprise market. Understanding the Multi-Cloud Network (MCN) architecture is a must for Network/Cloud architects and Transit Networking is one of the Cloud Core element of MCN architecture.

Aviatrix offer two distinct design patterns to build global transit in Azure and for cross-cloud connectivity. Both of them have their pros and cons that will be discussed later.

Azure Transit Network Design Patterns

The general recommendation and best practice from Aviatrix is to deploy “Aviatrix Transit with VNet GWs” design pattren.

Refer to my blog post to deploy Aviatrix Controller from the Azure Market Place.

Aviatrix Transit with VNet GWs

In this pattern, a transit network in Azure is built with a Transit gateway (aka hub gateway) in the centralized VNet (aka Transit VNet) and spoke gateways in the spoke VNets.

In this model Aviatrix Controller

  • Deploys the Aviatrix Azure Transit GW in the transit VNet
  • Deploys the Aviatrix Azure Spoke GW in the spoke VNets
  • Orchestrates vNET route creation and propagation
  • Connects spoke VNet to Transit/Hub GW and
  • Controls and steers the traffic accordion to the desired state
  • Provides life-cycle of management of the deployment
Aviatrix Transit with VNet GWs

Aviatrix recommendation is to use “Aviatrix Transit with VNet GW” design pattern

Aviatrix Transit with VNet GWs – Details

  • This model Provides encrypted connection between Spoke and Transit VNets
    • This is extremely important and in majority of the cases first requirement for enterprises
  • It can leverage Aviatrix ActiveMesh between Transit and Spoke VNets.
    • ActiveMesh provides huge advantages by building multiple Active/Active encrypted links between Hub and Spoke VNets.
    • It provides higher availability of service in case one or even two links go down.
    • ActiveMesh links actively participate in packer forwarding that results in increased throughput as compare to single encrypted link
  • Enterprises can leverage Advanced troubleshooting and visibility options provided by Aviatrix
    • For example Aviatrix GW allows enterprises to take tcpdump or packet capture of the traffic passing through
  • It allows enterprises to deploy consistent enterprise network architecture across multiple regions and multiple-clouds

Aviatrix Transit with VNet Peering

Aviatrix also offers building transit networks by natively peering the spoke VNets with the Aviatrix Transit GW. This model does not require any GWs in the spoke VNet.

In this model Aviatrix controller

  • Deploys the Aviatrix Azure Transit GW in the transit VNet
  • Orchestrates vNET route creation and propagation
  • Connects spoke VNet to Transit/Hub GW and
  • Controls and steers the traffic accordion to the desired state
  • Provides life-cycle of management of the deployment

Aviatrix recommendation is to use “Aviatrix Transit with VNet GW” design pattern

Aviatrix Transit with VNet Peering – Details

  • This model does not provide encryption between spoke and transit VNets
  • Less visibility into the traffic and overall operations
  • There is no option to take tcpdump or packet capture at the spoke VNet as compare to the other other model
  • Ops team depends on the Azure tools and options to troubleshoot rather than using Aviatrix’s rich set of troubleshooting and visibility tools
  • No gateways in the spoke VNets means no IPSec tunnel charges between spoke and transit VNets
  • The Aviatrix ActiveMesh behavior is different in this model as compare to the previous one. Since we do not have Aviatrix Spoke GW in the spoke VNet, the behavior is more like Primary/Backup links
    • If spoke VNet has multiple route tables, the Aviatrix controller will configure both primary and backup Transit GWs as the default gateway for different route tables. By doing so, we can achieve load balancing for Spoke VNet outbound traffic
    • If Spoke VNet has only one route table, this route table will point to the private IP of the primary Transit GW until that GW fails. In case of primary Transit GW failure, the controller will automatically update the Spoke VNet’s route table to point to the private IP of the backup Transit GW
  • The throughput between Transit and Spoke VNet is what Azure native VNet peering provides (At the time of writing this article, Azure did not publish those numbers)
  • Aviatrix has done performance testing using different Azure VM sizes. Refer to following results as a reference
Azure Throughput

Conclusion

Transit Network is an integral part of Multi-Cloud Network (MCN) architecture. It fits right into the Cloud Core pillar of MCN architecture. Aviatrix offer two different Azure Transit Network design patterns to cater various enterprise needs in this space.

In my next blog, I will discusses the Azure Transit Network deployment with native VNet peering in more details.

Azure Aviatrix Controller Deployment

Aviatrix Controller (AVX-CTRL) can be deployed in AWS, Azure, GCP or OCI. Only one single AVX-Controller needed for an enterprise multi-cloud deployment. A single AVX-Controller can control, manage and operate resources in an all the public clouds.

Recently I noticed that more and more enterprises are asking to deploy Aviatrix Controller in Azure. Hence I decided to write this short blog with screen shots.

Azure Cloud Portal

This blog assumes that you are somewhat familiar with the Azure Cloud Portal.

1- Login to Azure Portal @ https://portal.azure.com
2- Click on the Marketplace link (this could be in a different place depending on your customization) as shown in the screen-shot here

Azure Marketplace

3- Search Aviatrix in Azure Marketplace (as shown in the screen-shot below)

Search Aviatrix and select Aviatrix Platform Bundle – PAYG

Here you need to select the Aviatrix Bundle – PAYG.

After that, you will see multiple Aviatrix plans listed on Azure Marketplace page. These plans listed based on your enterprise needs and use-cases. In this deployment I have picked pick “Aviatrix Networking Platform Bundle”

Aviatrix Software planDescription
Multi-service units and SSL VPN users per BYOLEach FQDN deployment, site to cloud tunnel, or multi-cloud tunnel is a service unit. You can configure as many as SSL VPN users to access your private cloud with MFA and SMAL on Aviatrix Secure Networking Platform.
The description of the plan selected for this customer deployment

Deploy Aviatrix Controller VM in Azure

At this stage, Azure will created the Aviatrix Controller VM. All the steps onward are related to Azure’s Aviatrix VM creation.

Enter basic VM information. Select the default size for now.
Select default disk selection option

Select the Resource Group (RG) for Aviatrix Controller VM deployment. Aviatrix will create the NSG with proper security automatically

You can leave the default setting here

Leave this section with default config.

Tags are important – apply at least name tag to this VM

At this state the Aviatrix Controller VM deployment is underway. It will take about 3 to 5 min for this process to compelete.

Conclusion

Now that your Aviatrix Controller VM is ready, you can login to the UI by browsing the Public IP address of your controller. The default user name is admin and default password is the “Private IP Address” of Aviatrix Controller VM.

Aviatrix CloudWAN Deployment with AWS Global Accelerator

In this blog post I explained what Aviatrix CloudWAN solution is. Here let us actually deploy it and appreciate the simplicity of implementation.

Recently I worked with an enterprise (lets call is netJoints Inc., as I cannot share the actual name of my customer) and connected their branches (Cisco Routers) in various regions to Aviatrix Global Transit Network.

I will show how to connect a branch in Singapore.

Step1 – Register Cisco Router to Aviatrix Controller

Step2 – Attach Cisco Router to Public Cloud Transit

In this step Aviatrix Controller will automatically builds IPSec tunnel to connect branch router in Singapore to Public Cloud Transit Network. This Transit network could be

1- Aviatrix Transit GW (AVX-TGW)
2- AWS Transit GW (AWS-TGW)

AVX-TGW is preferred option as it allows to build a true Global Transit across multiple-regions and multiple-clouds. AWS-TGW is limited to only single region and obviously is only available in AWS, hence is not recommended for enterprise multi-cloud customers.

Prepare to attach:

Attach to cloud now:

Following diagram shows Singapore-Br1 attached to AVX-TGW

You can also get IPSec VPN tunnel details under Site2Cloud menu

Click on the tunnel to see the routes it learned via BGP

Cisco Router Configuration

Following is what Aviatrix Controller has configured in the background

IPSec Config

BGP Config

AWS Global Accelerator Configuration

Following is what Aviatrix Controller configured in the AWS