Over the past few months, I’ve had a similar recurring question come up from customers, “How do I access my IBM PowerVS LPARs through my IBM Cloud Direct Link”.
While there is documentation available that shows how to conceptually do it, a practical end-to-end guide does not exist.
Let’s first start by reviewing the target architecture we want to create and the cloud components necessary. Our goal is we want to have the source network of 10.240.128.0/18 in the on-premises data center reach the 172.16.0.0/18 network in PowerVS.
Where some complexity comes in is because PowerVS today does not directly connect into the Transit Gateway. So, we need to build the connectivity through the Juniper to reach the Direct Link connected to the Transit Gateway.
As you can see there are 4 BGP sessions (there could be more if there are redundant Direct Links) and 3 GRE tunnels that we need to configure.
BGP #1 – Between the Customer Edge Router and the Direct Link Gateway
BGP #2 & #3 – Between the Juniper vSRX and the PowerVS instances. These connections require GRE tunnels to be built.
BGP #4 – Between the Transit Gateway and the Juniper vSRX. This requires a GRE tunnel to be built between the Transit Gateway and the Juniper vSRX
Now that we understand what the architecture will look like and the pieces we need let’s start configuring.
Juniper Deployment
The first piece we want to deploy is the Juniper vSRX itself. It can be found under the Gateway Appliances tile in the catalog. There will be options for a Virtual Router Appliance/ Vyatta as the vendor. Ensure Juniper is listed under the vendor dropdown.
We want to deploy the Juniper vSRX into the same region and ideally the data center that the PowerVS instance will be. In my case, Dallas is the region and I have selected the DAL10 data center.
Once the Juniper is provisioned it can be found under the Classic Infrastructure > Gateway Appliances section. On the page there will be a private IP address listed, this is important to note as this address will be used to terminate the GRE tunnels in the subsequent sections.
Direct Link Provisioning
Next, we will order the Direct Link. This has the potential to take the longest if there is a need to get a new circuit brought to the data center/PoP location. The ordering part on the cloud portal is relatively quick. When ordering make sure the tile used is a Direct Link Connect (often referred to as a Direct Link Connect 2.0) or Direct Link Dedicated 2.0 as shown below. Do not use any Direct Link with the word classic in the name otherwise it will not be able to connect to a Transit Gateway and this architecture will not work.
The Direct Link order page will ask for which location to establish the connection, the provider being used, IP and BGP information for peering. This would be BGP session #1 that we referred to earlier. In the example screenshots, I have selected to provision the Direct Link in Washington with Global Routing to reach the Transit Gateway in Dallas. The Connection section at the end of the order form will ask if the Direct Link should be connected to ‘Direct resources’ or ‘Transit Gateway’. Select Transit Gateway then create the Direct Link. Depending on the type of Direct Link (Dedicated or Connect) it may require a physical cross-connect, or a virtual connection to be established through an exchange provider. This can happen in parallel while the rest of the environment is being set up.
Power Systems Virtual Server Cloud Connections
Next, we will want to provision the PowerVS service. Look for the Power Systems Virtual Server tile in the catalog.
We want to provision into the region (ideally the datacenter) that we provisioned the Juniper vSRX. Once the PowerVS resource is provisioned, it can be found in the resource from the resource list. We will then need to create a subnet in the PowerVS instance. This subnet is where our test LPAR will be provisioned to.
After we create the subnet, we will create our first GRE tunnel to connect PowerVS to Classic. From the Cloud Connections on the left-hand menu click Create Connection. Give it a name and select the speed of 5Gbps. If the Juniper vSRX is in the same region as the PowerVS instance the global routing toggle can be left off. Under Endpoint destinations, select the check box for Classic Infrastructure and enable the toggle for GRE. This will enable two form boxes:
GRE destination IP
This should be the private IP address of the Juniper vSRX. Our example is ‘10.94.176.70’. It can be found in the Classic Infrastructure > Gateway Appliances section.
GRE subnet
This will be used by PowerVS to create the local GRE IP addresses as well as the local/remote tunnel addresses. We will use 192.168.10.0/29 as our subnet.
PowerVS will automatically calculates the addresses as follows:
192.168.10.1 – Local GRE IP
192.168.10.5 – Local GRE tunnel address
192.168.10.6 – Remote GRE tunnel address
Next, in the subnets section click the attach existing button and select the subnet created earlier.
Click Create Connection. It will take a few minutes to create the connection. While it is creating repeat the process again to create the second Cloud Connection/GRE tunnel but this time use 192.168.20.0/29 as the GRE subnet.
It’s important to note that for these BGP sessions, PowerVS provides the ASN and they cannot be changed. So, the ASN information for BGP #2 & 3 would be:
- PowerVS: 64999 (except for WDC04 which uses 64995)
- vSRX: 64880
Once both Cloud Connections are completed, we can proceed to provision the Transit Gateway and BGP session #4.
Transit Gateway Provisioning
From the cloud catalog, provision the Transit Gateway into the same region as the Juniper vSRX and PowerVS instance.
Since the Transit Gateway is connecting to Classic Infrastructure, we can use Local Routing and avoid bandwidth charges. Enabling global Routing would allow the Transit Gateway to connect to VPCs that are in different regions.
On the order form under the Connections section, select Classic Infrastructure to provide the connection to classic infrastructure. Also, add a connection for the Direct Link created earlier. Then click create.
Once the Transit Gateway is provisioned click on the provisioned gateway and click Add Connection. In the Network connection dropdown, select GRE tunnel. In the Zone drop-down, select the zone that corresponds to the datacenter that the Juniper vSRX and the PowerVS instance are in. My Juniper was provisioned in DAL10 so I selected Zone 1 for the GRE connection. Look at the following doc page for the zone to datacenter mapping: https://cloud.ibm.com/docs/overview?topic=overview-locations#mzr-table.
Under base connection, select the Classic Infrastructure connection that was previously created. Then we must fill out the GRE tunnel information. It is a bit different from the PowerVS GRE page where all the IPs were calculated from the GRE subnet put in. With Transit Gateway we must explicitly state the IPs. I will use the 192.168.30.0/29 subnet and keep the format consistent with how PowerVS provisioned the GRE IPs.
Remote Gateway IP – Juniper vSRX private IP
Local Gateway IP – 192.168.30.1
Remote tunnel IP – 192.168.30.6
Local tunnel IP – 192.168.30.5
(optional) Remote BGP ASN – We will keep this the same as what PowerVS had assigned 64880 for BGP #2 & 3. Transit Gateway will have an ASN assigned automatically.
Once the GRE connection is provisioned make note of the assigned Transit Gateway ASN.
The last step is to configure the Juniper vSRX provisioned earlier and setup the routing between the GRE tunnels.
Juniper Configuration
The configuration below will setup everything needed with the example IPs discussed. Change the IPs, ASNs, and prefixes in the import/export maps as required. The bolded areas are what may need to be changed. SSH into the Juniper vSRX private IP using the admin credentials, type ‘configure’ to enter configuration mode and paste the commands. Type ‘commit’ to save and apply the configuration.
#Create GRE tunnels, change source address to transit IP, destination IP to TGW & power local gateway
set interfaces gr-0/0/0 unit 0 tunnel source 10.94.176.70
set interfaces gr-0/0/0 unit 0 tunnel destination 192.168.30.1
set interfaces gr-0/0/0 unit 0 family inet address 192.168.30.6/30
set interfaces gr-0/0/0 unit 1 tunnel source 10.94.176.70
set interfaces gr-0/0/0 unit 1 tunnel destination 192.168.10.1
set interfaces gr-0/0/0 unit 1 family inet address 192.168.10.6/30
set interfaces gr-0/0/0 unit 2 tunnel source 10.94.176.70
set interfaces gr-0/0/0 unit 2 tunnel destination 192.168.20.1
set interfaces gr-0/0/0 unit 2 family inet address 192.168.20.6/30
#Create BGP groups, match local address IPs to above net address, neighbour IP other side of tunnel IP
set protocols bgp group TGW1 local-address 192.168.30.6
set protocols bgp group TGW1 family inet unicast
set protocols bgp group TGW1 peer-as 4201065540
set protocols bgp group TGW1 neighbor 192.168.30.5 local-as 64880
set protocols bgp group TGW1 import tgw-import-policy
set protocols bgp group TGW1 export tgw-export-policy
set protocols bgp group PVSCC1 local-address 192.168.10.6
set protocols bgp group PVSCC1 family inet unicast
set protocols bgp group PVSCC1 peer-as 64999
set protocols bgp group PVSCC1 neighbor 192.168.10.5 local-as 64880
set protocols bgp group PVSCC1 import powervs-import-policy
set protocols bgp group PVSCC1 export powervs-export-policy
set protocols bgp group PVSCC2 local-address 192.168.20.6
set protocols bgp group PVSCC2 family inet unicast
set protocols bgp group PVSCC2 peer-as 64999
set protocols bgp group PVSCC2 neighbor 192.168.20.5 local-as 64880
set protocols bgp group PVSCC2 import powervs-import-policy
set protocols bgp group PVSCC2 export powervs-export-policy
#Allow only exact match to be imported/exported to BGP
set policy-options policy-statement tgw-import-policy term allowed_from_tgw from route-filter 10.240.128.0/18 exact
set policy-options policy-statement tgw-import-policy term allowed_from_tgw then accept
set policy-options policy-statement tgw-import-policy term others then reject
set policy-options policy-statement powervs-import-policy term allowed_from_power from route-filter 172.16.0.0/18 exact
set policy-options policy-statement powervs-import-policy term allowed_from_power then accept
set policy-options policy-statement powervs-import-policy term others then reject
set policy-options policy-statement powervs-export-policy term advertised_to_power from route-filter 10.240.128.0/18 exact
set policy-options policy-statement powervs-export-policy term advertised_to_power then accept
set policy-options policy-statement powervs-export-policy term others then reject
set policy-options policy-statement tgw-export-policy term advertised_to_tgw from route-filter 172.16.0.0/18 exact
set policy-options policy-statement tgw-export-policy term advertised_to_tgw then accept
set policy-options policy-statement tgw-export-policy term others then reject
#security zones/policies
set security zones security-zone TGW host-inbound-traffic system-services all
set security zones security-zone TGW host-inbound-traffic protocols all
set security zones security-zone TGW interfaces gr-0/0/0.0
set security zones security-zone POWERVS host-inbound-traffic system-services all
set security zones security-zone POWERVS host-inbound-traffic protocols all
set security zones security-zone POWERVS interfaces gr-0/0/0.1
set security zones security-zone POWERVS interfaces gr-0/0/0.2
set security policies from-zone TGW to-zone POWERVS policy allow match source-address any destination-address any application any
set security policies from-zone TGW to-zone POWERVS policy allow then permit
set security policies from-zone POWERVS to-zone TGW policy allow match source-address any destination-address any application any
set security policies from-zone POWERVS to-zone TGW policy allow then permit
set security policies from-zone POWERVS to-zone POWERVS policy allow match source-address any destination-address any application any
set security policies from-zone POWERVS to-zone POWERVS policy allow then permit
set security policies from-zone TGW to-zone TGW policy allow match source-address any destination-address any application any
set security policies from-zone TGW to-zone TGW policy allow then permit
#set static route so gateway IP that is terminating remote side of tunnel is reachable via transit ip in underlay. The next hop would be the gateway address of the subnet for the vsrx private IP
set routing-options static route 192.168.20.1/32 next-hop 10.94.176.65
set routing-options static route 192.168.10.1/32 next-hop 10.94.176.65
set routing-options static route 192.168.30.1/32 next-hop 10.94.176.65
#local firewall policies to allow ping on tunnel IPs
set firewall filter PROTECT-IN term PING from destination-address 192.168.10.6/32
set firewall filter PROTECT-IN term PING from destination-address 192.168.20.6/32
set firewall filter PROTECT-IN term PING from destination-address 192.168.30.6/32
#local firewall policies to allow BGP on tunnel IPs
set firewall filter PROTECT-IN term BGP from destination-address 192.168.10.6/32
set firewall filter PROTECT-IN term BGP from destination-address 192.168.20.6/32
set firewall filter PROTECT-IN term BGP from destination-address 192.168.30.6/32
set firewall filter PROTECT-IN term BGP from source-address 192.168.10.5/32
set firewall filter PROTECT-IN term BGP from source-address 192.168.20.5/32
set firewall filter PROTECT-IN term BGP from source-address 192.168.30.5/32
set firewall filter PROTECT-IN term BGP from protocol tcp
set firewall filter PROTECT-IN term BGP from port 179
set firewall filter PROTECT-IN term BGP then accept
After applying these commands, we should be able to run ‘Show bgp summary’ in the Juniper and see 3 BGP sessions over the GRE tunnels in established mode.
A ‘show route protocol bgp’ should also display the expected routes. It will be important the Direct Link is properly setup at this point otherwise we will not see the routes learned from the Direct Link.
Provision an LPAR and test a ping. Pings between our networks from the Direct Link to PowerVS should now work.
Troubleshooting
If connectivity isn’t working as expected here are a few things to check to help narrow down problems:
- Check that the expected routes are in the routing table
- Use the ‘Routes’ feature on the Transit Gateway to validate what subnets are being learned from which connections
- If the Transit Gateway does not have the routes learned from the Direct Link look at the Direct Link setup
- If the Transit gateway does not have the PowerVS routes learned from the vSRX GRE connection look at the vSRX/TGW connection
- If the vSRX does not have routes for PowerVS look at the PowerVS/vSRX connection
- If routes are not in the routing table check the policy maps for the BGP sessions and ensure masks length matches the prefixes specified
- Check the status of the BGP sessions on the Juniper and ensure they are established
- Check that the GRE remote tunnel interfaces are pingable
- Check that the GRE remote interfaces are pingable
- Double-check ASN settings
The Cloud Guy 