In general, we found most AWS users/customers struggle with setting up networking and DNS resolution in a multi account AWS environment. Establishing a solid platform for networking and DNS resolution is of utmost importance to ensure your AWS environment can easily scale efficiently while maintaining a lower operating cost and strong security posture. In this post we focus primarily on different strategies for setting up Hybrid DNS resolution using AWS Route53 resolvers. AWS re:Invent 2019 presented with a deep dive session on different approaches to setup AWS Hybrid DNS resolution in an enterprise environment setup. Our solutions mentioned below, resemble solutions 1,2 & 4 discussed in their deep dive session.

Nomenclature

• Hub AWS account – AWS account used as the central hub for DNS resolution. In all diagrams, this account is depicted as “org-core-network”.
• Hub VPC – VPC which reside in hub AWS account.
• Spoke AWS account – all AWS accounts that are not the hub AWS account.
• Spoke VPC – VPC which reside in a spoke AWS account.

Solution 1

Design Overview

The first solution uses a centralised model which directs DNS requests from all spoke VPCs directly to Route53 inbound resolver endpoint which is provisioned and configured in hub VPC. This is implemented by configuring the domain name servers property of VPC DHCP options set in spoke VPCs to the target IP address of inbound resolver endpoint interfaces. Route53 private hosted zones provisioned in all spoke AWS accounts are associated to hub VPC. There are Route53 resolver rules configured in hub AWS account and associated ONLY to hub VPC which forwards DNS query requests to on-premise DNS servers via Route53 outbound resolver endpoint to resolve any corporate domain entries.

Assumptions

• Access to hub and all spoke AWS accounts with sufficient IAM permissions to execute necessary tasks/commands.
• Network connectivity between hub VPC and all spoke VPCs either via AWS Transit Gateway (preferable) or AWS VPC peering connection. Please ensure VPC Network ACLs are configured to allow TCP/UDP port 53 from all spoke VPCs to hub VPC Route53 resolver endpoint interfaces.
• Network connectivity between hub VPC and on-premise network either via AWS Direct Connect and/or site-to-site VPN connection. Please ensure VPC Network ACL in hub VPC is configured to allow inbound & outbound TCP/UDP port 53 from & to on-premise network IP CIDR respectively for bi-directional DNS resolution.

Implementation

1. Associate AWS Route53 private hosed zones in all spoke AWS accounts (that require DNS resolution) with hub VPC. There is no CloudFormation support for associating a Route53 hosted zone with a VPC in another AWS account. Therefore, this can be only implemented via AWS CLI and/or SDK. Please refer to AWS Route53 documentation to create VPC association authorisation and associate VPC with hosted zone between two AWS accounts.
2. Provision AWS Route53 resolver endpoints both inbound and outbound in hub VPC. It is recommended to provision endpoints across all availability zones to achieve high availability of services. Specify a pre-determined IP address for each interface endpoint provisioned in a given availability zone.
3. Create following AWS Route53 resolver rule(s) in hub AWS account.
   1. Target IPs of on-premise DNS servers are configured as a forwarding rule to resolve any DNS queries for a target domain hosted on-premise. The resolver endpoint ID should be pointing to AWS Route53 outbound resolver endpoint.
4. Associate Route53 resolver rule(s) created in step 3 with hub VPC.
5. Create a VPC DHCP option set in all spoke VPCs with domain name servers property configured to IP address of AWS Route53 inbound resolver endpoint interfaces created in step 2.

Solution 2

Design Overview

Similar to solution 1, solution 2 also uses a centralised model which uses Route53 resolvers (both inbound & outbound) provisioned and configured in hub VPC. However, not all DNS requests are forwarded to AWS Route53 resolver endpoints. Instead DNS queries that targeted towards domains that are hosted on-premise corporate servers and/or Route53 private hosted zones are ONLY forwarded to resolver endpoints. This is achieved via AWS Route53 resolver rules which are provisioned and configured in hub AWS account. These rules are shared to all spoke AWS accounts via AWS RAM and thereafter, associated to all VPCs (including hub VPC). Every other DNS query will be resolved by it’s own corresponding VPC DNS resolver (VPC DNS IP).

Assumptions

• Access to hub and all spoke AWS accounts with sufficient IAM permissions to execute necessary tasks/commands.
• Network connectivity between hub VPC and all spoke VPCs either via AWS Transit Gateway (preferable) or AWS VPC peering connection. Please ensure VPC Network ACLs are configured to allow TCP/UDP port 53 from all spoke VPCs to hub VPC Route53 resolver endpoint interfaces.
• Network connectivity between hub VPC and on-premise network either via AWS Direct Connect and/or site-to-site VPN connection. Please ensure VPC Network ACL in hub VPC is configured to allow inbound & outbound TCP/UDP port 53 from & to on-premise network IP CIDR respectively for bi-directional DNS resolution.

Implementation

1. Associate AWS Route53 private hosed zones in all spoke AWS accounts (that require DNS resolution) with hub VPC. There is no CloudFormation support for associating a Route53 hosted zone with a VPC in another AWS account. Therefore, this can be only implemented via AWS CLI and/or SDK. Please refer to AWS Route53 documentation to create VPC association authorisation and associate VPC with hosted zone between two AWS accounts.
2. Provision AWS Route53 resolver endpoints both inbound and outbound in hub VPC. It is recommended to provision endpoints across all availability zones to achieve high availability of services. Specify a pre-determined IP address for each interface endpoint provisioned in a given availability zone.
3. Create following AWS Route53 resolver rule(s) in hub AWS account.
   1. Target IPs of on-premise DNS servers are configured as a forwarding rule to resolve any DNS queries for a target domain hosted on-premise. The resolver endpoint ID should be pointing to AWS Route53 outbound resolver endpoint.
   2. Target IPs of AWS Route53 inbound resolver endpoints (created in step 2) are configured as a forwarding rule to resolve any DNS queries for a target domain hosted in AWS Route53 private hosed zones, The resolver endpoint ID should be pointing to AWS Route53 outbound resolver endpoint.
4. Share AWS Route53 resolver rule(s) (created in step 3) via AWS RAM with all spoke AWS accounts that require DNS resolution.
5. Associate shared AWS Route53 rule(s) (implemented in step 4) with all VPCs that require DNS resolution.

Solution 3

Design Overview

The final solution is a bit complex in comparison to previous two solutions mentioned above. The model uses a decentralised approach where any AWS Route53 private hosted zone provisioned in an AWS account is associated with all VPCs that require DNS resolution. It is the most cost-effective solution but require a lot of administration and setup work due to the complex architecture. Both AWS Route53 inbound and outbound resolver endpoints are created in hub VPC. The outbound resolver endpoint forwards DNS requests to resolve entries hosted on-premise DNS servers. This is implemented via AWS Route53 resolver rules which are provisioned and configured in hub AWS account. These rules are then shared to all spoke AWS accounts via AWS RAM and consequently associated to all VPCs (including hub VPC). Meanwhile the inbound resolver endpoint is used to resolve DNS requests originating from on-premise DNS servers.

Assumptions

• Access to hub and all spoke AWS accounts with sufficient IAM permissions to execute necessary tasks/commands.
• Network connectivity between hub VPC and all spoke VPCs either via AWS Transit Gateway (preferable) or AWS VPC peering connection. Please ensure VPC Network ACLs are configured to allow TCP/UDP port 53 from all spoke VPCs to hub VPC Route53 resolver endpoint interfaces.
• Network connectivity between hub VPC and on-premise network either via AWS Direct Connect and/or site-to-site VPN connection. Please ensure VPC Network ACL in hub VPC is configured to allow inbound & outbound TCP/UDP port 53 from & to on-premise network IP CIDR respectively for bi-directional DNS resolution.

Implementation

1. Associate a AWS Route53 private hosed zones with all VPCs (that require DNS resolution). This is repeated for all AWS Route53 private hosted zones. There is no CloudFormation support for associating a Route53 hosted zone with a VPC in another AWS account. Therefore, this can be only implemented via AWS CLI and/or SDK. Please refer to AWS Route53 documentation to create VPC association authorisation and associate VPC with hosted zone between two AWS accounts.
2. Provision AWS Route53 endpoints both inbound and outbound in hub VPC. It is recommended to provision endpoints across all availability zones to achieve high availability of services. Specify a pre-determined IP address for each interface endpoint provisioned in a given availability zone.
3. Create following AWS Route53 resolver rule(s) in hub AWS account.
   1. Target IPs of on-premise DNS servers are configured as a forwarding rule to resolve any DNS queries for a target domain hosted on-premise. The resolver endpoint ID should be pointing to AWS Route53 outbound resolver endpoint.
4. Share AWS Route53 resolver rule(s) (created in step 3) via AWS RAM with all AWS accounts that require DNS resolution.
5. Associate shared AWS Route53 rule(s) (implemented in step 4) with all VPCs that require DNS resolution.

Comparison

Although implementation steps look identical in all solutions, there are key differences between them. As mentioned above, AWS re:Invent 2019 deep dive session covers pros and cons about each solution in detail. However, our comparison focuses on metrics such as cost, ease of setup/maintenance and resiliency/availability.

Conclusion

Solution 1 can be setup / managed relatively easily but it comes with a drawback of having a higher operating cost. Also, in certain circumstances during heavy network traffic scenarios, there might be instances where AWS Route53 resolver endpoints reach request limits which consequently result in downtime of services. This is because all DNS requests from all VPCs are forwarded to AWS Route53 resolver endpoints which creates a bottleneck and  heavy load on the interfaces.

Solution 3 was considered as best practice in AWS re:Invent 2019 deep dive session. However, implementation setup and/or maintenance is cumbersome which can easily lead to misconfiguration of resources overtime. Therefore, it is essential to have a mature automated framework to efficiently manage a stable environment. In an AWS organisation where there are AWS Route53 private hosted zones provisioned in hundreds of AWS accounts, this approach becomes a nightmare to manage even with sufficient automation scripts.

Solution 2 is relatively easier to setup / manage than solution 3 since it follows the same centralised approach as solution 1. It has a low-to-medium operating cost because DNS requests that are used to resolve entries in domains that are hosted on-premise DNS servers and/or AWS private hosted zones are ONLY forwarded to AWS resolver endpoints. Due to this fact, it also has a low-to-medium risk of hitting request limits even in heavy traffic scenarios. Therefore, solution 2 is our recommended solution and go to approach for most implementations.

If you require a fully automated framework which includes complete customisation for implementing either solution mentioned above, please check out our AWS Foundations service offering.