Advanced real-world Cisco Application Centric Infrastructure (ACI) monitoring and troubleshootingForewords written by Yusuf Bhaiji, Director of Certifications, Cisco Systems; and Ronak Desai, VP of Engineering for the Data Center Networking Business Unit, Cisco Systems.This expert guide and reference will help you confidently deploy, support, monitor, and troubleshoot ACI fabrics and components. It is also designed to help you prepare for your Cisco DCACIA (300-630) exam, earning Cisco Certified Specialist-ACI Advanced Implementation certification and credit toward CCNP Data Center certification if you choose.Authored by three leading Cisco ACI experts, it combines a solid conceptual foundation, in-depth technical knowledge, and practical techniques. It also contains proven features to help exam candidates prepare, including review questions in most chapters, and Key Topic icons highlighting concepts covered on the exam.The authors thoroughly introduce ACI functions, components, policies, command-line interfaces, connectivity, fabric design, virtualization and service integration, automation, orchestration, and more. Next, they introduce best practices for monitoring and management, including the use of faults, health scores, tools, the REST API, in-band and out-of-band management techniques, and monitoring protocols. Proven configurations are provided, with steps for verification. Finally, they present advanced forwarding and troubleshooting techniques for maximizing ACI performance and value.ACI Advanced Monitoring and Troubleshooting is an indispensable resource for every data center architect, engineer, developer, network or virtualization administrator, and operations team member working in ACI environments.

Foreword by Yusuf Bhaiji xxviii Foreword by Ronak Desai xxix Introduction xxx PART I: INTRODUCTION TO ACIChapter 1 Fundamental Functions and Components of Cisco ACI 1 ACI Building Blocks 8 Hardware Specifications 8 ACI Key Concepts 14 Control Plane 15 Data Plane 17 VXLAN 17 Tenant 18 VRF 19 Application Profile 20 Endpoint Group 21 Contracts 22 Bridge Domain 24 External Routed or Bridged Network 25 Summary 26 Review Key Topics 26 Review Questions 27 Chapter 2 Introduction to the ACI Policy Model 31 Key Characteristics of the Policy Model 32 Management Information Tree (MIT) 33 Benefits of a Policy Model 37 Logical Constructs 37 Tenant Objects 38 VRF Objects 39 Application Profile Objects 40 Endpoint Group Objects 41 Bridge Domain and Subnet Objects 43 Bridge Domain Options 45 Contract Objects 46 Labels, Filters, and Aliases 48 Contract Inheritance 49 Contract Preferred Groups 49 vzAny 50 Outside Network Objects 51 Physical Construct 52 Access Policies 52 Switch Policies 53 Interface Policies 54 Global Policies 55 Managed Object Relationships and Policy Resolution 57 Tags 58 Default Policies 58 How a Policy Model Helps in Diagnosis 60 Summary 63 Review Key Topics 63 Review Questions 64 Chapter 3 ACI Command-Line Interfaces 67 APIC CLIs 68 NX-OS-Style CLI 68 Bash CLI 74 ACI Fabric Switch CLIs 78 iBash CLI 78 VSH CLI 81 VSH_LC CLI 83 Summary 84 Reference 84 Chapter 4 ACI Fabric Design Options 85 Physical Design 85 Single- Versus Multiple-Fabric Design 87 Multi-Pod 97 Multi-Site 116 Remote Leaf 131 Hardware and Software Support 134 ACI Multi-Pod and Remote Leaf Integration 143 Logical Design 149 Design 1: Container-as-a-Service Using the OpenShift Platform and Calico CNI 149 Design 2: Vendor-Based ERP/SAP Hana Design with ACI 165 Design 3: vBrick Digital Media Engine Design with ACI 175 Summary 180 Review Key Topics 181 Review Questions 181 Chapter 5 End Host and Network Connectivity 185 End Host Connectivity 185 VLAN Pool 186 Domain 186 Attachable Access Entity Profiles (AAEPs) 186 Switch Policies 187 Interface Policies 188 Virtual Port Channel (VPC) 191 Port Channel 197 Access Port 201 Best Practices in Configuring Access Policies 206 Compute and Storage Connectivity 207 L4/L7 Service Device Connectivity 210 Network Connectivity 213 Connecting an External Bridge Network 213 Connecting an External Routed Network 218 Diagnosing Connectivity Problems 242 Summary 245 Review Questions 245 Chapter 6 VMM Integration 249 Virtual Machine Manager (VMM) 249 VMM Domain Policy Model 250 VMM Domain Components 250 VMM Domains 250 VMM Domain VLAN Pool Association 252 VMware Integration 257 Prerequisites for VMM Integration with AVS or VDS 257 Guidelines and Limitations for VMM Integration with AVS or VDS 257 ACI VMM Integration Workflow 258 Publishing EPGs to a VMM Domain 258 Connecting Virtual Machines to the Endpoint Group Port Groups on vCenter 259 Verifying VMM Integration with the AVS or VDS 259 Microsoft SCVMM Integration 260 Mapping ACI and SCVMM Constructs 261 Mapping Multiple SCVMMs to an APIC 262 Verifying That the OpFlex Certificate Is Deployed for a Connection from the SCVMM to the APIC 262 Verifying VMM Deployment from the APIC to the SCVMM 263 OpenStack Integration 263 Extending OpFlex to the Compute Node 264 ACI with OpenStack Physical Architecture 264 OpFlex Software Architecture 265 OpenStack Logical Topology 265 Mapping OpenStack and ACI Constructs 266 Kubernetes Integration 272 Planning for Kubernetes Integration 272 Prerequisites for Integrating Kubernetes with Cisco ACI 273 Provisioning Cisco ACI to Work with Kubernetes 274 Preparing the Kubernetes Nodes 277 Installing Kubernetes and Cisco ACI Containers 279 Verifying the Kubernetes Integration 280 OpenShift Integration 281 Planning for OpenShift Integration 282 Prerequisites for Integrating OpenShift with Cisco ACI 283 Provisioning Cisco ACI to Work with OpenShift 284 Preparing the OpenShift Nodes 287 Installing OpenShift and Cisco ACI Containers 290 Updating the OpenShift Router to Use the ACI Fabric 291 Verifying the OpenShift Integration 291 VMM Integration with ACI at Multiple Locations 292 Multi-Site 292 Remote Leaf 295 Summary 298 Chapter 7 L4/L7 Service Integration 299 Service Insertion 299 The Service Graph 300 Managed Mode Versus Un-Managed Mode 301 L4-L7 Integration Use Cases 302 How Contracts Work in ACI 303 The Shadow EPG 306 Configuring the Service Graph 307 Service Graph Design and Deployment Options 312 Policy-Based Redirect (PBR) 322 PBR Design Considerations 323 PBR Design Scenarios 324 Configuring the PBR Service Graph 325 Service Node Health Check 326 Common Issues in the PBR Service Graph 328 L4/L7 Service Integration in Multi-Pod and Multi-Site 332 Multi-Pod 332 Multi-Site 338 Review Questions 342 Chapter 8 Automation and Orchestration 343 The Difference Between Automation and Orchestration 343 Benefits of Automation and Orchestration 344 REST API 349 Automating Tasks Using the Native REST API: JSON and XML 351 API Inspector 351 Object (Save As) 353 Visore (Object Store Browser) 355 MOQuery 357 Automation Use Cases 364 Automating Tasks Using Ansible 372 Ansible Support in ACI 375 Installing Ansible and Ensuring a Secure Connection 378 APIC Authentication in Ansible 382 Automation Use Cases 384 Orchestration Through UCS Director 392 Management Through Cisco UCS Director 392 Automation and Orchestration with Cisco UCS Director 393 Automation Use Cases 395 Summary 402 Review Questions 402 PART II: MONITORING AND MANAGEMENT BEST PRACTICESChapter 9 Monitoring ACI Fabric 405 Importance of Monitoring 405 Faults and Health Scores 407 Faults 407 Health Scores 411 ACI Internal Monitoring Tools 415 SNMP 415 Syslog 420 NetFlow 426 ACI External Monitoring Tools 430 Network Insights 430 Network Assurance Engine 437 Tetration 453 Monitoring Through the REST API 473 Monitoring an APIC 475 Monitoring Leafs and Spines 482 Monitoring Applications 499 Summary 505 Review Questions 506 Chapter 10 Network Management and Monitoring Configuration 509 Out-of-Band Management 509 Creating Static Management Addresses 510 Creating the Management Contract 510 Choosing the Node Management EPG 513 Creating an External Management Entity EPG 513 Verifying the OOB Management Configuration 515 In-Band Management 517 Creating a Management Contract 517 Creating Leaf Interface Access Policies for APIC INB Management 518 Creating Access Policies for the Border Leaf(s) Connected to L3Out 520 Creating INB Management External Routed Networks (L3Out) 522 Creating External Management EPGs 524 Creating an INB BD with a Subnet 527 Configuring the Node Management EPG 529 Creating Static Management Addresses 530 Verifying the INB Management Configuration 530 AAA 533 Configuring Cisco Secure ACS 533 Configuring Cisco ISE 542 Configuring AAA in ACI 547 Recovering with the Local Fallback User 550 Verifying the AAA Configuration 550 Syslog 551 Verifying the Syslog Configuration and Functionality 555 SNMP 556 Verifying the SNMP Configuration and Functionality 562 SPAN 566 Access SPAN 567 Fabric SPAN 571 Tenant SPAN 572 Ensuring Visibility and Troubleshooting SPAN 575 Verifying the SPAN Configuration and Functionality 576 NetFlow 577 NetFlow with Access Policies 580 NetFlow with Tenant Policies 582 Verifying the NetFlow Configuration and Functionality 585 Summary 587 PART III: ADVANCED FORWARDING AND TROUBLESHOOTING TECHNIQUESChapter 11 ACI Topology 589 Physical Topology 589 APIC Initial Setup 593 Fabric Access Policies 595 Switch Profiles, Switch Policies, and Interface Profiles 595 Interface Policies and Policy Groups 596 Pools, Domains, and AAEPs 597 VMM Domain Configuration 601 VMM Topology 601 Hardware and Software Specifications 603 Logical Layout of EPGs, BDs, VRF Instances, and Contracts 605 L3Out Logical Layout 606 Summary 608 Review Key Topics 608 References 609 Chapter 12 Bits and Bytes of ACI Forwarding 611 Limitations of Traditional Networks and the Evolution of Overlay Networks 611 High-Level VXLAN Overview 613 IS-IS, TEP Addressing, and the ACI Underlay 615 IS-IS and TEP Addressing 615 FTags and the MDT 618 Endpoint Learning in ACI 626 Endpoint Learning in a Layer 2-Only Bridge Domain 627 Endpoint Learning in a Layer 3-Enabled Bridge Domain 635 Fabric Glean 640 Remote Endpoint Learning 641 Endpoint Mobility 645 Anycast Gateway 647 Virtual Port Channels in ACI 649 Routing in ACI 651 Static or Dynamic Routes 651 Learning External Routes in the ACI Fabric 656 Transit Routing 659 Policy Enforcement 661 Shared Services 664 L3Out Flags 668 Quality of Service (QoS) in ACI 669 Externally Set DSCP and CoS Markings 671 CoS Preservation in ACI 672 Multi-Pod 674 Multi-Site 680 Remote Leaf 684 Forwarding Scenarios 686 ARP Flooding 686 Layer 2 Known Unicast 688 ARP Optimization 690 Layer 2 Unknown Unicast Proxy 690 L3 Policy Enforcement When Going to L3Out 693 L3 Policy Enforcement for External Traffic Coming into the Fabric 695 Route Leaking/Shared Services 695 Consumer to Provider 695 Provider to Consumer 698 Multi-Pod Forwarding Examples 698 ARP Flooding 700 Layer 3 Proxy Flow 700 Multi-Site Forwarding Examples 703 ARP Flooding 703 Layer 3 Proxy Flow 705 Remote Leaf 707 ARP Flooding 707 Layer 3 Proxy Flow 710 Summary 713 Review Key Topics 713 References 714 Review Questions 714 Chapter 13 Troubleshooting Techniques 717 General Troubleshooting 717 Faults, Events, and Audits 718 moquery 722 iCurl 724 Visore 726 Infrastructure Troubleshooting 727 APIC Cluster Troubleshooting 727 Fabric Node Troubleshooting 734 How to Verify Physical- and Platform-Related Issues 737 Counters 737 CPU Packet Captures 743 SPAN 748 Troubleshooting Endpoint Connectivity 751 Endpoint Tracker and Log Files 752 Enhanced Endpoint Tracker (EPT) App 756 Rogue Endpoint Detection 758 Troubleshooting Contract-Related Issues 759 Verifying Policy Deny Drops 764 Embedded Logic Analyzer Module (ELAM) 765 Summary 769 Review Key Topics 769 Review Questions 769 Chapter 14 The ACI Visibility & Troubleshooting Tool 771 Visibility & Troubleshooting Tool Overview 771 Faults Tab 772 Drop/Stats Tab 773 Ingress/Egress Buffer Drop Packets 774 Ingress Error Drop Packets Periodic 774 Storm Control 774 Ingress Forward Drop Packets 775 Ingress Load Balancer Drop Packets 776 Contract Drops Tab 777 Contracts 777 Contract Considerations 778 Events and Audits Tab 779 Traceroute Tab 780 Atomic Counter Tab 782 Latency Tab 785 SPAN Tab 786 Network Insights Resources (NIR) Overview 787 Summary 790 Chapter 15 Troubleshooting Use Cases 791 Troubleshooting Fabric Discovery: Leaf Discovery 792 Troubleshooting APIC Controllers and Clusters: Clustering 795 Troubleshooting Management Access: Out-of-Band EPG 799 Troubleshooting Contracts: Traffic Not Traversing a Firewall as Expected 801 Troubleshooting Contracts: Contract Directionality 804 Troubleshooting End Host Connectivity: Layer 2 Traffic Flow Through ACI 807 Troubleshooting External Layer 2 Connectivity: Broken Layer 2 Traffic Flow Through ACI 812 Troubleshooting External Layer 3 Connectivity: Broken Layer 3 Traffic Flow Through ACI 814 Troubleshooting External Layer 3 Connectivity: Unexpected Layer 3 Traffic Flow Through ACI 816 Troubleshooting Leaf and Spine Connectivity: Leaf Issue 821 Troubleshooting VMM Domains: VMM Controller Offline 826 Troubleshooting VMM Domains: VM Connectivity Issue After Deploying the VMM Domain 829 Troubleshooting L4-L7: Deploying an L4-L7 Device 832 Troubleshooting L4-L7: Control Protocols Stop Working After Service Graph Deployment 834 Troubleshooting Multi-Pod: BUM Traffic Not Reaching Remote Pods 837 Troubleshooting Multi-Pod: Remote L3Out Not Reachable 839 Troubleshooting Multi-Site: Using Consistency Checker to Verify State at Each Site 841 Troubleshooting Programmability Issues: JSON Script Generates Error 844 Troubleshooting Multicast Issues: PIM Sparse Mode Any-Source Multicast (ASM) 846 Summary 860 Appendix A Answers to Chapter Review Questions 861Index 873