CCIE Data Center :: Storage

Multihop FCoE Configuration


Table of Contents
Course Files
  • 1 Introduction Closed Caption 0h 44m
    2 Storage Networking Hardware Architecture Closed Caption 0h 46m
    3 Fibre Channel Switching Overview Closed Caption 0h 51m
    4 Fibre Channel Switching Verification Closed Caption 1h 12m
    5 Nexus 5500UP Initialization & FC Interface Configuration Closed Caption 1h 08m
    6 SAN Port Channels Closed Caption 0h 13m
    7 Virtual SANs (VSANs) Closed Caption 0h 21m
    8 FC Fabric Services :: FC Domain Closed Caption 0h 31m
    9 FC Fabric Services :: FSPF Closed Caption 0h 21m
    10 FC Fabric Services :: FLOGI & FCNS Closed Caption 0h 19m
    11 FC Fabric Services :: Basic Zoning Closed Caption 1h 08m
    12 FC Fabric Services :: Enhanced Zoning & Aliases Closed Caption 1h 13m
    13 FC Fabric Services :: Q&A Closed Caption 0h 13m
    14 iSCSI :: Part 1 Closed Caption 0h 32m
    15 iSCSI :: Part 2 Closed Caption 0h 58m
    16 FCIP Closed Caption 0h 44m
    17 NPV & NPIV Closed Caption 1h 12m
    18 FCoE Overview Closed Caption 0h 58m
    19 FCoE Configuration :: Part 1 Closed Caption 0h 49m
    20 FCoE Configuration :: Part 2 Closed Caption 0h 43m
    21 FCoE Q&A Closed Caption 0h 14m
    22 Multihop FCoE Overview Closed Caption 0h 36m
    23 Multihop FCoE Configuration Closed Caption 1h 16m
    24 FCoE NPV Closed Caption 0h 37m
    25 Data Center Network Manager (DCNM) Closed Caption 0h 20m
    Total Duration   17h 59m
  • 0:00:17 So in our next section here with the FCoE, we're gonna talk about the multi-hop design.
    0:00:22 And we're gonna look at the configuration on the command line for getting the server 1 here to talk to the storage arrays,
    0:00:30 the JBOD 1 and JBOD 2 over the FCoE.
    0:00:33 So essentially here, we're gonna have FCoE again that's going from server 1 to 5K 2,
    0:00:38 from 5K 2 to 7K 1, 7K 1 to 5K 1 and then this is where it's changing back into native FC.
    0:00:46 So the demarcation point between the FCoE and the native Fibre Channel domain is gonna be on 5K 1.
    0:00:52 Now we'll see that there are some caveats here with the 7K that are different than the 5K.
    0:00:58 One of them first off is gonna be based on the particular modules that are required.
    0:01:03 Now if we look back at the Cisco's website for some of the 7K information, let's go to
    0:01:13 And let's go down to the 7K and compare models.
    0:01:19 And we wanna look at the F series modules.
    0:01:22 Okay. The F1 and the F2 modules, these both do support FCoE, where the M series modules do not.
    0:01:31 So even the F2 modules, or excuse me the M2 modules, none of them support FCoE.
    0:01:36 Okay. The F cards for the F1 module; this is supported on supervisor 1, supervisor 2 and 2E.
    0:01:45 The F2 cards is not supported with sup 1.
    0:01:49 So if you had supervisor 1, you're not gonna be able to run F2.
    0:01:52 And additionally, when you do run F2 it requires it's own virtual device context, and it requires the newest code.
    0:01:59 So whatever the 6.1 sub release is for the F2 card, that's the one that is just recently added FCoE.
    0:02:07 So it's literally within the past couple of weeks that the support was just added for this particular module.
    0:02:13 So in our particular case today, this is the module that we have is the F1/32 XP, which is the 10 GigE, actually 1 gig, 10 GigE 32 port.
    0:02:23 And we're gonna be using it to run the FCoE feature set on.
    0:02:27 Now from the actual configuration point of view, right now we're starting at a blink configuration on the 7K.
    0:02:35 So we're looking at this from the original initialization.
    0:02:38 So I raised the config, reload it, I basically just put the admin password in.
    0:02:44 And we're here at the prompt with no configuration in there.
    0:02:53 Okay. So from the switches config again, if we look at the Show run, basically this is just the basic default configuration.
    0:03:01 I haven't really changed anything yet.
    0:03:03 The only thing, other thing that I need to add first before we go any further is to power on the other modules if we look at the Show environment power.
    0:03:11 Right now, the M1 cards and the F2 card, these are being denied power because of the power redundancy mode.
    0:03:18 So it's trying to run the end redundant mode, but I don't actually have enough physical capacity for to do that.
    0:03:25 So I need to say that the power redundancy mode is combined.
    0:03:31 Okay. We talked about this a little bit in the Nexus class.
    0:03:33 Within the scope of the lab exam, it's highly unlikely that you're gonna have to deal with this, anything that's related to the physical chassis,
    0:03:41 but of course from a practical implementation point of view,
    0:03:44 this is important to know what are the specific power requirements of the supervisors, of the line cards, of the fan modules.
    0:03:52 Because it does draw a lot of power, and if you don't have the correct circuits,
    0:03:57 then if there's a failure of like one of the power supplies then you could potentially lose all of the line cards and the supervisor.
    0:04:03 Okay. So we're gonna say here that this device is N7K 1-1.
    0:04:11 And let me add my other username here.
    0:04:14 Let's say username Brian, password cciedc01.
    0:04:18 The password is weak, password should contain characters from at least 3 of the following classes.
    0:04:23 What I can say here instead is no password strength check.
    0:04:30 And my username Brian is in the role of network admin.
    0:04:37 Now, I mentioned this briefly yesterday.
    0:04:39 One of the issues with the Nexus 5K and the 7K from a storage versus LAN point of view, is from your day-to-day operations,
    0:04:47 normally you're gonna have your LAN team that's managing the,
    0:04:51 LAN team is managing the Ethernet stuff and then the SAN team is managing the storage configs.
    0:04:57 So with the default roles, in this particular version, we have network operator, network admin.
    0:05:02 Where at network admin is your super user.
    0:05:05 And then for the particular VDC's you have VDC admin and you have VDC operator.
    0:05:10 So these are gonna be for the particular VDC.
    0:05:13 The super user versus the read-only you.
    0:05:17 Now, the problem is then if you want to do custom roles with the role-based access control,
    0:05:23 it gets pretty complicated to do this with the manual configuration.
    0:05:30 And what I need to only find where the link is here.
    0:05:33 There is basically a template that Cisco has.
    0:05:48 Let me just search for it here.
    0:05:49 It should be the FCoE and RBAC, R-B-A-C template. This one.
    0:06:02 This is for 503, this should show the full template, here it is.
    0:06:12 So this is what you would paste in essentially for your LAN admin role.
    0:06:20 And then you have a separate one that is for your SAN admin role.
    0:06:25 Now in the newer versions of the 5K, the 5.2, the one that just came out, they actually had this built-in as a pre-defined role.
    0:06:32 So you can specify someone as a LAN admin or a SAN admin because as you can see here,
    0:06:37 this is not something that you can just know off the top of your head.
    0:06:40 You would there basically doing is accounting for every possible command that you would need to issue
    0:06:46 that is related to your day-to-day SAN operations versus your day-to-day LAN operations.
    0:06:52 But the problem is if you don't do something like this, you could have your LAN people mess up something in the storage.
    0:06:59 And like I mentioned before, especially if you're doing boot from SAN,
    0:07:03 if you accidentally shut one of the links down or cause convergence event in the Fabric,
    0:07:07 you could knock the devices, knock your N servers off of the, not really off of the network but off of their actual OS completely,
    0:07:15 like Windows could crash or Linux could crash if it doesn't have access to its actual boot drive.
    0:07:20 So I'm not gonna Show the demo of this but just be aware that this is available, it's under the FCoE and RBAC configuration in the Nexus 5K guide.
    0:07:30 And then under sample conifg, this is what they're saying is a LAN admin and then you have the SAN admin.
    0:07:35 Okay. Let's also go to the 7K configuration.
    0:07:38 So let's start from the main documentation page.
    0:07:42 So under the Cisco docs, let's go to products and to switches then to datacenter switches, Nexus 7K, configuration guides and the FCoE configuration.
    0:07:59 Now we'll see that the config from a logical stand point is very similar to what we did previously in the 5K.
    0:08:05 That we need to do a binding between our VLAN and our VSAN.
    0:08:09 We're gonna configure via a Fibre Channel interface that's bound to a physical link.
    0:08:13 The VFC is gonna be a member of the VSAN, and then we activate the interfaces.
    0:08:19 Okay. That's pretty much the extent of it.
    0:08:20 But there are some other specific caveats with the 7Ks.
    0:08:24 One of them is gonna be related to the storage VDC.
    0:08:28 So normally, with the sup 1, you can have up to 4 virtual device contexts.
    0:08:33 In the case of sup2 and sup2 E, if you have the additional VDC licensing,
    0:08:40 you can go up to 8 VDCs plus a separate management VDC for basically like SSHing into,
    0:08:48 and then you can change to the other context, change to the other VDCs.
    0:08:51 In our case we're using sup1, so we have 4.
    0:08:54 But out of these 4, one of them has to be dedicated just for storage.
    0:08:58 So the way you think of this logically is that the storage VDC is like the MDS switching code inside of the Nexus 7K.
    0:09:07 So it's just for SAN, it's not for any other operations.
    0:09:10 And it does count against one of the 4 VDCs that we have, or in the case of the sup2 or sup2 E,
    0:09:17 it does count against one of the 8 VDCs that you could potentially have.
    0:09:21 So that's my next step, is that I need to allocate the VDC.
    0:09:25 But to do that, I need to turn the feature on first.
    0:09:28 So let's say, install the feature set, the feature set is FCoE.
    0:09:35 Okay. Then let's look at the Show modules.
    0:09:40 My F1 card is in module 2, is in slot 2.
    0:09:44 So I need to assign the FCoE license to this specific module.
    0:09:50 I need to say, license FCoE, module 2.
    0:09:59 Okay. The license is not available in this case because I don't actually have it assigned if we Show license usage.
    0:10:07 I need to turn the Grace period on, because there's not an actual install license for FCoE.
    0:10:12 So let's say license Grace period, then assign the FCoE license to module 2.
    0:10:19 Now, I can actually start the VDC and then I'm gonna allocate the interfaces into that VDC.
    0:10:25 So VDC will give it a name, we'll say this is N7K 1-SAN.
    0:10:31 And the key here is that the type is going to be storage.
    0:10:35 Now, when we do this creation, depending on what the state of your configuration is up to this point.
    0:10:40 If I already had some other configuration going on with the owner VDC which is the default one, and then the other user VDCs.
    0:10:48 We can run into a case that the storage VDC starts to create and then fails.
    0:10:53 And one of the reasons could be that your QoS template is wrong, which actually it is in this case.
    0:10:59 Okay. It says, I cannot start the storage VDC.
    0:11:03 The reason why is the active template is not FCoE compatible.
    0:11:07 To enable the feature set FCoE, apply a non-8E network QoS policy.
    0:11:15 So what are they talking about here.
    0:11:16 What does non-8E network QoS mean?
    0:11:20 Okay, let's look at the Show run section system.
    0:11:26 And it doesn't actually show up, here it is the default.
    0:11:31 But we can see it if we look at the Show policy map type network QoS.
    0:11:38 We talked about this yesterday with the 5K, that we need to make sure that it is enforcing the no-drop policy.
    0:11:45 And it allows the priority flow control for the specific FCoE traffic.
    0:11:50 And it's doing that based on two things, based on the classes service 3 and then based on the Ether type which I believe is 8906 for the FCoE right its 8906.
    0:12:01 So the traffic comes in if it is 8906.
    0:12:04 We need to get it to land in this in-drop FCoE class, which is to say to pause it, and then to not drop the traffic.
    0:12:13 If we were to look at the Show class map type network QoS, and let's look for this class-NQ-7E-indrop-FCoE which is this one.
    0:12:30 It says match classes service 3 and match the protocol FCoE.
    0:12:33 So it's doing the classification here based on the Ether type and based on the layer 2 class of service marking.
    0:12:40 So really, what this, what the error message means, is that I need to take this policy that they already have predefined.
    0:12:47 And I just need to activate it.
    0:12:49 And the reason why it's giving me this error is that the network QoS and the 7K, is defined in the owner VDC or the default VDC.
    0:12:58 It's not assigned on the other user VDCs because basically, what this is talking about is the QoS on the Fabric back plane.
    0:13:06 So we need to go to global config here say, system QoS, then say service policy type is network QoS.
    0:13:14 Then if you issue the question mark here, you'll see what the default ones are in there.
    0:13:18 So this one, we want.
    0:13:21 It doesn't necessarily have to be this one.
    0:13:23 It just have to be one of, it has to be one of these 3.
    0:13:28 But I'm gonna say the one that is just talking about FCoE and then data traffic is in everything else.
    0:13:37 Okay. So we have the new policy.
    0:13:38 Let's try to recreate the VDC now.
    0:13:40 So let's say, VDC, the name is N7K 1-SAN, the type is storage.
    0:13:48 VDC is not in the active state.
    0:13:53 Let's see, can we remove it?
    0:13:57 Delete failed. VDC is not active.
    0:13:59 Let's just give it a different name, SAN 1.
    0:14:04 Only one VDC can run storage traffic.
    0:14:07 Let's say Show VDC usage, or just Show VDC.
    0:14:13 The create is in a progress.
    0:14:15 What I may need to do here, because basically it's an order of operations problem that I should've changed the QoS policy before I try to allocate the VDC.
    0:14:28 I think what I'm gonna have to do is to reload here, or just wait.
    0:14:31 Eventually it's gonna time-out but I think it takes about 10 minutes before it does time-out.
    0:14:35 So I'm gonna add a couple other minor changes here real quick.
    0:14:39 I'm gonna go to the management interface, and give it the IP address that I'm using for management, which here is, and then enable telnet.
    0:14:52 Okay, so I can login to it.
    0:14:54 And let me make sure my admin username is correct, username admin, password, copy run start.
    0:15:29 And then let's reload.
    0:15:30 Okay. In the meantime, while we're waiting for this to come back, we're gonna look at the rest of the config.
    0:15:34 So we need to, first let's look at the FC network.
    0:15:37 I wanna make sure that the storage arrays are actually registered to the Fabric.
    0:15:41 And that up to 5K 1, we have just our native Fibre Channel working.
    0:15:46 So let's go to the CLI of MDS 1, and let's look at what's the state of the native Fibre Channel network.
    0:15:54 Because of course, FCoE is not gonna work unless regular FC is working to the actual arrays.
    0:16:03 So let's go to MDS 1, let's look at the Show interface brief exclude SFP apps, and so I only wanna show my active links.
    0:16:11 And it says that interface is FC 1/17 and 1/18, these are both up in Fabric loop.
    0:16:19 Now, the particular VSAN that we're gonna be using is 1010.
    0:16:23 That's the same one that we were using yesterday for the FCoE.
    0:16:26 So right now, both of these are in VSAN 1010.
    0:16:32 This is ultimately what we're gonna be extending down to the server.
    0:16:36 Which means that on 5K 2, we are going to have a VFC A, Virtual Fibre Channel Interface that's in VSAN 1010.
    0:16:46 And that ultimately is gonna make the Fabric end-to-end for them.
    0:16:49 Okay. So it says the interfaces are up.
    0:16:51 Let's make sure that the disks are actually registered, that they sent FLOGI and that they're in the name server database.
    0:16:57 So let's Show FLOGI Database.
    0:17:00 Okay, we have the disks, and we could see some of these have device aliases from before.
    0:17:05 So there's disk 0 of JBOD 0.
    0:17:11 So as long, basically as long as we can get one of these disks to mount on the server, that's really all that matters.
    0:17:16 We don't care about the details of all of them potentially.
    0:17:19 But again, this is at least telling me that the devices have registered to the Fabric.
    0:17:23 Again, the Show FLOGI Database, this is showing the local registration.
    0:17:27 If I wanted to see it everywhere throughout the entire Fabric, we can look at the Show FCNS Database, the Show Fibre Channel Name Server database.
    0:17:38 And that would tell us, is the Fabric end-to-end between all of the switches.
    0:17:45 Okay there's a question here about the 7K: Is the VDC type storage mandatory whenever the 7K is involved in FCoE?
    0:17:54 Yes. It is mandatory.
    0:17:56 So this is the only case where you specify in the VDC, let's scroll back up here if you can see the syntax.
    0:18:02 That's the only time that you say, what's the type?
    0:18:05 So the type is normally just a regular user VDC.
    0:18:09 This is a special type of VDC that is just for the MDS config.
    0:18:13 So then, additionally I need to assign the license to the module.
    0:18:17 You could have multiple modules obviously in the chassis.
    0:18:20 And in that case you would have to license each of the modules.
    0:18:24 But then the issue is that we're gonna have to allocate Ports from the F1 module into that VDC.
    0:18:30 I could allocate all the ports if I wanted to, or the F2 is gonna have to be in its own VDC, which is one of the requirements.
    0:18:38 Okay. So I don't necessarily have to allocate all of the module though.
    0:18:41 It doesn't have to be every port; it has to be at least, in our case, that should be at least 2 ports,
    0:18:46 because we're going from the 7K to 1 switch then 7K to another switch.
    0:18:52 Okay. So let's look back at the Fabric design here.
    0:18:55 So the MDS 1 is working.
    0:18:57 Let's look at now what's going on with the trunk link that's supposed to be going down to 5K 1.
    0:19:05 Let's say Show Interface FC 1/1 to 2 Brief.
    0:19:11 It says, these are on.
    0:19:14 The admin mode is E but the operational mode is nothing.
    0:19:19 The status is not connected.
    0:19:22 So this means that there's something wrong most likely on the other side.
    0:19:25 So in my links, these should be not in the shutdown state.
    0:19:28 We Show run interface FC 1/1 to 2.
    0:19:33 Okay. I had this configured as E-Ports which is fine.
    0:19:35 Basically it just means that the 5K is not configured.
    0:19:38 So on 5K, let's say Show Interface FC 1/27 to 28 Brief, and these are down.
    0:19:47 So let's just go to those links and say, no shut.
    0:19:53 And if I'm gonna join their Fabric, I need to make sure that I'm agreeing on the VSAN numbers.
    0:19:58 So in this case we're using VSAN 1010.
    0:20:01 So let's create VSAN 1010 in the VSAN database. VSAN 1010.
    0:20:14 Okay. If we now look at the Show Interface FC 1/27 to 28 trunk VSAN.
    0:20:25 Ultimately I need to make sure here that VSAN 10/10 is trunking on both of the links.
    0:20:30 Okay. We can immediately see here that there's a problem here.
    0:20:32 It says the zoning modes do not match.
    0:20:35 So there's a rejection from the adjacent switch.
    0:20:38 We look at the Show Zone Status for VSAN 10/10.
    0:20:45 I'm in basic zoning mode, they're in enhanced mode.
    0:20:48 So let's say zone mode is enhanced for VSAN 1010.
    0:21:05 Then let's shut these link down and bring them back up.
    0:21:07 So that was FC 1/27 to 28, shut down: No Shutdown.
    0:21:24 And again what's our final verification gonna be?
    0:21:26 How can I know, is 5K 1 actually on the Fabric for VSAN 1010.
    0:21:31 Okay. We could look at the trunk, but the trunk doesn't necessarily mean that I know about the entire Fabric.
    0:21:37 I wanna look at the FCNS Database.
    0:21:39 So once I actually see the disks in there, then I know what their FCId to PWWN mapping is.
    0:21:45 It means that really, the Fabric should be working everywhere.
    0:21:48 So if we Show FCNS Database for VSAN 1010, okay, now the Fabric is up.
    0:21:54 Okay. Now, I know about those disks.
    0:21:56 And the device alias database, this is already being synchronized through CFS, through Cisco Fabric Services.
    0:22:03 So basically, throughout the entire Fabric, I should have these shortcut names.
    0:22:06 There's a question here: Are there any normal Nexus functions that the storage VDC cannot perform?
    0:22:12 It's actually all of them.
    0:22:14 So the only thing the storage VDC can perform is the SAN switching.
    0:22:19 And again, there's not native Fibre Channel on the 7K, it's only FCoE.
    0:22:23 So basically the only thing we're gonna be doing in there is creating the VSAN, the VLAN mappings.
    0:22:30 The VSAN, the VLAN mappings were also gonna be doing the potentially zoning,
    0:22:35 creating the Virtual Fibre Channel interfaces, and then binding those to the physical links.
    0:22:40 So it's a much smaller subset of the config that we have in the regular user VDCs, or the default or owner VDC.
    0:22:49 Okay. So let's see now is, okay so the switch is back. Now, let's login.
    0:22:55 And Show VDC.
    0:23:02 Let's say Show run section system, for system QoS.
    0:23:07 Okay. So we're now, we're running the correct policy now.
    0:23:09 So let's see if the VDC is gonna allocate.
    0:23:12 VDC, give it a name. I'll say this is -SAN is type storage.
    0:23:24 And let's let this sit there and think about it for a while.
    0:23:32 It takes a couple of minutes for it to actually start all the processes.
    0:23:37 Okay, so now up to this point, what I verified is working, is that the links from the storage arrays to the MDS is working.
    0:23:47 The trunk links between the MDS and the 5K is working.
    0:23:51 Now let's work on the actual initiator.
    0:23:53 So I need to get the FCoE link up between the virtual Node Port, and the Virtual Fabric Port.
    0:24:02 So the Virtual Fabric Port, that's the switch, the Fibre Channel forwarder.
    0:24:06 The virtual Node Port, that's our VE, or that's our E-Node, not VE. E-Node, or Ethernet Node Port.
    0:24:17 So let's take a look at the actual server.
    0:24:23 And right now, the server should say that it's not part of the Fabric, 'cause I don't have any other config done for FCoE yet.
    0:24:32 So let's go to our FCoE manager app, or just our interface manager.
    0:24:41 So it says that the LAN is up, but FCoE is not.
    0:24:45 So you could see the little FCoE icon there, it's red.
    0:24:48 It means that I'm not permanently joined to the Fabric.
    0:24:51 These targets here, these are what I had mapped previously.
    0:24:54 These are the disks that basically, we're gonna give them zoning access to.
    0:24:59 But FCoE is down, and we would see this under the port like if we went to the port information says there's no fiber attachment.
    0:25:11 Also if we looked at the, let's see where this is, under, looking for the DCB settings.
    0:25:25 It may not actually show the DCB settings until, it may not show this until I register what the Fabrics.
    0:25:42 So let's leave this off for a second.
    0:25:46 Okay. So now let's go down to the 5K 2, that's where our FCoE config is gonna go down to the actual end host.
    0:25:55 Okay. So I need to configure the VSAN to VLAN mapping to start.
    0:26:00 So let's create the VSAN. VSAN database.
    0:26:06 We have VSAN 1010 and we have the VLAN mapping.
    0:26:11 So for clarity here, I'm just saying that it is the same number.
    0:26:14 It doesn't necessarily have to be, but for simplicity, we would probably want it that way.
    0:26:18 So VLAN 1010 is gonna have the FCoE VSAN 1010.
    0:26:27 If we look at the physical link, the physical link here is going to be, and let's shut this down.
    0:26:36 So that's gonna be the last step is to enable it.
    0:26:39 So the physical link is gonna be a trunk, switchport mode trunk.
    0:26:42 The Spanning tree port type is edge trunk, because we need to make sure it's not running the listening and learning delay.
    0:26:50 Then normally, what you would want to do here is to limit the VLAN numbers, just to whatever you're actually using.
    0:26:57 So in this case, we'll say that the switchport trunk allowed list is going to be VLANs 10 and 1010, where my switchport trunk native VLAN is 10.
    0:27:07 That's basically my access VLAN.
    0:27:10 So this is the full extent of the config on the physical link.
    0:27:16 Okay. We're basically just making it a trunk and a port type edge.
    0:27:19 Okay. Then we have our Interface Virtual Fibre Channel.
    0:27:24 I'm going to bind the interface E1/17 to this link.
    0:27:31 This is going to be a trunking Fabric Port or a TF.
    0:27:36 And in this case, since there's only one particular VSAN that I'm using, I probably want to limit it just to that.
    0:27:43 You don't necessarily have to, but from a real design point of view, you would want to do that.
    0:27:46 You don't want other traffic that's unneeded on the port, could be going down to it.
    0:27:51 So let's say the switchport trunk allowed VSAN here is 1010.
    0:27:59 Then we'll say No Shutdown on the VFC.
    0:28:05 We go to the VSAN database and say the VSAN 1010 is then assigned to the interface, Virtual Fibre Channel.
    0:28:18 Okay. This is essentially the extent of our config.
    0:28:21 So if we go down to the physical link, let's say No Shutdown.
    0:28:30 It says the link is up.
    0:28:31 Let's look at the Show FLOGI Database, and the E-Node has registered.
    0:28:37 This would be the same as if we look at the Show FCoE database.
    0:28:43 But really, the key point here is that on Virtual Fibre Channel for 1/17 that we gave them an FCId.
    0:28:50 So they have a logical address and we know their port number, the WWPN.
    0:28:55 If we look at the Show, actually let's look at this port name, and then let's compare to what's on the MDS.
    0:29:02 And the MDS if we Show the device alias database, I previously configure this in here.
    0:29:08 Just to save us some time here.
    0:29:10 This is server 1, FCoE on SAN B.
    0:29:15 So that's their PWWN.
    0:29:17 So the PWWN, you're already gonna know this in a practical deployment.
    0:29:21 You're already gonna know what the value is because the server has it burned in to the card.
    0:29:27 So really, before you even put them on to the Fabric, you probably wanna configure all of this, the zoning and everything.
    0:29:32 If you just go to the management app of the card, and then look at the port, this number here, that's its PWWN.
    0:29:40 And we could see it's green now, it says, now we're joined to the Fabric.
    0:29:50 And where is my DCB info here, there it is.
    0:29:57 Okay. So DCBX is enabled, priority flow control is enabled, enhanced transmission selection is enabled.
    0:30:03 Okay this is the default policy of the 5K.
    0:30:05 We're giving half of the bandwidth for LAN, half of the bandwidth for SAN.
    0:30:11 Okay. And if I wanted to change this, I change it on the 5K, bounce the interface, which makes them renegotiate DCBX.
    0:30:18 And then the new percentage is whatever I'm defining is gonna be assigned down to card or advertise down to the card.
    0:30:31 Okay. There's a question: So in the practical deployment for the 7K, the type storage is actually to extend FCoE from the initiator to the target,
    0:30:40 while at the same time we have another VDC that is for LAN switching.
    0:30:44 That's correct. So basically, the storage VDC is just a logical one.
    0:30:48 You will have interfaces that are allocated to it, but you can actually share your interfaces from your regular VDC or LAN VDC to the storage VDC.
    0:30:58 And that's really the only case that the VDC has overlapping links that I can have the LAN traffic from the link come in and land on the regular LAN VDC.
    0:31:07 But then based on the Ether type, if the 7K sees 8906 come in which is the FCoE traffic.
    0:31:15 It knows to send it specifically to the storage VDC.
    0:31:19 But it's a special functionality, it's different than the other VDCs, it's just for the storage traffic.
    0:31:25 So let's go back to the 7K and see, so now VDC 2 is come online.
    0:31:31 So it just takes a little while for the initial allocation.
    0:31:35 Okay. Let's look at now the Show VDC.
    0:31:38 It says the N7K 1- SAN is active.
    0:31:43 This is the storage type of VDC.
    0:31:46 And I have my F1 module is gonna be assigned there.
    0:31:50 So a couple of other steps now, if we Show run VDC, what I now need to do is to actually allocate my links to it.
    0:31:59 So we're gonna go to this VDC, and I need to allocate my interfaces.
    0:32:06 Okay. It could be, if a link that were dedicating to the FCoE VDC, or could be shared between multiple ones.
    0:32:15 So on your normal VDCs you don't have this.
    0:32:18 It's just that this one is type storage, it allows you to overlap them, where you're sending the LAN traffic to the LAN VDC and then the storage traffic goes to the storage VDC.
    0:32:27 But in most designs you would want to use dedicated links.
    0:32:30 It just kind of simplifies the topology.
    0:32:34 So I'm gonna allocate the interfaces and that's based on the links that we're using here.
    0:32:38 So I need ports E2, 3, to 6, and it's gonna automatically take whatever ports are in the grouping that it needs.
    0:32:49 Okay. The next thing I need to do is to allocate some VLAN numbers that we are going to use for our VSANs.
    0:32:59 So these VLAN numbers, they basically need to be globally unique to the chassis.
    0:33:03 We don't wanna overlap them to other ones.
    0:33:05 So i'll say that the VLAN numbers, let's say they start at 1002 and they go to 1999, or let's say 1099.
    0:33:15 Because there's only a limited amount of VSANs you're gonna have anyways.
    0:33:18 Most of the time, you really only need 2, one just for SAN A, one for SAN B.
    0:33:23 So if you give them, like in this case it's almost 100.
    0:33:26 You never realistically gonna use that many.
    0:33:28 Okay. I think the limit of the platform is actually like 63 or so that I can't, it cannot use more than that.
    0:33:36 Okay. But remember whenever you create a new VSAN, what are you actually doing?
    0:33:41 You start a new copy of Fabric services.
    0:33:44 So it means that your domain ID which is the principal switch election is gonna happen.
    0:33:51 The FSPF routing, the FLOGI database, the FCNS database, it is additional overhead in processing power that you need for each of the VSANs.
    0:34:01 It's not as simple as like a VLAN, where the only thing you're doing is spawning an additional Spanning tree process.
    0:34:10 Okay. So we're allocating the VSANs.
    0:34:13 Let's say 1002-1099 and where is this coming from?
    0:34:21 I'll say this comes from my other current VDC.
    0:34:25 Right now, I only have one of them.
    0:34:27 So basically it means that in VDC's 7K 1-1, I cannot use these numbers anymore.
    0:34:35 Which is really not an issue because I still already have like 4000 of the VLANs I can use.
    0:34:40 It's just that I need to make sure not to use these numbers.
    0:34:42 So in your real deployment, you're gonna dedicate some portion of the VLAN numbers globally to the entire network that you wanna make sure it's just for storage.
    0:34:50 You don't wanna put any LAN traffic into these particular VLAN numbers.
    0:34:56 Okay. Now, we should be able, we're basically done with the owner VDC now.
    0:35:00 So let's look at what our config was here.
    0:35:04 So install the feature set, I specified which F1 or F2 modules are actually gonna get the license.
    0:35:12 Now, in this case, since I didn't actually have the license, permanent license installed.
    0:35:16 I had to turn the Grace period on, but in the lab exam probably you wouldn't have to do this.
    0:35:20 I'm assuming there would have their own chassiss licensed.
    0:35:23 They just generate their own keys.
    0:35:25 Then I have the QoS policy, which again here is making sure that I have the no-drop policy in the priority flow control pause for the FCoE traffic,
    0:35:37 which again is classifying based on the Ether type, 8906 and also the class of service 3.
    0:35:43 So the layer 2 header and the layer 2 markings.
    0:35:47 Okay, then I have these storage VDCs, so I said VDC, I gave it a name, said the type is storage.
    0:35:55 I allocated the physical links.
    0:35:57 These are the ones that are gonne be the FCoE trunks.
    0:36:00 And then I allocated the VLAN numbers which is, why is that not, so it's down here, a separate section of the config.
    0:36:10 Allocate the FCoE VLAN range.
    0:36:15 Okay. So now we can change to, to change to or switch to.
    0:36:19 Switch to VDC, N7K 1-SAN.
    0:36:25 So this is gonna be just like any other VDC, it's gonna initialize from scratch.
    0:36:29 So do I wanna enforce password security? No.
    0:36:32 Whatever my password is, okay, the default user is admin.
    0:36:36 No, I don't wanna go through the config dialog.
    0:36:38 And really the only other thing I wanna do is to, I'm gonna put a username here.
    0:36:44 So put my username. Username, Brian is gonna be the role of VDC admin.
    0:36:55 Okay. So I'm the super user here for the VDC.
    0:36:58 I'm gonna turn telnet on, feature telnet.
    0:37:04 Go to the management interface, and give it an address.
    0:37:18 Then I'm gonna come into this through telnet, not through the console.
    0:37:21 Now, the reason that I'm doing this is that from a logging point of view, I'm not sure if I mentioned this in the Nexus class
    0:37:29 but for whatever reason on the Nexus 5K and the 7K, when you connect to the console,
    0:37:35 it doesn't allow you to log syslog messages of a severity, I think it's 5 or above.
    0:37:42 Unless you change the console speed to higher, then at 9600 bits per second, or 9600 baud.
    0:37:49 So you could change it to 384, but the problem is then you have to edit like your access servers, your terminal servers to do that.
    0:37:58 So if you just telnet or SSH into the VDC or into the device over the management link, then you can do the logging.
    0:38:05 So if you say Show log here, I just wanna make sure that I can say logging monitor to basically everything.
    0:38:13 So I can see whatever the messages are about the FCoE as we're creating it.
    0:38:23 Okay. There's a question: I said that the interfaces can be shared between the storage VDCs and the traditional VDCs.
    0:38:28 Can you explain this in more detail why you would want to do this?
    0:38:32 So let's look at, let me draw this out as to why this might be a valid design.
    0:38:40 Let's say that we're using 7Ks as essentially our access switches.
    0:38:47 So we have 7K 1 and we have 7K 2.
    0:38:51 These are going down to our servers.
    0:38:56 So we have server 1, and server 1 has a converged network adapter.
    0:39:02 So we have 10 gig 1, and we have 10 gig 2.
    0:39:09 And this link is running both LAN and SAN traffic.
    0:39:16 The 7K is only gonna be able to do the FCoE for the storage VDCs.
    0:39:23 So basically internally, I have this VDC for SAN, for FCoE.
    0:39:33 But I also have the main one that is for the LAN traffic.
    0:39:37 So coming up from the server, let's put this a different color for clarity.
    0:39:42 So let's say that that's the LAN, is the blue traffic.
    0:39:45 So the LAN traffic is going both ways but as is the SAN traffic.
    0:39:52 So when the SAN traffic comes in, that has to go the storage VDC.
    0:39:56 When the LAN traffic comes in, that can go to the rest of the LAN.
    0:40:02 So whatever this physical link is that connects to the 7K, this needs to be shared.
    0:40:08 Where I'm sending the LAN traffic to the LAN VDC, and then I'm sending the storage traffic to the storage VDC.
    0:40:17 There's a question: So the VFC interface is gonna get assigned to the storage VDC.
    0:40:21 That's correct. Okay, anything that's related to Fibre Channel is always gonna go on the storage VDC.
    0:40:26 So if we have the VSAN database, the VLAN, the VSAN mapping, the Virtual Fibre Channel interfaces,
    0:40:33 and then whatever the physical links like the trunking that we need to do in order to get the VLANs assigned.
    0:40:38 That's all gonna be under the storage VDC.
    0:40:41 So in our particular case here, we're basically done with the default VDC or the owner VDC.
    0:40:46 Now that I have management access into this one, that was the only thing I need in the main VDC for.
    0:40:56 And actually let me make one more change here, which is no host name.
    0:41:08 No, not no host name. No VDC combined host name.
    0:41:16 Okay. So now it gets its own host name there.
    0:41:19 It doesn't have that prefix in front of it.
    0:41:22 Okay. So now, we're in the SAN VDC, let's look at the Show run and see what are the defaults in here.
    0:41:27 So we have some of our basic normal management stuff, I have my users in here.
    0:41:32 They have SNMPv3 configured for them.
    0:41:35 Okay. We'll see that this is gonna be used when we get to the GUI interface of the DCNM.
    0:41:42 DCNM is gonna be logging into the Nexus 5K, 7K and MDS using SNMPv3.
    0:41:47 I was gonna try to discover the topologies, so we need these logins there.
    0:41:52 Okay. The rest of it, it's just talking about trapping, logging, and then we have our interfaces.
    0:42:00 Okay. So the next thing we need to do is to configure the VSAN to now start to span between 5K 2 and the 7K.
    0:42:11 So as it stands now, let me start this diagram from scratch.
    0:42:17 We have VSAN 1010 assign to the JBOD, so that's working.
    0:42:28 The link between MDS and 5K is working, this is a TE Port, a Trunking Expansion Port.
    0:42:35 Then on server 1, we have the Virtual Fibre Channel 1/17 that is in VLAN 1010, which really means it's in VSAN 1010.
    0:42:51 So that's up, they're registered to the Fabric.
    0:42:53 On the server side, this is the E-Node, which is also the virtual Node Port, the VN port.
    0:43:03 On the switch side, this is the Virtual Fabric Port, the VF port.
    0:43:10 What we need to do now our final step is to get the Virtual Expansion Ports up between the 5K and the 7K.
    0:43:18 And these are what's allowing us to do our multi hop FCoE design.
    0:43:23 Because as I mentioned, it's not a simple as just normal Ethernet switching logic.
    0:43:27 You actually have to change the packet on a hop-by-hop basis.
    0:43:31 So the switches is doing, the switches are doing a layer 2 rewrite on a hop-by-hop basis.
    0:43:37 So similar to how your layer 3 IP router receives an IP packet in, but it changes the layer 2 encapsulation from the ingress to the egress interface.
    0:43:46 And give your coming in packet over sonnet and you're going to Ethernet.
    0:43:49 Then you have to change the format because it's different layer 2.
    0:43:52 Same logic here, but we're doing it for the SAN switching to change the Mac address header to the next Fibre Channel forwarder next hop.
    0:44:01 So it's gonna be rewritten 3 times here.
    0:44:03 The server's gonna generate the packet, 5K 2 is gonna rewrite it to say it gets to 7K 1.
    0:44:10 7K 1 is gonna have to rewrite it again to say it's going to 5K 1, then 5K 1 is going to decapsulate it and it goes out native Fibre Channel.
    0:44:23 Okay. So next, let's look at this from the 5K.
    0:44:26 The 5K is now going to say that these two links, E1/8 and E1/9.
    0:44:33 These are now gonna be Virtual Expansion Ports.
    0:44:37 So this then means, we need a new Interface Virtual Fibre Channel.
    0:44:43 Now, the Virtual Fibre Channel link can only be assigned to one physical interface at a time.
    0:44:50 So if I have two physical links from the Fibre Channel forwarder upstream, what do I probably want to do here.
    0:44:58 I gotta channel the links together.
    0:45:00 So we're gonna create a port channel.
    0:45:01 So it's a regular Ethernet port channel config, except then the VFC is gonna be bound to the port channel interface.
    0:45:09 So before we get to the VFC config, we're actually gonna do the port channel config first.
    0:45:14 So let's say, interface E1/8 to 9.
    0:45:19 This is in channel group, let's say that this is channel group 10.
    0:45:26 Channel group 10 mode active.
    0:45:30 And I want to start feature LACP.
    0:45:35 So E1/8 to 9 channel group 10 mode active.
    0:45:42 Okay. On interface port channel 10, the switchport mode is trunk.
    0:45:49 The switchport trunk allowed VLAN list is gonna be 1010.
    0:45:55 So just my VSAN to VLAN mapping, VLAN.
    0:46:01 Now if I wanted additional ones, like let's say we had 1020, I would just add that in there.
    0:46:06 But I need to make sure that I'm not running Ethernet VLANs across this link.
    0:46:11 Not running regular LAN Ethernet VLANs.
    0:46:13 I want this only to be for my storage traffic.
    0:46:16 So it's a dedicated wire that is just for storage, just for FCoE, it's not a converged link.
    0:46:21 It's only a converged link as it goes down to the CNA, down to the end host.
    0:46:26 Once it's in the core of the network, those are gonna be dedicated wires.
    0:46:30 Okay. So we're only allowing VLAN 1010 here.
    0:46:34 Okay. Then we have our Interface Virtual Fibre Channel.
    0:46:36 I'll say that this is interface VFC 10.
    0:46:40 And this is going to bind to the interface port channel 10.
    0:46:46 The switchport trunk allowed VSAN is 1010, because this is going to be a Trunking Expansion Port.
    0:46:55 I wanna limit it just for the ones that I'm actually using.
    0:46:58 And then No Shutdown.
    0:47:01 Okay. We'll come back to the full config and verify it afterwards.
    0:47:03 But we need to do the same thing on the other side.
    0:47:06 So 7K 1 is gonna create the VLAN, VLAN 1010.
    0:47:09 And under the VSAN database, we have VSAN 1010.
    0:47:17 On VLAN 1010, the FCoE VSAN is 1010.
    0:47:21 So we're associating the two of them.
    0:47:27 Okay. Next I have interfaces E2/5 to 6 and actually I need to turn LACP on first.
    0:47:37 On these links, they're gonna be in channel group 10 mode active.
    0:47:41 On interface port channel 10, the switchport mode is trunk.
    0:47:47 The switchport trunk allowed VLAN is 1010 which really means the VSAN 1010.
    0:47:54 Then interface VFC 10 is going to bind to the interface, port channel 10.
    0:48:02 And the switchport trunk allowed VSAN is 1010.
    0:48:07 Okay. Now, at this point, if we activate the port channel and the physical links.
    0:48:13 So E2/5 to 6, let's say No Shutdown.
    0:48:16 And the same thing on the other side, if this is gonna work, we should see.
    0:48:23 I think those are the wrong ports, let me double-check my physical diagram here.
    0:48:38 These should actually be 10 and 11, that's 10 and 11, that's not 8 and 9.
    0:48:51 That's why I got a different log message here on the 5K.
    0:48:56 So on 5K, what I need to do, Show run interface E1/8 to 9.
    0:49:02 I need to change these to 10 and 11.
    0:49:08 So on interface E1/10 to 11, so the same config as the other two ports.
    0:49:24 And then these links I may need to get rid of.
    0:49:52 Okay. So next, let's look at the Show interface brief.
    0:49:55 Let's just see is the port channel up.
    0:49:58 Okay. It says port channel 10 is up, it says that VFC 10 is up and trunking its operational mode is TF.
    0:50:12 Let's now look at the Show Interface VFC 10.
    0:50:17 It says that VSAN 1010 is currently initializing, which is bad.
    0:50:24 Okay. We should see that the VSAN is up.
    0:50:31 So let's see on the 7K, do we get any log messages.
    0:50:34 It says the port channel is up, it says we're waiting for Fabric Login.
    0:50:38 Let's look at the Show Port Channel Summary.
    0:50:43 Okay. The port channel is up.
    0:50:45 A capital P for the members, capital U, so they're in the links, that should be fine.
    0:50:50 Let's look at the same thing on the other side, Show Port Channel Summary.
    0:50:54 Okay. The port channel is fine.
    0:50:55 This is gonna be something related to our SAN config now.
    0:50:58 So let's say on this side, Show Interface Virtual Fibre Channel 10 trunk VSAN.
    0:51:06 VSAN 1010 is down, waiting for Fabric Login.
    0:51:10 Show Interface VFC 1010, or just 10.
    0:51:15 Says 1010 is allowed, 1010 is initializing.
    0:51:19 The trunking mode is on and the admin port mode is F.
    0:51:27 This is what our problem is gonna be.
    0:51:28 This is not a Fabric Port, what do we need this to be here?
    0:51:35 We need it to be an expansion port.
    0:51:37 We need it to be an E port.
    0:51:39 So on interface VFC 10, the switchport mode should be E, not F.
    0:51:45 And the same thing on the 5K, interface VFC 10, switchport mode should be E, not F.
    0:51:55 Okay. So let's bounce the logical link.
    0:51:57 Let's shut down the VFC, let's do this on both sides.
    0:52:28 Okay. Now let's look at the Show Interface VFC 10.
    0:52:32 Okay. Now, it's up.
    0:52:33 So VSAN 1010 is up.
    0:52:36 Says the admin port is now E, the port mode is trunking expansion.
    0:52:40 But it's actually virtual trunking expansion.
    0:52:42 It is now a VTE Port because there's a Virtual Fibre Channel interface that's bound to the physical link.
    0:52:49 And we're running the point to point adjacency essentially, the virtual link over the Virtual Fibre Channel.
    0:52:54 This is essentially what they mean by multi hop FCoE,
    0:53:00 that we now have a virtual expansion port that's' gonna allow us to do this switch-to-switch connectivity.
    0:53:06 Now, if everything is working in the Fabric, at least up to 7K1.
    0:53:11 What should we see when we do verifications here?
    0:53:18 How can I tell that the Fabric for VSAN 1010 is actually working up to 7K 1?
    0:53:25 Okay? Just on this side, just on the right side, 'cause we didn't configure 5K 1 to talk to 7K 1.
    0:53:32 I should be able to look at the FCNS Database.
    0:53:35 So if I see that the servers FCId and their PWWN is registered, it should be that the Fabric is end-to-end.
    0:53:43 So let's look at the 7K and look at the Show FCNS Database, and there we go.
    0:53:50 So I know that the Emulex end adapter that's the CNA.
    0:53:55 They are a Node Port. This is their Fibre Channel ID, this is their PWWN.
    0:54:01 If I were to dent, look at the Show FC route for unicast routes.
    0:54:08 It says, in VSAN 1010 I know about someone who is 8D.
    0:54:14 If we Show FC domain, domain list, 8D is the 5K.
    0:54:23 Okay. 5K 2 in this case is the principal switch.
    0:54:27 We don't really care again who this is though, because the only thing that principal switch is doing is giving us a domain ID.
    0:54:33 In a real design, we probably wanna set this just for, to make it easier for verification.
    0:54:38 And again, when you're setting that, basically what you're doing it's like setting your router ID in OSPF, or your net address in IS to IS.
    0:54:46 So you don't necessarily have to set it statistically, it's just kind of good practice for verification.
    0:54:51 But based on the fact that now, I know about the CNA in my FCNS Database, it means that FCoE is end-to-end from the server up to 7K 1.
    0:55:03 So now we have the last portion which is gonna be the virtual expansion port between 5K 1 and 7K 1.
    0:55:13 So before we do that, let's look at the config we have up to this point.
    0:55:16 On 5K 1, let's Show run and we channeled the links together.
    0:55:25 So let me start to to make a list of what we have here.
    0:55:29 I have the port channel and the Virtual Fibre Channel, which is then bound to this physical links, and that's actually it.
    0:55:49 Because the FCoE VLANs, those were already associated.
    0:55:56 And it's already allowed on the link.
    0:55:58 So the port channel itself, this config is unrelated.
    0:56:01 It does just normal Ethernet port channel that I'm putting the two physical links together.
    0:56:06 The physical links, I'm saying that they're, I'm allowing VSAN 1010 on it.
    0:56:11 And the way I'm allowing VSAN 1010 is by doing the association of the VSAN to VLAN and then the VLAN is allowed on the link.
    0:56:21 So it is kind of redundant that I have to keep doing this over and over, allowing it on the Virtual Fibre Channel and in the physical link.
    0:56:27 But the end goal is that you're trying to make the network is optimal forwarding as you can.
    0:56:33 You don't want to just send your storage traffic to every single link in the network, every single trunk link.
    0:56:38 Okay. Also need to make sure now for my rest of my LAN ports.
    0:56:43 I don't want VSAN, or excuse me, I don't want VLAN 1010 to trunk in any of those interfaces.
    0:56:50 I only want it going down to the CNA's and then going up on the VE Port up to the 7K.
    0:56:56 So 5K 2's uplinks to the regular Ethernet network.
    0:57:01 So if I'm going, let's say I may have a different physical switch that I'm going from here,
    0:57:06 or maybe 7K 2, I need to make sure that on this link 1010, is not allowed.
    0:57:14 Because otherwise, I'm gonna be sending the FCoE traffic to some random portion of the network, that's not gonna need it.
    0:57:26 Okay. So let's take this config now, and let's apply it to 5K 1.
    0:57:31 Really, the only thing that I need to do here is to change these port numbers.
    0:57:38 These are gonna be 8 and 9.
    0:57:44 Okay. I'm also gonna shut these links down just to make sure that we're activating everything at the very last portion.
    0:57:53 And let's give this a new port channel number.
    0:57:55 Let's say this is 20, port channel 20, channel group 20, channel group 20.
    0:58:03 Okay. So this is gonna be 5K 1's config.
    0:58:15 LACP process needs to be started before configuring active modes.
    0:58:18 So feature LACP, and let me paste that back in.
    0:58:26 Okay. So same thing on the 7K.
    0:58:27 Really, the only thing that's gonna change on the 7K is I need a new port channel for my physical links.
    0:58:34 And then I need a new Virtual Fibre Channel interface.
    0:58:40 And actually what I should have done on, what I should've done here on the 5K is use the different interface number.
    0:58:54 Okay. I don't necessarily have to but it's gonna make it more confusing if I don't use the same number.
    0:58:59 So let's change that to Virtual Fibre Channel 20.
    0:59:09 Okay. So it's VFC 20 now which correlates to port channel 20.
    0:59:17 So the 7K's config, you can see as even simpler because really the only thing that's in here is just related to SAN.
    0:59:22 There's nothing that's related to LAN switching.
    0:59:25 So let's open up a new Window, this is now gonna be for the 7K.
    0:59:30 The 7K already has the VSANs of configure, that already has LACP running.
    0:59:34 So I now have a new port channel, I have a new VFC, which is binding to a new port channel.
    0:59:42 Then these are gonna be on Ethernet 2 and 3, that is binding to the new channel number.
    0:59:55 So as you could see here, the FCoE configure is really not that involved.
    1:00:00 It's just that there's so many pieces that go into the puzzle.
    1:00:03 You have to know how to verify each of them individually before you can get the final overall picture.
    1:00:09 So the native Fibre Channel has to be working, we have to make sure that the,
    1:00:14 that everyone is logging in with the Fabric Login, that they get in to the FCNS Database.
    1:00:18 So the config is straight forward but it's just that there's so many pieces that you could potentially have problems with.
    1:00:31 So those are, did I use the wrong links there?
    1:00:36 E2, that should be 2/3 to 4.
    1:00:40 So 3 to 4, that's what the issue was.
    1:00:50 Okay. So those links are up, the Virtual Fibre Channel is up.
    1:00:53 Let's look at the Show Port Channel Summary.
    1:00:57 It says the ports are down which is correct because the other side is shut down.
    1:01:01 So let's go to 5K 1, and we're gonna activate the physical links.
    1:01:04 So interface E1/9 to 10, or actually 8 to 9.
    1:01:12 E1/8 to 9, No Shutdown.
    1:01:17 I should see that the physical links come up, that my port channel comes up.
    1:01:22 Then finally I should see that my Virtual Fibre Channel comes up.
    1:01:30 Let's look at the Show Port Channel Summary.
    1:01:35 So the port channel is up, let's look at the Show Interface VFC.
    1:01:41 Show Interface VFC 20, it says that VSAN 1010 is isolated.
    1:01:50 And why is that isolated?
    1:01:51 It says the zoning modes do not match.
    1:01:55 Let's look at the Show Zone Status for VSAN 1010, my mode is enhanced.
    1:02:07 Let's see what is their mode.
    1:02:11 Zoning mode does not match, receive rejection from adjacent switch.
    1:02:15 Show Zone Status VSAN 10/10.
    1:02:19 Okay, my mode is basic, so we cannot have some of the switches running enhanced modes, some of them running basic.
    1:02:23 So let's say the zone mode is enhanced for VSAN 10/10.
    1:02:29 This should now have changed this on 5K 2 as well if we Show Zone Status VSAN 1010.
    1:02:37 Okay they're running in enhanced, then let's bounce the link.
    1:02:43 Let's say on port channel 20, it's shut down.
    1:02:49 No Shutdown, that should bring the physical links up and then eventually the Virtual Fibre Channel should come up.
    1:03:02 Port channel is up, Show interface brief.
    1:03:09 We should see the VFC becomes a Trunking Expansion Port, that's actually a VTE, a Virtual Trunking Expansion Port.
    1:03:19 Now VSAN 1010 is up.
    1:03:23 If we now Show FCNS Database, now we see the disks.
    1:03:28 So this is now telling me that the Fabric is completely end-to-end.
    1:03:32 So that the disks sent Fabric Login.
    1:03:41 MDS gave them the FCId.
    1:03:45 They then registered with the FCNS.
    1:03:48 Okay. Now, everyone in the Fabric should know their FCId to PWWN mapping.
    1:03:57 We looked at this Show FCNS Database on 7K 1 and it saw the disks.
    1:04:02 This means that everything is end-to-end this way.
    1:04:05 So the Fabric is working on the left.
    1:04:08 If we look back at Show FCNS Database, we see also the CNA, the Converged Network Adapter.
    1:04:14 This means that the Fabric is end-to-end on this side.
    1:04:19 Now, if I go to MDS 1, and I see the initiator, and I go to 5K 2, and I see the target, now everything should be ready finally for the data plane.
    1:04:30 So in MDS 1, let's say Show FCSN Database.
    1:04:34 I know about the server.
    1:04:37 On 5K 2, Show FCNS Database, I know about the disks.
    1:04:42 Okay. So now, multi hop FCoE should be working end-to-end.
    1:04:45 Let's go to the actual server and see, did we get the targets?
    1:04:54 There are the mappings.
    1:04:56 Let's look at our disk management, and there's our volumes, our volumes mounted.
    1:05:04 If we look at here the volumes, there's JBOD 1 disk 0.
    1:05:18 Okay. There's a question: Can I show 5K 2's server port config again?
    1:05:23 Yes. So let's go through our full verification here.
    1:05:27 So on 5K 2, this is going down to the initiator which is E1/17.
    1:05:36 It's a trunk that is only allowing VLANs 10 and 1010.
    1:05:41 Okay 10 is my data VLAN, that's for my LAN traffic, 1010 is for my VSAN traffic.
    1:05:50 Okay, also it's Spanning tree port type edge trunk which means that we don't have to wait for the listening and learning phases.
    1:05:56 This is important to make sure that the Fabric Login is gonna work.
    1:05:59 Otherwise the FLOGI is gonna time-out.
    1:06:00 You won't be able to join the Fabric.
    1:06:03 Okay. Then I have the interface VFC 1/17.
    1:06:07 This is that VSAN 1010 is the only one on there.
    1:06:11 This is calling the physical link 1/17, then VFC 1/10, or 1/17, this needs to be assigned to VSAN 1010.
    1:06:21 So if we Show Run Section VSAN Database, Interface Virtual Fibre Channel 1/17 is part of that VSAN.
    1:06:36 If we Show FLOGI Database, now we have them registered.
    1:06:41 Okay. Next we have the ports that are going over to the 7K, which is E1/10 to 11.
    1:06:52 So let's Show run interface E1/10 to 11.
    1:06:57 These are in an Ethernet port channel, which is channel group 10.
    1:07:01 If we Show run interface port channel 10, this is allowing VLAN 1010.
    1:07:09 If we Show run Interface Virtual Fibre Channel 10, this is our expansion port, our Virtual Expansion port.
    1:07:17 This is what's allowing the multi hop FCoE.
    1:07:22 So now, when we look at the Show Interface VFC 10, we should see that the VSAN is up.
    1:07:30 The port mode is trunking expansion.
    1:07:34 If we Show interface brief, the port channel is up.
    1:07:43 And the Virtual Fibre Channel is up.
    1:07:49 Okay. Here, we could see the speed is 20 gig, because it's the aggregation of two of the physical 10 gig links.
    1:08:01 So again, really the overall configuration for this is not complicated as compared to some of the other stuff that we've seen on the 5K or the MDS's.
    1:08:09 It's really just to understand what's the overall logic that's going into this.
    1:08:14 How does the virtual Node Port correspond to the E-Node Port?
    1:08:17 How does the Virtual Fabric Port correspond to the Fabric Port?
    1:08:20 How does the virtual expansion correlate to the normal expansion port?
    1:08:24 That's essentially what we're configuring with the FCoE, is that we're taking the Virtual Fibre Channel interfaces,
    1:08:29 and we're making them look like normal regular NF or E-Ports.
    1:08:36 Now, again not all platforms support this currently.
    1:08:39 So when we look at later in the UCS class and talk about the Fabric interconnect, we're not gonna have these Virtual Expansion Ports.
    1:08:44 We're just gonna have virtual Node Ports.
    1:08:47 On another type we're gonna look at is the VNP port which is the proxy virtual Node Port,
    1:08:52 which is gonna be for FCoE NPV, Node Port Virtualization.
    1:08:58 So really the only thing that's changing from our previous FCoE example is that we have this virtual Trunking Expansion Ports,
    1:09:06 or Virtual Expansion Ports that are running between the switches.
    1:09:13 Okay, there's a question: Do we need to enable jumbo frames in QoS?
    1:09:17 Yes we do. The way that we enabled them was by applying the FCoE policy.
    1:09:24 So on the 5K's, if we look at the Show run section system, or system QoS, we already have the FCoE default policy.
    1:09:35 So if we Show policy map, type network QoS, and we look at the FCoE default NQ policy, this says the MTU is 2158.
    1:09:52 Then on the 7K, that's what went into the default VDC.
    1:09:56 So if we go back to the default VDC here, and Show run section system.
    1:10:10 This is service policy type network QoS default NQ 7E policy.
    1:10:15 If we Show policy map type network QoS, the NQ 7E policy says that if it's data traffic the MTU is 1500.
    1:10:24 If it's FCoE, we're allowing pause which is priority flow control and then the MTU is 2112.
    1:10:31 I'm not 100% sure why it shows these as different values here.
    1:10:34 2112, that's the payload size, that's the actual Fibre Channel FCP payload, is up to 2112.
    1:10:42 I think maybe what it's talking about here is that it's just not taking into account the overhead for the extra layer 2 encap.
    1:10:48 But I'll have to look into that in a little bit more detail and see what the difference is.
    1:10:54 But essentially by having the QoS policy, that's what's giving us the jumbo MTU.
    1:11:01 Okay. There's a question: Can we see how the Mac address is changing?
    1:11:07 There's, I don't believe there's actually a way that I can do that right now.
    1:11:13 What I would have to do is, let's see here.
    1:11:26 I'd have to put a physical tap on that link to do so, which means I'd have to rewire some stuff.
    1:11:34 For this question, I'm gonna come back to this.
    1:11:36 I'm gonna do a blog post on this to Show the actual data plane,
    1:11:39 because it is important to understand behind the scenes what does it actually look like when the data is flowing.
    1:11:45 Because in your real design, if you need to troubleshoot something,
    1:11:47 you're gonna troubleshoot it from the data plane and the control plane configuration.
    1:11:52 'Cause if it's an application level problem, that's what the actual data flow capture is gonna show you versus something like in the config.
    1:11:59 So I will come back to that, I have to some physical rewiring in order to get the port redirection to look at that.
    1:12:06 Because right now basically where my layer 1 tap is, is on this link.
    1:12:12 And capturing it there is just gonna show the last hop FCoE.
    1:12:15 It's not gonna show the multi hop rewrite that's going on the 7K's.
    1:12:26 There's a question: Can I enable FCoE over a link that has Fabric path for the data VLANs.
    1:12:32 I don't believe that's currently supported.
    1:12:35 I do remember reading that they're going to be adding functionality for it but I think as it stands right now, the VLAN if it is mapped to,
    1:12:46 and let's look at this on the 7K, if we look at the Show run here, and that's end up the wrong VDC.
    1:12:53 We want the storage one.
    1:12:55 Let's Show run here.
    1:12:57 I would have to go to this VLAN and say the mode is Fabric path, and I don't believe it's allowed in this case.
    1:13:11 There's a question: Would the Ethanalyzer work for capturing this FCoE packets?
    1:13:16 No, it would not. And the reason why is that you can only capture packets that are punted to the CPU.
    1:13:24 And the only way to punt them to the CPU is that either their control plane,
    1:13:28 like their OSPF routing protocol packets, or you could do an access list log on the interface.
    1:13:33 The problem is that the F1 and the F2 modules, you cannot put an access list directly at the interface to log it.
    1:13:39 So really, the only way to realistically capture this is to do a layer 1 tap.
    1:13:45 And if you need to do this in a production, there's a couple different companies that have these.
    1:13:50 If you search for 10 GigE, Ethernet tap, net optics is a popular one.
    1:14:01 If we look at here 10 GigE LC Fiber Tap.
    1:14:08 Basically what this is, it's a passive tap.
    1:14:12 So you go in this port and then out the other, and it makes a copy to this one.
    1:14:18 This is better than doing a switchport analyzer because it shows you the full layer 1 framing.
    1:14:24 So then if you need to do some really low level troubleshooting of, figure out is it a cable problem?
    1:14:29 Like is the fiber too long or is there some problem with the transceiver.
    1:14:34 The switchport analyzer is really not gonna tell you that.
    1:14:38 So we're capturing in line, but the issue is that you have to capture in line of 10 gig.
    1:14:42 Because these ports, most of them are 10 gig only ports, not 10/1.
    1:14:47 In the case of the F1 module, the F1 module is 1/10.
    1:14:52 And you can, you could use then just a 1 gig tap to do that which is, will be like this one.
    1:15:02 The problem is though your FCoE is supposed to be running on your 10 gig links.
    1:15:08 So between like the server and the converge network adapter and the first tap switch, the Fibre Channel forwarder, that has to run it 10 gig.
    1:15:18 Otherwise it's not going to, it's not gonna negotiate.
    1:15:23 Because FCoE is not supported on 1 GigEthernet, that's only on 10 gig.
    1:15:27 So if you are troubleshooting this in production and you need to see the data plane you probably need to get something like this to do in line 10 GigE monitoring.
CCIE Data Center :: Storage
Title: CCIE Data Center :: Storage
Duration: 17h 59m
Instructor: Brian McGahan, #8593 CCIEx4, CCDE #2013::13
Get instant access to our entire library!
Sign Up

© 2003 - 2015 INE All Rights Reserved