python 中文乱码问题深入分析
转载 & & 作者：
一直以来，python中的中文编码就是一个极为头大的问题，经常抛出编码转换的异常，python中的str和unicode到底是一个什么东西呢？
在本文中，以'哈'来解释作示例解释所有的问题，“哈”的各种编码如下： 1. UNICODE (UTF8-16)，C854； 2． UTF-8，E59388； 3． GBK，B9FE。 一、python中的str和unicode 一直以来，python中的中文编码就是一个极为头大的问题，经常抛出编码转换的异常，python中的str和unicode到底是一个什么东西呢？ 在python中提到unicode，一般指的是unicode对象，例如'哈哈'的unicode对象为 u'\u54c8\u54c8' 而str，是一个字节数组，这个字节数组表示的是对unicode对象编码(可以是utf-8、gbk、cp936、GB2312)后的存储的格式。这里它仅仅是一个字节流，没有其它的含义，如果你想使这个字节流显示的内容有意义，就必须用正确的编码格式，解码显示。 例如：
对于unicode对象哈哈进行编码，编码成一个utf-8编码的str－s_utf8,s_utf8就是是一个字节数组，存放的就是'\xe5\x93\x88\xe5\x93\x88'，但是这仅仅是一个字节数组，如果你想将它通过print语句输出成哈哈，那你就失望了，为什么呢？
因为print语句它的实现是将要输出的内容传送了操作系统，操作系统会根据系统的编码对输入的字节流进行编码，这就解释了为什么utf-8格式的字符串“哈哈”，输出的是“鍝堝搱”，因为 '\xe5\x93\x88\xe5\x93\x88'用GB2312去解释，其显示的出来就是“鍝堝搱”。这里再强调一下，str记录的是字节数组，只是某种编码的存储格式，至于输出到文件或是打印出来是什么格式，完全取决于其解码的编码将它解码成什么样子。
这里再对print进行一点补充说明：当将一个unicode对象传给print时，在内部会将该unicode对象进行一次转换，转换成本地的默认编码（这仅是个人猜测）
二、str和unicode对象的转换
str和unicode对象的转换，通过encode和decode实现，具体使用如下：
将GBK'哈哈'转换成unicode，然后再转换成UTF8
三、Setdefaultencoding
如上图的演示代码所示：
当把s(gbk字符串)直接编码成utf-8的时候，将抛出异常，但是通过调用如下代码：
import&sys
reload(sys)
sys.setdefaultencoding('gbk')
后就可以转换成功，为什么呢？在python中str和unicode在编码和解码过程中，如果将一个str直接编码成另一种编码，会先把str解码成unicode，采用的编码为默认编码，一般默认编码是anscii，所以在上面示例代码中第一次转换的时候会出错，当设定当前默认编码为'gbk'后，就不会出错了。
至于reload(sys)是因为Python2.5&初始化后会删除&sys.setdefaultencoding&这个方法，我们需要重新载入。
四、操作不同文件的编码格式的文件
建立一个文件test.txt，文件格式用ANSI，内容为:
用python来读取
# coding=gbk
print open("Test.txt").read()
结果：abc中文
把文件格式改成UTF-8：
结果：abc涓?枃
显然，这里需要解码：
# coding=gbk
import codecs
print open("Test.txt").read().decode("utf-8")
结果：abc中文
上面的test.txt我是用Editplus来编辑的，但当我用Windows自带的记事本编辑并存成UTF-8格式时，
运行时报错：
Traceback (most recent call last):
File "ChineseTest.py", line 3, in&
print open("Test.txt").read().decode("utf-8")
UnicodeEncodeError: 'gbk' codec can't encode character u'\ufeff' in position 0: illegal multibyte sequence
原来，某些软件，如notepad，在保存一个以UTF-8编码的文件时，会在文件开始的地方插入三个不可见的字符（0xEF 0xBB 0xBF，即BOM）。
因此我们在读取时需要自己去掉这些字符，python中的codecs module定义了这个常量：
# coding=gbk
import codecs
data = open("Test.txt").read()
if data[:3] == codecs.BOM_UTF8:
data = data[3:]
print data.decode("utf-8")
结果：abc中文
五、文件的编码格式和编码声明的作用
源文件的编码格式对字符串的声明有什么作用呢？这个问题困扰一直困扰了我好久，现在终于有点眉目了，文件的编码格式决定了在该源文件中声明的字符串的编码格式，例如：
str&=&'哈哈'
print&repr(str)
a.如果文件格式为utf-8，则str的值为：'\xe5\x93\x88\xe5\x93\x88'（哈哈的utf-8编码）
b.如果文件格式为gbk，则str的值为：'\xb9\xfe\xb9\xfe'（哈哈的gbk编码）
在第一节已经说过，python中的字符串，只是一个字节数组，所以当把a情况的str输出到gbk编码的控制台时，就将显示为乱码：鍝堝搱；而当把b情况下的str输出utf-8编码的控制台时，也将显示乱码的问题，是什么也没有，也许'\xb9\xfe\xb9\xfe'用utf-8解码显示，就是空白吧。&_&
说完文件格式，现在来谈谈编码声明的作用吧，每个文件在最上面的地方，都会用#&coding=gbk&类似的语句声明一下编码，但是这个声明到底有什么用呢？到止前为止，我觉得它的作用也就是三个：
声明源文件中将出现非ascii编码，通常也就是中文；
在高级的IDE中，IDE会将你的文件格式保存成你指定编码格式。
决定源码中类似于u'哈'这类声明的将‘哈'解码成unicode所用的编码格式，也是一个比较容易让人迷惑的地方，看示例：
#coding:gbk
ss&=&u'哈哈'
print&repr(ss)
print&'ss:%s'&%&ss
将这个些代码保存成一个utf-8文本，运行，你认为会输出什么呢？大家第一感觉肯定输出的肯定是：
u'\u54c8\u54c8'
但是实际上输出是：
u'\u935d\u581d\u6431'
为什么会这样，这时候，就是编码声明在作怪了，在运行ss&=&u'哈哈'的时候，整个过程可以分为以下几步：
1)&获取'哈哈'的编码：由文件编码格式确定，为'\xe5\x93\x88\xe5\x93\x88'（哈哈的utf-8编码形式）
2)&转成 unicode编码的时候，在这个转换的过程中，对于'\xe5\x93\x88\xe5\x93\x88'的解码，不是用utf-8解码，而是用声明编码处指定的编码GBK，将'\xe5\x93\x88\xe5\x93\x88'按GBK解码，得到就是''鍝堝搱''，这三个字的unicode编码就是u'\u935d\u581d\u6431'，至止可以解释为什么print&repr(ss)输出的是u'\u935d\u581d\u6431' 了。
好了，这里有点绕，我们来分析下一个示例：
#-*-&coding:utf-8&-*-
ss&=&u'哈哈'
print&repr(ss)
print&'ss:%s'&%&ss
将这个示例这次保存成GBK编码形式，运行结果，竟然是：
UnicodeDecodeError:&'utf8'&codec&can't&decode&byte&0xb9&in&position&0:&unexpected&code&byte
这里为什么会有utf8解码错误呢？想想上个示例也明白了，转换第一步，因为文件编码是GBK，得到的是'哈哈'编码是GBK的编码'\xb9\xfe\xb9\xfe'，当进行第二步，转换成 unicode的时候，会用UTF8对'\xb9\xfe\xb9\xfe'进行解码，而大家查utf-8的编码表会发现，utf8编码表（关于UTF- 8解释可参见字符编码笔记：ASCII、UTF-8、UNICODE）中根本不存在，所以会报上述错误。
您可能感兴趣的文章:
大家感兴趣的内容
12345678910
最近更新的内容
常用在线小工具Ethernet (IEEE802.3)
A Made IT project
Ethernet (IEEE802.3)General
This type of network was developed by Xerox. It was eventually standardized as the IEEE802.3 based on the Ethernet DIX standard (DIX= Digital, Intel, Xerox). The IEEE802.3 describes all eth both 10, 100 and 1000 Mbps networks. This means they have a lot in common and are easilly connectable to eachother. Between two different 10 Mbps standards a repeater with two different interfaces is enough and the same goes for two different 100 Mbps standards. However to connect a 100 Mbps network to a 10 Mbps you need a bridge. This discribes the difference. The difference between the different 10 Mbps standards is situated on Layer 1 of the OSI reference model, where the difference between 10 and 100 Mbps is situated on the MAC-layer which is part of Layer 2 of the OSI reference model.
Detailed description
According to the OSI layers an ethernet network looks like this:
2Data LinkLLC or LLC + SNAP
MAC
1PhysicalInterface + PHY
LLC
For SNAP and IEEE802.2 (LLC) see the corresponding documents.
MAC: IEEE802.3 CSMA/CD
The complete ethernet networking family is based on the CSMA/CD protocol. CSMA/CD stands for Carrier Sense Multiple Access with Collision Detetection, which means that a station that has something to send listens for a carrier (if someone is already sending something) if not it sends its data. At the moment multiple stations can decide to send their data, since they all heard no carrier, this is the multiple access. After sending the station keeps on listening to the carrier and when they detect that another station started sending too, collision detection, it backs off, waits a random time and starts the whole procedure from scratch.
The timing involved for sending and receiving is different for 10 and 100 Mbps ethernet:
1000 Mbps
Bit Time
1 ns
Interpacket gap
96 bit times or 9.6 &s
96 bit times or 0.96&s
96 bit times or 0.096&s
The Data-link-frame
[META]
Interface and PHY
PHY is an abbreviation for the tranceiver. A tranceiver can be a separate device or it can be integrated on the network card or on the motherboard of the PC. It is on this layer that the real difference between the different standards start. There fore they all have their own document to describe them.
GeneralThe PHY is the actual tranceiver which can be a separate device or it can be integrated on the network card. The tranceiver interface is called AUI (Attachment Unit Interface). When a network card doesn't contain the interface there will be a Sub-D15 female connector. On the cable will be a tranceiver with an male connector. This means that an AUI-cable will ALWAYS be male-female.
&MaleFemale
SubD15 (AUI)
AUI pinning specification
PinSignalPinSignal
1Control In Circuit Shield9Control In Circuit B
2Control In Circuit A10Data Out Circuit B
3Data Out Circuit Shield11Data Out Circuit Shield
4Data In Circuit Shield12Data In Circuit B
5Data In Circuit A13Voltage Plus
6Voltage Common14Voltage Shield
7Control Out Circuit A15Control Out Circuit B
8Control Out Circuit Shield&&
The difference between the different 10 Mbps topologies is in the PHY part. This section connects directly to the cable and is responsible for everything that is medium depended like: line encoding, transmission voltages, SQE, etc.
With AUI there are two ways power can be provided to the units. You eighter have a positive or negative polarity.
Positive polarity:Pin 13 (to shell), +11 V to +13 V; Pin 6 (to shell), about 0.05 V
Negative polarity:Pin 13 (to shell), about 0.5 V; Pin 6 (to shell), -11 to -16 V
Do not mix units with different polarities!
SQE
The Signal Quality Error signal is also called 'heartbeat' and is a kind of keepalive notification between the transceiver and the ethernet device. SQE can be ON or OFF between a transceiver and a workstation or file server. It MUST be set OFF between a transceiver and a Repeater.
10Base-5 Quick Overview
IEEE-spec802.3
Max. speed10 Mbps
CableStandard Ethernet Coax Cable
Connectors N-type
Terminators50 ohm
Max. length of a segment500m/1640ft
Max. number of taps per segment100
Max. number of stations per network1024
Min. distance between taps2.5m/8.3ft
Max. length of tranceiver cable50m/164ft
Max. number of repeaters4
TopologyBus
Maximum Topology
10Base-5 is a bus topology. A thick coaxial cable runs through the building and stations are attached to this cable by tranceivers. An closer look on cabling can be found below.
The maximum amount of repeaters in a network is four. Since a segment may be up to 500 meters the total network length can be 2500 meter. There is little catch in this because 2 of the total of 5 segments may not be occupied. This doesn't matter for the length, but it does for the way you position your computers. The 2 non-occupied segments are only ment for extending the network and are called IRLs (Inter Repeater Links). For larger distances you need fiber optic repeaters or bridges or routers.
10Base-5 uses standard coaxial cable, the cable is about 1cm thick and yellow (normal cable) or orange (plenum cable) coloured. The connector type used is N-type.
&MaleFemaleSide
N-Type
A segment can be one cable length with only two N-type connectors at the far ends, or it can be composed of several pre-terminated cables. The two last N-type connectors on the cable need to have a 50 ohm terminator installed. One of the terminaters should be grounded. A segment is defined as all the cable between two terminators.
Devices are attached to the segment (or backbone cable) by means of transceivers. A transceiver can be intrusive (N-type) or non-intrusive (vampire type). A transceiver always has a Sub-D15 male AUI (Attachment Unit Interface) connector. An IEEE802.3 10Base-5 compatible device has a female Sub-D15 female AUI (Attachment Unit Interface) connector that normally connects to a tranceiver by means of a tranceiver cable (also called AUI or drop cable). The transceiver cable is ALWAYS a male-female cable.
Tranceivers and Fan-Out Units
A vampire or non-intrusive transceiver is connected to the coax cable by opening up the jacket and shielding, drilling a hole in the insulation and inserting a 'centre probe' in the hole. Two 'braid picks' connect to the braided shielding and the centre probe connects the core of the cable to the transceiver body. An intrusive transceiver has N-type connectors and connects to cable runs terminated with N-type connectors. A tranceiver can have the possibility to send out SQEs. This is a kind of keep alive message to notify to the attached station that everything is still allright.
One tranceiver tap can support multiple devices but will still count as only one of the 100 taps. A unit that makes it possible to connect multiple devices to one tap is called a Fan-Out Unit. A Fan-Out Unit usually has one ethernet device port (female) and 2, 4 or 8 ethernet transceiver ports (male). A transceiver cable connects the female AUI connector of the Fan-out unit to the male AUI connector of the tranceiver on the segment. Transceiver cables are also used to connect the ethernet devices to the transceiver ports on the Fan-Out Unit.
Notes on 10Base-5
With AUI there are two ways power can be provided to the units. You eighter have a positive or negative polarity.
Positive polarity:Pin 13 (to shell), +11 V to +13 V; Pin 6 (to shell), about 0.05 V
Negative polarity:Pin 13 (to shell), about 0.5 V; Pin 6 (to shell), -11 to -16 V
Do not mix units with different polarities!
You cannot adapt to Thin Ethernet by using BNC-to-N-type adapters. You need a Thick/Thin Repeater or Media-Converter. In some networks with very short segments, using a BNC-to-N-type adapter appears to work. However, problems will occur when extending the segments or when adding new workstations or file servers. At first glance it looks that the problems are caused by the segment-extension or by the added network stations. Troubleshooting such a network will take a lot of time looking at the wrong causes.
10BASE-2 Quick Overview
IEEE-norm802.3
Maximum speed10 Mbps
CableRG58
ConnectorsBNC
Terminators50 ohm
Max. length of a segment185m/607ft
Max. number of taps per segment30
Max. amount of stations per network1024
Min. distance between taps0.5m/1.65ft
Max. number of repeaters4
TopologyBus
Maximum Topology
10Base-2 is a bus topology. The cable runs from computer to computer, like a daisy-chain. All devices are connected to the cable through a T-connector. The tranceiver is on the ethernet card in the device. This means that no cable is allowed between the T-connector and the device.
A complete 10BASE-2 network (one collision domain) may consist of five segments interconnected by four repeaters. Only three of those five segments may have network devices connected to them (populated). The other two segments function as Inter Repeater Links (IRLs) and
their only function is to extend the network. This allows for a maximum of 925m/3035ft (5x185m) of network cable if you stick to 10Base-2 cable. For larger distances you need 10Base-5 or fiber optic repeaters or bridges or routers.
Cabling
Thin Ethernet is also called Cheapernet. Thin Ethernet uses RG58 coaxial cable and BNC connectors.
&MaleFemaleSide
BNC
An IEEE802.3 10Base-2 compatible device has a female BNC connector that connects to the coax cable by means of a FMF BNC T-connector. The T-connector connects directly to the device. It is not allowed to have any length of cable between the BNC T-connector and the device. The two last BNC T-connectors need to have a 50 ohm terminator installed on the un-used (open) side. One of the terminaters should be grounded. A segment is defined as all the cable plus T-connectors between two terminators.
Notes on 10BASE-2
When measuring with an ohmmeter between the center and shell of the male connector of a BNC T-connector you should read 25 Ohm. This equals the resistance of the two 50 Ohm terminators in parallel. When measuring with the ohmmeter on one end of the cable where the terminator is unscrewed, you should measure 50 ohm. This equals the 50 Ohm terminator at the other end.
RG58 PVC cable is 5 mm in diameter, a plenum cable is a little thicker.
You cannot adapt to Thicknet cable by using BNC-to-N-type adapters. You need a Thick/Thin Repeater or Media-Converter. In some networks with very short segments, using a BNC-to-N-type adapter appears to work. However, problems will occur when extending the segments or when adding new workstations or file servers. At first glance it looks that the problems are caused by the segment-extension or by the added network stations. Troubleshooting such a network will take a lot of time looking at the wrong causes.
One BNC tap can support multiple devices but will still count as only one of the 30 taps. For example you can use one BNC T-connector to connect a BNC Multi-Port Repeater.
The maximum length of a Thin Ethernet segment is 185m according to the specs. In practice you may run into trouble with shorter lengths of cable. Due to the influence of the environment and due to different cable types (although they're all RG58) and due to less than perfect connector-to-cable connections you will loose segment length.
10Base-T Quick Overview
Wire speed
Cable type
UTP CAT 3, 4 and 5
Connector type
Used pins1 & 2, 3 & 6
Max. length of a segment
100m/328ft
Max. number of taps per segment
Max. amount of stations per network
Max. amount of repeaters
Maximum Topology
A segment is defined as the cable between the hub and a workstation. According to the EIA/TIA this length has a maximum of 100m separated in: 5m from HUB to patchpanel, 90 meters from patchpanel to wall outlet, and 5 meters from wall outlet to the workstation.
A complete 10Base-T network (one collision domain) may consist of 4 repeaters between the two far most workstations. Meaning the maximum length of a 10Base-T Network can be 500m/1500ft. To exceed this maximum you need Fiber Optic Repeaters or Bridges or Routers.
10Base-T uses Category 3, 4 or 5 UTP cable and RJ45 connectors.
&MaleFemale
RJ45
Any IEEE802.3 10Base-T compatible device has a female RJ45 connector that normally connects to a hub or concentrator using UTP cable. The cabling is more or less a DTE/DCE situation. The workstation is a DTE and the HUB is a DCE. Connecting a workstation to an HUB requires a straight cable. Connecting two hubs or two workstations togather requires a crossed cable.
10Base-T only uses 4 wires. In general the cable that is installed will be an 8-wire cable. The pins 4, 5, 7 and 8 are simply not used.
RJ45 pinning
PinSignal
1Transmit (positive)
2Transmit (negative)
3Receive (positive)
6Receive (negative)
Straight cable
Crossed cable
HUBs and repeaters
Repeaters in a 10Base-T network can have 2 or more ports. The most common word for those repeaters with more than two ports that act as the center of the network is HUB. HUBs may be connected to each other. There are three ways to do this:
1) The HUBs are stackable, this means that a stacked HUB counts as one repeater in the network.
2) Through un uplink port. This way a straight cable (1:1) can be used
3) Connecting two station ports with a crossed cable.
Error detection
A collision within a 10Base-T network is detected by the simultaneous occurance of signals on the transmit and receive pairs.
Link Integrity Test
By checking the receive signals a HUB monitors if a link is working correctly. On idle networks HUBs send a link test signal to one-another to verify the link integrity.
If a HUB has a link test LED this makes it easier to monitor if the link is working allright. If the link is allright (LEDs on both sides have to be lit) you know your cable wiring is correct, but it doesn't say anything about the quality of the cabling.
Notes on 10Base-T
The used UTP cable varies between 0,404mm and 0,643mm diameter.
A common mistake with 10Base-T is the use of flat satin cable instead of twisted pair cable (flat satin is the kind of cable you use for your telephone cabling). The flat satin cable is NOT twisted. This results in excessive signal crosstalk which causes phantom collisions.
You cannot adapt 10Base-2 to 10Base-T by using a balun. Baluns always use two-wire UTP. 10Base-T uses 4-wire UTP. The only way you can use a 10Base-2 balun is in pairs. A pair of baluns and a length of two-wire UTP cable replaces a length of thin coax cable. That is why you cannot use a balun to convert between 10Base-2 and 10Base-T. You need a Repeater or Media-Converter.
The smallest network possible is two stations connected by a cross cable.
It is not advisable to use the non-used wires (4, 5, 7, 8) for other kinds of datacommunication like telehone, etc. The telephone signals can cause errors on the LAN connections. And even ruin your HUBs.
10Base-F Quick Overview
Max. speed10 Mbps
50, 62.5, or 100 &m (62.5 &m most common)
Connectors
dual ST or dual SMA 905
Max. length of a segment
up to 2 km, depending on link budget
Max. number of taps per segment
Max. amount of stations per network
Max. amount of repeaters
Maximum Topology
Between any pair of devices in a 10Base-F-only network (one collision domain) you can have a maximum of four repeaters. This means that five fiber point-to-point segments are connected by four repeaters. Populated or non-populated segments is not an issue here. Only when you start mixing 10Base-2, 10Base-5 and 10Base-F that becomes important.
Repeaters can be two-port or multi-port (usually 4 or 8).
10Base-F uses two strands of optical fiber. In general multi-mode fiber of 62.5 &m diameter. The connectors used are most of the time ST, SC or SMA905/906.
&MaleFemaleSide
ST
10Base-F is a star topology and not a bus like Thin and Thick ethernet. A device can have a fiber port or an AUI port. A fiber port has two connectors, one for transmit and one for receive. A device with an AUI port can be connected to a fiber network by using a Fiber Optic Transceiver.
Notes on 10Base-F
General
According to the OSI layers an ethernet network looks like this:
OSI Layer 1
Repeater
Reconciliation&
MIIBaseband Repeater Unit
PCSPCS and PMA
PMAPMA
PMD
AUTONEG
The Physical Medium
From the MII down to the physical medium is part of the PHY (tranceiver). The PHY may be integrated on the network card or it may be a separate device.
Reconciliation
This sub-layer translates the MAC terminology into MII terminology. It is a transparent, functionless sub-layer. Signals just pass through.
MII
The Media Independend Interface is the connection between the MAC layer and the PHY. It has the same function as the AUI connector in
networks. The connector is, when the tranceiver is not integrated on the circuit board, a MII40 connector, which actually is 40 pin Sub-D connector.
MaleFemale
PinFunctionPinFunction
1+5 Vdc/ 3.3 Vdc21+5 Vdc/ +3.3 Vdc
2MDIO22Ground
3MDC23Ground
4Rx Data24Ground
5Rx Data25Ground
6Rx Data26Ground
7Rx Data27Ground
8Rx Data Valid28Ground
9Rx Clock29Ground
10Rx Error30Ground
11Tx Error31Ground
12Tx Clock32Ground
13Tx Enable33Ground
14Tx Data34Ground
15Tx Data35Ground
16Tx Data36Ground
17Tx Data37Ground
18Collision38Ground
19Carrier Sense39Ground
20+5 Vdc/ +3.3 Vdc40+5 Vdc/ +3.3 Vdc
The data path is 4 bits wide (one nibble) and operates at 25 MHz, which leads to the throughput of 4 x 25 = 100 Mbps.
There are several options on the MII for different functions:
There is the different clock speed for 10/100 Mbps selection. For 100 Mbps the clock speed is 25 MHz and for 10 Mbps it is 2.5 MHz.
If a PHY (tranceiver) supports full-duplex transmission, it can be enabled on the MII
Also a lot of management features are possibly set on this level. If you want to know more about those read the IEEE 802.3u specifications.
With MII there are two ways power can be provided to the units. You eighter have +5V or +3.3V power. The latest one is for use with notebooks. Do not mix units with different voltages!
The maximum cable length between the MII and the actual tranceiver is 0.5 meter.
PHY
The difference between the different 100 Mbps topologies is in the PHY part. This section connects directly to the cable and is responsible for everything that is medium depended like: line encoding, transmission voltages, etc.
The different physical media standards:
The reason for a Class I and II repeater
The reason for a Class I repeater doesn't seem obvious, but there is a reason. The translation between 100Base-T4 and 100Base-Tx or Fx takes more time than translation between 100Base-Tx and 100Base-Fx. The tranceiver of the Tx and Fx standards look very much alike and therefor translation is easy. Where the tranceiver of T4 differs a lot from the other two.
A repeater with a T4 and another 100Base interface will almost certainly be a Class I repeater.
Another reason for Class I repeaters is that they might facilitate some additional features, which can not be implemented in a Class II repeater because of the tight timing delay that restricts a Class II repeater. An example of an additional feature is: stacking.
100Base-Tx Quick Overview
Wire speed
Cable type
Connector type
Used pins1 & 2, 3 & 6
Max. length of a segment
100m/328ft
Max. number of taps per segment
Max. amount of stations per network
Max. amount of repeaters
How it fits into OSI-Layer 1
OSI Layer 1
DTE Repeater
Reconciliation&
MIIBaseband Repeater Unit
PCSPCS and PMA
PMAPMA
PMD
AUTONEG
MDI
&The Physical Medium
PHY
The combination of the PCS to MDI is called a Tx tranceiver. PHY is a weird abbreviation for Physical Layer Entity. The actual tranceiver is based on the FDDI standard (X3T9.5).
PCS
The Physical Coding Sub-layer is as the name implies responsible for the encoding and decoding of the signals so they can be understood by the lower or upper layers. The coding scheme used is a 4B5B coding, which means that every nibble received from the MAC sub-layer is coded in to a 5 bit symbol.
Tx uses after the coding a MLT-3 signaling, which is a three-level signal. In the MLT-3 a change of level is marked as a logical one and a non-change is a logical zero. The data rate per pair is 100 Mbps. The transition rate on each pair is 5/4 of that rate, or 125 MHz.
Responsiblities:
Data encoding
Error checking
Collision detection
PMA
The Physical Medium Attachement is responsible for all the analog functions like transmit wave-shaping and receive data discrimination.
Responsiblities:
Link monitor
Carrier detect
Far end fault
PMD
The Physical Medium Dependend sub-layer for Tx is borrowed from the FDDI standard (ANSI X3.263 TP-PMD, Revision 2.2: 1 March 1995). It is not a 100% copy, but it comes close.
Responsiblities:
Analog functions
Clock recovery
Bit coding
Scrambling (Tx)
AUTONEG
Auto-Negotiation (also called NWAY) is a setting which can be turned on only on UTP networks (not STP). When turned on it provides the posibility of autodetecting the far ends capabilities. This way it can detect a 10 Mbps card and make connectivity possible with the 10/100 Mbps card.
MDI
The Media Dependend Interface for Tx is the RJ45 connector for UTP or the Sub-D9 for STP.
MaleFemale
PinSignal
1TX+
2TX-
3RX+
4&
5&
6RX-
7&
8&
MaleFemale
PinSignal
1RX+
2&
3&
4&
5TX+
6RX-
7&
8&
9RX-
ShieldChassis
The Medium
Tx networks transmit their data over 4-wire (2 pair) Unshielded Twisted Pair or Shielded Twisted Pair. This can only be Cat. 5 (for UTP) or Type 1 (for STP) cabling with a maximum distance of 100 meter. The entire collision domain may be 200 meters. Adding repeaters will not get you greater distances, adding fiber repeaters will get you a little more distance between two end stations. Adding bridges (switches) or routers is the only real way you can extend your network over larger distances.
100Base-T4 Quick Overview
Wire speed
Cable type
UTP CAT 3 or better
Connector type
Used pins1 & 2, 3 & 6, 4 & 5, 7 & 8
Max. length of a segment
100m/328ft
Max. number of taps per segment
Max. amount of stations per network
D Max. amount of repeaters
How it fits into OSI-Layer 1
OSI Layer 1
Repeater
Reconciliation&
MIIBaseband Repeater Unit
PCSPCS and PMA
PMAPMA
MDI
&The Physical Medium
PHY
The combination of the PCS and PMA, and the MDI, is called a T4 tranceiver. PHY is a weird abbreviation for Physical Layer Entity.
PCS
As the name Physical Coding Sub-layer implies, this layer is responsible for the coding and decoding of the data.
The PCS receives nibbles from the MII and froms them into octets (2 nibbles). An 8B6T coding is used on these octets, which makes sure that every octet is coded into a group of six three level symbols, which makes it possible to code more that one bit into a clock cycle. Now the symbols are send in a round robin-fashion to de three transmit pairs. Since there are three transmit and receive pairs the speed per pair 33.33333 Mbps. The transistion rate is 3/4 of that rate or 25 MHz.
Responsiblities:
Data encoding
Error checking
Collision detection
PMA
The Physical Medium Attachement is responsible for all the analog functions like transmit wave-shaping and receive data discrimination.
Responsiblities:
Link integrity
Carrier detect
Pair skew alignment
Clock recovery
MDI
The Media Dependend Interface for T4 is the RJ45 connector.
MaleFemale
PinSignal
1TX_D1+
2TX_D1-
3RX_D2+
4BI_D3+
5BI_D3-
6RX_D2-
7BI_D4+
8BI_D4-
There is one dedicated transmit and one receive pair and two bidirectional pairs, which can be used for transmit and receive.
The Medium
T4 networks transmit their data over 8-wire (4 pair) Unshielded Twisted Pair. This may be Cat. 3 or better cabling with a maximum distance of 100 meter. The entire collision domain may be 200 meters. Adding repeaters will not get you greater distances, even if you use fiber repeaters the largest distance between two end stations will be 200 meters. Adding bridges (switches) or routers is the only way you can extend your network over larger distances.
100Base-Fx Quick Overview
Wire speed
Cable type
Connector type
Max. length of a segment half-duplex 412m/1300ft
Max. length of a segment full-duplex ft
Max. number of taps per segment
Max. amount of stations per network
Max. amount of repeaters
How it fits into OSI-Layer 1
OSI Layer 1
Repeater
Reconciliation&
MIIBaseband Repeater Unit
PCSPCS and PMA
PMAPMA
PMD
AUTONEG
MDI
&The Physical Medium
PHY
The combination of the PCS to MDI is called a Fx tranceiver. PHY is a weird abbreviation for Physical Layer Entity. The actual tranceiver is based on the FDDI standard (X3T9.5).
PCS
The Physical Coding Sub-layer is as the name implies responsible for the encoding and decoding of the signals so they can be understood by the lower or upper layers. The coding scheme used is a 4B5B coding, which means that every nibble received from the MAC sub-layer is coded in to a 5 bit symbol.
Fx uses after the coding a NRZI signaling, which is a two-level signal. According to NRZI a logical one is a change of state and a logical zero is not. The reason for a less complex signaling scheme in the Fx standard is due to the fact that fiber is not receptable to noise. The data rate per strain is 100 Mbps. The transition rate on each strain is 5/4 of that rate, or 125 MHz.
Responsiblities:
Data encoding
Error checking
Collision detection
PMA
The Physical Medium Attachement is responsible for all the analog functions like transmit wave-shaping and receive data discrimination.
Responsiblities:
Link monitor
Carrier detect
Far end fault
PMD
The Physical Medium Dependend sub-layer for Fx is borrowed from the FDDI standard. The actual tranceiver is based on the FDDI standard ISO 90.
Responsiblities:
Analog functions
Clock recovery
Bit coding
Scrambling (Tx)
AUTONEG
Auto-Negotiation (also called NWAY) is a setting which can be turned on. When turned on it provides the posibility of autodetecting the far ends capabilities. This way it can detect a 10 Mbps card and make connectivity possible between the 10/100 Mbps card and the 10 Mbps card.
MDI
The Media Dependend Interface for Fx is not just one connector. It can be transported over various connetors.
&MaleFemaleSide
ST
The Medium
Fx networks transmit their data over 2 strands of fiber. This might be single mode or multimode. The actual design was for 62.5/125 &m multimode fiber and a wavelength of 1300 nm. Single mode fiber is NOT part of the standard. There for an exact estimate of the distance is not given, but distances of 20 km/ 660000 feet should be possible.
Introduction
In July 1996 the IEEE802.3 formed the 802.3z task force responsible for the development of a Gigabit Ethernet standard. In March 1997 a split was made resulting in the original IEEE802.3z task force and a IEEE802.3ab task force.
The IEEE802.3z standard will result in a standard (around March of 1998) that deals with the MAC-layer specifications and the physical-layer specifications for fiber (1000BASE-SX and 1000BASE-LX) and a short copper cable run (1000BASE-CX).
The IEEE802.3ab standard only handles the compatibility with the already installed cabling (UTP CAT5) with cable runs of up to 100 mtr (according to the EIA/TIA 586-A spec). This standard will be finished near the end of 1998 and is only a physical-layer specification called 1000BASE-T.
A rough overview of Gigabit Ethernet and OSI would look like this:
MAC (Full Duplex/ Half Duplex)
&GMII
1
1000BASE-X Encode/Decode
1000BASE-T Encode/Decode
1000BASE-SX
1000BASE-LX
1000BASE-CX
1000BASE-T
This document will describe the MAC-layer and the GMII. At the end of this document you will find links to the IEEE802.3z and IEEE802.3ab PHY-layer pages.
MAC
Some enhancements had to made to the CSMA/CD protocol to maintain a 200 meter collision domain when using gigabit speeds.
The carrier time and Ethernet slot time needed to be extended from their original 64 bytes to 512 bytes. Packets larger then 512 bytes will not be extended, but packets shorter then 512 bytes will use the extended time. To prevent the performance lost in networks with a large amount of small packets a new concept is added called packet bursting. This allows devices to send bursts of small packets to fully utilize the bandwidth.
Note: Full-duplex connections are NOT subject to these changes.
Reconciliation
The reconciliation sub-layer is actually a transparent interface between the MAC sub-layer and the PHY.
GMII
The Gigabit Media Independent Interface connects the reconciliation sub-layer to the PHY. It includes a 8-bit data bus which operates at 125 MHz. It also has clock signals, carrier indicators and error conditions. At the time of writing this document this is all the information that we have.
The different physical connections
Notes on Gigabit Ethernet:
A new repeater is being defined by some vendors and is called a buffered distributor. This box is a full-duplex, multi-port repeater, which may buffer some frames before forwarding.
How it fits into OSI-Layer 1
OSI Layer 1GMII
PCS
PMA
PMD
AUTONEG
The Physical Medium
From the GMII down to the physical medium is part of the PHY (tranceiver). The PHY may be integrated on the network card or it may be a separate device.
PCS
The PCS (Physical Coding Sublayer) is as the name indicates responsible for encoding the data for transmission. The coding scheme used is 8B/10B. The transmission code is DC balanced.
PMA
The Physical Medium Attachment converts the 10-bit encoded data from the PCS into serialized data, and vice versa.
PMD
This layer (Physical Media Depended sub-layer) is responsible for the transmission of the signals on the various media. For fiber this is 850 and 1300 nm.
For copper this could be short links (25 meter) over 150 ohm balanced, shielded, cabling also called Twinax. The 1000Base-SX solution is borrowed from Fiber Channel and uses a serial line speed of 1.25 GHz with a 8B/10B encoding.
The different media have different distance specifications.
NameMediumDriverDistance
1000Base-SXMultimode Fiber 50&m or 62,5 &m850 nm300 m/ 990 ft or 500 meters/ 1500 feet
1000Base-LXSingle Mode Fiber1300 nm3 kilometers/ 9000 feet
1000Base-CXShort Link Copper, twinax or quad&25 meter/ 75 feet
How it fits into OSI-Layer 1
OSI Layer 1GMII
PCS
PMA
PMD
AUTONEG
The Physical Medium
From the GMII down to the physical medium is part of the PHY (tranceiver). The PHY may be integrated on the network card or it may be a separate device.
PCS
The PCS for horizontal copper (1000Base-T) is not yet defined.
PMA
[META]
PMD
Horizontal copper (100 meter) over Cat. 5 UTP cable is not yet defined.
STANDARD EXPECTED IN 1999
Resources:
Ethernet -- Access to Ethernet (IEEE 802.3) Information
IEEE ComSoc TC on Gigabit Networking
Gigabit Ethernet over Cat5
IOL training page}