D=1 2 1 4 0 -1 2 -1 4 7 2 1 0 2 1 3
点击联系发帖人
时间：2018-06-04 11:51
centos7.4+udev+12.2.0.1+gi+asm+rac双节点安装详细教程
vm: oracle vm 5.2.8 r121009 (Qt5.6.2)
os: centos7.4
db: oracle 12.2.0.1 + gi + asm
192.168.56.100
192.168.56.200
192.168.165.100
192.168.165.200
192.168.56.101
192.168.56.201
192.168.56.110
step 1：准备工作
centos74_12.2.0.1_rac1
centos74_12.2.0.1_rac2
oracle越来越大，越来越耗内存，本次的配置为 2cpu+6G
创建共享磁盘,分配给两台虚拟机
oracle virtualbox创建共享磁盘:
进入 E:\oraclevm\centos74_rac1_rac2_sharedisk文件夹
1、在目录下创建共享磁盘，大小为10g、10g、40g
D:\\VirtualBox\VBoxManage createmedium disk --filename share01.vdi --size 10000 --format VDI --variant Fixed
Medium created. UUID: 204e-4b93-9fb1-f6eb823829fd
D:\Oracle\VirtualBox\VBoxManage createmedium disk --filename share02.vdi --size 10000 --format VDI --variant Fixed
Medium created. UUID: a364d619-a7ca-4c18-abb7-4c2a43aa5f91
D:\Oracle\VirtualBox\VBoxManage createmedium disk --filename share03.vdi --size 40000 --format VDI --variant Fixed
Medium created. UUID: 8-4c47-97d7-48ba
2、给虚拟机（区别大小写）添加共享磁盘
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac1 --storagectl SATA --port 1 --device 0 --type hdd --medium 204e-4b93-9fb1-f6eb823829fd --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac2 --storagectl SATA --port 1 --device 0 --type hdd --medium 204e-4b93-9fb1-f6eb823829fd --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac1 --storagectl SATA --port 2 --device 0 --type hdd --medium a364d619-a7ca-4c18-abb7-4c2a43aa5f91 --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac2 --storagectl SATA --port 2 --device 0 --type hdd --medium a364d619-a7ca-4c18-abb7-4c2a43aa5f91 --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac1 --storagectl SATA --port 3 --device 0 --type hdd --medium 8-4c47-97d7-48ba --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac2 --storagectl SATA --port 3 --device 0 --type hdd --medium 8-4c47-97d7-48ba --mtype shareable
&storagectl &SATA& ：在虚拟机看到的名称，如果是英文版的，可能名称不一样
&port 1 ：第一个SATA接口
&device 0 ：IDE接口的辅盘，主盘为0
3、启用共享磁盘：
D:\Oracle\VirtualBox\VBoxManage modifymedium disk 204e-4b93-9fb1-f6eb823829fd --type shareable
D:\Oracle\VirtualBox\VBoxManage modifymedium disk a364d619-a7ca-4c18-abb7-4c2a43aa5f91 --type shareable
D:\Oracle\VirtualBox\VBoxManage modifymedium disk 8-4c47-97d7-48ba --type shareable
nodea、nodeb配置网络
[root@nodea ~]# ip a
mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:4d:bc:9c brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 86049sec preferred_lft 86049sec
inet6 fe80::b61e:73ca:34d6:20ad/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:9b:3a:a5 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.100/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::c:d0aa/64 scope link
valid_lft forever preferred_lft forever
4: enp0s9:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d6:e8:2e brd ff:ff:ff:ff:ff:ff
inet 192.168.165.100/24 brd 192.168.165.255 scope global enp0s9
valid_lft forever preferred_lft forever
inet6 fe80::d3e9:3afe: scope link
valid_lft forever preferred_lft forever
[root@nodeb ~]# ip a
mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:91:f3:46 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 85943sec preferred_lft 85943sec
inet6 fe80::7db3:6c82:36a3:f295/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ea:5e:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.200/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::b62d::d819/64 scope link
valid_lft forever preferred_lft forever
4: enp0s9:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:eb:82:db brd ff:ff:ff:ff:ff:ff
inet 192.168.165.200/24 brd 192.168.165.255 scope global enp0s9
valid_lft forever preferred_lft forever
inet6 fe80::e89d:cdaa:ec18:29e9/64 scope link
valid_lft forever preferred_lft forever
删除多个默认网关
# route del default enp0s8
# route del default enp0s9
nodea、nodeb修改/etc/hosts
# vi /etc/hosts
192.168.56.100 nodea
192.168.56.200 nodeb
192.168.165.100 nodea-priv
192.168.165.200 nodeb-priv
192.168.56.101 nodea-vip
192.168.56.201 nodeb-vip
192.168.56.110 node-scan
nodea、nodeb修改主机名
# vi /etc/hostname
# vi /etc/hostname
nodea、nodeb检查 shm 和 swap
shm 是oracle 的amm需要的
# df -h /dev/shm
# vi /etc/fstab
defaults,size=3072m
关闭selinux：
临时修改：
set enforce 0
永久修改：
vi /etc/selinux/config
SELINUX=DISABLED
vi /etc/sysconfig/selinux
SELINUX=DISABLED
查看两个文件是ln -s 的关系：
ls -l /etc/sysconfig/selinux
/etc/sysconfig/selinux -& ../selinux/config
nodea、nodeb修改内核参数
vi /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
# For more information, see sysctl.conf(5) and sysctl.d(5).
#ORACLE SETTING
#kernel.shmall = 2097152
#kernel.shmmax =
kernel.shmmni = 4096
fs.aio-max-nr = 1048576
fs.file-max =
kernel.sem = 250 8
net.ipv4.ip_local_port_range =
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
RESOURCE DESCRIPTION
LIMIT USED
Number of message queues
Max size of message (bytes)
Default max size of queue (bytes)
Shared memory segments
Shared memory pages
Max size of shared memory segment (bytes)
Min size of shared memory segment (bytes)
Number of semaphore identifiers
Total number of semaphores
Max semaphores per semaphore set.
Max number of operations per semop(2)
Semaphore max value
# sysctl -a |grep -i shm
kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall =
kernel.shmmax =
kernel.shmmni = 4096
vm.hugetlb_shm_group = 0
# sysctl -a |grep -i sem
kernel.sem = 250
kernel.sem_next_id = -1
vi /etc/pam.d/login
#ORACLE SETTING
/lib64/security/pam_limits.so
nodea、nodeb禁用linux 的透明大页
# vi /etc/rc.local
echo 'never' & /sys/kernel/mm/transparent_hugepage/enabled
echo 'never' & /sys/kernel/mm/transparent_hugepage/defrag
# vi /etc/default/grub
transparent_hugepage=never
nodea、nodeb禁用numba
vi /etc/default/grub
nodea、nodeb修改io scheduler
vi /etc/default/grub
elevator=deadline
nodea、nodeb执行下面命令,生成新的grub
grub2-mkconfig -o /boot/grub2/grub.cfg
nodea、nodeb 使用udev绑定硬盘,参考另外一篇博客
获取RESULT,在
7下，可以使用如下命令：
/usr/lib/udev/scsi_id -g -u /dev/sdb
1ATA_VBOX_HARDDISK_VB204e4375-fd293882
/usr/lib/udev/scsi_id -g -u /dev/sdc
1ATA_VBOX_HARDDISK_VBa364d619-915faa43
/usr/lib/udev/scsi_id -g -u /dev/sdd
1ATA_VBOX_HARDDISK_VBd97810
vi /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL==&sd*&, SUBSYSTEM==&block&, PROGRAM==&/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name&, RESULT==&1ATA_VBOX_HARDDISK_VB204e4375-fd293882&, SYMLINK+=&asm-diskb&, OWNER=&grid&, GROUP=&asmadmin&, MODE=&0660&
KERNEL==&sd*&, SUBSYSTEM==&block&, PROGRAM==&/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name&, RESULT==&1ATA_VBOX_HARDDISK_VBa364d619-915faa43&, SYMLINK+=&asm-diskc&, OWNER=&grid&, GROUP=&asmadmin&, MODE=&0660&
KERNEL==&sd*&, SUBSYSTEM==&block&, PROGRAM==&/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name&, RESULT==&1ATA_VBOX_HARDDISK_VBd97810&, SYMLINK+=&asm-diskd&, OWNER=&grid&, GROUP=&asmadmin&, MODE=&0660&
重新加载分区
/sbin/partprobe /dev/sdb
/sbin/partprobe /dev/sdc
/sbin/partprobe /dev/sdd
用udevadm进行测试
udevadm test /sys/block/sdb
udevadm info --query=all --path=/sys/block/sdb
udevadm info --query=all --name=asm-diskb
/usr/sbin/udevadm control --reload-rules
systemctl status systemd-udevd.service
systemctl enable systemd-udevd.service
检查设备是否正确绑定
# ls -l /dev/asm* /dev/sdb /dev/sdc /dev/sdd
lrwxrwxrwx 1 root root
3 Mar 20 21:58 /dev/asm-diskb -& sdb
lrwxrwxrwx 1 root root
3 Mar 20 22:05 /dev/asm-diskc -& sdc
lrwxrwxrwx 1 root root
3 Mar 20 22:05 /dev/asm-diskd -& sdd
brw-rw---- 1 grid asmadmin 8, 16 Mar 20 21:58 /dev/sdb
brw-rw---- 1 grid asmadmin 8, 32 Mar 20 21:58 /dev/sdc
brw-rw---- 1 grid asmadmin 8, 48 Mar 20 22:05 /dev/sdd
修改 resolv.conf
# mv /etc/resolv.conf /etc/resolv.conf.
禁用 zeroconf
# vi /etc/sysconfig/network
NOZEROCONF=yes
创建用户组
groupadd -g 10000 oinstall
groupadd -g 10001 dba
groupadd -g 10002 oper
groupadd -g 10003 asmadmin
groupadd -g 10004 asmdba
groupadd -g 10005 asmoper
groupadd -g 10006 backupdba
groupadd -g 10007 dgdba
groupadd -g 10008 kmdba
groupadd -g 10009 racdba
useradd -u 10000 -g oinstall -G dba,oper,backupdba,dgdba,kmdba,asmdba,racdba oracle
useradd -u 10001 -g oinstall -G dba,asmadmin,asmdba,asmoper,racdba grid
uid=10000(oracle)
gid=10000(oinstall)
groups=10000(oinstall),
10001(dba),
10002(oper),
10004(asmdba),
10006(backupdba),
10007(dgdba),
10008(kmdba),
10009(racdba)
uid=10001(grid)
gid=10000(oinstall)
groups=10000(oinstall),
10001(dba),
10003(asmadmin),
10004(asmdba),
10005(asmoper),
10009(racdba)
passwd oracle
passwd grid
mkdir -p /u01/app/gridbase/12.2.0/grid_1
mkdir -p /u01/app/grid/product/12.2.0/grid_1
mkdir -p /u01/app/oracle/product/12.2.0/db_1
chown -R grid:oinstall /u01
chown -R grid:oinstall /u01/app/gridbase
chown -R grid:oinstall /u01/app/grid
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
grid 用户环境变量
vi /home/grid/.bash_profile
PS1=&[`whoami`@`hostname`:&'$PWD]$'
export ORACLE_BASE=/u01/app/gridbase/12.2.0/grid_1
export ORACLE_HOME=/u01/app/grid/product/12.2.0/grid_1
export ORACLE_SID=+ASM1
#export ORACLE_SID=+ASM2
export NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'
export TNS_ADMIN=$ORACLE_HOME/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
oracle 用户环境变量
vi /home/oracle/.bash_profile
PS1=&[`whoami`@`hostname`:&'$PWD]$'
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/12.2.0/db_1
export ORACLE_SID=rac01
#export ORACLE_SID=rac02
export TNS_ADMIN=/u01/app/grid/product/12.2.0/grid_1/network/admin
#export ORA_NLS10=$ORACLE_HOME/nls/data
export NLS_LANG=AMERICAN_AMERICA.AL32UTF8
export NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
修改资源限制
vi /etc/security/limits.conf
#ORACLE SETTING
#at least 90 percent of the current RAM when HugePages memory is enabled and
#at least 3145728 KB (3 GB) when HugePages memory is disabled
#at least 90 percent of the current RAM when HugePages memory is enabled and
#at least 3145728 KB (3 GB) when HugePages memory is disabled
yum 安装12.2.0.1 的依赖包
The following packages (or later versions) must be installed:
binutils-2.23.52.0.1-12.el7 (x86_64)
compat-libcap1-1.10-3.el7 (x86_64)
compat-libstdc++-33-3.2.3-71.el7 (i686)
compat-libstdc++-33-3.2.3-71.el7 (x86_64)
glibc-2.17-36.el7 (i686)
glibc-2.17-36.el7 (x86_64)
glibc-devel-2.17-36.el7 (i686)
glibc-devel-2.17-36.el7 (x86_64)
libaio-0.3.109-9.el7 (i686)
libaio-0.3.109-9.el7 (x86_64)
libaio-devel-0.3.109-9.el7 (i686)
libaio-devel-0.3.109-9.el7 (x86_64)
libgcc-4.8.2-3.el7 (i686)
libgcc-4.8.2-3.el7 (x86_64)
libstdc++-4.8.2-3.el7 (i686)
libstdc++-4.8.2-3.el7 (x86_64)
libstdc++-devel-4.8.2-3.el7 (i686)
libstdc++-devel-4.8.2-3.el7 (x86_64)
libxcb-1.9-5.el7 (i686)
libxcb-1.9-5.el7 (x86_64)
libX11-1.6.0-2.1.el7 (i686)
libX11-1.6.0-2.1.el7 (x86_64)
libXau-1.0.8-2.1.el7 (i686)
libXau-1.0.8-2.1.el7 (x86_64)
libXi-1.7.2-1.el7 (i686)
libXi-1.7.2-1.el7 (x86_64)
libXtst-1.2.2-1.el7 (i686)
libXtst-1.2.2-1.el7 (x86_64)
make-3.82-19.el7 (x86_64)
net-tools-2.0-0.17.git.el7 (x86_64) (for Oracle RAC and Oracle Clusterware)
nfs-utils-1.3.0-0.21.el7.x86_64 (for Oracle ACFS)
smartmontools-6.2-4.el7 (x86_64)
sysstat-10.1.5-1.el7 (x86_64)
unixODBC-2.3.1 or later
yum install binutils compat-libcap1 compat-libstdc++-33 compat-libstdc++-33.i686 gcc
gcc-c++ glibc glibc.i686 glibc-devel glibc-devel.i686 ksh libgcc libgcc.i686 libstdc++ libstdc++.i686
libstdc++-devel libstdc++-devel.i686 libaio libaio.i686 libaio-devel libaio-devel.i686 libXext
libXext.i686 libXtst libXtst.i686 libX11 libX11.i686 libXau libXau.i686 libxcb libxcb.i686 libXi
libXi.i686 make sysstat unixODBC unixODBC-devel readline libtermcap-devel pdksh net-tools nfs-utils smartmontools
禁止ntp服务
systemctl stop chronyd
systemctl disable chronyd
systemctl stop ntpd.service
systemctl disable ntpd.service
systemctl stop ntpdate.service
systemctl disable ntpdate.service
mv /etc/chrony.conf /etc/chrony.conf.bak
mv /etc/ntp.conf /etc/ntp.conf.bak
禁止avahi-daemon服务
avahi-daemon 服务会影响 oracle的多波通信，进而导致节点重启
因此，oracle环境下不能启用 avahi-daemon 服务
systemctl stop avahi-daemon.service
systemctl disable avahi-daemon.service
禁用其它一些服务
systemctl disable iptables.service
systemctl disable firewalld.service
systemctl disable cups.service
systemctl disable ip6tables.service
systemctl disable firstboot-graphical.service
systemctl disable postfix.service
systemctl disable NetworkManager.service
最后确保以下服务正常
systemctl enable crond.service
systemctl enable sshd.service
systemctl enable sysstat.service
systemctl enable network.service
step 3：安装 gi
下载linuxx64_12201_grid_home.zip 上传到 nodea 的 /tmp 下
切换到grid用户，解压到grid用户下的$ORACLE_HOME
安装 cvuqdisk
# cd /u01/app/grid/product/12.2.0/grid_1/cv/rpm
# rpm -ivh cvuqdisk-1.0.10-1.rpm
运行预检查
$ cd /u01/app/grid/product/12.2.0/grid_1/
$ ./runcluvfy.sh stage -pre crsinst -n nodea,nodeb -verbose
Verifying Single Client Access Name (SCAN) ...FAILED
Verifying DNS/NIS name service 'node-scan' ...FAILED
PRVG-1101 : SCAN name &node-scan& failed to resolve
SCAN 如果使用hosts方式的话，可以忽略
修正后，运行 gridSetup.sh
Configure Oracle Grid Infrastructure for a New Cluster
Configure an Oracle Standalone Cluster
Cluster Name : node-cluster
SCAN Name : node-scan
SCAN Port : 1521
role:HUB LEAF
at least 39,152MB
最后以root运行两个脚本
/u01/app/gridbase/12.2.0/oraInventory/orainstRoot.sh
/u01/app/grid/product/12.2.0/grid_1/root.sh
运行后再次检查
$ cd /u01/app/grid/product/12.2.0/grid_1/
$ ./runcluvfy.sh stage -post crsinst -n nodea,nodeb -verbose
step 3：安装 db
下载 linuxx64_12201_database.zip 解压后安装
Install database software only
Oracle Real Application Cluster database installation
最后以root运行脚本
/u01/app/oracle/product/12.2.0/db_1/root.sh
接下来用 dbca 创建
Global database name : rac0
SID Prefix: rac0
em port：5500
URL=https://node-scan:5500/em
多节点安装前后都检测一下
./runcluvfy.sh stage -pre crsinst -n nodea,nodeb -verbose
./runcluvfy.sh stage -pre dbinst -n nodea,nodeb -verbose
./runcluvfy.sh stage -post crsinst -n nodea,nodeb -verbose
./runcluvfy.sh stage -post dbinst -n nodea,nodeb -verbose
./cluvfy stage -pre dbinst -n nodea,nodeb -verbose
./cluvfy stage -post dbinst -n nodea,nodeb -verbose
查看一些信息
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css
crsctl get css
Displays the value of a Cluster Synchronization Services parameter
clusterguid
disktimeout
reboottime
noautorestart
crsctl get css ipmiaddr
Displays the IP address of the local IPMI device as set in the Oracle registry
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css diagwait
CRS-4678: Successful get diagwait 0 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css reboottime
CRS-4678: Successful get reboottime 3 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl lsmodules crs
List CRSD Debug Module: AGENT
List CRSD Debug Module: AGFW
List CRSD Debug Module: CLSCAL
List CRSD Debug Module: CLSCEVT
List CRSD Debug Module: CLSFRAME
List CRSD Debug Module: CLSINET
List CRSD Debug Module: CLSO
List CRSD Debug Module: CLSVER
List CRSD Debug Module: CLUCLS
List CRSD Debug Module: COMMCRS
List CRSD Debug Module: COMMNS
List CRSD Debug Module: CRSAPP
List CRSD Debug Module: CRSCCL
List CRSD Debug Module: CRSCEVT
List CRSD Debug Module: CRSCOMM
List CRSD Debug Module: CRSD
List CRSD Debug Module: CRSEVT
List CRSD Debug Module: CRSMAIN
List CRSD Debug Module: CRSOCR
List CRSD Debug Module: CRSPE
List CRSD Debug Module: CRSPLACE
List CRSD Debug Module: CRSRES
List CRSD Debug Module: CRSRPT
List CRSD Debug Module: CRSRTI
List CRSD Debug Module: CRSSE
List CRSD Debug Module: CRSSEC
List CRSD Debug Module: CRSTIMER
List CRSD Debug Module: CRSUI
List CRSD Debug Module: CLNT
List CRSD Debug Module: OCRAPI
List CRSD Debug Module: OCRASM
List CRSD Debug Module: OCRCAC
List CRSD Debug Module: OCRCLI
List CRSD Debug Module: OCRMAS
List CRSD Debug Module: OCRMSG
List CRSD Debug Module: OCROSD
List CRSD Debug Module: OCRRAW
List CRSD Debug Module: OCRSRV
List CRSD Debug Module: OCRUTL
List CRSD Debug Module: SuiteTes
List CRSD Debug Module: UiServer
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl check has
CRS-4638: Oracle High Availability Services is online
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl stat res -t
--------------------------------------------------------------------------------
State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ora.DG_DATA01.dg
ora.LISTENER.lsnr
ora.MGMT.dg
ora.net1.network
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
ora.MGMTLSNR
OFFLINE OFFLINE
Started,STABLE
Started,STABLE
OFFLINE OFFLINE
ora.nodea.vip
ora.nodeb.vip
ora.qosmserver
ora.rac0.db
Open,HOME=/u01/app/o
racle/product/12.2.0
/db_1,STABLE
Open,HOME=/u01/app/o
racle/product/12.2.0
/db_1,STABLE
ora.scan1.vip
--------------------------------------------------------------------------------
查看一些进程
# ps -fu root|grep -i init
00:00:00 /usr/sbin/alsactl -s -n 19 -c -E ALSA_CONFIG_PATH=/etc/alsa/alsactl.conf --initfile=/lib/alsa/ini/00main rdaemon
0 10:19 pts/0
00:00:00 grep --color=auto -i init
00:00:00 /bin/sh /etc/init.d/init.tfa run &/dev/null 2&&1 /dev/null 2&&1
# ps -fu root|grep -i grid_1 root
0 10:17 pts/0 00:00:00 grep --color=auto -i grid_1 root
09:31 ? 00:00:25 /u01/app/grid/product/12.2.0/grid_1/jdk/jre/bin/java -Xms128m -Xmx512m oracle.rat.tfa.TFAMain /u01/app/grid/product/12.2.0/grid_1/tfa/nodea/tfa_home root
09:33 ? 00:00:08 /u01/app/grid/product/12.2.0/grid_1/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8 root
09:33 ? 00:00:04 /u01/app/grid/product/12.2.0/grid_1/bin/orarootagent.bin root
09:33 ? 00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/cssdmonitor root
09:33 ? 00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/cssdagent root
09:34 ? 00:00:07 /u01/app/grid/product/12.2.0/grid_1/bin/octssd.bin reboot root
09:34 ? 00:00:15 /u01/app/grid/product/12.2.0/grid_1/bin/osysmond.bin root
09:34 ? 00:00:16 /u01/app/grid/product/12.2.0/grid_1/bin/crsd.bin reboot root
09:34 ? 00:00:12 /u01/app/grid/product/12.2.0/grid_1/bin/orarootagent.bin root
09:34 ? 00:00:13 /u01/app/grid/product/12.2.0/grid_1/bin/ologgerd -M root
0 09:35 ? 00:00:03 /u01/app/grid/product/12.2.0/grid_1/perl/bin/perl /u01/app/grid/product/12.2.0/grid_1/bin/diagsnap.pl start
# ps -fu grid
C STIME TTY
00:00:00 /u01/app/grid/product/12.2.0/grid_1/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/scriptagent.bin
00:00:07 /u01/app/grid/product/12.2.0/grid_1/jdk/bin/java -server -Xms128M -Xmx384M -Djava.awt.headless=tru
00:00:03 /u01/app/grid/product/12.2.0/grid_1/jdk/bin/java -classpath /u01/app/grid/product/12.2.0/grid_1/jd
00:00:14 /u01/app/grid/product/12.2.0/grid_1/bin/oraagent.bin
00:00:00 /u01/app/grid/product/12.2.0/grid_1/opmn/bin/ons -d
00:00:00 /u01/app/grid/product/12.2.0/grid_1/opmn/bin/ons -d
00:00:00 /u01/app/grid/product/12.2.0/grid_1/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
00:00:00 /u01/app/grid/product/12.2.0/grid_1/bin/tnslsnr LISTENER -no_crs_notify -inherit
00:00:07 /u01/app/grid/product/12.2.0/grid_1/bin/oraagent.bin
00:00:01 /u01/app/grid/product/12.2.0/grid_1/bin/mdnsd.bin
00:00:07 /u01/app/grid/product/12.2.0/grid_1/bin/evmd.bin
00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/gpnpd.bin
00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/evmlogger.bin -o /u01/app/grid/product/12.2.0/grid_1/log/[
00:00:05 /u01/app/grid/product/12.2.0/grid_1/bin/gipcd.bin
00:00:08 /u01/app/grid/product/12.2.0/grid_1/bin/ocssd.bin
00:00:00 asm_pmon_+ASM1
00:00:00 asm_clmn_+ASM1
00:00:00 asm_psp0_+ASM1
00:00:27 asm_vktm_+ASM1
00:00:00 asm_gen0_+ASM1
00:00:00 asm_mman_+ASM1
00:00:00 asm_gen1_+ASM1
00:00:01 asm_diag_+ASM1
00:00:00 asm_ping_+ASM1
00:00:00 asm_pman_+ASM1
00:00:04 asm_dia0_+ASM1
00:00:02 asm_lmon_+ASM1
00:00:02 asm_lmd0_+ASM1
00:00:05 asm_lms0_+ASM1
00:00:02 asm_lmhb_+ASM1
00:00:00 asm_lck1_+ASM1
00:00:00 asm_dbw0_+ASM1
00:00:00 asm_lgwr_+ASM1
00:00:00 asm_ckpt_+ASM1
00:00:00 asm_smon_+ASM1
00:00:00 asm_lreg_+ASM1
00:00:00 asm_pxmn_+ASM1
00:00:00 asm_rbal_+ASM1
00:00:00 asm_gmon_+ASM1
00:00:00 asm_mmon_+ASM1
00:00:00 asm_mmnl_+ASM1
00:00:03 asm_imr0_+ASM1
00:00:00 asm_lck0_+ASM1
00:00:02 asm_gcr0_+ASM1
00:00:00 asm_ppa7_+ASM1
00:00:00 asm_asmb_+ASM1
00:00:01 oracle+ASM1_crf (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
# ps -fu oracle
C STIME TTY
00:00:00 ora_w005_rac02
00:00:00 ora_w004_rac02
00:00:10 /u01/app/grid/product/12.2.0/grid_1
00:00:00 ora_pmon_rac02
00:00:00 ora_clmn_rac02
00:00:01 ora_psp0_rac02
00:00:01 ora_ipc0_rac02
00:01:02 ora_vktm_rac02
00:00:00 ora_gen0_rac02
00:00:00 ora_mman_rac02
00:00:01 ora_gen1_rac02
00:00:04 ora_diag_rac02
00:00:00 ora_ofsd_rac02
00:00:02 ora_dbrm_rac02
00:00:00 ora_vkrm_rac02
00:00:00 ora_ping_rac02
00:00:00 ora_sb_rac02
00:00:00 ora_acms_rac02
00:00:00 ora_pman_rac02
00:00:12 ora_dia0_rac02
00:00:08 ora_lmon_rac02
00:00:22 ora_lms0_rac02
00:00:09 ora_lmd0_rac02
00:00:01 ora_rmv0_rac02
00:00:00 ora_rms0_rac02
00:00:06 ora_lmhb_rac02
00:00:00 ora_lck1_rac02
00:00:00 ora_dbw0_rac02
00:00:00 ora_lgwr_rac02
00:00:02 ora_ckpt_rac02
00:00:00 ora_smon_rac02
00:00:00 ora_smco_rac02
00:00:00 ora_reco_rac02
00:00:00 ora_lreg_rac02
00:00:00 ora_pxmn_rac02
00:00:00 ora_rbal_rac02
00:00:00 ora_asmb_rac02
00:00:05 ora_fenc_rac02
00:00:14 ora_mmon_rac02
00:00:01 ora_mmnl_rac02
00:00:00 ora_d000_rac02
00:00:00 ora_s000_rac02
00:00:08 ora_imr0_rac02
00:00:03 ora_scm0_rac02
00:00:01 ora_lck0_rac02
00:00:00 ora_mark_rac02
00:00:00 ora_rsmn_rac02
00:00:00 ora_tmon_rac02
00:00:09 ora_gcr0_rac02
00:00:00 ora_tt00_rac02
00:00:00 ora_tt01_rac02
00:00:00 ora_tt02_rac02
00:00:00 ora_gtx0_rac02
00:00:00 ora_rcbg_rac02
00:00:00 ora_aqpc_rac02
00:00:03 ora_p000_rac02
00:00:03 ora_p001_rac02
00:00:00 ora_p002_rac02
00:00:00 ora_p003_rac02
00:00:00 ora_qm02_rac02
00:00:03 ora_qm05_rac02
00:00:00 ora_q004_rac02
00:00:07 ora_cjq0_rac02
00:00:00 ora_w003_rac02
00:00:00 ora_w000_rac02
00:00:00 ora_w001_rac02
00:00:00 ora_w006_rac02
00:00:00 ora_ppa7_rac02
00:00:05 ora_o000_rac02
00:00:00 ora_w002_rac02
00:00:00 ora_w007_rac02
00:00:00 ora_gcr1_rac02
00:00:00 ora_q005_rac02
00:00:05 ora_o001_rac02
00:00:00 ora_p004_rac02
00:00:00 ora_p005_rac02
00:00:01 oraclerac02 (DESCRIPTION=(LOCAL=YES
00:00:00 oraclerac02 (DESCRIPTION=(LOCAL=YES
00:00:00 oraclerac02 (DESCRIPTION=(LOCAL=YES}
vm: oracle vm 5.2.8 r121009 (Qt5.6.2)
os: centos7.4
db: oracle 12.2.0.1 + gi + asm
192.168.56.100
192.168.56.200
192.168.165.100
192.168.165.200
192.168.56.101
192.168.56.201
192.168.56.110
step 1：准备工作
centos74_12.2.0.1_rac1
centos74_12.2.0.1_rac2
oracle越来越大，越来越耗内存，本次的配置为 2cpu+6G
创建共享磁盘,分配给两台虚拟机
oracle virtualbox创建共享磁盘:
进入 E:\oraclevm\centos74_rac1_rac2_sharedisk文件夹
1、在目录下创建共享磁盘，大小为10g、10g、40g
D:\\VirtualBox\VBoxManage createmedium disk --filename share01.vdi --size 10000 --format VDI --variant Fixed
Medium created. UUID: 204e-4b93-9fb1-f6eb823829fd
D:\Oracle\VirtualBox\VBoxManage createmedium disk --filename share02.vdi --size 10000 --format VDI --variant Fixed
Medium created. UUID: a364d619-a7ca-4c18-abb7-4c2a43aa5f91
D:\Oracle\VirtualBox\VBoxManage createmedium disk --filename share03.vdi --size 40000 --format VDI --variant Fixed
Medium created. UUID: 8-4c47-97d7-48ba
2、给虚拟机（区别大小写）添加共享磁盘
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac1 --storagectl SATA --port 1 --device 0 --type hdd --medium 204e-4b93-9fb1-f6eb823829fd --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac2 --storagectl SATA --port 1 --device 0 --type hdd --medium 204e-4b93-9fb1-f6eb823829fd --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac1 --storagectl SATA --port 2 --device 0 --type hdd --medium a364d619-a7ca-4c18-abb7-4c2a43aa5f91 --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac2 --storagectl SATA --port 2 --device 0 --type hdd --medium a364d619-a7ca-4c18-abb7-4c2a43aa5f91 --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac1 --storagectl SATA --port 3 --device 0 --type hdd --medium 8-4c47-97d7-48ba --mtype shareable
D:\Oracle\VirtualBox\VBoxManage storageattach centos74_12.2.0.1_rac2 --storagectl SATA --port 3 --device 0 --type hdd --medium 8-4c47-97d7-48ba --mtype shareable
&storagectl &SATA& ：在虚拟机看到的名称，如果是英文版的，可能名称不一样
&port 1 ：第一个SATA接口
&device 0 ：IDE接口的辅盘，主盘为0
3、启用共享磁盘：
D:\Oracle\VirtualBox\VBoxManage modifymedium disk 204e-4b93-9fb1-f6eb823829fd --type shareable
D:\Oracle\VirtualBox\VBoxManage modifymedium disk a364d619-a7ca-4c18-abb7-4c2a43aa5f91 --type shareable
D:\Oracle\VirtualBox\VBoxManage modifymedium disk 8-4c47-97d7-48ba --type shareable
nodea、nodeb配置网络
[root@nodea ~]# ip a
mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:4d:bc:9c brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 86049sec preferred_lft 86049sec
inet6 fe80::b61e:73ca:34d6:20ad/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:9b:3a:a5 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.100/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::c:d0aa/64 scope link
valid_lft forever preferred_lft forever
4: enp0s9:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:d6:e8:2e brd ff:ff:ff:ff:ff:ff
inet 192.168.165.100/24 brd 192.168.165.255 scope global enp0s9
valid_lft forever preferred_lft forever
inet6 fe80::d3e9:3afe: scope link
valid_lft forever preferred_lft forever
[root@nodeb ~]# ip a
mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:91:f3:46 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 85943sec preferred_lft 85943sec
inet6 fe80::7db3:6c82:36a3:f295/64 scope link
valid_lft forever preferred_lft forever
3: enp0s8:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:ea:5e:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.200/24 brd 192.168.56.255 scope global enp0s8
valid_lft forever preferred_lft forever
inet6 fe80::b62d::d819/64 scope link
valid_lft forever preferred_lft forever
4: enp0s9:
mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:eb:82:db brd ff:ff:ff:ff:ff:ff
inet 192.168.165.200/24 brd 192.168.165.255 scope global enp0s9
valid_lft forever preferred_lft forever
inet6 fe80::e89d:cdaa:ec18:29e9/64 scope link
valid_lft forever preferred_lft forever
删除多个默认网关
# route del default enp0s8
# route del default enp0s9
nodea、nodeb修改/etc/hosts
# vi /etc/hosts
192.168.56.100 nodea
192.168.56.200 nodeb
192.168.165.100 nodea-priv
192.168.165.200 nodeb-priv
192.168.56.101 nodea-vip
192.168.56.201 nodeb-vip
192.168.56.110 node-scan
nodea、nodeb修改主机名
# vi /etc/hostname
# vi /etc/hostname
nodea、nodeb检查 shm 和 swap
shm 是oracle 的amm需要的
# df -h /dev/shm
# vi /etc/fstab
defaults,size=3072m
关闭selinux：
临时修改：
set enforce 0
永久修改：
vi /etc/selinux/config
SELINUX=DISABLED
vi /etc/sysconfig/selinux
SELINUX=DISABLED
查看两个文件是ln -s 的关系：
ls -l /etc/sysconfig/selinux
/etc/sysconfig/selinux -& ../selinux/config
nodea、nodeb修改内核参数
vi /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
# For more information, see sysctl.conf(5) and sysctl.d(5).
#ORACLE SETTING
#kernel.shmall = 2097152
#kernel.shmmax =
kernel.shmmni = 4096
fs.aio-max-nr = 1048576
fs.file-max =
kernel.sem = 250 8
net.ipv4.ip_local_port_range =
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
RESOURCE DESCRIPTION
LIMIT USED
Number of message queues
Max size of message (bytes)
Default max size of queue (bytes)
Shared memory segments
Shared memory pages
Max size of shared memory segment (bytes)
Min size of shared memory segment (bytes)
Number of semaphore identifiers
Total number of semaphores
Max semaphores per semaphore set.
Max number of operations per semop(2)
Semaphore max value
# sysctl -a |grep -i shm
kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall =
kernel.shmmax =
kernel.shmmni = 4096
vm.hugetlb_shm_group = 0
# sysctl -a |grep -i sem
kernel.sem = 250
kernel.sem_next_id = -1
vi /etc/pam.d/login
#ORACLE SETTING
/lib64/security/pam_limits.so
nodea、nodeb禁用linux 的透明大页
# vi /etc/rc.local
echo 'never' & /sys/kernel/mm/transparent_hugepage/enabled
echo 'never' & /sys/kernel/mm/transparent_hugepage/defrag
# vi /etc/default/grub
transparent_hugepage=never
nodea、nodeb禁用numba
vi /etc/default/grub
nodea、nodeb修改io scheduler
vi /etc/default/grub
elevator=deadline
nodea、nodeb执行下面命令,生成新的grub
grub2-mkconfig -o /boot/grub2/grub.cfg
nodea、nodeb 使用udev绑定硬盘,参考另外一篇博客
获取RESULT,在
7下，可以使用如下命令：
/usr/lib/udev/scsi_id -g -u /dev/sdb
1ATA_VBOX_HARDDISK_VB204e4375-fd293882
/usr/lib/udev/scsi_id -g -u /dev/sdc
1ATA_VBOX_HARDDISK_VBa364d619-915faa43
/usr/lib/udev/scsi_id -g -u /dev/sdd
1ATA_VBOX_HARDDISK_VBd97810
vi /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL==&sd*&, SUBSYSTEM==&block&, PROGRAM==&/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name&, RESULT==&1ATA_VBOX_HARDDISK_VB204e4375-fd293882&, SYMLINK+=&asm-diskb&, OWNER=&grid&, GROUP=&asmadmin&, MODE=&0660&
KERNEL==&sd*&, SUBSYSTEM==&block&, PROGRAM==&/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name&, RESULT==&1ATA_VBOX_HARDDISK_VBa364d619-915faa43&, SYMLINK+=&asm-diskc&, OWNER=&grid&, GROUP=&asmadmin&, MODE=&0660&
KERNEL==&sd*&, SUBSYSTEM==&block&, PROGRAM==&/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/$name&, RESULT==&1ATA_VBOX_HARDDISK_VBd97810&, SYMLINK+=&asm-diskd&, OWNER=&grid&, GROUP=&asmadmin&, MODE=&0660&
重新加载分区
/sbin/partprobe /dev/sdb
/sbin/partprobe /dev/sdc
/sbin/partprobe /dev/sdd
用udevadm进行测试
udevadm test /sys/block/sdb
udevadm info --query=all --path=/sys/block/sdb
udevadm info --query=all --name=asm-diskb
/usr/sbin/udevadm control --reload-rules
systemctl status systemd-udevd.service
systemctl enable systemd-udevd.service
检查设备是否正确绑定
# ls -l /dev/asm* /dev/sdb /dev/sdc /dev/sdd
lrwxrwxrwx 1 root root
3 Mar 20 21:58 /dev/asm-diskb -& sdb
lrwxrwxrwx 1 root root
3 Mar 20 22:05 /dev/asm-diskc -& sdc
lrwxrwxrwx 1 root root
3 Mar 20 22:05 /dev/asm-diskd -& sdd
brw-rw---- 1 grid asmadmin 8, 16 Mar 20 21:58 /dev/sdb
brw-rw---- 1 grid asmadmin 8, 32 Mar 20 21:58 /dev/sdc
brw-rw---- 1 grid asmadmin 8, 48 Mar 20 22:05 /dev/sdd
修改 resolv.conf
# mv /etc/resolv.conf /etc/resolv.conf.
禁用 zeroconf
# vi /etc/sysconfig/network
NOZEROCONF=yes
创建用户组
groupadd -g 10000 oinstall
groupadd -g 10001 dba
groupadd -g 10002 oper
groupadd -g 10003 asmadmin
groupadd -g 10004 asmdba
groupadd -g 10005 asmoper
groupadd -g 10006 backupdba
groupadd -g 10007 dgdba
groupadd -g 10008 kmdba
groupadd -g 10009 racdba
useradd -u 10000 -g oinstall -G dba,oper,backupdba,dgdba,kmdba,asmdba,racdba oracle
useradd -u 10001 -g oinstall -G dba,asmadmin,asmdba,asmoper,racdba grid
uid=10000(oracle)
gid=10000(oinstall)
groups=10000(oinstall),
10001(dba),
10002(oper),
10004(asmdba),
10006(backupdba),
10007(dgdba),
10008(kmdba),
10009(racdba)
uid=10001(grid)
gid=10000(oinstall)
groups=10000(oinstall),
10001(dba),
10003(asmadmin),
10004(asmdba),
10005(asmoper),
10009(racdba)
passwd oracle
passwd grid
mkdir -p /u01/app/gridbase/12.2.0/grid_1
mkdir -p /u01/app/grid/product/12.2.0/grid_1
mkdir -p /u01/app/oracle/product/12.2.0/db_1
chown -R grid:oinstall /u01
chown -R grid:oinstall /u01/app/gridbase
chown -R grid:oinstall /u01/app/grid
chown -R oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
grid 用户环境变量
vi /home/grid/.bash_profile
PS1=&[`whoami`@`hostname`:&'$PWD]$'
export ORACLE_BASE=/u01/app/gridbase/12.2.0/grid_1
export ORACLE_HOME=/u01/app/grid/product/12.2.0/grid_1
export ORACLE_SID=+ASM1
#export ORACLE_SID=+ASM2
export NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'
export TNS_ADMIN=$ORACLE_HOME/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$ORACLE_HOME/bin:$PATH
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
oracle 用户环境变量
vi /home/oracle/.bash_profile
PS1=&[`whoami`@`hostname`:&'$PWD]$'
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/12.2.0/db_1
export ORACLE_SID=rac01
#export ORACLE_SID=rac02
export TNS_ADMIN=/u01/app/grid/product/12.2.0/grid_1/network/admin
#export ORA_NLS10=$ORACLE_HOME/nls/data
export NLS_LANG=AMERICAN_AMERICA.AL32UTF8
export NLS_DATE_FORMAT='yyyy-mm-dd hh24:mi:ss'
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export TEMP=/tmp
export TMP=/tmp
export TMPDIR=/tmp
修改资源限制
vi /etc/security/limits.conf
#ORACLE SETTING
#at least 90 percent of the current RAM when HugePages memory is enabled and
#at least 3145728 KB (3 GB) when HugePages memory is disabled
#at least 90 percent of the current RAM when HugePages memory is enabled and
#at least 3145728 KB (3 GB) when HugePages memory is disabled
yum 安装12.2.0.1 的依赖包
The following packages (or later versions) must be installed:
binutils-2.23.52.0.1-12.el7 (x86_64)
compat-libcap1-1.10-3.el7 (x86_64)
compat-libstdc++-33-3.2.3-71.el7 (i686)
compat-libstdc++-33-3.2.3-71.el7 (x86_64)
glibc-2.17-36.el7 (i686)
glibc-2.17-36.el7 (x86_64)
glibc-devel-2.17-36.el7 (i686)
glibc-devel-2.17-36.el7 (x86_64)
libaio-0.3.109-9.el7 (i686)
libaio-0.3.109-9.el7 (x86_64)
libaio-devel-0.3.109-9.el7 (i686)
libaio-devel-0.3.109-9.el7 (x86_64)
libgcc-4.8.2-3.el7 (i686)
libgcc-4.8.2-3.el7 (x86_64)
libstdc++-4.8.2-3.el7 (i686)
libstdc++-4.8.2-3.el7 (x86_64)
libstdc++-devel-4.8.2-3.el7 (i686)
libstdc++-devel-4.8.2-3.el7 (x86_64)
libxcb-1.9-5.el7 (i686)
libxcb-1.9-5.el7 (x86_64)
libX11-1.6.0-2.1.el7 (i686)
libX11-1.6.0-2.1.el7 (x86_64)
libXau-1.0.8-2.1.el7 (i686)
libXau-1.0.8-2.1.el7 (x86_64)
libXi-1.7.2-1.el7 (i686)
libXi-1.7.2-1.el7 (x86_64)
libXtst-1.2.2-1.el7 (i686)
libXtst-1.2.2-1.el7 (x86_64)
make-3.82-19.el7 (x86_64)
net-tools-2.0-0.17.git.el7 (x86_64) (for Oracle RAC and Oracle Clusterware)
nfs-utils-1.3.0-0.21.el7.x86_64 (for Oracle ACFS)
smartmontools-6.2-4.el7 (x86_64)
sysstat-10.1.5-1.el7 (x86_64)
unixODBC-2.3.1 or later
yum install binutils compat-libcap1 compat-libstdc++-33 compat-libstdc++-33.i686 gcc
gcc-c++ glibc glibc.i686 glibc-devel glibc-devel.i686 ksh libgcc libgcc.i686 libstdc++ libstdc++.i686
libstdc++-devel libstdc++-devel.i686 libaio libaio.i686 libaio-devel libaio-devel.i686 libXext
libXext.i686 libXtst libXtst.i686 libX11 libX11.i686 libXau libXau.i686 libxcb libxcb.i686 libXi
libXi.i686 make sysstat unixODBC unixODBC-devel readline libtermcap-devel pdksh net-tools nfs-utils smartmontools
禁止ntp服务
systemctl stop chronyd
systemctl disable chronyd
systemctl stop ntpd.service
systemctl disable ntpd.service
systemctl stop ntpdate.service
systemctl disable ntpdate.service
mv /etc/chrony.conf /etc/chrony.conf.bak
mv /etc/ntp.conf /etc/ntp.conf.bak
禁止avahi-daemon服务
avahi-daemon 服务会影响 oracle的多波通信，进而导致节点重启
因此，oracle环境下不能启用 avahi-daemon 服务
systemctl stop avahi-daemon.service
systemctl disable avahi-daemon.service
禁用其它一些服务
systemctl disable iptables.service
systemctl disable firewalld.service
systemctl disable cups.service
systemctl disable ip6tables.service
systemctl disable firstboot-graphical.service
systemctl disable postfix.service
systemctl disable NetworkManager.service
最后确保以下服务正常
systemctl enable crond.service
systemctl enable sshd.service
systemctl enable sysstat.service
systemctl enable network.service
step 3：安装 gi
下载linuxx64_12201_grid_home.zip 上传到 nodea 的 /tmp 下
切换到grid用户，解压到grid用户下的$ORACLE_HOME
安装 cvuqdisk
# cd /u01/app/grid/product/12.2.0/grid_1/cv/rpm
# rpm -ivh cvuqdisk-1.0.10-1.rpm
运行预检查
$ cd /u01/app/grid/product/12.2.0/grid_1/
$ ./runcluvfy.sh stage -pre crsinst -n nodea,nodeb -verbose
Verifying Single Client Access Name (SCAN) ...FAILED
Verifying DNS/NIS name service 'node-scan' ...FAILED
PRVG-1101 : SCAN name &node-scan& failed to resolve
SCAN 如果使用hosts方式的话，可以忽略
修正后，运行 gridSetup.sh
Configure Oracle Grid Infrastructure for a New Cluster
Configure an Oracle Standalone Cluster
Cluster Name : node-cluster
SCAN Name : node-scan
SCAN Port : 1521
role:HUB LEAF
at least 39,152MB
最后以root运行两个脚本
/u01/app/gridbase/12.2.0/oraInventory/orainstRoot.sh
/u01/app/grid/product/12.2.0/grid_1/root.sh
运行后再次检查
$ cd /u01/app/grid/product/12.2.0/grid_1/
$ ./runcluvfy.sh stage -post crsinst -n nodea,nodeb -verbose
step 3：安装 db
下载 linuxx64_12201_database.zip 解压后安装
Install database software only
Oracle Real Application Cluster database installation
最后以root运行脚本
/u01/app/oracle/product/12.2.0/db_1/root.sh
接下来用 dbca 创建
Global database name : rac0
SID Prefix: rac0
em port：5500
URL=https://node-scan:5500/em
多节点安装前后都检测一下
./runcluvfy.sh stage -pre crsinst -n nodea,nodeb -verbose
./runcluvfy.sh stage -pre dbinst -n nodea,nodeb -verbose
./runcluvfy.sh stage -post crsinst -n nodea,nodeb -verbose
./runcluvfy.sh stage -post dbinst -n nodea,nodeb -verbose
./cluvfy stage -pre dbinst -n nodea,nodeb -verbose
./cluvfy stage -post dbinst -n nodea,nodeb -verbose
查看一些信息
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css
crsctl get css
Displays the value of a Cluster Synchronization Services parameter
clusterguid
disktimeout
reboottime
noautorestart
crsctl get css ipmiaddr
Displays the IP address of the local IPMI device as set in the Oracle registry
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css diagwait
CRS-4678: Successful get diagwait 0 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl get css reboottime
CRS-4678: Successful get reboottime 3 for Cluster Synchronization Services.
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl lsmodules crs
List CRSD Debug Module: AGENT
List CRSD Debug Module: AGFW
List CRSD Debug Module: CLSCAL
List CRSD Debug Module: CLSCEVT
List CRSD Debug Module: CLSFRAME
List CRSD Debug Module: CLSINET
List CRSD Debug Module: CLSO
List CRSD Debug Module: CLSVER
List CRSD Debug Module: CLUCLS
List CRSD Debug Module: COMMCRS
List CRSD Debug Module: COMMNS
List CRSD Debug Module: CRSAPP
List CRSD Debug Module: CRSCCL
List CRSD Debug Module: CRSCEVT
List CRSD Debug Module: CRSCOMM
List CRSD Debug Module: CRSD
List CRSD Debug Module: CRSEVT
List CRSD Debug Module: CRSMAIN
List CRSD Debug Module: CRSOCR
List CRSD Debug Module: CRSPE
List CRSD Debug Module: CRSPLACE
List CRSD Debug Module: CRSRES
List CRSD Debug Module: CRSRPT
List CRSD Debug Module: CRSRTI
List CRSD Debug Module: CRSSE
List CRSD Debug Module: CRSSEC
List CRSD Debug Module: CRSTIMER
List CRSD Debug Module: CRSUI
List CRSD Debug Module: CLNT
List CRSD Debug Module: OCRAPI
List CRSD Debug Module: OCRASM
List CRSD Debug Module: OCRCAC
List CRSD Debug Module: OCRCLI
List CRSD Debug Module: OCRMAS
List CRSD Debug Module: OCRMSG
List CRSD Debug Module: OCROSD
List CRSD Debug Module: OCRRAW
List CRSD Debug Module: OCRSRV
List CRSD Debug Module: OCRUTL
List CRSD Debug Module: SuiteTes
List CRSD Debug Module: UiServer
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl check has
CRS-4638: Oracle High Availability Services is online
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
# /u01/app/grid/product/12.2.0/grid_1/bin/crsctl stat res -t
--------------------------------------------------------------------------------
State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ora.DG_DATA01.dg
ora.LISTENER.lsnr
ora.MGMT.dg
ora.net1.network
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
ora.MGMTLSNR
OFFLINE OFFLINE
Started,STABLE
Started,STABLE
OFFLINE OFFLINE
ora.nodea.vip
ora.nodeb.vip
ora.qosmserver
ora.rac0.db
Open,HOME=/u01/app/o
racle/product/12.2.0
/db_1,STABLE
Open,HOME=/u01/app/o
racle/product/12.2.0
/db_1,STABLE
ora.scan1.vip
--------------------------------------------------------------------------------
查看一些进程
# ps -fu root|grep -i init
00:00:00 /usr/sbin/alsactl -s -n 19 -c -E ALSA_CONFIG_PATH=/etc/alsa/alsactl.conf --initfile=/lib/alsa/ini/00main rdaemon
0 10:19 pts/0
00:00:00 grep --color=auto -i init
00:00:00 /bin/sh /etc/init.d/init.tfa run &/dev/null 2&&1 /dev/null 2&&1
# ps -fu root|grep -i grid_1 root
0 10:17 pts/0 00:00:00 grep --color=auto -i grid_1 root
09:31 ? 00:00:25 /u01/app/grid/product/12.2.0/grid_1/jdk/jre/bin/java -Xms128m -Xmx512m oracle.rat.tfa.TFAMain /u01/app/grid/product/12.2.0/grid_1/tfa/nodea/tfa_home root
09:33 ? 00:00:08 /u01/app/grid/product/12.2.0/grid_1/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8 root
09:33 ? 00:00:04 /u01/app/grid/product/12.2.0/grid_1/bin/orarootagent.bin root
09:33 ? 00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/cssdmonitor root
09:33 ? 00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/cssdagent root
09:34 ? 00:00:07 /u01/app/grid/product/12.2.0/grid_1/bin/octssd.bin reboot root
09:34 ? 00:00:15 /u01/app/grid/product/12.2.0/grid_1/bin/osysmond.bin root
09:34 ? 00:00:16 /u01/app/grid/product/12.2.0/grid_1/bin/crsd.bin reboot root
09:34 ? 00:00:12 /u01/app/grid/product/12.2.0/grid_1/bin/orarootagent.bin root
09:34 ? 00:00:13 /u01/app/grid/product/12.2.0/grid_1/bin/ologgerd -M root
0 09:35 ? 00:00:03 /u01/app/grid/product/12.2.0/grid_1/perl/bin/perl /u01/app/grid/product/12.2.0/grid_1/bin/diagsnap.pl start
# ps -fu grid
C STIME TTY
00:00:00 /u01/app/grid/product/12.2.0/grid_1/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/scriptagent.bin
00:00:07 /u01/app/grid/product/12.2.0/grid_1/jdk/bin/java -server -Xms128M -Xmx384M -Djava.awt.headless=tru
00:00:03 /u01/app/grid/product/12.2.0/grid_1/jdk/bin/java -classpath /u01/app/grid/product/12.2.0/grid_1/jd
00:00:14 /u01/app/grid/product/12.2.0/grid_1/bin/oraagent.bin
00:00:00 /u01/app/grid/product/12.2.0/grid_1/opmn/bin/ons -d
00:00:00 /u01/app/grid/product/12.2.0/grid_1/opmn/bin/ons -d
00:00:00 /u01/app/grid/product/12.2.0/grid_1/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
00:00:00 /u01/app/grid/product/12.2.0/grid_1/bin/tnslsnr LISTENER -no_crs_notify -inherit
00:00:07 /u01/app/grid/product/12.2.0/grid_1/bin/oraagent.bin
00:00:01 /u01/app/grid/product/12.2.0/grid_1/bin/mdnsd.bin
00:00:07 /u01/app/grid/product/12.2.0/grid_1/bin/evmd.bin
00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/gpnpd.bin
00:00:02 /u01/app/grid/product/12.2.0/grid_1/bin/evmlogger.bin -o /u01/app/grid/product/12.2.0/grid_1/log/[
00:00:05 /u01/app/grid/product/12.2.0/grid_1/bin/gipcd.bin
00:00:08 /u01/app/grid/product/12.2.0/grid_1/bin/ocssd.bin
00:00:00 asm_pmon_+ASM1
00:00:00 asm_clmn_+ASM1
00:00:00 asm_psp0_+ASM1
00:00:27 asm_vktm_+ASM1
00:00:00 asm_gen0_+ASM1
00:00:00 asm_mman_+ASM1
00:00:00 asm_gen1_+ASM1
00:00:01 asm_diag_+ASM1
00:00:00 asm_ping_+ASM1
00:00:00 asm_pman_+ASM1
00:00:04 asm_dia0_+ASM1
00:00:02 asm_lmon_+ASM1
00:00:02 asm_lmd0_+ASM1
00:00:05 asm_lms0_+ASM1
00:00:02 asm_lmhb_+ASM1
00:00:00 asm_lck1_+ASM1
00:00:00 asm_dbw0_+ASM1
00:00:00 asm_lgwr_+ASM1
00:00:00 asm_ckpt_+ASM1
00:00:00 asm_smon_+ASM1
00:00:00 asm_lreg_+ASM1
00:00:00 asm_pxmn_+ASM1
00:00:00 asm_rbal_+ASM1
00:00:00 asm_gmon_+ASM1
00:00:00 asm_mmon_+ASM1
00:00:00 asm_mmnl_+ASM1
00:00:03 asm_imr0_+ASM1
00:00:00 asm_lck0_+ASM1
00:00:02 asm_gcr0_+ASM1
00:00:00 asm_ppa7_+ASM1
00:00:00 asm_asmb_+ASM1
00:00:01 oracle+ASM1_crf (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
# ps -fu oracle
C STIME TTY
00:00:00 ora_w005_rac02
00:00:00 ora_w004_rac02
00:00:10 /u01/app/grid/product/12.2.0/grid_1
00:00:00 ora_pmon_rac02
00:00:00 ora_clmn_rac02
00:00:01 ora_psp0_rac02
00:00:01 ora_ipc0_rac02
00:01:02 ora_vktm_rac02
00:00:00 ora_gen0_rac02
00:00:00 ora_mman_rac02
00:00:01 ora_gen1_rac02
00:00:04 ora_diag_rac02
00:00:00 ora_ofsd_rac02
00:00:02 ora_dbrm_rac02
00:00:00 ora_vkrm_rac02
00:00:00 ora_ping_rac02
00:00:00 ora_sb_rac02
00:00:00 ora_acms_rac02
00:00:00 ora_pman_rac02
00:00:12 ora_dia0_rac02
00:00:08 ora_lmon_rac02
00:00:22 ora_lms0_rac02
00:00:09 ora_lmd0_rac02
00:00:01 ora_rmv0_rac02
00:00:00 ora_rms0_rac02
00:00:06 ora_lmhb_rac02
00:00:00 ora_lck1_rac02
00:00:00 ora_dbw0_rac02
00:00:00 ora_lgwr_rac02
00:00:02 ora_ckpt_rac02
00:00:00 ora_smon_rac02
00:00:00 ora_smco_rac02
00:00:00 ora_reco_rac02
00:00:00 ora_lreg_rac02
00:00:00 ora_pxmn_rac02
00:00:00 ora_rbal_rac02
00:00:00 ora_asmb_rac02
00:00:05 ora_fenc_rac02
00:00:14 ora_mmon_rac02
00:00:01 ora_mmnl_rac02
00:00:00 ora_d000_rac02
00:00:00 ora_s000_rac02
00:00:08 ora_imr0_rac02
00:00:03 ora_scm0_rac02
00:00:01 ora_lck0_rac02
00:00:00 ora_mark_rac02
00:00:00 ora_rsmn_rac02
00:00:00 ora_tmon_rac02
00:00:09 ora_gcr0_rac02
00:00:00 ora_tt00_rac02
00:00:00 ora_tt01_rac02
00:00:00 ora_tt02_rac02
00:00:00 ora_gtx0_rac02
00:00:00 ora_rcbg_rac02
00:00:00 ora_aqpc_rac02
00:00:03 ora_p000_rac02
00:00:03 ora_p001_rac02
00:00:00 ora_p002_rac02
00:00:00 ora_p003_rac02
00:00:00 ora_qm02_rac02
00:00:03 ora_qm05_rac02
00:00:00 ora_q004_rac02
00:00:07 ora_cjq0_rac02
00:00:00 ora_w003_rac02
00:00:00 ora_w000_rac02
00:00:00 ora_w001_rac02
00:00:00 ora_w006_rac02
00:00:00 ora_ppa7_rac02
00:00:05 ora_o000_rac02
00:00:00 ora_w002_rac02
00:00:00 ora_w007_rac02
00:00:00 ora_gcr1_rac02
00:00:00 ora_q005_rac02
00:00:05 ora_o001_rac02
00:00:00 ora_p004_rac02
00:00:00 ora_p005_rac02
00:00:01 oraclerac02 (DESCRIPTION=(LOCAL=YES
00:00:00 oraclerac02 (DESCRIPTION=(LOCAL=YES
00:00:00 oraclerac02 (DESCRIPTION=(LOCAL=YES}
我要回帖
更多关于 D/A 的文章
- ·CNG套件D06GT喷射电脑适用于什么车型
- ·先锋50U600D的PPTV PAD为什么不能用了
- ·pspice多个地仿真不出现 而且弹出A/D demo 那个对话框 怎么解决 谢谢了
- ·高中化学在线采纳求解 看D威慑呢用冷却结晶不行这两种结晶区别是什么
- ·为啥不为什么会选择我D
- ·Skechers2016年新款D'lites2情侣熊猫鞋 复古休闲跑步鞋99999111怎么鉴别正品
- ·D的一双球鞋，怎么样哈哈
- ·600w是多少v足球心水资料有没有免费的？
- ·减压孔板减压8.71米，D=51mm
- ·Ъ ê τ 3 6 5 d 7. c о м#UTrwe 为什么央视现在看不了 中超
- ·Ъ ê τ 3 6 5 d ⒌ c ом#wereds中超中甲足球联赛什么时候能有安徽足球队
- ·在梯形图中加上[LD M8000,MOV C0 D0,MOV D9 D1]这三条指令是什么意思?
- ·以DAC0832为例，说明D/A的单缓冲与双缓冲有何不同
- ·思科SG100D-08和思科SG90D-08和思科SG95D-08哪个好
- ·这个D选项的式子是怎么得出来的
- ·FOREVER永久QJ006-D山地车好不好？永久山地车值得选吗
- ·北京时间今天凌晨，同处在欧冠小组赛D组的（ ）队和尤文图斯队分别轻取赫罗纳队和（
- ·Cr/D/Fail这个有什么特别之处吗
- ·后卫怎么打1V2的情况下 大D们都怎么防守的
- ·CDND more exceptional c++pdf下载，分享一下百度云，谢谢！！！
- ·D=1 2 1 4 0 -1 2 -1 4 7 2 1 0 2 1 3
- ·今年竞猜世界 D组排名的情况？
- ·耐克zoom wil Dhorse 4为什么网上买的没有
- ·LR-ZP-P-D-1和LR-ZP-A-D-1有什么不同
- ·普菜达手机型D236，语音找手机资料丢失，想复制同手机型号语音找手机软件安装，能够解决吗？谢谢
- ·D：FUSE（成新世界怎么样店）怎么样，好不好的默认
- ·我的D照明年7月份到期,但我在北京打工回不来怎么办?
- ·YG701D型全自动织物缩水率试验机能当冼衣机用吗
- ·文丘里管推动活塞A上升,活塞直径D质量m，d1=20mm,d2=10mm,h=100mm
- ·Hub75-D是08还是12的插口
- ·D.cd5-10.DC是什么表面处理方式？
- ·001D0F32374E无线网卡的驱动在哪下
- ·星宝TD15一38彩电高压包23一D201xB(华强电视配件公司)初级对地电阻为零是原设计还是击穿
- ·kFR一65w/D一TD和KFR一65W/D360一TD有区别吗
- ·这个是D有什么影响吗？
- ·IF(D4>=600012,B4*1.5%,IF(D4<=60000>=30001,B4*0.9%),IF(D4<=30000>=20001,B4*0.5%))这个公式错在
- ·您好，能给发一下MSVCP71D.dll 在win7 64位操作系统下的 发到705020841@qq.com就好
- ·有骨质增生，买了康百力的钙 D和康百力鲨鱼软骨粉，每天怎么吃几粒比较好，饭前吃还是饭后吃？
- ·htt D://minban.sz.edu.cn
- ·为什么选项D是对的？
- ·据ℒุิิDุ็็官方报导巨星接连倒下腹股沟真的如此凶残吗？
- ·ℒุิ是什么意思;ิ是什么意思;Dุ็็传奇球星，你认识几个?
- ·ℒ ̶D想知道，有哪些规划球员值得一提的?
- ·富莱仕DOF隼-150D和爱图仕120D比性能怎么样？
- ·第八题的D选项如何求最短半径求给出证明为何与CD,bc相切时是最短半径。
- ·3506.2OO =( )D？
- ·朵唯D1如何开启语音助手？
- ·高压锅上标的D一2丶24一28是什么意思意思谢谢了！
- ·D一苎檬烯怎样运用到水基清洗剂中
- ·李さんが来ない（D）は、决してはじめない。 Aこと Bつもり Cまま Dかぎり 本句怎么译じめない什么意思
- ·自动变速器车挂D位时，车有轻微振动
- ·NW一BUS|NESS一VAL|D一ERROR一TxYZM一1254是什么
- ·OPPO Reno2 Z4D震感模式怎么打开
- ·chateau tour D'Auron 2015年份的多少钱
- ·BT1200DQT—2A倍特电动车是多少V多少A
- ·D乚AsSllLSASERp○DucT
- ·怎么突然东南D x 3后窗关不上
- ·你好，请问这道题的D选项，fx的单增区间为什么不是（1，+∞）谢谢！
- ·Hz50E5D怎么识别画面出现的人或者物海信为什么不能给个完整的操作说明
- ·高中物理一题为什么D选项有两种说法 哪个对
- ·杀毒之后Dbugview停止工作 怎么重新安装回去
- ·自动变速器车挂D位时，车有轻微振动
- ·D.从女排世界杯的赛场上传来惊心动魄的消世界杯冠军。有没有错
- ·钉钉DlNG发送时最多多少字符
- ·精益生产自动化中什么是人机分离程度D等级
- ·高中物理，D选项详解
- ·请问R9126D各管脚用万表怎样检测
- ·MB65-3508D内胆如何清洗
- ·口罩AIR和POR.D有什么区别
- ·K46的节拍条件时值一定要比D更多或跟D一样吗 K46的时值比D时值短可以吗
- ·LGWD-丅12410D主版少两个排插口有什么用
- ·杨氏两小孔d=1mm,小孔D=50cm,n=1.58的厚板挡住一个小孔条纹移动45cm,求薄片厚度
- ·英业达怎么样服务器部分备件上标注的D/C代表什么
- ·苹果6s更新ios13.4三D塌池会取消吗
- ·c语言直接选择排序选择D那里错了
- ·0xD热熔性聚烯复合防水卷材是什么材料
- ·请问可不可以发一下风A月D大陆无删减版给我，737657245@qq.com
- ·监控录像机的标签上8109D是什么意思
- ·银威JBY-D-201(C)点钞机多少钱
- ·SN18000D如何制母卡
- ·设D表示由全体能被3除尽的整数组成的集合,则d= 时,d D A -12 B 2 C -13 D 19
- ·英杰电熔焊机DPS3一IGPT怎样才能调到套筒的参数
- ·ARcSABERD-8什么球拍
- ·宏碁v3-771和772的D面通用吗
- ·八脚贴片第一行显示D134033 第二行显示1112A3第三行显示•638AB代表什么
- ·淘比乐D20如何配对
更多推荐
- ·百度怎样登录知道，怎么不能登录❓？
- ·家里养鱼几条最吉利风水讲究是什么？
- ·鱼缸放置在客厅哪个位置好的问题？怎样处理才能不影响风水？
- ·本田XR–V价格一般多少钱-V和思域这两款价格差不多，拿不准要哪个，感觉思域配置高？
- ·请问谁有关东情事这部看日本电视剧的软件片源?
- ·不浪费每一寸阳光是词吗?
- ·怎样可以找出自身你能找出图中的错误吗？因为有些错误自己意识不到。
- ·在一个减法算式里被减数减数,被减数,减数与差这三个数之和是498,减数比差大19,求减数是
- ·初三英语上册语法总结填词
- ·show hand是什么意思
- ·写鹤庆西龙潭的作文
- ·凉菜培训哪里好？口味要正宗凉拌菜技术培训的？
- ·化学离子共存问题问题。
- ·青铜葵花里你我最喜欢的一本书青铜葵花谁？为什么？举一个例子
- ·拟采用套管式换热器用120℃的饱和水蒸气的饱和蒸汽压（汽化热 r=2205kJ/kg）加热空气。空气以150k
- ·开8去年开高分数线是多少2019
- ·高数第二类曲面积分公式分
- ·67×109+109×21+12×109180÷15用简便方法计算计算
- ·我初中生想要一个手表，简约的，100左右
- ·D=1 2 1 4 0 -1 2 -1 4 7 2 1 0 2 1 3
- ·10千瓦49极发电机机用两个5千瓦2极的电机能带动吗？
- ·今年全球航空业净利润如何？
- ·我国地面站接收到高分六号西昌卫星发射基地首轨数据了吗？
- ·红米和nubia的长哪个手机电池续航最好机型怎么样
- ·问超级本，轻薄本和笔记本的区别，有什么区别
- ·电脑weixinfile文件为什么打不开
- ·苹果六 K歌上如何把字体放大
- ·求Linux0.11的源代码及中文分析PDF，
- ·谁能介绍下网络营销的主要内容有？
- ·沙特对上埃及队网络电视可以看嘛？
- ·有会用Dreamweaver CS6做企业网业的么？
- ·软件日语实时翻译软件和人工日语实时翻译软件哪个好
- ·人工加粉的可靠吗翻译比软件翻译更可靠吗
- ·51单片机excel查找相同单元格30-3F单元中的相同数个数
- ·winform有哪些好用的控件做小说阅读器的时候用什么控件好可以实现手机阅读器的效果
