Technology Blog by Ajit: End to end configuration of SDP listener

I have been doing few SDP listener configuration in quarter to full rack Exadata. SDP is one of the performance enhancement done by oracle in Exadata. This uses infiniband, so Exalogic racks which are connected in the same infiniband fabric make use of this SDP.
SDP is easy to set and you can do it in few steps. For easy understanding I'm using 2 node Exadata.
tnsnames.ora file is also added in this page for reference.

Pre-setup

It is better to use dcli to propagate these changes to all the nodes.

Two files have to be edited to enable SDP setup in each node and those nodes have to be rebooted.

create a text file called node with all the node name

ed05db01

ed05db02

Check if /etc/infiniband/openib.conf
SDP_LOAD=yes

Use the below script to do presetup

#!/bin/bash
# Script name :preset.sh

sed -i "/^use both server/c\use both server * :" /etc/ofed/libsdp.conf
sed -i "/^use both client/c\use both client * :" /etc/ofed/libsdp.conf

grep sdp_zcopy_thresh /etc/modprobe.conf

if [ $? = 0 ] ; then

sed -i "/sdp_zcopy_thresh/c\options ib_sdp sdp_zcopy_thresh=0 recv_poll=0 sdp_apm_enable=0" /etc/modprobe.conf

else

echo "options ib_sdp sdp_zcopy_thresh=0 recv_poll=0 sdp_apm_enable=0" >> /etc/modprobe.conf

chmod 750 preset.sh
dcli -g node -l root -x preset.sh
Some cases I had seen that above changes took effect without rebooting the node. I have no explanation for that. So at the end if you don't see sdp not working, reboot the node

dcli -g node -l root reboot
Now you need internal IPs for each node, so for us we need 2 IPs.
NOTE: These IPs shouldnt not be pingable as of now, you should get 100% pocket loss.
All ips should be part of your internal subnet. by default it'll be configured in bondib0

Setup

[root@ed05db01 admin]# ifconfig bondib0|grep "inet addr"
inet addr:192.168.169.41 Bcast:192.168.175.255 Mask:255.255.248.0

Use a IP calculator available via google. I usually use http://jodies.de/ipcalc
provide your ip: 192.168.169.41 and netmask: 255.255.248.0 and pick any two unpingable ip from range between "HostMin" and "HostMax" and add those entry into /etc/hosts of each node of your rack

192.168.170.207 ed05db01-ibvip.comp.com ed05db01-ibvip
192.168.170.208 ed05db02-ibvip.comp.com ed05db02-ibvip

Now create network with IP "Network" IP you got from the calculator

/u01/app/11.2.0.4/grid/bin/srvctl add network -k 2 -S 192.168.168.0/255.255.248.0/bondib0

You can validate this by

/u01/app/11.2.0.4/grid/bin/crsctl stat res -t | grep net

Based on /etc/hosts entry now you can create vip for each node, but can be executed from one node

/u01/app/11.2.0.4/grid/bin/srvctl add vip -n ed05db01 -A ed05db01-ibvip/255.255.248.0/bondib0 -k 2

/u01/app/11.2.0.4/grid/bin/srvctl add vip -n ed05db02 -A ed05db02-ibvip/255.255.248.0/bondib0 -k 2

Now add and start SDP listener from one node, run it as GI owner:

/u01/app/11.2.0.4/grid/bin/srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522
/u01/app/11.2.0.4/grid/bin/srvctl start listener -l LISTENER_IB

Now to know what VIP is configured for each node, execute

srvctl status vip –i ed05db01

srvctl status vip –i ed05db02

Now register your remote listener

alter system set remote_listener="ed05db-scan1:1521" scope=both;

After this tnsnames.ora have to be edited with these information tnsnames.ora file

After this register the sdp with the database
alter system set listener_networks='((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))', '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;