Uploading image thru Docker in OCI

Upload of image thru GUI OCI console or you can either user cloudshell or thru OCI executable.

To upload you need

- auth key

- tag

- registry name

Auth key generation

You can have maximum of 2 auth keys for the user and make sure you save key when its generated.

oci ce cluster generate-token --cluster-id ocid1.cluster.oc1.iad.....

This should generate less than 20chars and keep it safe.

Docker login

Login into docker using your id. If you are using your corporate SSO, you have to use your OCI federated id and auth token is the password.

List of region is available in

https://docs.cloud.oracle.com/en-us/iaas/Content/Registry/Concepts/registryprerequisites.htm

docker login us-ashburn-1.ocir.io

Username: <tenancy>/oracleidentitycloudservice/ajit.solomon

Password:

Login Succeeded

Now from your local docker repo you can upload to OCI repository by tagging it.

Docker push

To tag, you have provide region/tenanency/registry_name

docker tag cd14cecfdb3a us-ashburn-1.ocir.io/<tenanency>/jenkins:latest

Prerequisite for pushing an image is to create manually an empty repo/image thru OCI gui. For example, if you want to push the above tagged image jenkins, then go to Container Registry --> Create Registry as jenkins

Now you can push image to OCI registry

docker push us-ashburn-1.ocir.io/<tenanency>/jenkins:latest

If it doesn't find the registry, then OCI will attempt to create it under root compartment.

You can view if it's successful in Developer Services --> Registry(OCIR)

OCI - Manual creation of Instance from scratch

All the action provided here can be done from OCI console thru clicks. This is when you want to scripting thru OCI cli. You can either use cloudshell or install OCI module for your desktop.

Most of values we use is either configured in config or oci_cli_rc files.

As I use some of the resource very frequently, I had put, 

- user

- fingerprint

- key_file

- tenancy

- region

- compartment-id

- availability-domain

I had used CLI to configure from compartment to instance creation.

oci iam compartment create --name 'Ajit' --description "Testing OCI"

{

  "data": {

    "compartment-id": "ocid1.tenancy.oc1.....",

  ------------------------------------------------------------------------------------------------------------------------------

With that you have place to create your resources with assumption that proper quota is allocated.

Each compartment needs to have VCN for proper seperation.

oci network vcn create --cidr-block 192.168.0.0/16 --display-name AjitVcn1 --dns-label AjitDns1

This will allow us to carve out subnet in the vcn

oci network subnet create --cidr-block 192.168.10.0/24 --vcn-id ocid1.vcn.oc1.iad.... --security-list-ids '["ocid1.securitylist.oc1...."]'

Both vcn-id and security-list-ids is taken from "oci network vcn create" output

  ------------------------------------------------------------------------------------------------------------------------------

Now we can move to connect the vcn to connect to outside the instance vcn.

oci network internet-gateway create --is-enabled true --vcn-id ocid1.vcn.oc1.iad.... --display-name AjitGW

After opening up the Gateway for the vcn, we need to streamline who can connect to our GW.

Route table info can be seen in "oci network subnet create" output.

oci network route-table update --rt-id ocid1.routetable.oc1.iad.... --route-rules '[{"cidrBlock":"0.0.0.0/0","networkEntityId":"ocid1.internetgateway.oc1.iad...."}]'

  ------------------------------------------------------------------------------------------------------------------------------

All our setup to spin up compute is ready. Make sure you generate ssh key before you spin up the compute.

oci compute instance launch --display-name AjitVm --image-id ocid1.image.oc1.iad.... --subnet-id ocid1.subnet.oc1.iad.... --shape VM.Standard2.1 --assign-public-ip true --metadata '{"ssh_authored_keys":"ajit"}'

You can confirm the creation using "oci compute instance get"

Using the key, you can login thru opc user.

To prefix zero in front of digits

To prefix zero in front of digits

In bash, prefixing zero needs more of formatting, than direct way. This example shows how to prefix zero at the same time maintain the digit width.

#!/bin/bash
export TXT=Number
for LP in `seq -f "$TXT%02g" 12`
do
        echo $LP
done

The output will have the same width.

Number01

Number02

Number03

Number04

Number05

Number06

Number07

Number08

Number09

Number10

Number11

Number12

SDP tnsnames.ora entry for 2 node rack

Model tns entry in the first node, next node will be other way

DBM =
(DESCRIPTION =
  (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = ed05-scan1)(PORT = 1521))
  )
  (CONNECT_DATA =
    (SERVICE_NAME = dbm)
   )
)

DBM_IB =
(DESCRIPTION =
  (ADDRESS_LIST =
    (LOAD_BALANCE=on)
    (ADDRESS = (PROTOCOL = SDP)(HOST = ed05db01-ibvip)(PORT = 1522))
    (ADDRESS = (PROTOCOL = SDP)(HOST = ed05db02-ibvip)(PORT = 1522))
   )
  (CONNECT_DATA =
     (SERVICE_NAME = dbm)
   )
)

LISTENER_IBREMOTE =
(DESCRIPTION =
   (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = SDP)(HOST = ed05db02-ibvip.oracle.com)(PORT = 1522))
   )
)

LISTENER_IBLOCAL =
(DESCRIPTION =
   (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = ed05db01-ibvip)(PORT = 1522))
      (ADDRESS = (PROTOCOL = SDP)(HOST = ed05db01-ibvip)(PORT = 1522))
   )
)

LISTENER_IPLOCAL =
(DESCRIPTION =
    (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = ed0501-vip)(PORT = 1521))
    )
)

LISTENER_IPREMOTE =
(DESCRIPTION =
  (ADDRESS_LIST =
     (ADDRESS = (PROTOCOL = TCP)(HOST = ed0502-vip)(PORT = 1521))
  )
)

Changing password in non-interactive

Password change in Linux is done thru "passwd" command, but the bottleneck on this is interactive. Particularly it'll be difficult to handle thru scripting.
Here is the one liner for creation of useradd and changing its password

useradd -u 5313 -g dba -d /home/ajit -m ajit

To change the password for the user ajit,

echo "ajit:newpass"|chpasswd

These password can also be done thru encryption.

IP address validation in linux/bash

Finding an IP address is valid or not.

In Linux 5, instead of finding IP is valid or not thru script, we can just execute a command to find. This helps to decrease number of lines we write to find its validity.

If you provide correct IP, the return code will be zero. If a not-a-valid IP provided, then the return value will be 1

# ipcalc -c 10.10.10.10

# echo $?

0

# ipcalc -c 10.10.10.1011

ipcalc: bad IPv4 address: 10.10.10.1011

# echo $?

1

First time passwordless ssh setup without manually entering password

First time passwordless ssh automation

     Passwordless ssh setup for one or two servers will be OK to do in the traditional way, where you enter password for the first time. If it comes to many server, it'll be cumbersome.
    This is the same thing happened to one of my friend, where she has to do over 50 servers for her important work. She was entering password manually. 

I wanted to avoid manual entry of password and finally I came across "expect" scripting. By this you can do passwordless ssh without entering it manually. 

There could be two scenario, one you have the same password in all the servers/nodes or different password in each server/node. If you want to do many, just have a masterfile with servers/nodes and password and you can loop thru this.

As you can easily write a wrapper loop script, I'm just providing what is required.  I'm new to expect script and in the process of enhancing the code.
--------------------------------------------------------------------------------------------------------------------------

One time or optional
rm -rf /root/.ssh/id_rsa
ssh-keygen -q -f /root/.ssh/id_rsa -N ''

--------------------------------------------------------------------------------------------------------------------------

Add your remote servername to known_hosts as ssh-keyscan doesnt have option of -o 
sed -i /^"<servername>/d >> /root/.ssh/known_hosts 2>&1
ssh-keyscan -t rsa <servername> 2>&1 >> /root/.ssh/known_hosts 

--------------------------------------------------------------------------------------------------------------------------

Have the below in a separate file.
#!/usr/bin/expect

# This script pushes the key to the remote host. ssh-copy-id doesn't check for duplication, so displaying an message also in the script.

set host [lrange $argv 0 0]
set pass [lrange $argv 1 1]
set timeout -1
spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$host
match_max 100000

# Looking for all possible string. Options might change in your environment
expect {
        "password" {
                send -- "$pass\r"
                send -- "\r"
                sleep 1
                send_user "        Ssh key copied to $host\n"
                send_user "     **** Check for duplication entry in $host:/root/.ssh/authorized_keys ****\n"
        }
        "$host's" {
                send_user "        Ssh key copied to $host\n"
                send_user "     **** Check for duplication entry in $host:/root/.ssh/authorized_keys ****\n"
        }
        "Password:*" {
                send -- "$pass\r"
                send -- "\r"
                sleep 1
                send_user "        Ssh key copied to $host\n"
                send_user "     **** Check for duplication entry in $host:/root/.ssh/authorized_keys ****\n"
        }
        "machine" {
                sleep 1
                send_user "        Ssh key copied to $host\n"
                send_user "     **** Check for duplication entry in $host:/root/.ssh/authorized_keys ****\n"
        }
}
interact

In case if your expect script fails, try to troubleshoot thru expect debugger "expect -d <script>"

man-in-the-middle ssh issue

This common "man-in-the-middle" can be seen whenever the ssh key got changed. This particularly annoys when it fails the automation process.
As a manual process you can open ~/.ssh/known_hosts and delete the host file you are trying to access.
More clean way of doing and which can also be part of the automation, use  ssh-keygen with -R

             ssh-keygen -R 10.10.10.10

You might have to look for both hostname, hostname with domain and ip address. This will avoid your "man-in-the-middle" issue.
After this create a fresh key.

LFI-01518: write error

Message and corrective action provided by far looks like a serious issue. It could be in few occasion, but most common issues are

• Filesystem full
• No privilege to write.

CLSD: A file system error occurred while attempting to write to file "/u01/app/11.2.0.3/grid/log/ed05db02/alerted05db02.log" during alert write processing for process "client". Additional diagnostics: LFI-00004: Call to lfibwrt() failed.

LFI-01518: write() failed(OSD return value = 28) in slfiwl.

Common occasion are when you are trying to start ASM instance thru automatic or manual start.

How to avoid Blob corruption

Blob corruption

   Oracle database have Basic and Secure blob type. Both store images, but former one have many bugs and development stopped long time ago. So its better to convert existing Basic blob to Secure and avoid corruption.

One of the error you get when Basic blob got corrupted is

ORA-01555: snapshot too old: rollback segment 

The above error usually comes when you execute DML statements, but you'll get this for blob corruption too. This blog describes how you can avoid this. The conversion wont affect your application functionality.

Fresh creation

We can force all new blob creation in the database to secure, by executing 

alter system set db_securefile=always scope=both;

The above statement will force Secure or Basic blob to be created as Secure.

Existing Basic to Secure conversion

 This can be done in few steps using redefinition.

• create a dummy table with same table structure with securefile clause

 create table blob_tab
(eid varchar2(40), details blob, primary key(eid))LOB (details) STORE AS SECUREFILE (tablespace users)  tablespace users;

• Start the redef proces

dbms_redefinition.start_redef_table('scott', 'old_table', 'blob_tab', 'eid eid, details details');

• If you have any primary key in the original table drop it temporarily.

alter table scott.old_table drop primary key;

• Copy all relevant information, declare a variable "error" as number

dbms_redefinition.copy_table_dependents ('scott','old_table', 'blob_tab',1,true, true, true, false, error);

• Again create the primary key which was dropped before

alter table scott.old_table add primary key(eid);

• Move back all the information back to the original table

dbms_redefinition.finish_redef_table ('scott', 'old_table', 'blob_tab')

You can confim this with by,
select securefile from user_lobs where table_name='OLD_TABLE'; 

Rolling back

In case you want to rollback, create a interim table without "securefile" option and follow the same procedure. 

Remember your db_securefile parameter overrides your create table option for blob.

Troubleshooting

You could face few issues.

If you abort/error out in the middle of redef and start, you'll get the below error.

ORA-23539: table "SCOTT"."OLD_TABLE" currently being redefined

You can delete "materialized view log" and drop original table and recreate interim table.

Or, you can simply

dbms_redefinition.abort_redef_table('scott','old_table','blob_tab');
-----------------------------------------------------------------------------------------------------------------------------

If you have primary key you have to drop and recreate it, just like the above steps, else you'll get

ORA-01408: such column list already indexed

Check emkey is copied to the repository

 Emkey is kept secured,but upgrade process demand that it should be available. So we have to expose emkey to the installer. After that the key can be removed from repository and it make the emkey secured again.

So if you get an error message for the prerequisite "Check emkey is copied to the repository". For all this activity you need SYSMAN password.

Go to the OMS server and ORACLE_HOME/bin

• ./emctl config emkey -copy_to_repos

Now proceed again with the upgrade. Once the upgrade is finished, remove the key from the repository.

• ./emctl config emkey -remove_from_repos

To know the status of the emkey, execute

• ./emctl status emkey

The key can be seen in $ORACLE_HOME/sysman/config/emkey.ora

Mail/sendmail activation in Linux

Mail setup has to be activated in each linux server. So it can monitored for our regular use. Do mailx/sendmail from that linux server and if you don't receive email, check the basic setup requirement. This could be the probable cause if it's a new system. For this you have to know your internal mail server.

Change the file /etc/sysconfig/sendmail with the below information

DAEMON=no

Now you need your internal mail information

In file /etc/mail/submit.cf look for D{MTAHost}. Change that with your internal mail server

D{MTAHost}<internal-mail.company.com>

Use your mailx/sendmail to check the activation.

Missing id_rsa file

One of the important component in passwordless setup is id_rsa key file. This file contains the key for the current server. Scenario of missing id_rsa is usually a new setup. id_rsa file can be created by ssh-keygen.

                                       ssh-keygen -q -f /root/.ssh/id_rsa -N ''

Now if you start your passwordless process, it should go thru.

Changing owership of a Listener

This page deals with handling permission for the listener. Starting or managing a listener needs os user privilege. This works just like chmod, but executing with crsctl command. If improperly configured others can take advantage of the listener.

To check what is the current permission.
crsctl status resource ora.LISTENER_IB.lsnr -p | grep ACL=
The output will look like below..
ACL=owner:root:rwx,pgrp:root:r-x,other::r--

Here it shows who is the owner, primary group and others.
owner = root with permission rwx
pgrp = root with permission r-x
other = permission is only read.

To change the primary group
crsctl setperm resource ora.LISTENER_IB.lsnr -u 'pgrp:oinstall:rwx'

This can be seen with

crsctl status resource ora.LISTENER_IB.lsnr -p | grep ACL=

ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--

Connecting OVMM to EMCC

Many steps are involved in Configuring and Connecting OVMM. I'm sharing this as I passed thru many hurdles. This blog only details the steps for setting the authentication and connect string.

You need to have java and location where you can create the key. You need access to both root or sudo and the id where agent is installed.

Login into ovmm vServer

export JAVA_HOME=/u01/oem_agent/core/12.1.0.3.0/jdk

export OMS_AGENT_HOME=/u01/oem_agent

As root

$JAVA_HOME/jre/bin/keytool -keystore /u01/app/oracle/ovm-manager-3/ovmmCoreTcps.ks -exportcert -alias ovmm -file  $OMS_AGENT_HOME/ovmmCoreTcps.ks

As oracle, execute the following. Default password is "welcome"

$OMS_AGENT_HOME/agent_inst/bin/emctl secure  add_trust_cert_to_jks -trust_certs_loc $OMS_AGENT_HOME/ovmmCoreTcps.ks -alias ovmm
Once the key is created, you have to enable the remote access. With the current ovmm version, non-secure port 54321 is disabled, instead use the secured port 54322.
Username and password is your ovmm admin account and its password, usually it is "admin"

provide full dns server name

cd /u01/app/oracle/ovm-manager-3/ovm_shell

sh ovm_shell.sh --url=tcps://exl-ovmm.us.oracle.com:54322 --username=admin --password=welcome1
ovm = OvmClient.getOvmManager ()
f = ovm.getFoundryContext ()

j = ovm.createJob ( 'AK00012345' );<-The number can be seen in the vDC first node, /var/exalogic/info/em-context.info 
j.begin ();

f.setAsset ( "EXALOGIC_ID",'AK00012345'); 

j.commit ();

ctrl d

cd /u01/app/oracle/ovm-manager-3/bin

Execute the below scripts, which are self explanatory

./secureOvmmTcpGenKeyStore.sh

./secureOvmmTcp.sh

Restart your ovmm service

/sbin/service ovmm stop

/sbin/service ovmm start

After this you can provide these info and your ovmm username and password in EMCC

tcps://el01-ovmm:54322

https://el01-ovmm:7002/ovm/console/faces/login.jspx

Re-configuring SDP IPs

In case of a situation where you need to re-configure the existing IPs for SDP, then these steps can be done. This includes cleanup of existing setup and re-configuring the sdp.

1. Remove your IB listener, by the grid software
/u01/app/11.2.0.4/grid/bin/srvctl remove listener -l LISTENER_IB

2. Remove your vip resources, better to use force option "-f". This has to be done each node
/u01/app/11.2.0.4/grid/bin/srvctl remove vip -i ed05db01-ibvip -f
..
..
/u01/app/11.2.0.4/grid/bin/srvctl remove vip -i ed05db08-ibvip -f

3. Remove your network, better to use force option "-f"
/u01/app/11.2.0.4/grid/bin/srvctl remove network -k 2 -f

To redo the setup, you can follow this link from "Setup"
http://ajitsolomon.blogspot.com/p/end-to-end-configuration-of-sdp-listener.html

Setting up and destroying passwordless ssh to sun storage

Setting passwordless ssh to sun storage

Setting up of storage passwordless ssh is different from server to server method. Sun storage uses restricted shell, which makes the storage safe and exposes what its needed for us to manage.
Below script can be written to Sun storage or just copy paste the commands.
We have to transfer id_rsa token of the current server to the storage which you need passwordless ssh.
Based on your system, use RSA or DSA keys. Here I'm using RSA keys

If you dont have a key, generate once with
ssh-keygen -q -f ~/.ssh/id_rsa -N ''

#!/bin/bash

export key=`cat ~/.ssh/id_rsa.pub |cut -d " " -f2`
export key=`echo "\""$key"\""`

ssh exasn01<<EOF
script
run('configuration preferences keys')
run('create')
run('set type=RSA')
try {
run('set key=$key')
run('set comment="Exa"')
run('commit')
}
catch(err){
if (err.code == EAK_BADARG){
        printf ('Key already exist\n ');
 }
                }

EOF

End to end configuration of SDP listener

I have been doing few SDP listener configuration in quarter to full rack Exadata. SDP is one of the performance enhancement done by oracle in Exadata. This uses infiniband, so Exalogic racks which are connected in the same infiniband fabric make use of this SDP.
SDP is easy to set and you can do it in few steps. For easy understanding I'm using 2 node Exadata.
tnsnames.ora file is also added in this page for reference. 

Pre-setup

It is better to use dcli to propagate these changes to all the nodes.

Two files have to be edited to enable SDP setup in each node and those nodes have to be rebooted.

create a text file called node with all the node name

ed05db01

ed05db02

Check if /etc/infiniband/openib.conf
SDP_LOAD=yes

Use the below script to do presetup

#!/bin/bash
# Script name :preset.sh

sed -i "/^use both server/c\use both server * :" /etc/ofed/libsdp.conf
sed -i "/^use both client/c\use both client * :" /etc/ofed/libsdp.conf

grep sdp_zcopy_thresh /etc/modprobe.conf

if [ $? = 0 ] ; then

        sed -i "/sdp_zcopy_thresh/c\options ib_sdp sdp_zcopy_thresh=0 recv_poll=0 sdp_apm_enable=0" /etc/modprobe.conf

else

        echo "options ib_sdp sdp_zcopy_thresh=0 recv_poll=0 sdp_apm_enable=0" >> /etc/modprobe.conf

fi

ORA-16649 with Observer

When observer is running it takes full control of your db status. ORA-16649 will be given if you  start primary database after it crashed. This is not an issue when the observer is running.

Check if the observer and fast_start failover is running. If you find it as running, just wait for few minutes based on how long and how active the new primary database.
The database will be put on to standby mode, it'll get reinstated and it'll start to roll forward.

SQL> startup
ORACLE instance started.

Total System Global Area 5016387584 bytes
Fixed Size     2934696 bytes
Variable Size  1107298392 bytes
Database Buffers  3892314112 bytes
Redo Buffers    13840384 bytes
Database mounted.
ORA-16649: possible failover to another database prevents this database from being opened

Scripting oracle wallet creation - mkstore

This post is not about creating wallet manually, which I had seen in google search. One thing I didn't find is how do you script these to avoid human interaction.

In usual method, you can execute as
mkstore -wrl . -create -nologo
This will make you enter and re-enter password.

To avoid this,

echo -e "Newpass\nNewpass" | mkstore -wrl . -create -nologo

Unable to start observer due to DGM-16979 in oracle 12c

dgmgrl will allow you to do many activity thru sys user, but certain activity needs non sys user.

This error is due to non-sys user. Authentication issue is due to the account sysdg status. This account should be open in both primary and standby
Database comes with sysdg account with locked and expired, unlock and set the password.
Now the observer should start.

DGMGRL> start observer
[P003 01/14 15:53:45.14] Authentication failed.
DGM-16979: Unable to log on to the primary or standby database as SYSDBA
Failed.

DGM-16979 in oracle 11g

DGM-16979 is an common error and sometimes is misleading. One of the issue I faced in oracle 11g is, observer starts if you do manually even after login thru wallet.

dgmgrl /@wallet_sys
DGMGRL for Linux: Version 11.2.0.4.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL>

When you try to start observer, it'll spit out errors like

Could not verify magic words.
DGM-16979: Unable to log on to the primary or standby database as SYSDBA

Even if you are using wallet successfully for other activities, you might still have to add default user and password to your existing wallet.

mkstore -wrl . -createEntry oracle.security.client.default_username sys
mkstore -wrl . -createEntry oracle.security.client.default_password PassWord

Now your observer should start.