Transcript
Related Sap Note No. 391 Archive Stuck Situation and Troubleshooting Archive Stuck Situation Symptom: When in SAP R/3, archive stuck situation arise, R3stops responding, no user will be able to login to the system, update process will be hang up,all users who are logged in system will also be hang. Physical Symptom: At os level /oracle/<sid>/oraarch or /oracle/<sid>/saparch directory willbe full of archive log files. There will not be free space available in above volumes. Situation when archive stuck can arise: When there is big client copy is under progress orbig background batch input job is running or any major database update related process isrunning, that time archive log will be generated rapidly. In such situation if archive logbackup is not taken at regular interval, then the archive volume will be filled up and archivestuck situation arise. Action to be taken when archive stuck situation arise For Unix System: 1. Login with ora<sid> user in system2. Find out the location of archive log files. Generally it is /oracle/<SID>/oraarch orsaparch volume. How to find out the location of archive log file You will find <SID>arch1_xxxx.dbf file in archive directory (For Example:SRQarch1_2833.dbf file is reside in /oracle/SRQ/oraarch volume)3. Check for how much free space is available in oraarch directory#bdf | grep oraarch /dev/vg12/oraarch 87785472 87785472 0 100% /oracle/SRQ/oraarchIn above example oraarch directory is 100% utilized. We need to move files from oraarchvolume to other directory.4. Immediately create ticket with concern database backup team to take thearchive log backup of system for until further notice.5. Find out the free volume in system where you can move the archive log file. Thevolume must be from sapdata volume, where you can copy the archive log files. How to find out free volume in system: srqadm@hg20449:/ > bdf | grep sapdataFilesystem kbytes used avail %used Mounted on /dev/vg11/sapdata9 52428800 40963976 11375328 78% /oracle/SRQ/sapdata9 /dev/vg11/sapdata8 52428800 40963976 11375328 78% /oracle/SRQ/sapdata8 /dev/vg11/sapdata7 52428800 40963976 11375328 78% /oracle/SRQ/sapdata7 /dev/vg11/sapdata6 52428800 40963976 11375328 78% /oracle/SRQ/sapdata6 /dev/vg11/sapdata5 57671680 55300216 2353008 96% /oracle/SRQ/sapdata5 /dev/vg11/sapdata4 52428800 40963976 11375328 78% /oracle/SRQ/sapdata4 /dev/vg11/sapdata3 52428800 40963976 11375328 78% /oracle/SRQ/sapdata3 /dev/vg10/sapdat20 52428800 51204024 1215280 98% /oracle/SRQ/sapdata20 /dev/vg11/sapdata2 52428800 40963976 11375328 78% /oracle/SRQ/sapdata2 /dev/vg10/sapdat19 52428800 40963976 11375328 78% /oracle/SRQ/sapdata19 /dev/vg10/sapdat18 52428800 40963976 11375328 78% /oracle/SRQ/sapdata18 /dev/vg10/sapdat17 52428800 40963976 11375328 78% /oracle/SRQ/sapdata17 /dev/vg11/sapdat16 104857600 75781696 28848816 72% /oracle/SRQ/sapdata16 /dev/vg11/sapdat15 104857600 83973664 20720856 80% /oracle/SRQ/sapdata15 /dev/vg11/sapdat14 52428800 40963976 11375328 78% /oracle/SRQ/sapdata14 /dev/vg11/sapdat13 57671680 40964136 16577088 71% /oracle/SRQ/sapdata13 /dev/vg11/sapdat12 52428800 42486840 9864360 81% /oracle/SRQ/sapdata12 /dev/vg11/sapdat11 57671680 56324184 1337040 98% /oracle/SRQ/sapdata11 /dev/vg11/sapdat10 52428800 50180024 2231280 96% /oracle/SRQ/sapdata10 /dev/vg11/sapdata1 52428800 43020008 9335360 82% /oracle/SRQ/sapdata1In above output /oracle/SRQ/sapdata16 is having the maximum free space availableupto 28 Gb, here you can move upto 24 Gb of archive log files from /oracle/SRQ/oraarchvolume.6. Create one directory with name movedarchfiles into /oracle/SRQ/sapdata16 directory#cd /oracle/SRQ/sapdata16#mkdir movedarchfiles7. Copy *.dbf file from /oracle/SRQ/oraarch directory to the /oracle/SRQ/sapdata16/movedarchfiles directory#cd /oracle/SRQ/sapdata16/movedarchfiles #/oracle/SRQ/sapdata16/movedarchfiles > mv /oracle/SRQ/oraarch/*.dbf . Caution: Do not move all *.dbf file together, it will fill up the sapdata volume if the totalsize of all dbf file is greater than available size in sapdata volume, sapdata volume will be100% utilized. Only move 15 to 20 dbf file to the sapdata volume in one single action. Note: Only move the file upto that level by which oraarch directory usage comesbelow threshold volume. 8. After moving archive log file to the above volume, go to /oracle/SRQ/saparch volumeand rename the arch<SID>.log file to arch<SID>.log.old file as per following.Note: arch<SID>.log file can be in saparch or oraarch directory.#cd /oracle/SRQ/saparch# /oracle/SRQ/saparch > mv arch<SID>.log arch<SID>.log.old9. Keep an eye on archive log backup process and be in touch with database backupteam in order to take continuous archive log backup.10. Once archive directory usage come down to 20% to 30%, start moving the archivefiles from the /oracle/SRQ/sapdata16/movedarchfiles to /oracle/SRQ/oraarchdirectory with following command # mv /oracle/SRQ/sapdata16/movedarchfiles/*.dbf /oracle/SRQ/oraarch/. Note: oraarch directory should note be utilized above threshold value after movingthe archive log file, if it is so, then only move some files and then get archive logbackup and then once again move some files. 11. After moving archive log file to the above volume, go to /oracle/SRQ/saparch volumeand rename the arch<SID>.log file to arch<SID>.log.old1 file as per following. Note: This action is required at each and every movement action of archive log files.arch<SID>.log file can be in saparch or oraarch directory. # /oracle/SRQ/saparch > mv arch<SID>.logarch<SID>.log.old1
