All your script in this lab must use the bash shell in the interpreter line. In
ID: 2246685 • Letter: A
Question
All your script in this lab must use the bash shell in the interpreter line.
In this lab, we’ll work with one of the files created in the previous lab.
First create 2 directories called SFTP and Enrollment.
The SFTP directory will act as the client-facing server; i.e. where all the client upload their enrollment files.
The Enrollment directory will act as the backend server (behind firewall) where the files must be transferred before it can be decrypted and processed.
So for the purpose of this lab, the terms SFTP server and SFTP directory (similarly Enrollment server and Enrollment directory) will be used interchangeably.
Place all the files (arch_files, elig_files, name_files, decrypt_files, conv_files, disney_files, gate_files) in a directory of your choice in the SFTP server.
In this lab, we’ll write a script to process the “Eligibility” files (elig_files).
Task 1: Partition elig_files
Take the elig_files and partition them per clients. The table below shows the client directories associated with the pathnames of 4 eligibility files
File pathname
Corresponding Client directory
/home/oeb-1165-sftp/home/oeb-1165-sftp/inbound/OEB_To_InfoArmor_Eligibility_20170728.csv.pgp
oeb-1165-sftp
/home/spgi-1490-sftp/home/spgi-1490-sftp/inbound/SPGI_To_InfoArmor_Eligibility_20170728.csv.pgp
spgi-1490-sftp
/home/tpadv-sftp/home/tpadv-sftp/inbound/TripAdvisor_To_InfoArmor_Eligibility_20170802.csv.pgp
tpadv-sftp
/home/spgi-1490-sftp/home/ spgi-1490-sftp/inbound/ SPGI_To_InfoArmor_Eligibility_File_20170630.csv.gpg
spgi-1490-sftp
The above example (4 files) will result in 3 partitions, since there are 3 distinct clients.
Please name elig_partition.script
Task 2: File transfer
You are to write a script to transfer the files from each partition created above into the Enrollment server. To do so, follow the below steps:
Each partition is associated with a particular SFTP client directory; such as oeb-1165-sftp. We need to get rid of the “-sftp” to obtain the corresponding Enrollment server directory; such as oeb-1165. The path to each client’s directory in the Enrollment server is “Enrollment/Customers/<client directory>” as illustrated in the table below:
SFTP pathname
Enrollment pathname
/home/oeb-1165-sftp/home/oeb-1165-sftp/inbound/OEB_To_InfoArmor_Eligibility_20170728.csv.pgp
Enrollment/Customers/oeb-1165/ OEB_To_InfoArmor_Eligibility_20170728.csv.pgp
/home/spgi-1490-sftp/home/spgi-1490-sftp/inbound/SPGI_To_InfoArmor_Eligibility_20170728.csv.pgp
Enrollment/Customers/spgi-1490/ SPGI_To_InfoArmor_Eligibility_20170728.csv.pgp
/home/tpadv-sftp/home/tpadv-sftp/inbound/TripAdvisor_To_InfoArmor_Eligibility_20170802.csv.pgp
Enrollment/Customers/tpadv/ TripAdvisor_To_InfoArmor_Eligibility_20170802.csv.pgp
/home/spgi-1490-sftp/home/ spgi-1490-sftp/inbound/ SPGI_To_InfoArmor_Eligibility_File_20170630.csv.gpg
Enrollment/Customers/spgi-1490/ SPGI_To_InfoArmor_Eligibility_File_20170630.csv.gpg
Write a script, called elig_transfer.script, to
Create, for each partition, the corresponding directory tree in the Enrollment server
Create in that directory 2 subfolders called inbound and archive. For the client oeb-1165-sftp, we will have Enrollment/Customers/oeb-1165/inbound and Enrollment/Customers/oeb-1165/archive
Transfer the files for that partition to the client’s inbound directory. For example, the file /home/oeb-1165-sftp/home/oeb-1165-sftp/inbound/OEB_To_InfoArmor_Eligibility_20170728.csv.pgp will be transferred into Enrollment/Customers/oeb-1165/inbound/OEB_To_InfoArmor_Eligibility_20170728.csv.pgp
Task 3: Processing
An eligibility file contain at least one column of data called EID (i.e. Employee ID).
First, you’ll create a few eligibility files and place them in an appropriate inbound directory in the Enrollment server.
From Excel, create a .csv file called SPGI_To_InfoArmor_Eligibility_File_20170630.csv with 1 column of 15 imaginary EID. Be sure to include the header (i.e. EID). Include a blank row
From Excel, create a .csv file called Enrollment/Customers/oeb-1165/ OEB_To_InfoArmor_Eligibility_20170728.csv with 5 columns of 10 imaginary data corresponding to the following header:
EID: Employee IDs
Status: Terminated or Active
Eligibility: INELIG or ELIG
Name: Full name
DOB: date of birth
Now, write a script called elig_proc.script to process the .csv files as follow:
If there is a “Status” field, remove all records with “Terminated” status
If there is a “Eligibility” field, remove all “INELIG” records
Extract only the column of EID data (including header) for eligible employees in active status. There must be no empty record, no empty column, etc.
Save the new file as <initial filename>.pgp
For example, the processed file for SPGI_To_InfoArmor_Eligibility_File_20170630.csv will be called SPGI_To_InfoArmor_Eligibility_File_20170630.csv.pgp
Transfer the processed file into the archive folder
File pathname
Corresponding Client directory
/home/oeb-1165-sftp/home/oeb-1165-sftp/inbound/OEB_To_InfoArmor_Eligibility_20170728.csv.pgp
oeb-1165-sftp
/home/spgi-1490-sftp/home/spgi-1490-sftp/inbound/SPGI_To_InfoArmor_Eligibility_20170728.csv.pgp
spgi-1490-sftp
/home/tpadv-sftp/home/tpadv-sftp/inbound/TripAdvisor_To_InfoArmor_Eligibility_20170802.csv.pgp
tpadv-sftp
/home/spgi-1490-sftp/home/ spgi-1490-sftp/inbound/ SPGI_To_InfoArmor_Eligibility_File_20170630.csv.gpg
spgi-1490-sftp
Explanation / Answer
elig_partition.script : This file needs to be called repeatedly with parameter as complete pathname for all the files which need to be partitioned in the SFTP directory. It will create the specified directory within a "result" directory one level above SFTP and copy the file to the newly created directory for further processing
VAR1=$1
VAR2=$VAR1|cut -d "/" -f 3
if [! -d ../result/$VAR2]; then
mkdir ../result/$VAR2
fi
cp $VAR1 ../result/$VAR2
---------------------------------------------------------------------------------------------------------------------------------------------
elig_transfer.script: This script will reside in the 'result' directory created above and will identify all the directories created under 'result' in step 1 and carry out the file copy operation to Enrollment/Customers
for i in $(ls -d */);
do
TEMP=${i%%/};
DIRECTORY=$TEMP|grep -o '.*-'|rev | cut -c 2- | rev
if [! -d ../Enrollment/Customers/$DIRECTORY]; then
mkdir ../Enrollment/Customers/$DIRECTORY
mkdir ../Enrollment/Customers/$DIRECTORY/incoming
mkdir ../Enrollment/Customers/$DIRECTORY/archive
cp -r ./$TEMP/* ../Enrollment/Customers/$DIRECTORY/incoming
fi
done
----------------------------------------------------------------------------------------------------------------------------------------------------
elig_proc.script: This file uses an awk script called elig_proc.awk
elig_proc.awk:
BEGIN {FS = ","}
{
if($1 !="" && $2!="Terminated" && $3!="INELIG") print $1
}
elig_proc.script: This script takes as argument the path to csv file to be processed
awk -f elig_proc.awk $1> $1.pgp
A=dirname $1
B="/../archive"
cp $1.pgp $A$B