Easy way to do AWS Glue Job creation using AWS CLI
In this post, I will demonstrate how to deploy AWS Glue Job using AWS cli on a local windows laptop.
Follow the below steps to deploy AWS glue job on a ‘Windows 10’ environment:
1. Download and Install AWS CLI Setup
2. Configure AWS credentials
3. Create Glue job using AWS cli
Step 1: Download and Install AWS CLI Setup
Download AWS cli version 2 from this link:
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html
Click on https://awscli.amazonaws.com/AWSCLIV2.msi to download
Click on the downloaded file and install the setup.
Once Installed, open command prompt and type command: aws --version
On successful Installation, you can see the aws-cli version details
Step 2: Configure AWS Credentials
In command prompt, type the command: aws configure
Please enter the access key, secret access key, region name(optional), default output format (optional) to configure AWS credentials
Step 3: Create Glue job using AWS cli
Firstly, we need to create json with these following parameters:
code snippet :
{
"Name": "sample_glue_job",
"Role": "arn:aws:iam::xxxxxxxxxxxx:role/AWSGlueServiceRole",
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://test/sample_glue_job.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--TempDir": "s3://test/temp",
"--enable-metrics": "",
"--job-bookmark-option": "job-bookmark-disable",
"--job-language": "python"
},
"MaxRetries": 0,
"Timeout": 2880,
"WorkerType": "Standard",
"NumberOfWorkers": 10,
"GlueVersion": "2.0"
}
The above code snippet is for reference.
You need to create a similar json with required job parameters
Please replace the fields high-lighted(yellow color) with the properties as per the requirements in your AWS Account.
Name this json as sample.json and store it in any directory.
Example: I am storing my sample.json in this location: C:\Users\Harsha\ sample.json
Run the following command in cmd:
aws glue create-job --cli-input-json file://C:\Users\Harsha\sample.json
This command will take the json from the given file location and will create a gluejob with specified properties.
You can login to AWS Glue console and can find the created job “sample_glue_job” in the list of jobs.
