Skip to content

Learn. Share. Repeat.

iam

Cross-account AWS Glue Data Catalog access with Glue ETL

Posted on May 16, 2020May 19, 2020 by Anand

To process data in AWS Glue ETL, DataFrame or DynamicFrame is required. A DataFrame is similar to a table and supports functional-style (map/reduce/filter/etc.) along with SQL operations. The AWS Glue DynamicFrame is similar to DataFrame, except that each record is self-describing, so no schema is required initially. It computes a schema on-the-fly when required, and … Continue reading Cross-account AWS Glue Data Catalog access with Glue ETL

Posted in AWS, Big Data, Data Engineering, Glue Catalog, iamTagged AWS, Big Data, cross-account, Data Engineering, glue, Glue Catalog, glue etl, iam7 Comments

Scan Dynamodb table from AWS Glue in different account

Posted on March 28, 2020May 19, 2020 by Anand

In this blog post I will list down the steps required to setup the AWS Glue job to scan the dynamodb table in another account. In my setup, I scan the dynamodb table in Account A (us-west-2), perform glue transformations in Account B (us-east-1) and write it to S3 in Account B. Account A - … Continue reading Scan Dynamodb table from AWS Glue in different account

Posted in AWS, boto3, Data Engineering, Dynamodb, iamTagged AWS, aws glue, cross account, Data Engineering, Dynamodb, iam, sts1 Comment
DISCLAIMER: The views expressed are mine and may not necessarily reflect that of my employers.The views,opinions and comments expressed by visitors are theirs alone and may not reflect my opinion.
May 2022
S M T W T F S
1234567
891011121314
15161718192021
22232425262728
293031  
« Feb    

Archives

My Tweets

  • @WANatlParkFund @MtRainierWatch Colleague and friend @iamJohnmgray is climbing! Wishing him and all others best of luck. 2 days ago
  • And Liverpool wins! Fun final #FACupFinal 2 days ago
  • Chelsea vs. Liverpool - start of extra time #FACupFinal 2 days ago
  • @driscollis A) ([1, 2], ['three', 'four', 'five']) 2 days ago
  • RT @DalianaLiu: How to do outlier detection in 2022: Outlier detection is a common use case in tech. Here are 3 techniques you should know… 5 days ago

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 1,037 other followers

Recent Posts

  • Use Amazon MSK Connect with Lenses plugin to sink data from Amazon MSK to Amazon S3
  • Guide on PySpark DataFrame Functionality
  • Amazon Athena – CTAS, UNLOAD, Parameterized Prepared statements, Partition Projection
  • Apache Kafka – Introduction & Basic Commands
  • How to retrieve credentials stored in AWS Secrets Manager from AWS Lambda running in VPC

Blog Stats

  • 1,872,227 hits
Website Powered by WordPress.com.
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
  • Follow Following
    • Learn. Share. Repeat.
    • Join 1,037 other followers
    • Already have a WordPress.com account? Log in now.
    • Learn. Share. Repeat.
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...