Add new partitions in AWS Glue Data Catalog from AWS Glue Job

Posted on by Anand

Given that you have a partitioned table in AWS Glue Data Catalog, there are few ways in which you can update the Glue Data Catalog with the newly created partitions. Run MSCK REPAIR TABLE <database>.<table_name> in AWS Athena service.Rerun the AWS Glue crawler . Recently, AWS Glue service team has added a new feature (or … Continue reading Add new partitions in AWS Glue Data Catalog from AWS Glue Job

Advertisement

New Features in Amazon DynamoDB – PartiQL, Export to S3, Integration with Kinesis Data Streams

Posted on by Anand

Every time with AWS re:Invent around, AWS releases many new features over a period of month. In this blog post I will touch on 3 new features which were introduced for Amazon DynamoDB. DynamoDB is a non-relational managed database with single digit millisecond performance at any scale. New Features in Amazon DynamoDB - PartiQL - SQL-compatible … Continue reading New Features in Amazon DynamoDB – PartiQL, Export to S3, Integration with Kinesis Data Streams

Usecase with RDS Snapshot Export to S3

Posted on by Anand

AWS recently announced "Amazon RDS Snapshot Export to S3" feature wherein you can now export Amazon Relational Database Service (Amazon RDS) or Amazon Aurora snapshots to Amazon S3 as Apache Parquet, an efficient open columnar storage format for analytics. I had a use-case to refresh Athena tables daily with full data set in Account B(us-east-1) … Continue reading Usecase with RDS Snapshot Export to S3