How to retrieve credentials stored in AWS Secrets Manager from AWS Lambda running in VPC

AWS Secrets Manager is a secrets management service that enables you to store credentials and retrieve it dynamically when you need them. It helps protect access to your applications and services. With AWS Secret Manager you can - Programmatically retrieve encrypted secret values at runtimeStore different types of secretsEncrypt secret dataAutomate secret's rotation When you …

Advertisement

Sequential counter with groupby – Pandas DataFrame

Pandas DataFrame is a 2-dimensional tabular data structure with labeled axes. For this blog, we have a table "person" in database containing name, age and city column. As dml transactions are performed on this table, the new image of the record along with the dml operation type is captured and stored in json file. The …

Reading Parquet files with AWS Lambda

I had a use case to read data (few columns) from parquet file stored in S3, and write to DynamoDB table, every time a file was uploaded. Thinking to use AWS Lambda, I was looking at options of how to read parquet files within lambda until I stumbled upon AWS Data Wrangler. From the document …

Merge json files using Pandas

Quick demo for merging multiple json files using Pandas - import pandas as pd import glob import json file_list = glob.glob("*.json") >>> file_list ['b.json', 'c.json', 'a.json'] Use enumerate to assign counter to files. allFilesDict = {v:k for v, k in enumerate(file_list, 1)} >>> allFilesDict {1: 'b.json', 2: 'c.json', 3: 'a.json'} Append the data into list …

Pandas – ValueError: If using all scalar values, you must pass an index

Reading json file using Pandas read_json can fail with "ValueError: If using all scalar values, you must pass an index". Let see with an example - cat a.json { "creator": "CaptainAmerica", "last_modifier": "NickFury", "title": "Captain America: The First Avenger", "view_count": 12000 } >>> import pandas as pd >>> import glob >>> for f in glob.glob('*.json'): …

Python – sort() vs sorted(list)

You can compare list using sort() or sorted(list), but be careful with sort() - >>> c = [('d',4), ('c',3), ('a',1), ('b', 2)] >>> a = [('a',1), ('b', 2), ('c',3), ('d',4)] >>> a.sort() == c.sort() True >>> >>> a = [('a',1), ('b', 2), ('c',3), ('d',4)] >>> b = [('b',2), ('c', 3), ('a',1)] >>> >>> a.sort() == …

Python – str.maketrans()

Working on a Python code, I had a requirement for removing the single/double quotes and open/close brackets from the string of below format -- >>> text = """with summary as (select ' ... 'p.col1,p.col2,p.col3, ROW_NUMBER() ' ... 'OVER(PARTITION BY p.col1,p.col3 ORDER BY ' ... 'p.col2) AS rk from (select * from (select ' ... 'col2, …

S3 bucket size using aws cli

To get the total objects count and total size for folder within S3 bucket recursively - aws s3 ls --summarize --human-readable --recursive s3://content-dev/mywork/dumb | grep 'Total' Total Objects: 1922 Total Size: 297.4 GiB

namedtuple to JSON – Python

In pgdb - PostgreSQL DB API, the cursor which is used to manage the context of a fetch operation returns list of named tuples. These named tuples contain field names same as the column names of the database query. An example of a row from the list of named tuples - Row(log_time=datetime.datetime(2019, 3, 20, 5, …

Python List

This blog post is about appending data elements to list in Python. Suppose we have a simple list "x", we will look at different ways to append elements to this list. x = [1, 2, 3] The "append" method appends only a single element >>> x [1, 2, 3] >>> x.append(4) >>> x [1, 2, …