FastAI : Training a Digit Classifier - [Chapter-4] - Part II

Metric is to drive human understanding and the loss is to drive automated learning. Stochastic Gradient Descent - As Arthur Samuel had mentioned the description of machine learning Suppose we arrange for some automatic means of testing the effectiveness of any current weight assignment in terms of actual performance and provide a mechanism for altering …

Advertisement

fastai : Training a Digit Classifier - Baseline Model [Chapter-4] - Part I

Chapter 1 Notes - https://aprakash.wordpress.com/2021/07/05/fastai-just-go-out-and-play-chapter-1/ I want to start this blog with some of my personal notes before jumping on the book. Below is the notebook with examples of 1-dimensional, 2-dimensional and 3-dimensional array using numpy. 2. What is a norm? A norm is a measure of distance and have three properties All distances are …

fastai : Just Go out and Play [Chapter -1]

For quite some time I have been thinking to start writing about data science and machine learning, but somehow it wasn't happening. Recently, I joined 25 weeks long fastbook reading session organized by Weights & Biases and lead by Aman Arora ( awesome instructor 👏). This is my first time participating in a reading session …

Export table from Aurora PostgreSQL to Amazon S3

In this blog post I discuss how to export 100GB non-partitioned table from Aurora PostgreSQL to Amazon S3. I will walk you through two approaches that you can use to export the data. Firstly I will demonstrate using aws_s3, a PostgreSQL extension which Aurora PostgreSQL provides and then using AWS Glue service. The post also …

Build a basic UI for uploading data to s3 using AWS Amplify

Disclaimer: I am not a web developer and I do not have knowledge of JavaScript. The code snippet is mostly taken from different sources and tweaked for my specific use case. 🙂 I was looking for a way to enable a user to upload bunch of small size (< 10 MB) csv files to a …

Handling missing values in Pandas to Spark DataFrame conversion

Data transformation is an important aspect of Data Engineering and can be a challenging task depending on the dataset and the transformation requirements. A bug in data transformation can have a severe impact on the final data set generated leading to data issues. In this blog I am going to share my experience of having …

AWS Glue Job Fails with CSV data source does not support map data type error

AWS Glue is a serverless ETL service to process large amount of datasets from various sources for analytics and data processing. Recently I came across "CSV data source does not support map data type" error for a newly created glue job. In a nutshell, the job was performing below steps: Read the data from S3 …

Connect to AWS Aurora PostgreSQL/Amazon Redshift Database from AWS Lambda

In this blog post I will discuss following scenarios to connect to databases from AWS Lambda function: Connecting to Amazon Aurora PostgreSQL database in private subnet with public accessibility set to No in same AWS account.Connecting to cross account Amazon Redshift database in public subnet with public accessibility set to Yes. Connect to Amazon Aurora …

Add new partitions in AWS Glue Data Catalog from AWS Glue Job

Given that you have a partitioned table in AWS Glue Data Catalog, there are few ways in which you can update the Glue Data Catalog with the newly created partitions. Run MSCK REPAIR TABLE <database>.<table_name> in AWS Athena service.Rerun the AWS Glue crawler . Recently, AWS Glue service team has added a new feature (or …

Use redshift-data api with AWS Glue Python Shell job

AWS Glue is a fully managed extract, transform, and load (ETL) service to process large amount of datasets from various sources for analytics and data processing. When you add a AWS Glue job, you can choose the job to be either Spark or Spark Streaming or Python shell type. For one of my use-case I …