Amazon DynamoDB Time to Live (TTL) allows you to define a per-item timestamp to determine when an item is no longer needed.
Shortly after the date and time of the specified timestamp, DynamoDB deletes the item from your table without consuming any write throughput.
TTL is provided at no extra cost as a means to reduce stored data volumes by retaining only the items that remain current for your workload’s needs.

In this video , the concept of TTL is explained with a detail practical demo.

Documentation Link :
[ Ссылка ]

Watch the Part 2 of AWS DynamoDB TTL from this below link:
---------------------------------------------------------------------------------------------------
[ Ссылка ]

Prerequisite:
--------------------------
Load JSON file to Dynamo table using boto3 session
[ Ссылка ]
Change Data Capture for DynamoDB Streams using Amazon Kinesis in-depth intuition
[ Ссылка ]

Code to ingest data in DynamoDB:
------------------------------------------------------------
import boto3
import json
access_key="{}"
secret_access_key="{}"
session=boto3.Session(aws_access_key_id=access_key,aws_secret_access_key=secret_access_key, region_name='us-east-1')
client_dynamo=session.resource('dynamodb')
table=client_dynamo.Table('ttl_data')
records=""
default_ttl = 300
with open('/content/sample_data/anscombe.json','r') as datafile:
records=json.load(datafile)
count=0;
for i in records:
i['rool_no']=count
i['X']=str(i['X'])
i['Y']=str(i['Y'])
i['ttl']= int(time.time()) + default_ttl
print(i)
response=table.put_item(Item=i)
count+=1

Lambda Code used to transform Firehose Data:
---------------------------------------------------------------------------------
import json
import boto3
import base64

output = []

def lambda_handler(event, context):
print(event)
for record in event['records']:
payload = base64.b64decode(record['data']).decode('utf-8')
print('payload:', payload)

row_w_newline = payload + "\n"
print('row_w_newline type:', type(row_w_newline))
row_w_newline = base64.b64encode(row_w_newline.encode('utf-8'))

output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': row_w_newline
}
output.append(output_record)

print('Processed {} records.'.format(len(event['records'])))

return {'records': output}



Check this playlist for more Data Engineering related videos:
[ Ссылка ]


Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
[ Ссылка ]

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL

свернуть