How to retain 7 days old files in container?

Question

I want to maintain last 7 days data in container and want to delete remaining old folders.

I have situation like container name is input and maintaining folder structure in format of yyyy/MM/dd. 2024 has months folders of 03 & 04. Month 03 has days folders from 24 to 31. and Month 04 has days folders from 01 to 05.

As per scenario want to retain last 7 days data @adddays-7

means wants to delete days folders from 24 to 28. How can i achieve this?

Currently i used logic in IF condition @less(item().name, substring(addDays(utcNow(), -7, 'yyyy/MM/dd'), 8,2)) and pipeline as Get Metadata | ForEach | IF | Delete|.

Answer 1

Hi Anuganti Suresh

This is a bit complicated logic to implement only with ADF activities. One workaround is to find out all the months in existing folders in the following format (MM-dd):

0324

0325

0326

0327

0328

0329

0330

0331

0401

0402

0403

0404

0405

Insert these values to a Az. SQL table or a temp table. From there, it will be very easy to find out the latest 7 folders by sorting, For instance, select top 7 monthdays from tablename order by 1 desc

Based on the result, you can delete the rest of the folders.

Hope this helps.

Answer 2

Hello @Anuganti Suresh

To achieve your requirement of retaining the last 7 days data and deleting the remaining old folders, you can use the logic that you have mentioned in the IF condition. The logic seems correct and should work fine. Here is a step-by-step guide to implement this logic in Azure Data Factory:

1. Create a pipeline and add a "Get Metadata" activity to it.
2. In the "Get Metadata" activity, set the folder path to the root folder of your container, i.e., "input".
3. Add a "ForEach" activity to the pipeline and connect it to the "Get Metadata" activity.
4. In the "ForEach" activity, set the "Items" property to the output of the "Get Metadata" activity. This will iterate over all the folders in the container.
5. Inside the "ForEach" activity, add an "IF" activity and connect it to the "ForEach" activity.
6. In the "IF" activity, set the condition to @less(item().name, substring(addDays(utcNow(), -7, 'yyyy/MM/dd'), 8,2)). This will check if the folder name is older than 7 days.
7. If the condition is true, add a "Delete" activity inside the "IF" activity and connect it to the "IF" activity.
8. In the "Delete" activity, set the folder path to input/@{item().name}. This will delete the folder that is older than 7 days. This pipeline will iterate over all the folders in the container and delete the folders that are older than 7 days.

I hope this helps! Let me know if you have any further questions.

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.