Share via

Error when uploading CSV files to Azure blob storage using ManagedIdentity with C# and GraphQLAPI. "Total size of resourceFiles is using 147% of available space after processing."

Siddhesh Chavan 0 Reputation points
2024-05-15T08:40:14.42+00:00

Hello All,

I have created a console app using C# where I try to fetch the data from GraphQLAPI and store the data in csv files. post that it would be uploaded to the Azure blob storage using ManagedIdentity.

This complete code work fine in my local VS till creation of CSV file ManagedIdentity doesn't work as the IP for my local is different not the AZURE IP. That is expected.

When I try to call this code using Azure batch in ADF I get this error
User's image

When I debug my solution in VS it generates 71 files and I think that is the issue why it's not executing. If I try to delete some files The code would start to execute but would fail due to missing .dll or .xml files. So those 71 files are necessary, those files are type of .exe, .pdb, .xml or.dll files.

Libraries I am using are:-

using System;
using System.IO;
using System.Threading.Tasks;
using AEAP.HR.HoxhuntEmail.Helper;
using GraphQL;
using GraphQL.Client.Http;
using GraphQL.Client.Serializer.Newtonsoft;
using System.Collections.Generic;
using CsvHelper.Configuration;
using Azure.Identity;
using Azure.Storage.Blobs;
using System.Reflection;
using Newtonsoft.Json;

Please let us know if you have any alternative for this.

ASP.NET Core
ASP.NET Core
A set of technologies in the .NET Framework for building web applications and XML web services.
4,274 questions
ASP.NET
ASP.NET
A set of technologies in the .NET Framework for building web applications and XML web services.
3,355 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,850 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,452 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 38,301 Reputation points Microsoft Employee
    2024-05-15T15:56:22.2233333+00:00

    Hi Siddhesh Chavan,

    Thank you for posting query in Microsoft Q&A Platform.
    The error message you are seeing indicates that the total size of the resource files you are using is exceeding the available space after processing. This could be due to the large number of files that are being generated by your application, as you mentioned.

    Any specific reason, why we are using Azure Batch service here?

    I would suggest to use Azure Functions or Azure App Service to host your application instead of Azure Batch. These services provide a more lightweight and scalable way to run your application in the cloud, and can help you avoid issues with resource constraints.

    Hope this helps. Please try same and let me know how it goes.