Rename PySpark Result File

 Due to the distributed nature of Apache Spark, when writing result, we can't specify name for the result file. This makes the result file hard to predict which I need for my process orchestration. In my case, I need to write the result to S3 and I finally found a way to do this within a reasonable amount of time by utilizing aws wrangler, Panda, and optionally Arrow. I basically feed Spark dataframe to aws wrangler and have it write to S3 using a specific name.

Here's link to my sample: https://github.com/nik-yo/PySparkFilename

Comments

Post a Comment

Popular posts from this blog

Sentinel One Strikes Again. No internet connection. Uninstall Sentinel One Agent.

This happened to a co-worker of mine a while back when his test application file was marked as suspicious by Sentinel One antivirus and had his internet on his laptop disabled. Today, it happened to me without any suspicious file. Probably suspicious activity, who knows. On Microsoft Edge, it says "Hmmm... your Internet access is blocked.", "Firewall or antivirus software may have blocked the connection", and "ERR_NETWORK_ACCESS_DENIED". So, I worked with my IT to uninstall the agent, but uninstalling is not without a fight. Here are the steps that I took: Since it is a Windows 11 machine with Bitlocker, I have to first get the Bitlocker key. From command prompt run: manage-bde -protectors -get C: After I verified it is the same key that the IT has, I saved the key outside of the machine. Then go to system configuration by searching for "sysconfig" or run msconfig. Under "boot" tab, check the "Safe boot" option, then click ...
Read more

A2 Hosting Let's Encrypt Can't Install Certificate on ASP.NET Core Application

I have an ASP.NET Core app hosted by A2 Hosting. A2 Hosting provides free SSL certificate through Let's Encrypt, so I decided to install it for my application. Since the portal is by Plesk, I follow the guide in: https://support.plesk.com/hc/en-us/articles/115000165013 However, it didn't work as expected. My application still has a web.config file. After trying different ways, I commented out aspNetCore module in the web.config and then the certificate was installed successfully. By commenting the module, ASP.NET Core processing by IIS is disabled, so the validation request by Let's Encrypt is treated like a regular request towards a static file.
Read more

NuGet Package Reference NU6105 Publish Error

Some of my .NET Core applications are already using PackageReference which is a very nice idea. However, through a combination of packages, Visual Studio did not allow me to publish my project although it built fine. During publish, it threw error on NU6105 warning. Along with that most of it comes with the following message: Detected package downgrade Some developers solve it by finding which package caused the issue and manually added them through NuGet, but I find them troublesome until I found the following article: https://docs.microsoft.com/en-us/nuget/reference/errors-and-warnings/nu1605 In my case, all I need to do to solve it is to install the following NuGet package:  Microsoft.NETCore.Targets
Read more