AWS Cognito Error on Sign Up

I was exploring AWS Cognito for authentication. It works great, but I got the following error message after I tested the sign up process: An error was encountered with the requested page. I found out later that I misunderstood the AutoVerifiedAttributes field in my CloudFormation. I thought it would mark an email or phone number as verified without actually verifying them. Apparently, it means it will try to verify either email or phone number. So, when I set it to email, it sent a verification email and the sign up process went without error.

ASP.NET Application Crashed without Error Message

I encountered a strange error with ASP.NET Web API application. It runs fine locally, but when we deployed to Kubernetes cluster, it crashed as soon as it starts. And no error message was thrown.

So, I pulled the application to my local and it crashed as well no matter how I run it, dotnet cli, Docker Desktop, Visual Studio debug. The only one that runs fine is the version from the repo. At this point, there are only two possibilities, either the environment is the issue or the application is the issue, so I decided to deploy it to a different environment and it's still not working, so it must be something with the application.

Since it is the application, I tried to change the log level to Trace to get more information but no new error message that provides a hint on what's going on. Memory dump didn't work as the collector didn't have enough time to collect before the application crashed.

At the end, I decided to approach this the hard way. So, in my local, there are two versions. One is freshly pulled from the environment which is not working. The other one is from the source control repository which works. The most obvious difference between the two is the configuration, in this case, we use appsettings.json. But it looks fine.

My next guess is one of the dlls is probably the issue, so I started to try to break the one that works by substituting the file with different size one-by-one from the broken application. However, all the suspicious dlls don't seem to cause a problem.

As I run out of dlls, I start substituting the appsettings.json. It has different size, why not. And that's when the working version stops working. To zero in on the cause, I start removing fields in the appsettings.json file to see if I can make it to work. Finally, there's one field that's the cause. It looks similar to the following:

{
  "Settings": {
     "Key": "Value"
  }
}

Looking into the code that read that field, it is immediately obvious. In ASP.NET, there's a way to deserialize the field to an object and we can use enum for the value. Let say:

{
  "Settings": {
     "Pet": "Dog"
  }
}

The code to retrieve the setting will be similar to the following:

public enum Pets {Cat, Fish}

public class TheSettings {
  public Pets Pet { get; set; }
}

TheSettings settings = configuration.GetSection("Settings").Get<TheSettings>();

All looks great, except, the enum doesn't have the value specified in the appsettings.json file. In the example above, Dog is not an enum of Pets.  So, when it tries to deserialize the config, it breaks and somehow it didn't throw an error.

Fix either the enum or the value in the field in appsettings.json finally fixed the issue and the application can start without crashing.

JWT is not well formed in ASP.NET Web API JwtBearer .NET 8

 It never caused a problem for me to implement JwtBearer token validator, but this time it is really take my time to troubleshoot what's going on. Long story short, there's a breaking change going to .NET 8 and on top of that, the default package version doesn't solve the issue.

Here's how I implement my service:

services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(JwtBearerDefaults.AuthenticationScheme, options => ...removed for brevity);
services.AddAuthorization();

...

app.UseAuthentication();
app.UseAuthorization();

But checking the bearer token, it was a completely valid token. I retrieved the token using a quick custom middleware.

app.Use(async (context, next) =>
{
    await next.Invoke();
    Debug.WriteLine(context.Request.Headers.Authorization);
});

app.UseAuthentication();

...

Then I validate the token in https://jwt.io.

The error that I received contains:

IDX14100: JWT is not well formed, there are no dots (.). The token needs to be in JWS or JWE Compact Serialization Format.

On top of that, the browser response header has the following header:

WWW-Authenticate: Bearer error="invalid_token"

Searching online, the most helpful hint is probably this thread: https://github.com/dotnet/aspnetcore/issues/52075

There are some suggestions in there, but the one that finally solves my problem is the fact that the following package version doesn't work:

Microsoft.IdentityModel.Protocols.OpenIdConnect 7.1.2

It was a transitive package and I have to upgrade it by installing the latest version, as of this time 8.1.0.

And that fixed my issue without any code change.

Swagger .NET 8 Error

Swashbuckle CLI was able to output schema of my API before, but this time, it throws this error message:

System.InvalidOperationException: A type named 'StartupProduction' or 'Startup' could not be found in assembly

I used top level statement with minimal API on .NET 8 and nothing is changed on that, so I was not able to find anything to do with Startup type.

After I investigate further by commenting line by line, I found out that the issue is on my switch statement.

So it looks like the following:

return config.Section?.Key switch
{
  Value1 => services.AddSingleton<Handler1>(),
  Value2 => services.AddSingleton<Handler2>(),
  _ => throw new InvalidOperationException();
}

Problem is the Section is pulled from appsettings.json and when the CLI runs, it doesn't have value, so it never returned the services object.

Changing the above to the following fixed the issue:

return config.Section == null ? services : config.Section.Key switch
{
  Value1 => services.AddSingleton<Handler1>(),
  Value2 => services.AddSingleton<Handler2>(),
  _ => throw new InvalidOperationException();
}

Background Image on WordPress Editor

I realized that I don't have the Layout option under Styles menu in the Editor. I found out later that I have a bare minimum theme.json. And adding appearanceTools: true field cause it to show up. My theme.json became:

{
	"version": 3,
	"$schema": "https://schemas.wp.org/wp/6.6/theme.json",
	"settings": {
		"appearanceTools": true
	}	
}

WordPress Create Block Theme Plugin

 I'm working on a custom WordPress theme and I saw a very helpful plugin called "Create Block Theme" which is supposed to help developer create the theme.

So, for starting, I tried to edit one of my templates, but when I hit "Save Changes" under "Save Changes to Theme" section, I expected it to overwrite my template html file, but it didn't. I was checking permissions and potential bugs, but seems like everything is good.

After playing around a little, apparently, I need to hit "Save" first, so it records the customization in the database and then click the "Save Changes to Theme" will use the value from the database and modified the html file itself.

Delete Git Branches by Days Ago

Often my branches piled up and I need a way to automatically delete them based on how many days ago. I can't find an easy way online, so I ended up writing my own scripts in PowerShell and Bash. In this repo, I have the script to clean up branches that are 90 days or older based on last committed date.

Repo: https://github.com/nik-yo/DeleteGitBranchesByDaysAgo

Rename PySpark Result File

 Due to the distributed nature of Apache Spark, when writing result, we can't specify name for the result file. This makes the result file hard to predict which I need for my process orchestration. In my case, I need to write the result to S3 and I finally found a way to do this within a reasonable amount of time by utilizing aws wrangler, Panda, and optionally Arrow. I basically feed Spark dataframe to aws wrangler and have it write to S3 using a specific name.

Here's link to my sample: https://github.com/nik-yo/PySparkFilename

Connecting Pod in Minikube to Kafka or any Services Running in Docker Desktop

I'm working on a demo where I need to subscribe my application to Kafka locally in Docker Desktop. I have 3 use cases:

1. Connecting from a different container in Docker desktop, so in the same network as the Kafka container.
2. Connecting from the application running on the host, so outside of Docker desktop for debugging purposes.
3. Connecting from a pod inside Minikube running in Docker Desktop.

Same Docker Network

On the first case, I actually need to connect AKHQ container to Kafka, my Kafka container env variable for advertised listeners looks like the following:

environment:
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092, ... (removed for brevity)

Since AKHQ running in Docker desktop as well, it can use kafka:29092.

From the Host (my computer) Outside of Docker Network

Next is my application that runs outside of Docker desktop, since it won't resolve the kafka host, it has to use the 2nd entry of the advertised listener. In my case, I had to change the port from 9092 to prevent conflict with other Kafka instance, but for it to work, I had to change the port mapping, so the Kafka container configuration looks like the following:

ports:
   - 19092:19092
environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:19092, ...

To prevent conflict, the port mapping on the host and container has to be modified, so my application connects using localhost:19092

From Pod Inside Minikube

This one is confusing me, but I found out that kube-dns is installed by default and Minikube provides a convenient hostname. For more details: https://minikube.sigs.k8s.io/docs/handbook/host-access/

host.minikube.internal

So updating my configuration a little, I can connect from the pod using host.minikube.internal:19094

ports:
    - 19092:19092
    - 19094:19094
environment:
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: ...,PLAINTEXT_POD://host.minikube.internal:19094
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: ...,PLAINTEXT_POD:PLAINTEXT

Multiple BackgroundService or IHostedServices but Only One Works

 In my worker app, I attempted to add multiple hosted services as follow:

builder.Services
    .addSingleton(HostedService1)
	.addSingleton(HostedService2)
	.addSingleton(HostedService3);

All the hosted services are added, but when the application run, only 1 is executing.

Thanks to Stephen Cleary, apparently issue with synchronous call. https://blog.stephencleary.com/2020/05/backgroundservice-gotcha-startup.html.

I ended up using Task.Run for code that executes for a long time. Inside the ExecuteAsync:

await Task.Run(async () => await LongRunningProcess());

Use Multiple Git Accounts on One Computer

I was looking for a way to use two git accounts in a single machine. Apparently, there are multiple ways to do that:

• Use different protocols for different accounts. One account can use HTTP, another account uses SSH.
• Use different SSH keys. One per account.
• Use HTTP and PAT. This might be GitHub specific but per my experience with PAT in Azure DevOps, this is not feasible as PAT has expiration and needs to be renewed. https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-personal-account-on-github/managing-your-personal-account/managing-multiple-accounts

I ended up using different protocols since that will save me effort in configuring one of the accounts.

First, I need to create an ssh key. https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#generating-a-new-ssh-key.

Since I'm in Windows, I can use Git Bash.

1. Launch Git Bash
2. Run: ssh-keygen -t ed25519 -C "your_email@example.com"
3. If prompted to enter passphrase, make sure you note it somewhere (I saved mine in password manager).
4. If you need to change the passphrase, follow the steps in this link: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/working-with-ssh-key-passphrases#adding-or-changing-a-passphrase
5. Also note the location of the generated key. In my case, it is ~/.ssh/id_ed25519 or %USERPROFILE%/.ssh/id_ed25519

Once the key is generated, we need to add it to ssh_agent. This way, local git will know to use the key. https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent

1. Launch PowerShell in elevated admin mode.
2. Make sure the ssh-agent is running:
• Get-Service -Name ssh-agent | Set-Service -StartupType Manual
• Start-Service ssh-agent
3. Launch a separate PowerShell terminal without admin mode.
4. Run the following command: ssh-add c:/Users/{your_user}/.ssh/id_ed25519

After that, we need to add the key to GitHub: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account#adding-a-new-ssh-key-to-your-account.

1. In GitHub, under your profile menu on the top right of the page, select Settings.
2. Then on the left menu, select SSH and GPG keys.
3. Add a new key and use a descriptive name it under Title box. In my case, I use the name of my machine since the key resides in my machine.
4. Go to the key location. In my case: %USERPROFILE%/.ssh
5. Copy the content of the {key}.pub (note the .pub extension for public key, don't copy the one without it as it is the private key). In my case, it is id_ed25519.pub
6. Paste the content of the key to GitHub and click Add SSH key.

Then I added ssh config as follow:

1. In the key location, create file with name config. In my case, the file path will be: %USERPROFILE%/.ssh/config
2. For the content, it will be:
   

   Host github.com
     HostName github.com
     User git
     IdentityFile ~/.ssh/id_ed25519

3. If there are multiple ssh keys for multiple accounts, it will contain multiple entries where the Host can be different while the HostName can stay the same.

Afterwards, I had to configure global .gitconfig in %USERPROFILE% by adding the includeIf section. This is so that I can use different user name and email for different accounts. The content looks like the following:

[user]
   name = {username}
   email = {email}

[includeIf "gitdir:~/{other_folder}/"]
    path = ~/{other_folder}/.gitconfig

As you noticed, the includeIf will need a separate directory with its own .gitconfig file. It works for me as I have a separate directory for repositories that use a separate git account. The content of the other .gitconfig file will be:

[user]
   name = {other_username}
   email = {other_email}

By now, it is pretty much done. If I do a git clone on ssh path, it will prompt for my passphrase and it will work.

git clone git@github.com:{account}/{repo}.git

But in my case, I have existing repositories that needs to be updated to use ssh, so for each repo, I had to run:

git remote set-url origin git@github.com:{account}/{repo}.git

Thus, that's the end of my multi accounts journey.