Friday, July 7, 2017

AWSPowerShell Module - Unable to load type from assembly


We are recently starting to get involved with AWS at the company where I work, so of course I started looking at the AWSPowerShell module, and following the documentation it lead me to install the module, and then run the Set-AWSCredential cmdlet to save my credentials into a profile.

When I entered the cmdlet I received the following error:

Get-AWSCredential : Could not load type Amazon.Runtime.CredentialManagement.CredentialProfile" from assembly "AWSSDK.Core, Version=, Culture=neutral, PublicKeyToken=885c28607f98e604".
At line:1 char:1
+ Get-AWSCredential
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Get-AWSCredential], TypeLoadException
    + FullyQualifiedErrorId : System.TypeLoadException,Amazon.PowerShell.Common.GetCredentialCmdlet

While trying different things out I received this error on a number of other cmdlets as well. (For example, Get-AWSCredential, Get-EC2Instance, etc) It seemed that pretty much anything that required credentials would give me that error.

I started googling the error, but there was nothing, not one post that I found that might lead me to anything. That was a very scary moment, so I started troubleshooting :
  1. I redid my steps on a different machine to see if I had the same issue.
  2. I loaded up an AWS Instance and ran the cmdlet and it worked! What the hell!? 
    1. I thought maybe it is my server image? Maybe some configuration? So I took that module and loaded it on my server and that worked. What? Well I looked further into that module and found out that the AMI had a really outdated powershell module, to be exact and the most recent update is; Now I couldn't stay with that old version! I'd be missing out on a lot of cool new stuff, so I took that one out and continued with my quest.
  3. I was running out of options so I jumped on a chat with AWS to see if they could help; and he recommended trying the last version, as the newest version had just been released today, so maybe there was a bug.
    1. I removed the SDK for .NET and all of the Windows tools and related files, then rebooted and tried installing directly from psGallery without installing the SDK. 
    2. Then again with the SDK and not the Install-Package from psGallery. exit
    3. I went back 3 versions, and nothing. 
Nothing was working. There had to be something else wrong. It couldn't be that two of my machines were messing up at the same time, right?

At this point I was getting pretty desperate, so I went onto PowerShell and loaded up the assembly and tried to load the type manually, to my surprise it also failed. I tried looking for the type, maybe there was some typo? Nothing. The Namespace Amazon.Runtime.CredentialManager could not be found. 

I asked the AWS rep about it, I was still on the chat with him, and he pointed me to the source and the namespace and the class were indeed public, so there is no reason why i shouldn't be able to see it, but they weren't there! What is going on here? A few more hours went by trying different things on different machines we ended the chat with escalating to the dev team to try to get someone with even more knowledge to see if they could help, but I was determined. I just felt like I was close to a breakthrough. 

I started looking at common development issues for when a type couldn't be loaded from a dll, and that was a bust too.

I still had my console open with the assembly loaded, below is the code I used to load the assembly, and that's when i noticed something

[System.Reflection.Assembly]::LoadFile('C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSSDK.Core.dll'); 

In the results for the Fullname property of the assembly, the Location didn't match the path that I was giving when loading the assembly.

Now I probably saw this results about a million times, and never thought anything of it. Then it I wondered Why? Why is the Location, not the same? So I followed the path to find a different AWSSDK.Core.dll file with an old version. 

Here are both dll files details side by side:

So now I'm sitting here wondering what the heck is this file doing there, and why is the cmdlet trying to read that file instead of the one that I'm passing it? I bust out google and i search for the path where the dll was location: C:\Windows\Microsoft.NET\assembly\GAC_MSIL

This lead me to find what the GAC was, Global Assembly Cache, and how it is .NET uses it. Well now what to do? How do I update this file? Well it turns out you can't, you can uninstall the application and reinstall it, something I had already done multiple times, or you can manually remove the files so I renamed the folder.

I then restarted PowerShell, Imported the AWSPowerShell module, tried the cmdlet and boom! Great Success.

Now the kicker is that I don't know how those old dll files were cached. We work mostly out of Terminal Servers, so i'm not the only one that uses these machines, so maybe someone installed the AWS SDK back a while ago, but as you can see the last modified date on that file is 6/6/2017, and the kicker is that I checked if the AWS SDK/Module was installed before attempting to install it and it wasn't there, so who knows exactly what happened. I'm going to drop a bug report in the source code but not very hopeful that anything will come from it.

Anyways, I wanted to share my fun adventure on my first day playing with AWS and PowerShell. Lets hope not every day is like today, or maybe that they are i don't know I had fun.

Kind regards,

Update: Since I wrote this article i have found in a few occasions old assemblies on the GAC. I'm not sure what is causing this though as the assemblies that we are running with the powershell tools are up to date. This has been particularly notorious with the AWSSDK.S3.dll assembly.