This post has only a very limited audience, looking at a worldwide scale, since it talks about the quirks of integrating with an API of our local Flemish government. Still, my colleague and I struggled too much to not write about it, as limited as the audience may be.
Background knowledge
Let's start with a small overview of the parts involved:
- MAGDA is the data sharing platform of the Flemish government (part of Belgium), and it consists of SOAP-based service and most recently also some REST-based services (since March 2020 or so).
- All these MAGDA services are protected by a central authentication service, which they call GeoSecure. GeoSecure offers two protection methods: one based on the WS-Federation standard, and the other one being OAuth.
- Both the interactions with GeoSecure and the MAGDA services rely heavily on the use of certificates. These certificates can be requested on the VO-DCB platform (Vlaamse Overheid - Digitaal Certifcatenbeheer).
For a customer of ours, we were asked to integrate with one specific MAGDA service that was using token-based authentication. These token-based authenticated services are brand new, and the information on them very sparse. The documentation that existed was only showing Java code samples (we use C#), and they also require the use of HTTP signing. HTTP Signing is currently a draft IETF standard. I'm no fan of using draft standards, since you can never be sure what version of the draft is followed on both the client side (with the C# library we used) and the server side (which MAGDA implemented).
Let's walk through the process!
The process
Connection request for a MAGDA service
First things first, when one of the departments of the Flemish government wants to make use of the MAGDA services they need to send a connection request.
Upon approval of this request, they are provided with the following information:
- a client id to be provided when fetching an access token from GeoSecure
- a URI identifier for which the services are configured, this identifier needs to be in the client certificate used for authentication and signing
Certificate
Like mentioned, having a correct certificate is crucial for success. Getting a key is a process in itself, but can be split in three main parts:
- Create a private key and a Certificate Signing Request (CSR)
- Sign the CSR and receiving the certificate
- Transform the certificate in the necessary formats
1. Create a private key and a CSR
These steps are done through OpenSSL. Easiest way to use OpenSSL on Windows, is to use the one included with your GIT installation. This is most probably located at C:\Program Files\Git\usr\bin\openssl.exe
Generate a private key by executing:
./openssl.exe genrsa -out c:\temp\private_key.pem 4096Use that newly generated private key to generate your CSR:
./openssl.exe req -new -sha256 -key c:\temp\private_key.pem -out c:\temp\testapp.csrIt will prompt for some information, for which the department can provide the values. Most important field is when it asks for the Common Name, you need to fill in the URI identifier received upon approval of the connection request.
2. Sign the CSR and receive the certificate
The CSR from previous step now needs to be added in the VO-DCB admin panel. This process is normally driven by someone in the department of the Flemish Government, not by the integrator (us).
They upload the CSR, fill in some information and select the certificate type which in this case needs to be "SSL Signing Client". They will get a .der certificate file.
3. Transform the certificate in the necessary formats
Using certificates with a private key in Windows is mostly done through the .pfx format, which contains both in one file. Since VO-DCB delivers just a .der file and we already have the private key, we now need to merge them into one .pfx.
This is a two step process: first create a .crt and then a .pfx. Both are done with OpenSSL:
./openssl.exe x509 -in "c:\temp\certificate.der" -inform DER -out "c:\temp\certificate.crt"./openssl.exe pkcs12 -inkey "c:\temp\private_key.pem" -in "c:\temp\certificate.crt" -export -out "C:\temp\certificate.pfx"The last command will ask for a password to protect the private key.
GeoSecure JSON Web Key
GeoSecure is the token endpoint that will provide us our access tokens to call the MAGDA API's. We'll be requesting a token using the Client Credentials Grant using a JSON Web Key (they support client id/client secret only for older implementations).
As part of the connection request, the department received a client id and access to an admin panel to manage this client. In this admin panel, you have connect your certificate or JSON Web Key to the client so GeoSecure can validate your request.
Their documentation provides a .NET Core utility to generate a JSON Web Key (JWK) from a certificate, that you can go and upload in the admin panel.
The generate JWK looks like this:
{
"kid": "",
"kty": "",
"n": "",
"e": "",
"use": "",
"alg": "",
"x5t": "",
"x5c": ""
}If you put this in the admin panel, you get an error message: Fout bij lezen van JSON Web Key. Much, much later (when it was too late already) I learned that you only need to put in the value of the x5c parameter.
If that is the case, their UI doesn't make sense and could be made a lot more clear. The whole JSON body is the JWK, not just the x5c parameter inside it, so I don't blame us for not figuring this out on our own...
Although the documentation makes no mention of this, we tried another option we saw. In this interface we provide the certificate (without private key, thus the .cert or .der file) and let the admin panel generate the JWK for us.
This finally concludes the preparation steps, and we can go write some code!
The code
The complete example class to achieve success can be found on a Gist. I'll walk through it part by part in the next sections.
GeoSecure Access Token
First step is to fetch an access token that we can use to call the MAGDA Token-Based API Services. This process of fetching an access token is clearly described in the documentation, we only needed to transform it into C# code.
In essence, you need to do an application/x-www-form-urlencoded post to an endpoint. This needs to contain a set of parameters with some predefined values, and a client assertion parameter. The code we used is the following:
private async Task<MagdaAccessToken> GetAccessToken()
{
var request = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = new Uri(this._authUrl)
};
request.Headers.Add("Accept", "application/json");
request.Headers.Add("Accept-Encoding", "gzip, deflate, br");
List<string> scopes = new List<string>() {
"msg_mailbox_v1_P",
"msg_msg_v1_P",
"msg_statuses_v1_P"
};
Dictionary<string, string> postParams = new Dictionary<string, string>();
postParams.Add("grant_type", "client_credentials");
postParams.Add("scope", string.Join(" ", scopes));
postParams.Add("client_assertion_type", "urn:ietf:params:oauth:client-assertion-type:jwt-bearer");
postParams.Add("client_assertion", this.GetAuthJwtToken(string.Join(" ", scopes)));
request.Content = new FormUrlEncodedContent(postParams);
var response = await this._httpClient.SendAsync(request);
var responseContent = await response.Content.ReadAsStringAsync();
switch (response.StatusCode)
{
case HttpStatusCode.OK:
return JsonConvert.DeserializeObject<MagdaAccessToken>(responseContent);
default:
throw new Exception($"HTTP call GetToken failed with status code {response.StatusCode} and message {responseContent}");
}
} private async Task<MagdaAccessToken> GetAccessToken()
{
var request = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = new Uri(this._authUrl)
};
request.Headers.Add("Accept", "application/json");
request.Headers.Add("Accept-Encoding", "gzip, deflate, br");
List<string> scopes = new List<string>() {
"msg_mailbox_v1_P",
"msg_msg_v1_P",
"msg_statuses_v1_P"
};
Dictionary<string, string> postParams = new Dictionary<string, string>();
postParams.Add("grant_type", "client_credentials");
postParams.Add("scope", string.Join(" ", scopes));
postParams.Add("client_assertion_type", "urn:ietf:params:oauth:client-assertion-type:jwt-bearer");
postParams.Add("client_assertion", this.GetAuthJwtToken(string.Join(" ", scopes)));
request.Content = new FormUrlEncodedContent(postParams);
var response = await this._httpClient.SendAsync(request);
var responseContent = await response.Content.ReadAsStringAsync();
switch (response.StatusCode)
{
case HttpStatusCode.OK:
return JsonConvert.DeserializeObject<MagdaAccessToken>(responseContent);
default:
throw new Exception($"HTTP call GetToken failed with status code {response.StatusCode} and message {responseContent}");
}
}The client_assertion parameter is a JWT token signed with the private key from the certificate we generated above. We ended up with the following code:
private string GetAuthJwtToken(string scopes)
{
X509SecurityKey privateKey = new X509SecurityKey(this._clientCertificate);
ClaimsIdentity claimsIdentity = new ClaimsIdentity();
claimsIdentity.AddClaim(new Claim("sub", this._clientId));
claimsIdentity.AddClaim(new Claim("scope", scopes));
claimsIdentity.AddClaim(new Claim(JwtRegisteredClaimNames.Jti, Guid.NewGuid().ToString()));
return this._tokenHandler.CreateEncodedJwt(this._clientId, this._authUrl, claimsIdentity, null, DateTime.UtcNow.AddMinutes(10), DateTime.UtcNow, new SigningCredentials(privateKey, SecurityAlgorithms.RsaSha256Signature));
}MAGDA Token-Based API Services
With the access token from the previous step, we can go call the actual API to do our work. Additional hurdle here is the usage of HTTP Signing, a draft specification.
First method makes the actual call to the endpoint. Nothing spectacular here: fetch an access token, call a method to sign the HttpRequestMessage and post the actual call:
public async Task<HttpResponseMessage> SendMessage()
{
if (this._accessToken == null || (this._accessToken != null && this._accessToken.ExpireDate <= DateTime.Now.AddMinutes(1)))
this._accessToken = await this.GetAccessToken();
var request = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = new Uri(this._messagesEndpointUrl)
};
await SignHttpRequestMessage(request);
// Add additional payload properties here
// Like MessageToSend
return await this._httpClient.SendAsync(request);
}Then there is the code that does the actual signing of the message:
private async Task SignHttpRequestMessage(HttpRequestMessage request)
{
request.Headers.Add("signature-public-key", WrapCertificateInJWK(this._clientCertificate, _keyId));
request.Headers.Date = DateTimeOffset.UtcNow;
SigningSettings settings = new SigningSettings
{
SignatureAlgorithm = SignatureAlgorithm.CreateForSigning(this._clientCertificate, HashAlgorithmName.SHA512),
DigestHashAlgorithm = HashAlgorithmName.SHA512,
EnableNonce = false,
Headers = new[]
{
new HeaderName("signature-public-key")
},
UseDeprecatedAlgorithmParameter = true
};
var requestSigner = this._requestSignerFactory.Create(_keyId, settings);
await requestSigner.Sign(request);
request.Headers.Add("Signature", request.Headers.Authorization.Parameter);
request.Headers.Add("x-correlation-id", Guid.NewGuid().ToString());
request.Headers.Add("Accept", "application/json");
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", this._accessToken.AccessToken);
}We use the signing library provided by David Lievrouw (v5.0.0) to do the actual signing. Not sure if he doesn't follow the standard or MAGDA doesn't (since I didn't read it), but some tweaks were required to make it work.
The library signs the request and then puts the signing information in the Authorization header with a value of Signature. Our API expects to receive an Authorization header of type Bearer with the access token from the previous step, and expects the signing information in a header parameter called Signature. So we needed to copy some stuff around.
The default settings of the library don't match the requirements of the API anymore, so we needed to tweak them too. It needed to use our certificate to sign the message, make sure it uses all the required headers for signing and used a deprecated algorithm parameter because that's what the API expected.
Last thing we needed to generate was our certificate wrapped in a JWK, and put it in the signature-public-key header. In essence, this isn't too complex since C# has a JsonWebKey class for which you just need to provide the correct parameters.
Too bad the MAGDA API is a bit picky about additional properties when serializing the JsonWebKey, so we needed to create our own anonymous object with just the needed properties and serialize that one instead. In the end, this is how our code ended up:
private string WrapCertificateInJWK(X509Certificate2 certificate, string keyName)
{
var x509SigningKey = new X509SigningCredentials(certificate);
var x509Key = new X509SecurityKey(certificate);
var cert64 = Convert.ToBase64String(x509Key.Certificate.RawData);
var pubKey = x509Key.PublicKey as RSA;
var parameters = pubKey.ExportParameters(false);
var exponent = Convert.ToBase64String(parameters.Exponent);
var modulus = Convert.ToBase64String(parameters.Modulus);
var webKey = new JsonWebKey
{
Kty = "RSA",
Use = "sig",
Kid = keyName,
X5t = x509Key.Certificate.Thumbprint,
E = exponent,
N = modulus,
Alg = x509SigningKey.Algorithm
};
webKey.X5c.Add(cert64);
var simplifiedWebKey = new { kty = webKey.Kty, kid = webKey.Kid, n = webKey.N, e = webKey.E, x5c = webKey.X5c };
return JsonConvert.SerializeObject(simplifiedWebKey);
}All's Well That Ends Well
It was an enormous adventure getting this to work, given all the moving pieces and less then ideal documentation. In the end we figured it out, not in the least because we also got some engineers on the MAGDA side involved. They were responsible for the Java code, but could point us in the good direction looking at server-side logs and providing insights so we could get the C# code working.
Let's hope this helps improve the documentation so that the suffering is less for the next one that needs to go through this 😊.