In the previous post, we built a JWT authentication server. Now we will build on that server and create an initial instance of the application that I am working on. I have been working on a education engine called Varahamihir. We will talk about this engine at a later date, but for the scope of this post, we will define this engine as below.
The engine consists of an authorization and routing service. We have taken our authorization server from the last post and added the spring cloud gateway routing functionality to that. This will be the only micro-service in our architecture that will be exposed to internet while all other micro-services will be strictly internal services.
The authorization services maintains all the users that exist in the system but these users are provided access to different resources based on their role. When designing micro-services based solutions, people have following two options.
- Authorization server performs necessary authentication and authorization and then for communication between micro-services specific client tokens are used.
- The authorization token received from the user is passed to each of the micro-services. This requires us to make sure the token is passed around for completing the life-cycle of the request.
I always prefer option 2 for better audit and tracing capabilities. Let's look all the things that we need to make this solution work.
Gateway Changes
As we have already mentioned, we are re-purposing our authorization server to also double up as routing gateway.
As we can see in the yaml file, we define two routes, one for our student-service, and another for our guardian-service. Since we are going to use Kubernetes Service Discovery, these names have to match the service names declared in our pod configuration. Please be careful to only used exposed service names.
As we can see in the yaml file, we define two routes, one for our student-service, and another for our guardian-service. Since we are going to use Kubernetes Service Discovery, these names have to match the service names declared in our pod configuration. Please be careful to only used exposed service names.
In the yaml file above, we see two filters defined. These are the hooks to add/remove something to all the requests that are being handled by Spring Cloud Gateway.
For now these filters are just printing the request headers and are not performing anything useful. We also need to make changes to our SpringApplication to enable discovery for the services.
Look at the annotations on the class. The most important annotation is @EnableDiscoveryClient. Since we have to build a number of micro-services and many of the functions are common, we have created couple of modules which we will include in all micro services. The authentication path for identity-server is different than other micro-services because other micro-services will just validate the JWT token received from the gateway.
Spring does not recursively import all the application.properties files from child modules, we have to collect really required properties from any child module and manually import them to the application. That is the reason, we have @PropertySource("classpath:jwt.properties"), in the application because this is the properties file where some required properties are stored in one of the child modules. I would have liked if there was some way to instruct Spring to import all the child module's application.properties files recursively.
Apart from these changes, the gateway application is very similar to what we saw in previous post, we have to just pay attention to list of whitelisted URLs in VarahamihirWebServerSecurityConfig. We have basically whitelisted actuator and user registration URLs.
Micro-services
The gateway service (identity-server) is sitting on the edge and is the only entity exposed to internet. It performs complete authentication and authorization cycle. Once the request is accepted by the gateway and is being passed on the other micro-services, they only verify the token and install a security context so that appropriate role based access control can be applied on the endpoints.
We have following roles defined in our system.
Most of the roles defined in the system are self-explanatory. Some of the roles of interest are as below.
- ADMIN -- This is like the super-user of the system, will have complete access to all the functions of the system
- TENANT_ADMIN -- This is like the above role but will only have access to resources of a tenant. This is the admin of a particular tenant
- REFRESH -- This is an internal role that is assigned to the refresh token issued by the Authorization Server. This role is only allowed to call the refresh endpoint.
- AUTHENTICATE -- This is the role that is assigned when the user requests a new token using the username and password.
There are other specific roles for specific types of users and provide them access to their resources.
Authentication
The authentication flow of the microservice starts with few initial filters. Let's look at the order in the security config code.
The code is self-explanatory, we first whitelist a bunch of URLs and options and then we add three filters. The first is our good old TenantHeaderFilter that extracts and installs tenant into the SecurityContext. This is very similar to what we saw in previous post as well.
The next filter is the AuthorizationHeaderFilter, which extracts the Authorization header from the request and installs it into the SecurityContext.
Now the AuthenticationWebFilter provided by Spring is called which triggers the authentication. This filters requires a authentication manager, for which we use a standard UserDetailsRepositoryReactiveAuthenticationManager. We need to provide a UserDetailService to instantiate the authentication manager.
A standard implementation of UserDetailService requires one to fetch username and password from some persistent store. In our scenario, we already have an authenticated token, so we just provide a dummyPassword that is defined in the above class. All the other parameters to construct a proper UseDetails object can be extracted from the existing token and authentication can proceed.
In this flow, we are only worried about decoding and verifying the JWT token.
The main authentication function is handled by VarahamihirJWTClientAuthWebFilter.
This class uses a class VarahamihirJWTClientAuthConverters to perform the actual authentication. Both these classes are below. The filter function in VarahamihirJWTClientAuthWebFilter decided if the authentication needs to be done and then authenticate function performs actual authentication.
We use a utility class VarahamihirJWTBaseUtil. In the previous post, we had talked about VarahamihirJWTUtil. Since then, the class is refactored into two classes where the base class has all the decoding related functions and the child class has additional functionality. Many of the beans that we need to create a token are not needed when we want to validate and decode it.
We use a utility class VarahamihirJWTBaseUtil. In the previous post, we had talked about VarahamihirJWTUtil. Since then, the class is refactored into two classes where the base class has all the decoding related functions and the child class has additional functionality. Many of the beans that we need to create a token are not needed when we want to validate and decode it.
We use a utility class VarahamihirJWTBaseUtil. In the previous post, we had talked about VarahamihirJWTUtil. Since then, the class is refactored into two classes where the base class has all the decoding related functions and the child class has additional functionality. Many of the beans that we need to create a token are not needed when we want to validate and decode it.
Communication
Since we will be using OpenFeign for communication across micro-services, we define this in this common module as well.
We have three micro-services and we expose the interfaces related to each of them that are required for communication.
This the interface to identity server and provides two services, given a username, it can get you the details of that user and it can create a user. The second is required because when the other micro-services need to create a student or a guardian, they first need to create a user and then create student or guardian. The other micro-services also make sure appropriate roles are applied to users based on their function. The external endpoint doesn't allow one to provide roles.
The student service interface provides two interfaces, one to get the details of a student and another to create a student. This will be called from the guardian endpoint since we envisage that a guardian will create students.
The guardian service just provides a single interface and that allows one to query details of a guardian.
We also need to provide a config for all the Feign clients.
Implementing micro-service
Now that we have the common module completed, we can create our micro-services modules and implement the actual business logic. Here I present one sample of how the micro-service looks like. We take the example of Student micro-service.We first define an endpoint class, in this class our endpoint has two interfaces, a GET and a POST.
Now we define a service layer that will implement the functionality required by the endpoint.
Next we define a repository.
Next we define a mapper class that can convert entities to Pojo. This is not really required but is useful if what is exposed over JSON is remarkably different than what is stored in database.
Finally, our application. Make sure all the annotations are in place.
Deployment
First thing we need to do is the bundle the application in docker images. We need to make sure our pom.xml has following directive.
The build-image goal in spring-boot-maven-plugin makes sure that on mvn clean package -DskipTests command creates a docker image. We have prefixed the image with gcr.io because we want to use google cloud and if we push that image, it will be available in google container registery.
Now we need to generate yaml files for each of our applications. Even though I used dekorate to automatically generate yaml files, I always had to manually modify them. So I finally decided to create my own yaml files.
Since I am using Google Cloud SQL, I am using the sidecar approach to deploy the cloud sql proxy. Since the database requires authentication, I am using the kubernetes secret to define that.
We use Workload Identity method for the application to work in google cloud. Here are the steps that need to be followed for that.
Go to google cloud container page and create a cluster for yourself. You will also need to create a google project before that. Make a note of the cluster name.
Set your google cloud project in your shell and create a service account.
$ gcloud iam service-accounts create my_gsa_name $ kubectl create namespace my_namespace $ kubectl create serviceaccount --namespace my_namespace my_ksa
Now associate policy bindings and annotate the account
$ gcloud config set project my_project $ gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:cluster_project.svc.id.goog[my_namespace/my_ksa]" \ my_gsa_name@gsa_project.iam.gserviceaccount.comYou can leave my_namespace to default if you wish. my_ksa is your kubernetes service account which is tied to your my_gsa_name google account. Since we are going to use Cloud SQL, you need to make sure the my_gsa_name has one of the Cloud roles into it. Otherwise you will get errors related to permissions for using Cloud SQL.
Now we need to complete the binding between KSA and GSA.
$ kubectl annotate serviceaccount \ --namespace my_namespace \ my_ksa \ iam.gke.io/gcp-service-account=my_gsa_name@gsa_project.iam.gserviceaccount.comYou also need to create a secret for database that the yaml file will use to authenticate with cloud sql. This secret is used within the yaml file for service deployment.
$ kubectl create secret generic my-secret \ --from-literal=username=my-db-user \ --from-literal=password=MyDbPassword \ --from-literal=database=my-db-name
Here is the yaml file for one of the micro-services. Similarly one can write for other micro-services as well.
Now that we are ready with all the stuff, we just need to call mvn clean package -DskipTests. This will create all the docker images. Now we need to push these docker images to Google Container Registry. For this, we need to call docker push on each of the images. I use following script to automate the complete deployment operation.
#!/bin/bash
IMAGES=`docker images --format "table {{.Repository}},{{.ID}},{{.Repository}},{{.Tag}},{{.Size}}" | grep 'identity-server\|student\|guardian' |cut -d ',' -f 2`
[ $? -eq 0 ] || exit 1
docker image rm -f $IMAGES
mvn clean package -DskipTests
[ $? -eq 0 ] || exit 1
docker push gcr.io/varahamihir-cloud/student
[ $? -eq 0 ] || exit 1
docker push gcr.io/varahamihir-cloud/guardian
[ $? -eq 0 ] || exit 1
docker push gcr.io/varahamihir-cloud/identity-server
[ $? -eq 0 ] || exit 1
kubectl -n service rollout restart --namespace=varahamihir-k8s-ns deployment student
[ $? -eq 0 ] || exit 1
kubectl -n service rollout restart --namespace=varahamihir-k8s-ns deployment guardian
[ $? -eq 0 ] || exit 1
kubectl -n service rollout restart --namespace=varahamihir-k8s-ns deployment identity-server
[ $? -eq 0 ] || exit 1
Now that the servers are deployed, we can check the condition of all the pods.$ kubectl get pods --namespace=my-namespace NAME READY STATUS RESTARTS AGE guardian-7c75fcb9c8-txg4q 2/2 Running 0 3h6m identity-server-67d7878db8-47lqq 2/2 Running 0 3h6m student-68485fb749-l65gc 2/2 Running 1 3h6mAs we can see, each of the pods are running two workloads because the cloud_sql_proxy is running as a sidecar for each of the pods. The benefit of this is that we can use the cloud sql database as localhost. We can see the logs of any of the processes by following command.
$ kubectl logs --namespace=varahamihir-k8s-ns pods/student-5bf4ccd6cc-5g29q --container student Error from server (NotFound): pods "student-5bf4ccd6cc-5g29q" not found vavasthi@VinayLinux-Desktop:~/work/varahamihir$ kubectl logs --namespace=varahamihir-k8s-ns pods/student-68485fb749-l65gc --container student Container memory limit unset. Configuring JVM for 1G container. Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=216643K -XX:ReservedCodeCacheSize=240M -Xss1M -Xmx524732K (Head Room: 0%, Loaded Class Count: 35835, Thread Count: 50, Total Memory: 1073741824) Adding 127 container CA certificates to JVM truststore . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.1.RELEASE)Now that we are ready, we can see our micro-services in action. First let's create a guardian in the system.
$ curl --request POST --url http://35.225.145.174/default/registration/guardian --header cache-control: no-cache --header content-type: application/json --header x-tenant: default --data {"username" : "testguardian","password" : "testguardian1123","fullname" : "My Guardian","email":"testguardian1@varahamihir.com"}
{
"tenantId": "738f6863-32dc-4a92-9f3a-cad3ef38fe6f",
"user": {
"email": "testguardian1123",
"grantedAuthorities": [
"GUARDIAN",
"USER"
],
"id": "5d6a6e08-0a88-4c5f-9bc1-b220ece79397",
"mask": 0,
"tenantId": "738f6863-32dc-4a92-9f3a-cad3ef38fe6f",
"username": "testguardian"
},
"userId": "5d6a6e08-0a88-4c5f-9bc1-b220ece79397"
}
Now we can authenticate using this guardian.$ curl --request POST \
--url http://35.225.145.174/default/oauth/token \
--header 'authorization: Basic c3VwZXJzZWNyZXRjbGllbnQ6c3VwZXJzZWNyZXRjbGllbnQxMjM=' \
--header 'cache-control: no-cache' \
--header 'content-type: application/x-www-form-urlencoded' \
--header 'postman-token: 26951e8d-bbe2-f617-896d-9a31658fe142' \
--header 'x-tenant: default' \
--data 'username=testguardian&password=testguardian1123&audience=self&grant_type=password'
{
"auth_token": "eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiZGlyIn0..6p7meCoT0JyW9dKjeM_h3w.1bjKuj_5nVS_a7tC8zX6euEXyarrrQNCm62ImgrTxCw68oc7leQsAPh5TYyNpyGSE3pdsKCAie67c8WW13dBpRcw22BbitGTKdabpo8L3z5Y51YB7wWx_lQuSfXkTfd9haa2OmnpUjYyNMdHlu6QqGx4fI7iIOC0a4Ir4SkeF6jLMlpxyqpMkPyeZuQZHZDyjT9Cyo8bTFkaNZ8wPy_PX-nCWZROjQiXUQ33ChQ5Uy3QkBz0CCvAOMUjDyIuIA-yH2O35HkCsX44yZ2w-DDjzNyYe5WJIF1iuTpSQfLtkG0jvqkZUp5HyD6wwB5WcvbDzj-wOV7cuVEvY1BoFfUXEjfeZ-pH45_WWV05PeKyiCA.2xBZDLa9VQqiOUGK_pESuA",
"expiry": 500,
"refreshToken": "eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiZGlyIn0..FttlYRPM4nwViMyZELXnsQ.ZRklt6EClyJCI3lxaIr_W84mtMfdoZUlzWVkM-85tDz8-IwhRjtzieRfErzcuWJB1MB2Y2m4QtVRuLMagilWqc_CbLYb31SOLlMDfHN6TOJck-qpFg_qziaofFOFVY6idlUP9GO2-nuDjROXJLhjFC47mbwOrfqdGoWXAqoaBJzhgKRnKcWJKG_COjN5uXhnirCYM_bc3bI8HJG_1i_Ca_7wgDadSOIyXwS9-DqpqMjMBLZUyIe9S_jqwG7RZox1Ll_MWv7DZvYvhk8ouFZSkB5tL7qI2tNoEPNre_rICfSxNIgJmx_YeI-4IgvQyAbG6SanEDqEfEmRowjbQJ9GCw.-QPY9_YOFllm9yqb-B1WuQ",
"refreshTokenExpiry": 1000,
"scope": ",",
"tokenType": "Bearer"
}
Now using the auth_token, the guardian can add one student to himself.$ curl --request POST \
--url http://35.225.145.174/default/guardian/testguardian/student \
--header 'authorization: Bearer eyJlbmMiOiJBMTI4Q0JDLUhTMjU2IiwiYWxnIjoiZGlyIn0..6p7meCoT0JyW9dKjeM_h3w.1bjKuj_5nVS_a7tC8zX6euEXyarrrQNCm62ImgrTxCw68oc7leQsAPh5TYyNpyGSE3pdsKCAie67c8WW13dBpRcw22BbitGTKdabpo8L3z5Y51YB7wWx_lQuSfXkTfd9haa2OmnpUjYyNMdHlu6QqGx4fI7iIOC0a4Ir4SkeF6jLMlpxyqpMkPyeZuQZHZDyjT9Cyo8bTFkaNZ8wPy_PX-nCWZROjQiXUQ33ChQ5Uy3QkBz0CCvAOMUjDyIuIA-yH2O35HkCsX44yZ2w-DDjzNyYe5WJIF1iuTpSQfLtkG0jvqkZUp5HyD6wwB5WcvbDzj-wOV7cuVEvY1BoFfUXEjfeZ-pH45_WWV05PeKyiCA.2xBZDLa9VQqiOUGK_pESuA' \
--header 'cache-control: no-cache' \
--header 'content-type: application/json' \
--header 'postman-token: 2e9901da-ef01-a3bb-ba28-1f0dff5f7381' \
--header 'x-tenant: default' \
--data '{\n "username" : "teststudent1",\n "password" : "teststudent123",\n "fullname" : "Test Student",\n "email":"teststudent@varahamihir.com",\n "guardianName":"testguardian"\n}'
"userId": "45fe4888-2a64-48bd-bea7-ad512f0ef724",
"tenantId": "738f6863-32dc-4a92-9f3a-cad3ef38fe6f",
"guardianId": "5d6a6e08-0a88-4c5f-9bc1-b220ece79397",
"user": {
"id": "45fe4888-2a64-48bd-bea7-ad512f0ef724",
"tenantId": "738f6863-32dc-4a92-9f3a-cad3ef38fe6f",
"fullname": "Test Student",
"username": "teststudent1",
"email": "teststudent@varahamihir.com",
"mask": 0,
"grantedAuthorities": [
"STUDENT",
"USER"
]
}
}
So we can see the system coordinating across three micro-services. The communication can also be effective since the communication is happening across local ip addresses and doesn't have to go across the external load balancer. Each of the micro-services are also not aware of the ip addresses of each of the pieces and Kubernetes Discovery Service takes care of it.Complete code synchronized with this blog post is available in my repository tagged as v1.1.
