Merge pull request #50 from alpaca4j/feature/vaultIntegration+customV…

…ault

Adding custom safe definition

README.md

@@ -4,25 +4,25 @@
 
 # Rokku STS
 
-STS stands for Short Token Service. The Rokku STS performs operations that are specific to managing service tokens. 
+STS stands for Short Token Service. The Rokku STS performs operations that are specific to managing service tokens.
 For a higher level view of purpose of the Rokku STS service, please view the [Rokku](https://github.com/ing-bank/rokku) project.
 
 The Rokku STS simulates the following STS actions:
  * [GetSessionToken](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html)
  * [AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)
- 
+
 This is the internal endpoint that is exposed:
 
 
  * **Checks if a user credentials are active**
- 
+
         /isCredentialActive?accessKey=userAccessKey&sessionToken=userSessionToken
-   
+
    Response status:
-   
+
    * _FORBIDDEN_
    * _OK_
-      
+
        * With the following body response (for status OK) :
    ```json
      {
@@ -33,8 +33,8 @@ This is the internal endpoint that is exposed:
      "userRole": "userRole"
      }
    ```
- 
-   
+
+
 ## Quickstart
 #### What Do You Need
 
@@ -45,13 +45,13 @@ To get a quickstart on running the Rokku STS, you'll need the following:
 1. Launch the Docker images which contain the dependencies for Rokku STS:
 
         docker-compose up
-        
+
 2. When the docker services are up and running, you can start the Rokku STS:
 
         sbt run
-     
+
 3. Have fun requesting tokens
- 
+
 ## Architecture
 
 [MVP1](docs/mvp1-flow.md)
@@ -62,11 +62,11 @@ The STS service is dependant on two services:
 * [Keycloak](https://www.keycloak.org/) for MFA authentication of users.
 * A persistence store to maintain the user and session tokens issued, in the current infrastructure that is [MariaDB](https://mariadb.org).
 
-For the persistence, Rokku STS does not autogenerate the tables required. So if you launch your own MariaDB database, 
-you will need to create the tables as well. You can find the script to create the database, and the related tables 
+For the persistence, Rokku STS does not autogenerate the tables required. So if you launch your own MariaDB database,
+you will need to create the tables as well. You can find the script to create the database, and the related tables
 [here](https://github.com/ing-bank/rokku-dev-mariadb/blob/master/database/rokkudb.sql).
 
- 
+
 ## Test (mock version)
 
 `docker run -p 12345:12345 wbaa/rokku-sts:latest`
@@ -147,21 +147,24 @@ aws sts get-session-token  --endpoint-url http://localhost:12345 --region localh
 aws sts assume-role --role-arn arn:aws:iam::account-id:role/admin --role-session-name testrole --endpoint-url http://localhost:12345 --token-code validToken
 ```
 
-### NPA S3 users 
+### NPA S3 users
 
 STS allows NPA (non personal account) access, in cases where client is not able to authenticate
-with Keycloak server. 
+with Keycloak server.
 In order to notify STS that user is NPA user, below steps needs to be done:
 
 1. User needs to be in administrator groups (user groups are taken from Keycloak)
 
-2. Check settings of the value `STS_ADMIN_GROUPS` in application.conf and set groups accordingly. Config accepts 
+2. Check settings of the value `STS_ADMIN_GROUPS` in application.conf and set groups accordingly. Config accepts
 coma separated string: "testgroup, othergroup"
 
-3. Use postman or other tool of choice to send x-www-form-urlencoded values:
+3. A safe needs to exists with the correct name in vault, otherwise secrets will not be written to vault (404 in logs is an indication of that)
+
+4. Use postman or other tool of choice to send x-www-form-urlencoded values:
 
 ```
 npaAccount = value
+safeName = vaule
 awsAccessKey = value
 awsSecretKey = value
 ```
@@ -170,15 +173,15 @@ as POST:
 
 ```
 curl -X POST \
-     -d "npaAccount=${NPA_ACCOUNT}&awsAccessKey=${NPA_ACCESS_KEY}&awsSecretKey=${NPA_SECRET_KEY}" \
+     -d "npaAccount=${NPA_ACCOUNT}&safeName=${SAFE_NAME}&awsAccessKey=${NPA_ACCESS_KEY}&awsSecretKey=${NPA_SECRET_KEY}" \
      -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" \
      http://127.0.0.1:12345/admin/npa
 ```
 
 NPA user access key and account names must be unique, otherwise adding NPA will fail.
 
 User must also:
-- be allowed in Ranger Sever policies to access Ceph S3 resources 
+- be allowed in Ranger Sever policies to access Ceph S3 resources
 
 When accessing Rokku with aws cli or sdk, just export `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
 with NO `AWS_SESSION_TOKEN`
@@ -190,7 +193,7 @@ STS user account details are taken from Keycloak, but additionally one can mark
 by running:
 ```
 Enable:
-curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X PUT http://localhost:12345/admin/account/{USER_NAME}/enable 
+curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X PUT http://localhost:12345/admin/account/{USER_NAME}/enable
 
 Disable:
 curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X PUT http://localhost:12345/admin/account/{USER_NAME}/disable
@@ -204,4 +207,4 @@ If you plan to run rokku-sts in non-dev mode, make sure you at least set ENV val
 
 ```
 STS_MASTER_KEY = "radomKeyString"
-``` 
+```

docker-compose.yml

@@ -18,6 +18,8 @@ services:
 
   vault:
     image: vault:1.4.2
+    environment:
+      - VAULT_DEV_ROOT_TOKEN_ID=admin
     cap_add:
       - IPC_LOCK
     ports:

src/main/resources/reference.conf

@@ -47,8 +47,8 @@ db-dispatcher {
 
 vault {
     url = "http://127.0.0.1:8200"
-    path = "secret/npa"
-    token = "s.APXT5ze6knsgMGqYfb9WkHQj"
+    path = "secret"
+    token = "admin"
     retries = 3
     read-timeout= 10
     open-timeout = 5

src/main/scala/com/ing/wbaa/rokku/sts/api/AdminApi.scala

@@ -35,7 +35,7 @@ trait AdminApi extends LazyLogging with Encryption with JwtToken {
 
   protected[this] def insertAwsCredentials(username: UserName, awsCredential: AwsCredential, isNpa: Boolean): Future[Boolean]
 
-  protected[this] def insertNpaCredentialsToVault(username: UserName, awsCredential: AwsCredential): Future[Boolean]
+  protected[this] def insertNpaCredentialsToVault(username: UserName, safeName: String, awsCredential: AwsCredential): Future[Boolean]
 
   protected[this] def setAccountStatus(username: UserName, enabled: Boolean): Future[Boolean]
 
@@ -50,13 +50,13 @@ trait AdminApi extends LazyLogging with Encryption with JwtToken {
   def addNPA: Route = logRequestResult("debug") {
     post {
       path("npa") {
-        formFields((Symbol("npaAccount"), Symbol("awsAccessKey"), Symbol("awsSecretKey"))) { (npaAccount, awsAccessKey, awsSecretKey) =>
+        formFields((Symbol("npaAccount"), Symbol("safeName"), Symbol("awsAccessKey"), Symbol("awsSecretKey"))) { (npaAccount, safeName, awsAccessKey, awsSecretKey) =>
           authorizeToken(verifyAuthenticationToken) { keycloakUserInfo =>
             if (userInAdminGroups(keycloakUserInfo.userGroups)) {
               val awsCredentials = AwsCredential(AwsAccessKey(awsAccessKey), AwsSecretKey(awsSecretKey))
               onComplete(insertAwsCredentials(UserName(npaAccount), awsCredentials, isNpa = true)) {
                 case Success(true) =>
-                  insertNpaCredentialsToVault(UserName(npaAccount), awsCredentials)
+                  insertNpaCredentialsToVault(UserName(npaAccount), safeName, awsCredentials)
                   logger.info(s"NPA: $npaAccount successfully created by ${keycloakUserInfo.userName}")
                   complete(ResponseMessage("NPA Created", s"NPA: $npaAccount successfully created by ${keycloakUserInfo.userName}", "NPA add"))
                 case Success(false) =>
@@ -78,13 +78,13 @@ trait AdminApi extends LazyLogging with Encryption with JwtToken {
   def addServiceNPA: Route = logRequestResult("debug") {
     post {
       path("service" / "npa") {
-        formFields((Symbol("npaAccount"), Symbol("awsAccessKey"), Symbol("awsSecretKey"))) { (npaAccount, awsAccessKey, awsSecretKey) =>
+        formFields((Symbol("npaAccount"), Symbol("safeName"), Symbol("awsAccessKey"), Symbol("awsSecretKey"))) { (npaAccount, safeName, awsAccessKey, awsSecretKey) =>
           headerValueByName("Authorization") { bearerToken =>
             if (verifyInternalToken(bearerToken)) {
               val awsCredentials = AwsCredential(AwsAccessKey(awsAccessKey), AwsSecretKey(awsSecretKey))
               onComplete(insertAwsCredentials(UserName(npaAccount), awsCredentials, isNpa = true)) {
                 case Success(true) =>
-                  insertNpaCredentialsToVault(UserName(npaAccount), awsCredentials)
+                  insertNpaCredentialsToVault(UserName(npaAccount), safeName, awsCredentials)
                   logger.info(s"NPA: $npaAccount successfully created")
                   complete(ResponseMessage("NPA Created", s"NPA: $npaAccount successfully created", "NPA add"))
                 case Success(false) =>

src/main/scala/com/ing/wbaa/rokku/sts/vault/VaultService.scala

@@ -4,15 +4,15 @@ import java.nio.charset.StandardCharsets
 
 import akka.actor.ActorSystem
 import com.bettercloud.vault.response.VaultResponse
-import com.bettercloud.vault.{Vault, VaultConfig}
+import com.bettercloud.vault.{ Vault, VaultConfig }
 import com.ing.wbaa.rokku.sts.config.VaultSettings
 import com.ing.wbaa.rokku.sts.data.UserName
 import com.ing.wbaa.rokku.sts.data.aws.AwsCredential
 import com.typesafe.scalalogging.LazyLogging
 
-import scala.concurrent.{ExecutionContext, Future}
+import scala.concurrent.{ ExecutionContext, Future }
 import scala.jdk.CollectionConverters._
-import scala.util.{Failure, Success, Try}
+import scala.util.{ Failure, Success, Try }
 
 trait VaultService extends LazyLogging {
 
@@ -33,19 +33,29 @@ trait VaultService extends LazyLogging {
     vault
   }
 
-  def insertNpaCredentialsToVault(username: UserName, awsCredential: AwsCredential): Future[Boolean] = Future {
-    val secretsToSave: Map[String, AnyRef] = Map("accessKey" -> awsCredential.accessKey.value, "secretKey" -> awsCredential.secretKey.value)
+  def insertNpaCredentialsToVault(username: UserName, safeName: String, awsCredential: AwsCredential): Future[Boolean] = Future {
 
+    if (safeName.equalsIgnoreCase(vaultSettings.vaultPath)) {
+      writeSingleVaultEntry(username, safeName, awsCredential)
+    } else {
+      writeSingleVaultEntry(username, vaultSettings.vaultPath, awsCredential)
+      writeSingleVaultEntry(username, safeName, awsCredential)
+    }
+
+  }(executionContext)
+
+  private def writeSingleVaultEntry(username: UserName, safeName: String, awsCredential: AwsCredential) = {
+    val secretsToSave: Map[String, AnyRef] = Map("accessKey" -> awsCredential.accessKey.value, "secretKey" -> awsCredential.secretKey.value)
     logger.info(s"Performing vault write operation to ${vaultSettings.vaultPath} for ${username.value}")
     Try {
       vault.withRetries(vaultSettings.retries, 500)
         .logical()
-        .write(vaultSettings.vaultPath + "/" + username.value, secretsToSave.asJava)
+        .write(safeName + "/" + username.value, secretsToSave.asJava)
     } match {
       case Success(writeOperation) => reportOnOperationOutcome(writeOperation, username)
       case Failure(e: Throwable)   => reportOnOperationOutcome(e, username)
     }
-  }(executionContext)
+  }
 
   private def reportOnOperationOutcome(s: VaultResponse, name: UserName): Boolean = {
     val status = s.getRestResponse.getStatus

src/test/scala/com/ing/wbaa/rokku/sts/api/AdminApiTest.scala

@@ -2,8 +2,8 @@ package com.ing.wbaa.rokku.sts.api
 
 import akka.actor.ActorSystem
 import akka.http.scaladsl.model.headers.RawHeader
-import akka.http.scaladsl.model.{FormData, StatusCodes}
-import akka.http.scaladsl.server.{AuthorizationFailedRejection, MissingFormFieldRejection, MissingHeaderRejection, Route}
+import akka.http.scaladsl.model.{ FormData, StatusCodes }
+import akka.http.scaladsl.server.{ AuthorizationFailedRejection, MissingFormFieldRejection, MissingHeaderRejection, Route }
 import akka.http.scaladsl.testkit.ScalatestRouteTest
 import com.auth0.jwt.JWT
 import com.auth0.jwt.algorithms.Algorithm
@@ -41,7 +41,7 @@ class AdminApiTest extends AnyWordSpec
     override protected[this] def setAccountStatus(username: UserName, enabled: Boolean): Future[Boolean] = Future.successful(true)
     override protected[this] def getAllNPAAccounts: Future[NPAAccountList] = Future(NPAAccountList(List(NPAAccount("testNPA", true))))
 
-    override protected[this] def insertNpaCredentialsToVault(username: UserName, awsCredential: AwsCredential): Future[Boolean] = Future(true)
+    override protected[this] def insertNpaCredentialsToVault(username: UserName, safeName: String, awsCredential: AwsCredential): Future[Boolean] = Future(true)
   }
 
   private[this] val testRoute: Route = new testAdminApi().adminRoutes
@@ -60,7 +60,7 @@ class AdminApiTest extends AnyWordSpec
   "Admin Api" should {
     "check response" that {
       "return OK if user is in admin groups and all FormFields are posted" in {
-        Post("/admin/npa", FormData("npaAccount" -> "testNPA", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> validOAuth2TokenHeader ~> testRoute ~> check {
+        Post("/admin/npa", FormData("npaAccount" -> "testNPA", "safeName" -> "vault", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> validOAuth2TokenHeader ~> testRoute ~> check {
           assert(status == StatusCodes.OK)
         }
       }
@@ -70,7 +70,7 @@ class AdminApiTest extends AnyWordSpec
         }
       }
       "return Rejected if user is not in admin groups" in {
-        Post("/admin/npa", FormData("npaAccount" -> "testNPA", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> notAdminOAuth2TokenHeader ~> testRoute ~> check {
+        Post("/admin/npa", FormData("npaAccount" -> "testNPA", "safeName" -> "vault", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> notAdminOAuth2TokenHeader ~> testRoute ~> check {
           assert(rejections.contains(AuthorizationFailedRejection))
         }
       }
@@ -80,29 +80,29 @@ class AdminApiTest extends AnyWordSpec
         }
       }
       "return Rejected if user presents no token" in {
-        Post("/admin/npa", FormData("npaAccount" -> "testNPA", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> testRoute ~> check {
+        Post("/admin/npa", FormData("npaAccount" -> "testNPA", "safeName" -> "vault", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> testRoute ~> check {
           assert(rejections.contains(AuthorizationFailedRejection))
         }
       }
       "return Rejected if service token is missing" in {
-        Post("/admin/service/npa", FormData("npaAccount" -> "testNPA", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> testRoute ~> check {
+        Post("/admin/service/npa", FormData("npaAccount" -> "testNPA", "safeName" -> "vault", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey")) ~> testRoute ~> check {
           assert(rejections.contains(MissingHeaderRejection("Authorization")))
         }
       }
       "return OK if service token is correct" in {
-        Post("/admin/service/npa", FormData("npaAccount" -> "testNPA", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey"))
+        Post("/admin/service/npa", FormData("npaAccount" -> "testNPA", "safeName" -> "vault", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey"))
           .addHeader(RawHeader("Authorization", bearerToken("rokku"))) ~> testRoute ~> check {
             assert(status == StatusCodes.OK)
           }
       }
       "return Rejected if service token is not correct" in {
-        Post("/admin/service/npa", FormData("npaAccount" -> "testNPA1", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey"))
+        Post("/admin/service/npa", FormData("npaAccount" -> "testNPA1", "safeName" -> "vault", "awsAccessKey" -> "SomeAccessKey", "awsSecretKey" -> "SomeSecretKey"))
           .addHeader(RawHeader("Authorization", bearerToken("rokku1"))) ~> testRoute ~> check {
             assert(status == StatusCodes.InternalServerError)
           }
       }
       "return Rejected if user FormData is invalid" in {
-        Post("/admin/npa", FormData("npaAccount" -> "testNPA", "awsAccessKey" -> "SomeAccessKey")) ~> validOAuth2TokenHeader ~> testRoute ~> check {
+        Post("/admin/npa", FormData("npaAccount" -> "testNPA",  "safeName" -> "vault", "awsAccessKey" -> "SomeAccessKey")) ~> validOAuth2TokenHeader ~> testRoute ~> check {
           assert(rejections.contains(MissingFormFieldRejection("awsSecretKey")))
         }
       }