The microservices containerized components consist of the foundation of three ZooKeeper instances, three Solr instances, three Kakfa instances, and the following AcceleratorKMS microservices: SolrService, OutboxMonitor and ServiceMonitor.
Follow the steps presented here to install and deploy microservices, such as SOLR, Zoo Keeper, and Kafka.
Here's a network diagram illustrating where and how microservices fit with the other components required.
Before you begin
AcceleratorKMS microservices and overall networking components allow for various solutions, such as Docker, AWS ECS, and Azure Container. In addition to a container platform, you must be familiar with deploying the following microservices:
Procedure
-
Identify a VPC and two subnets where the microservices will reside.
-
Create a private namespace, so that the components can be registered.
-
Allow the components to communicate with each other.
-
Create a file system for the Solr and Kafka microservices where their content can be stored.
-
Create a security group for these components.
-
Depending on your requirements and configurations, specify each component with the precise access it requires (port 2049 for file system, port 8983 for SOLR and 80/443 for most components) or create a security group that has access to itself on the required ports and assign it to all components to allow communication between components.
Sample deployment of microservices
The following cloud formation templates provide an example deployment of AcceleratorKMS microservices. These templates are intended to serve as an example only. You may need to adjust the templates if they do not meet your security or configuration requirements. You must ensure that you have the Amazon Resource Names (ARNs) for the containers in a registry and the ARNs for task and execution roles. These CloudFormation templates are for an AWS environment, but you can use them as a base to understand what would be required in Azure, GC, or some other platform.
Copies of each sample template are included with a docker image for you to import into your repository for each microservice,
VPC / Core Setup
Identify the VPC, subnets and the security group used by the application server. The template will create a security group to be used by the micro services and will also create a private namespace for these components to register. Instead of using one security group for all components, you can alternatively create separate security groups if you would like to take a more granular approach.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
Vpc:
Description: select the VPC to be used by the acceleratorkms components
Type: 'AWS::EC2::VPC::Id'
Subnet1:
Description: Subnet1 to deploy the acceleratorkms components
Type: 'AWS::EC2::Subnet::Id'
Subnet2:
Description: Subnet2 to deploy the acceleratorkms components
Type: 'AWS::EC2::Subnet::Id'
AppServerSG:
Description: The SecurityGroup ID used by the AKMS app server
Type: String
Resources:
SecurityGroup:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupDescription: Security group used by all components
VpcId: !Ref Vpc
Tags:
- Key: Name
Value: 'AKMS-MicroServices-SG'
SecurityGroupIngress:
Type: 'AWS::EC2::SecurityGroupIngress'
Properties:
Description: allow communication from/to all components
GroupId: !Ref SecurityGroup
IpProtocol: '-1'
SourceSecurityGroupId: !Ref SecurityGroup
AppServerSecurityGroupIngress:
Type: 'AWS::EC2::SecurityGroupIngress'
Properties:
Description: allow communication from app server SG
GroupId: !Ref SecurityGroup
IpProtocol: '-1'
SourceSecurityGroupId: !Ref AppServerSG
PrivateDns:
Type: 'AWS::ServiceDiscovery::PrivateDnsNamespace'
Properties:
Name: !Sub 'AKMS-microservices.local'
Description: service map for microservice containers
Vpc: !Ref Vpc
Outputs:
VpcExport:
Description: acceleratorkms vpc
Value: !Ref Vpc
Export:
Name: !Sub '${AWS::StackName}-VPC'
SubnetsExport:
Description: acceleratorkms vpc subnets
Value: !Join
- ','
- - !Ref Subnet1
- !Ref Subnet2
Export:
Name: !Sub '${AWS::StackName}-Subnets'
SecurityGroupExport:
Description: acceleratorkms security group
Value: !Ref SecurityGroup
Export:
Name: !Sub '${AWS::StackName}-SecurityGroup'
PrivateDnsExport:
Description: acceleratorkms private dns namespace
Value: !Ref PrivateDns
Export:
Name: !Sub '${AWS::StackName}-PrivateDns'
File Systems
The only prompt required for the file system template is to enter the name of the VPC template (so that it can use the security groups, subnets, etc. identified in that template). This template will add rules to the security group to allow communication to the EFS file systems, and it will create 2 file systems and the required mount points for them – 1 file system for Solr and 1 for Kafka. Keep in mind that the rule added to allow access to the EFS is fully open. You can change the cidrIp entry to a specific range (or security group) or ensure that there is some other protection in place that does not allow EFS to be open to the entire network.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
VpcStackName:
Description: vpc stack name
Type: String
Resources:
SecurityGroupTcp2049Ingress:
Type: 'AWS::EC2::SecurityGroupIngress'
Properties:
Description: efs access permission
CidrIp: 0.0.0.0/0
FromPort: 2049
GroupId: !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
IpProtocol: TCP
ToPort: 2049
FileSystem:
Type: 'AWS::EFS::FileSystem'
Properties:
FileSystemTags:
- Key: Name
Value: AKMS-Solr
BackupPolicy:
Status: ENABLED
Encrypted: true
FileSystemPolicy:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'elasticfilesystem:ClientMount'
- 'elasticfilesystem:ClientRootAccess'
- 'elasticfilesystem:ClientWrite'
Principal:
AWS: '*'
LifecyclePolicies:
- TransitionToIA: AFTER_30_DAYS
PerformanceMode: generalPurpose
ThroughputMode: bursting
MountTargetSubnet1:
Type: 'AWS::EFS::MountTarget'
Properties:
FileSystemId: !Ref FileSystem
SubnetId: !Select
- 0
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
MountTargetSubnet2:
Type: 'AWS::EFS::MountTarget'
Properties:
FileSystemId: !Ref FileSystem
SubnetId: !Select
- 1
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
KafkaFileSystem:
Type: 'AWS::EFS::FileSystem'
Properties:
FileSystemTags:
- Key: Name
Value: AKMS-Kafka
BackupPolicy:
Status: ENABLED
Encrypted: true
FileSystemPolicy:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'elasticfilesystem:ClientMount'
- 'elasticfilesystem:ClientRootAccess'
- 'elasticfilesystem:ClientWrite'
Principal:
AWS: '*'
LifecyclePolicies:
- TransitionToIA: AFTER_30_DAYS
PerformanceMode: generalPurpose
ThroughputMode: bursting
KafkaMountTargetSubnet1:
Type: 'AWS::EFS::MountTarget'
Properties:
FileSystemId: !Ref KafkaFileSystem
SubnetId: !Select
- 0
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
KafkaMountTargetSubnet2:
Type: 'AWS::EFS::MountTarget'
Properties:
FileSystemId: !Ref KafkaFileSystem
SubnetId: !Select
- 1
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Outputs:
FileSystemExport:
Description: file system resource
Value: !Ref FileSystem
Export:
Name: !Sub '${AWS::StackName}-FileSystem'
KafkaFileSystemExport:
Description: file system resource
Value: !Ref KafkaFileSystem
Export:
Name: !Sub '${AWS::StackName}-KafkaFileSystem'
Elastic Container Service
This template also only required the name of the first VPC stack to reference its resources. This template will add security group rules to allow traffic in for Solr, it will create a log group for the components, and two load balancers (and their target groups) – one for Solr and one to act as a reverse proxy. As with the EFS template, the rule being added (SecurityGroupTcp8983Ingress) to allow access to the Solr is open to the world. You can modify this section to change the range, as only the app server(s) is required to have access to these load balancers.
The load balancer used for Solr is a network load balancer, with one listener on port 8983. There is one target group behind this load balancer that contains all the Solr instances.
The load balancer used for the Reverse Proxy is an application load balancer. It has one listener on port 80, but has 3 rules associated with that listener, each rule using a separate target group. The rules in the listener are:
If Path Pattern is */Services/Content/* OR */Services/Statements* then forward to the SolrService target group
If Path Pattern is */Services/ConsumeError/* then forward to the OutboxMonitor target group
-
Otherwise, as the final/default rule, forward traffic to the ServiceMonitor target group
The app server will require the FQDNs for each of these load balancers to be entered into the their configuration.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
VpcStackName:
Description: vpc stack name
Type: String
Resources:
SecurityGroupTcp8983Ingress:
Type: 'AWS::EC2::SecurityGroupIngress'
Properties:
Description: loadbalancer access permission
CidrIp: 0.0.0.0/0
FromPort: 8983
GroupId: !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
IpProtocol: TCP
ToPort: 8983
LogGroup:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: /AKMS
Cluster:
Type: 'AWS::ECS::Cluster'
Properties:
ClusterName: AKMS
LoadBalancer:
Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'
Properties:
Type: network
Scheme: internal
Subnets:
- !Select
- 0
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
- !Select
- 1
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
LoadBalancerAttributes:
- Key: load_balancing.cross_zone.enabled
Value: 'true'
Tags:
- Key: Name
Value: AKMS
ReverseProxy:
Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'
Properties:
Type: application
Name: AKMS-ReverseProxy
Scheme: internal
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Subnets:
- !Select
- 0
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
- !Select
- 1
- !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
LoadBalancerAttributes:
- Key: load_balancing.cross_zone.enabled
Value: 'true'
TaskExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- 'sts:AssumeRole'
Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
- 'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'
Tags:
- Key: Name
Value: AKMS-ecs-execution-role
TaskRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- 'sts:AssumeRole'
Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Version: 2012-10-17
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AmazonElasticFileSystemClientFullAccess'
Tags:
- Key: Name
Value: AKMS-ecs-execution-role
LoadBalancerTargetGroup:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
Port: 8983
Protocol: TCP
TargetType: ip
VpcId: !ImportValue
'Fn::Sub': '${VpcStackName}-VPC'
LoadBalancerListener:
Type: 'AWS::ElasticLoadBalancingV2::Listener'
Properties:
DefaultActions:
- ForwardConfig:
TargetGroups:
- TargetGroupArn: !Ref LoadBalancerTargetGroup
Type: forward
LoadBalancerArn: !Ref LoadBalancer
Port: 8983
Protocol: TCP
RevProxyServiceMonTG:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
Port: 8080
Protocol: HTTP
TargetType: ip
HealthCheckPath: /api/HealthCheck
VpcId: !ImportValue
'Fn::Sub': '${VpcStackName}-VPC'
Name: AKMS-RevProxyServiceMonTG
RevProxySolrSvcTG:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
Port: 8080
Protocol: HTTP
TargetType: ip
HealthCheckPath: /api/HealthCheck
VpcId: !ImportValue
'Fn::Sub': '${VpcStackName}-VPC'
Name: AKMS-RevProxySolrSvcTG
RevProxyOutboxMonTG:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
Properties:
Port: 8080
Protocol: HTTP
TargetType: ip
HealthCheckPath: /api/HealthCheck
VpcId: !ImportValue
'Fn::Sub': '${VpcStackName}-VPC'
Name: AKMS-RevProxyOutboxMonTG
RevProxyListener8080:
Type: 'AWS::ElasticLoadBalancingV2::Listener'
Properties:
DefaultActions:
- ForwardConfig:
TargetGroups:
- TargetGroupArn: !Ref RevProxyServiceMonTG
Type: forward
LoadBalancerArn: !Ref ReverseProxy
Port: 8080
Protocol: HTTP
ServiceMonRule80801:
Type: 'AWS::ElasticLoadBalancingV2::ListenerRule'
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref RevProxySolrSvcTG
Conditions:
- Field: path-pattern
PathPatternConfig:
Values:
- '*/Services/Content/*'
- '*/Services/Statements*'
ListenerArn: !Ref RevProxyListener8080
Priority: 1
ServiceMonRule80802:
Type: 'AWS::ElasticLoadBalancingV2::ListenerRule'
Properties:
Actions:
- Type: forward
TargetGroupArn: !Ref RevProxyOutboxMonTG
Conditions:
- Field: path-pattern
PathPatternConfig:
Values:
- '*/Services/ConsumeError/*'
ListenerArn: !Ref RevProxyListener
Priority: 2
Outputs:
ExecutionRoleExport:
Description: acceleratorkms task execution role
Value: !Ref TaskExecutionRole
Export:
Name: !Sub '${AWS::StackName}-ExecutionRole'
TaskRoleExport:
Description: acceleratorkms task role
Value: !Ref TaskRole
Export:
Name: !Sub '${AWS::StackName}-TaskRole'
ClusterExport:
Description: acceleratorkms ecs cluster
Value: !GetAtt Cluster.Arn
Export:
Name: !Sub '${AWS::StackName}-Cluster'
TargetGroupExport:
Description: SOLR load balancer target group
Value: !Ref LoadBalancerTargetGroup
Export:
Name: !Sub '${AWS::StackName}-TargetGroup'
RevProxyServiceMonTGExport:
Description: ServiceMonitor Target Group
Value: !Ref RevProxyServiceMonTG
Export:
Name: !Sub '${AWS::StackName}-ServceMonTG'
RevProxySolrSvcTGExport:
Description: SolrSvc Target Group
Value: !Ref RevProxySolrSvcTG
Export:
Name: !Sub '${AWS::StackName}-RevProxySolrSvcTG'
RevProxyOutboxMonTGExport:
Description: SolrSvc Target Group
Value: !Ref RevProxyOutboxMonTG
Export:
Name: !Sub '${AWS::StackName}-RevProxyOutboxMonTG'
LogGroupExport:
Description: acceleratorkms log group
Value: !Ref LogGroup
Export:
Name: !Sub '${AWS::StackName}-LogGroup'
DnsName:
Description: load balancer dns name
Value: !GetAtt LoadBalancer.DNSName
RevProxyDnsName:
Description: Reverse Proxy load balancer dns name
Value: !GetAtt ReverseProxy.DNSName
Containers - ZooKeeper
This template will prompt for the VPC and ECS stack names, and also prompts for the ZooKeeperID to be used. This template will have to be run three times, once each with an ID of 1, 2 and 3 to create the ZooKeeper ensemble as each node has a separate config. The CPU and Memory settings can be adjusted based on your requirements. This template registers the ZK instances to the private namespace, so that each of the components can reference each other regardless if the container has been replaced and has a new IP.
The separate configurations allow for each instance to refer to themselves as 0.0.0.0, so the list of ZooKeeper servers stored in the environment variable in each config would be:
ZKID:1
server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
ZKID:2
server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181
ZKID:3
server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
AWSTemplateFormatVersion: 2010-09-09
Parameters:
VpcStackName:
Description: vpc stack name
Type: String
EcsStackName:
Description: ecs stack name
Type: String
Default: Custname-ECS
ZookeeperId:
Description: Zookeeper id used by ZOO_MY_ID env variable
Type: Number
Default: 1
AllowedValues:
- 1
- 2
- 3
Conditions:
ZK1: !Equals
- !Ref ZookeeperId
- 1
ZK2: !Equals
- !Ref ZookeeperId
- 2
ZK3: !Equals
- !Ref ZookeeperId
- 3
Resources:
ServiceDiscoveryEntry:
Type: 'AWS::ServiceDiscovery::Service'
Properties:
Name: !Sub 'zoo${ZookeeperId}'
Description: !Sub '"$zoo${ZookeeperId}" service discovery entry in the private dns'
DnsConfig:
DnsRecords:
- TTL: 60
Type: A
RoutingPolicy: MULTIVALUE
HealthCheckCustomConfig:
FailureThreshold: 1
NamespaceId: !ImportValue
'Fn::Sub': '${VpcStackName}-PrivateDns'
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
Family: !Sub 'zoo${ZookeeperId}'
Cpu: '256'
Memory: '2048'
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ExecutionRoleArn: !ImportValue
'Fn::Sub': '${EcsStackName}-ExecutionRole'
ContainerDefinitions:
- Name: !Sub 'zoo${ZookeeperId}-searchdomain'
Image: 'docker/ecs-searchdomain-sidecar:1.0'
Essential: false
Command:
- !Sub '${AWS::Region}.compute.internal'
- !Sub AKMS-microservices.local
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: searchdomain
- DependsOn:
- Condition: SUCCESS
ContainerName: !Sub 'zoo${ZookeeperId}-searchdomain'
Name: !Sub 'zoo${ZookeeperId}'
Image: imagelocation/zookeeperimage@sha256:sha
Essential: true
Environment:
- Name: ZOO_STANDALONE_ENABLED
Value: false
- Name: ZOO_4LW_COMMANDS_WHITELIST
Value: 'mntr, conf, ruok'
- Name: ZOO_CFG_EXTRA
Value: >-
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true
- Name: ZOO_MY_ID
Value: !Ref ZookeeperId
- Name: ZOO_SERVERS
Value: !If
- ZK1
- !Sub >-
server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181
server.3=zoo3:2888:3888;2181
- !If
- ZK2
- !Sub >-
server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181
server.3=zoo3:2888:3888;2181
- !Sub >-
server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181
server.3=0.0.0.0:2888:3888;2181
PortMappings:
- ContainerPort: 2181
HostPort: 2181
- ContainerPort: 2888
HostPort: 2888
- ContainerPort: 3888
HostPort: 3888
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: zookeeper
Service:
Type: 'AWS::ECS::Service'
Properties:
ServiceName: !Sub 'zoo${ZookeeperId}'
Cluster: !ImportValue
'Fn::Sub': '${EcsStackName}-Cluster'
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentController:
Type: ECS
DesiredCount: 1
EnableECSManagedTags: true
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Subnets: !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
PlatformVersion: 1.4.0
PropagateTags: SERVICE
SchedulingStrategy: REPLICA
ServiceRegistries:
- RegistryArn: !GetAtt ServiceDiscoveryEntry.Arn
TaskDefinition: !Ref TaskDefinition
Containers - Solr
This template to deploy Solr will prompt for the names of the VPC, EFS and ECS stacks. Similiarly to the ZooKeeper template, this template has to be run three times, using three different SolrIds (1, 2 and 3). This allows for each instance to register to the private name space as solr1, solr2 or solr3. CPU and memory assignments can be adjusted as required. Instances will register with the private namespace, and will also connect to the Solr EFS. The solr containers are set to listen on port 8983 and are attached to a target group used by their load balancer.
AWSTemplateFormatVersion: 2010-09-09
Parameters:
VpcStackName:
Description: vpc stack name
Type: String
Default: CustName-VPC
EcsStackName:
Description: ecs stack name
Type: String
Default: CustName-ECS
EfsStackName:
Description: efs stack name
Type: String
Default: CustName-EFS
SolrId:
Description: Solr id used by the service
Type: Number
Default: 1
CPUAssign:
Description: Choose number of cores
Type: Number
Default: 2048
AllowedValues:
- 1024
- 2048
MemAssign:
Description: Choose amount of ram
Type: Number
Default: 16384
AllowedValues:
- 8192
- 16384
JavaMemMax:
Description: Java ram assignment
Type: Number
Default: 14
AllowedValues:
- 7
- 14
Resources:
AccessPoint:
Type: 'AWS::EFS::AccessPoint'
Properties:
FileSystemId: !ImportValue
'Fn::Sub': '${EfsStackName}-FileSystem'
PosixUser:
Uid: '8983'
Gid: '8983'
RootDirectory:
CreationInfo:
OwnerGid: '8983'
OwnerUid: '8983'
Permissions: '770'
Path: !Sub '/node${SolrId}'
ServiceDiscoveryEntry:
Type: 'AWS::ServiceDiscovery::Service'
Properties:
Name: !Sub 'solr${SolrId}'
Description: !Sub '"solr${SolrId}" service discovery entry in the private dns'
DnsConfig:
DnsRecords:
- TTL: 60
Type: A
RoutingPolicy: MULTIVALUE
HealthCheckCustomConfig:
FailureThreshold: 1
NamespaceId: !ImportValue
'Fn::Sub': '${VpcStackName}-PrivateDns'
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
Family: !Sub 'solr${SolrId}'
Cpu: !Sub '${CPUAssign}'
Memory: !Sub '${MemAssign}'
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ExecutionRoleArn: !ImportValue
'Fn::Sub': '${EcsStackName}-ExecutionRole'
TaskRoleArn: !ImportValue
'Fn::Sub': '${EcsStackName}-TaskRole'
Volumes:
- Name: solr-data
EFSVolumeConfiguration:
FilesystemId: !ImportValue
'Fn::Sub': '${EfsStackName}-FileSystem'
TransitEncryption: ENABLED
AuthorizationConfig:
AccessPointId: !Ref AccessPoint
IAM: ENABLED
ContainerDefinitions:
- Name: !Sub 'solr${SolrId}-searchdomain'
Image: 'docker/ecs-searchdomain-sidecar:1.0'
Essential: false
Command:
- !Sub '${AWS::Region}.compute.internal'
- !Sub AKMS-microservices.local
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: searchdomain
- DependsOn:
- Condition: SUCCESS
ContainerName: !Sub 'solr${SolrId}-searchdomain'
Name: !Sub 'solr${SolrId}'
Image: imagelocation/solrimage@sha256:sha
Essential: true
Environment:
- Name: SOLR_JAVA_MEM
Value: !Sub '-Xms1g -Xmx${JavaMemMax}g'
- Name: ZK_HOST
Value: !Sub 'zoo1:2181,zoo2:2181,zoo3:2181'
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: solr
PortMappings:
- ContainerPort: 8983
HostPort: 8983
Protocol: tcp
MountPoints:
- ContainerPath: /var/solr
ReadOnly: false
SourceVolume: solr-data
Service:
Type: 'AWS::ECS::Service'
Properties:
ServiceName: !Sub 'solr${SolrId}'
Cluster: !ImportValue
'Fn::Sub': '${EcsStackName}-Cluster'
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentController:
Type: ECS
DesiredCount: 1
EnableECSManagedTags: true
LaunchType: FARGATE
LoadBalancers:
- ContainerName: !Sub 'solr${SolrId}'
ContainerPort: 8983
TargetGroupArn: !ImportValue
'Fn::Sub': '${EcsStackName}-TargetGroup'
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Subnets: !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
PlatformVersion: 1.4.0
PropagateTags: SERVICE
SchedulingStrategy: REPLICA
ServiceRegistries:
- RegistryArn: !GetAtt ServiceDiscoveryEntry.Arn
TaskDefinition: !Ref TaskDefinition
Containers - Kafka
The Kafka template needs to have the names of the VPC, EFS and ECS stacks as well as the KafkaId number (1, 2 or 3). This template needs to be run three times, once for each of the three KafkaIds. The containers will require ports 9091, 9092, and 29092 open. There are several environment variables that are required for a Kafka cluster to be created:
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR : 3
KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR : 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR : 2
KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR : 3
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR : 3
As with the other containers, these containers will register with the private namespace. They will also connect to the Kafka EFS volume.
AWSTemplateFormatVersion: 2010-09-09
Description: The template used to create an ECS Service from the ECS Console.
Parameters:
VpcStackName:
Description: vpc stack Name
Type: String
Default: CustName-VPC
KafkaId:
Description: Kafka ID
Type: Number
EfsStackName:
Description: efs stack Name
Type: String
Default: CustName-EFS
EcsStackName:
Description: ecs stack Name
Type: String
Default: CustName-ECS
Resources:
ServiceDiscoveryEntry:
Type: 'AWS::ServiceDiscovery::Service'
Properties:
Name: !Sub 'kafka${KafkaId}'
Description: !Sub '"kafka${KafkaId}" service discovery entry in the private dns'
DnsConfig:
DnsRecords:
- TTL: 60
Type: A
RoutingPolicy: MULTIVALUE
HealthCheckCustomConfig:
FailureThreshold: 1
NamespaceId: !ImportValue
'Fn::Sub': '${VpcStackName}-PrivateDns'
AccessPoint:
Type: 'AWS::EFS::AccessPoint'
Properties:
FileSystemId: !ImportValue
'Fn::Sub': '${EfsStackName}-KafkaFileSystem'
PosixUser:
Uid: '8983'
Gid: '8983'
RootDirectory:
CreationInfo:
OwnerGid: '8983'
OwnerUid: '8983'
Permissions: '770'
Path: !Sub '/node${KafkaId}'
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Name: !Sub 'kafka${KafkaId}'
Image: imagelocation/kafkaimage@sha256:sha
PortMappings:
- Name: akmskafka-9091-tcp
ContainerPort: 9091
HostPort: 9091
Protocol: tcp
- Name: akmskafka-9092-tcp
ContainerPort: 9092
HostPort: 9092
Protocol: tcp
- Name: akmskafka-29092-tcp
ContainerPort: 29092
HostPort: 29092
Protocol: tcp
Environment:
- Name: KAFKA_BROKER_ID
Value: !Sub '${KafkaId}'
- Name: KAFKA_ZOOKEEPER_CONNECT
Value: >-
zoo1.AKMS-microservices.local:2181,zoo2.AKMS-microservices.local:2181,zoo3.AKMS-microservices.local:2181
- Name: CONFLUENT_METRICS_ENABLE
Value: 'true'
- Name: KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR
Value: '3'
- Name: CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS
Value: 'localhost:29092'
- Name: KAFKA_METRIC_REPORTERS
Value: io.confluent.metrics.reporter.ConfluentMetricsReporter
- Name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
Value: 'false'
- Name: KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR
Value: '3'
- Name: KAFKA_ADVERTISED_LISTENERS
Value: !Sub 'PLAINTEXT://kafka${KafkaId}.AKMS-microservices.local:29092'
- Name: KAFKA_TRANSACTION_STATE_LOG_MIN_ISR
Value: '2'
- Name: KAFKA_JMX_HOSTName
Value: localhost
- Name: KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR
Value: '3'
- Name: CONFLUENT_SUPPORT_CUSTOMER_ID
Value: anonymous
- Name: CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS
Value: '1'
- Name: KAFKA_JMX_PORT
Value: '9101'
- Name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
Value: '3'
- Name: KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL
Value: 'http://schema-registry:8081'
- Name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
Value: 'PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
- Name: KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS
Value: '0'
MountPoints:
- SourceVolume: kafka_data
ContainerPath: /var/lib/kafka/data
ReadOnly: false
- SourceVolume: kafka_secrets
ContainerPath: /etc/kafka/secrets
ReadOnly: false
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: kafka
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Family: !Sub 'kafka${KafkaId}'
TaskRoleArn: 'arn:aws:iam::accountnumber:role/taskrolename
ExecutionRoleArn: >-
arn:aws:iam::accountnumber:role/executionrolename
Volumes:
- Name: kafka_data
EFSVolumeConfiguration:
FilesystemId: !ImportValue
'Fn::Sub': '${EfsStackName}-KafkaFileSystem'
TransitEncryption: ENABLED
AuthorizationConfig:
AccessPointId: !Ref AccessPoint
IAM: ENABLED
- Name: kafka_secrets
EFSVolumeConfiguration:
FilesystemId: !ImportValue
'Fn::Sub': '${EfsStackName}-KafkaFileSystem'
TransitEncryption: ENABLED
AuthorizationConfig:
AccessPointId: !Ref AccessPoint
IAM: ENABLED
Cpu: '1024'
Memory: '8192'
ECSService:
Type: 'AWS::ECS::Service'
Properties:
Cluster: !ImportValue
'Fn::Sub': '${EcsStackName}-Cluster'
TaskDefinition: !Ref TaskDefinition
LaunchType: FARGATE
ServiceName: !Sub 'kafka${KafkaId}'
SchedulingStrategy: REPLICA
DesiredCount: 1
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Subnets: !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
PlatformVersion: LATEST
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentCircuitBreaker:
Enable: true
Rollback: true
DeploymentController:
Type: ECS
ServiceConnectConfiguration:
Enabled: false
Tags:
- Key: 'ecs:service:stackId'
Value: !Ref 'AWS::StackId'
EnableECSManagedTags: true
ServiceRegistries:
- RegistryArn: !GetAtt ServiceDiscoveryEntry.Arn
Outputs:
ECSService:
Description: The created service.
Value: !Ref ECSService
Containers - SolrService
The SolrService template requires the name of the VPC and ECS stacks, a SolrSvcID, and a ConnectionString. The SolrSvcId can be 1,2 or 3, but typically just one instance is required. This container listens on port 80. It registers to a targetgroup for the SolrService that sits behind the ReverseProxy load balancer. Only the app server(s) will access this container, via the load balancer. The ConnectionString is what the container will use to connect to the SQL database. An example of the connection string is:
Data Source=fqdn.of.the.database.server;Initial Catalog=database.name;User ID=username;Password=password;MultipleActiveResultSets=False
Communication to the SolrService container will come only from the app server(s) via port 80, and the SolrService container will communicate with the SQL server over port 1433.
AWSTemplateFormatVersion: 2010-09-09
Description: SolrService 4.0.4.27
Parameters:
VpcStackName:
Description: vpc stack name
Type: String
Default: CustName-VPC
SolrSvcId:
Description: SolrService ID
Type: Number
Default: 1
AllowedValues:
- 1
- 2
- 3
EcsStackName:
Description: ECS stack name
Type: String
Default: CustName-ECS
ConnectionString:
Description: Connection String Used to Access DB
Type: String
NoEcho: true
Resources:
ServiceDiscoveryEntry:
Type: 'AWS::ServiceDiscovery::Service'
Properties:
Name: !Sub 'SolrService${SolrSvcId}'
Description: !Sub '"SolrService${SolrSvcId}" service discovery entry in the private dns'
DnsConfig:
DnsRecords:
- TTL: 60
Type: A
RoutingPolicy: MULTIVALUE
HealthCheckCustomConfig:
FailureThreshold: 1
NamespaceId: !ImportValue
'Fn::Sub': '${VpcStackName}-PrivateDns'
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Name: !Sub 'solrsvc${SolrSvcId}'
Image: imagelocation/solrserviceimage@sha256:sha
PortMappings:
- Name: akmssolrservice-80-tcp
ContainerPort: 80
HostPort: 80
Protocol: tcp
Environment:
- Name: AKMS_CONNECTION_STRING
Value: !Sub '${ConnectionString}'
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: solrsvc
Family: !Sub 'SolrSvc${SolrSvcId}'
TaskRoleArn: 'arn:aws:iam::accountnumber:role/taskrolename
ExecutionRoleArn: >-
arn:aws:iam::accountnumber:role/executionrolename
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: '1024'
Memory: '8192'
ECSService:
Type: 'AWS::ECS::Service'
Properties:
Cluster: !ImportValue
'Fn::Sub': '${EcsStackName}-Cluster'
TaskDefinition: !Ref TaskDefinition
LaunchType: FARGATE
ServiceName: !Sub 'solrsvc${SolrSvcId}'
SchedulingStrategy: REPLICA
DesiredCount: 0
LoadBalancers:
- ContainerName: !Sub 'solrsvc${SolrSvcId}'
ContainerPort: 80
TargetGroupArn: !ImportValue
'Fn::Sub': '${EcsStackName}-RevProxySolrSvcTG'
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Subnets: !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
PlatformVersion: LATEST
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentCircuitBreaker:
Enable: true
Rollback: true
DeploymentController:
Type: ECS
ServiceConnectConfiguration:
Enabled: false
Tags:
- Key: 'ecs:service:stackId'
Value: !Ref 'AWS::StackId'
EnableECSManagedTags: true
ServiceRegistries:
- RegistryArn: !GetAtt ServiceDiscoveryEntry.Arn
Outputs:
ECSService:
Description: The created service.
Value: !Ref ECSService
Containers - OutboxMonitor
The OutboxMonitor template requires the name of the VPC and ECS stacks, an OutboxMonID, and a ConnectionString. The OutboxMonID can be 1,2 or 3, but typically just one instance is required. This container listens on port 80. It registers to a targetgroup for the OutBoxMon that sits behind the ReverseProxy load balancer. Only the app server(s) will access this container, via the load balancer. The ConnectionString is what the container will use to connect to the SQL database. An example of the connection string is:
Data Source=fqdn.of.the.database.server;Initial Catalog=database.name;User ID=username;Password=password;MultipleActiveResultSets=False
Communication to the OutBoxMonitor container will come only from the app server(s) via port 80, and the OutBoxMonitor container will communicate with the SQL server over port 1433.
AWSTemplateFormatVersion: 2010-09-09
Description: OutboxMonitor 4.0.4.27
Parameters:
OutboxMonId:
Description: OutboxMonitor ID
Type: Number
Default: 1
AllowedValues:
- 1
- 2
- 3
EfsStackName:
Description: efs stack Name
Type: String
Default: CustName-EFS
EcsStackName:
Description: ecs stack Name
Type: String
Default: CustName-ECS
VpcStackName:
Description: vpc stack name
Type: String
Default: CustName-VPC
ConnectionString:
Description: Connection String Used to Access DB
Type: String
NoEcho: true
Resources:
ServiceDiscoveryEntry:
Type: 'AWS::ServiceDiscovery::Service'
Properties:
Name: !Sub 'outboxmon${OutboxMonId}'
Description: !Sub '"outboxmon${OutboxMonId}" service discovery entry in the private dns'
DnsConfig:
DnsRecords:
- TTL: 60
Type: A
RoutingPolicy: MULTIVALUE
HealthCheckCustomConfig:
FailureThreshold: 1
NamespaceId: !ImportValue
'Fn::Sub': '${VpcStackName}-PrivateDns'
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Name: !Sub 'outboxmon${OutboxMonId}'
Image: imagelocation/outboxmonitorimage@sha256:sha
PortMappings:
- Name: akmsoutboxmonitor-80-tcp
ContainerPort: 80
HostPort: 80
Protocol: tcp
Essential: true
Environment:
- Name: AKMS_CONNECTION_STRING
Value: !Sub '${ConnectionString}'
LogConfiguration:
logDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: OutboxMon
Family: !Sub 'outboxmon${OutboxMonId}'
TaskRoleArn: 'arn:aws:iam::accountnumber:role/taskrolename
ExecutionRoleArn: >-
arn:aws:iam::accountnumber:role/executionrolename
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: '1024'
Memory: '4096'
ECSService:
Type: 'AWS::ECS::Service'
Properties:
Cluster: !ImportValue
'Fn::Sub': '${EcsStackName}-Cluster'
TaskDefinition: !Ref TaskDefinition
LaunchType: FARGATE
ServiceName: !Sub 'outboxmon${OutboxMonId}'
SchedulingStrategy: REPLICA
DesiredCount: 0
LoadBalancers:
- ContainerName: !Sub 'outboxmon${OutboxMonId}'
ContainerPort: 80
TargetGroupArn: !ImportValue
'Fn::Sub': '${EcsStackName}-RevProxyOutboxMonTG'
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Subnets: !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
PlatformVersion: LATEST
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentCircuitBreaker:
Enable: true
Rollback: true
DeploymentController:
Type: ECS
ServiceConnectConfiguration:
Enabled: false
Tags:
- Key: 'ecs:service:stackId'
Value: !Ref 'AWS::StackId'
EnableECSManagedTags: true
ServiceRegistries:
- RegistryArn: !GetAtt ServiceDiscoveryEntry.Arn
Outputs:
ECSService:
Description: The created service.
Value: !Ref ECSService
Containers - ServiceMonitor
The ServiceMonitor template requires the name of the VPC and ECS stacks, a ServiceMonID, and a ConnectionString. The ServiceMonID can be 1,2 or 3, but typically just one instance is required. This container listens on port 80. It registers to a targetgroup for the ServiceMonitor that sits behind the ReverseProxy load balancer. Only the app server(s) will access this container, via the load balancer. The ConnectionString is what the container will use to connect to the SQL database. An example of the connection string is:
Data Source=fqdn.of.the.database.server;Initial Catalog=database.name;User ID=username;Password=password;MultipleActiveResultSets=False
Communication to the ServiceMonitor container will come only from the app server(s) via port 80, and the ServiceMonitor container will communicate with the SQL server over port 1433.
AWSTemplateFormatVersion: 2010-09-09
Description: ServiceMonitor 4.0.4.27
Parameters:
SrvMonId:
Description: ServiceMonitor ID
Type: Number
Default: 1
AllowedValues:
- 1
- 2
- 3
EfsStackName:
Description: efs stack Name
Type: String
Default: CustName-EFS
EcsStackName:
Description: ecs stack Name
Type: String
Default: CustName-ECS
VpcStackName:
Description: vpc stack name
Type: String
Default: CustName-VPC
ConnectionString:
Description: Connection String Used to Access DB
Type: String
NoEcho: true
Resources:
ServiceDiscoveryEntry:
Type: 'AWS::ServiceDiscovery::Service'
Properties:
Name: !Sub 'servicemon${SrvMonId}'
Description: !Sub '"servicemon${SrvMonId}" service discovery entry in the private dns'
DnsConfig:
DnsRecords:
- TTL: 60
Type: A
RoutingPolicy: MULTIVALUE
HealthCheckCustomConfig:
FailureThreshold: 1
NamespaceId: !ImportValue
'Fn::Sub': '${VpcStackName}-PrivateDns'
TaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
Properties:
ContainerDefinitions:
- Name: !Sub 'servicemon${SrvMonId}'
Image: imagelocation/servicemonitorimage@sha256:sha
PortMappings:
- Name: akmsservicemonitor-80-tcp
ContainerPort: 80
HostPort: 80
Protocol: tcp
Essential: true
Environment:
- Name: AKMS_CONNECTION_STRING
Value: !Sub '${ConnectionString}'
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !ImportValue
'Fn::Sub': '${EcsStackName}-LogGroup'
awslogs-region: !Ref 'AWS::Region'
awslogs-stream-prefix: SrvMon
Family: !Sub 'servicemon${SrvMonId}'
TaskRoleArn: 'arn:aws:iam::accountnumber:role/taskrolename
ExecutionRoleArn: >-
arn:aws:iam::accountnumber:role/executionrolename
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: '1024'
Memory: '4096'
ECSService:
Type: 'AWS::ECS::Service'
Properties:
Cluster: !ImportValue
'Fn::Sub': '${EcsStackName}-Cluster'
TaskDefinition: !Ref TaskDefinition
LaunchType: FARGATE
ServiceName: !Sub 'servicemon${SrvMonId}'
SchedulingStrategy: REPLICA
DesiredCount: 0
LoadBalancers:
- ContainerName: !Sub 'servicemon${SrvMonId}'
ContainerPort: 80
TargetGroupArn: !ImportValue
'Fn::Sub': '${EcsStackName}-ServceMonTG'
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- !ImportValue
'Fn::Sub': '${VpcStackName}-SecurityGroup'
Subnets: !Split
- ','
- !ImportValue
'Fn::Sub': '${VpcStackName}-Subnets'
PlatformVersion: LATEST
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 100
DeploymentCircuitBreaker:
Enable: true
Rollback: true
DeploymentController:
Type: ECS
ServiceConnectConfiguration:
Enabled: false
Tags:
- Key: 'ecs:service:stackId'
Value: !Ref 'AWS::StackId'
EnableECSManagedTags: true
ServiceRegistries:
- RegistryArn: !GetAtt ServiceDiscoveryEntry.Arn
Outputs:
ECSService:
Description: The created service.
Value: !Ref ECSService