cheatsheet
Comprehensive Deployment Guide with Expected Outcomes
- Introduction
- Prerequisites
- Troubleshooting Guide
Introduction
The Krateo V2 Template Fireworks App provides a complete solution for deploying and managing fireworks applications on Kubernetes using Krateo's Composition system. This guide covers the entire lifecycle from initial deployment to advanced management scenarios.
Prerequisites
Before beginning, ensure you have:
- A Kubernetes cluster (v1.20+ recommended)
- Helm installed (v3.0+)
- kubectl configured to access your cluster
- A GitHub account with repository creation permissions
- A GitHub personal access token with repo scope
Initial Setup
1. Adding Krateo Helm Repository
helm repo add krateo https://charts.krateo.io
What to Expect:
- The command will register Krateo's chart repository with your local Helm installation
- Upon success, you'll see confirmation that "krateo" was added to your repositories
- This enables you to install Krateo charts using the
krateo/
prefix
2. Updating Helm Repositories
helm repo update krateo
What's Happening:
- Helm contacts the repository URL to fetch the latest chart information
- It updates the local cache of available charts and versions
- The success message indicates you now have access to the most recent charts
Core Platform Installation
3. Installing Krateo Platform
helm upgrade installer installer \
--repo https://charts.krateo.io \
--namespace krateo-system \
--create-namespace \
--install \
--version 2.4.2 \
--wait
Expected Behavior:
- Helm will create the krateo-system namespace if it doesn't exist
- The installer chart (version 2.4.2) will be deployed
- The --wait flag ensures command completes only when resources are ready
- Output shows deployment status and namespace information
4. Verifying Platform Readiness
kubectl wait krateoplatformops krateo --for condition=Ready=True --namespace krateo-system --timeout=660s
What This Does:
- Polls the Krateo platform status until "Ready" condition is True
- Times out after 660 seconds if not ready
- Successful output means core components are operational
5. Install required providers:
helm install github-provider krateo/github-provider --namespace krateo-system
helm install git-provider krateo/git-provider --namespace krateo-system
helm install argocd argo/argo-cd --namespace krateo-system --create-namespace --wait
These installations set up the necessary providers for GitHub and ArgoCD, enabling integration with your deployment process of the fireworks application.
CompositionDefinition Deployment
1. Creating the Application Namespace
kubectl create namespace fireworksapp-system
Expected Outcome:
- Creates a dedicated namespace for your Fireworks App resources
- Isolates your application resources from other deployments
- Simple confirmation message shows creation success
2. Deploying the CompositionDefinition
cat <<EOF | kubectl apply -f -
apiVersion: core.krateo.io/v1alpha1
kind: CompositionDefinition
metadata:
name: fireworksapp-cd
namespace: fireworksapp-system
spec:
chart:
repo: fireworks-app
url: https://charts.krateo.io
version: 1.1.13
EOF
What Happens Next:
- Krateo processes the definition and generates a Custom Resource Definition (CRD)
- A dedicated controller pod is deployed to manage compositions
- The system prepares to accept FireworksApp custom resources
3. Verifying CompositionDefinition Status
kubectl wait compositiondefinition fireworksapp-cd --for condition=Ready=True --namespace fireworksapp-system --timeout=600s
System Behavior:
- Command waits until the CompositionDefinition is fully processed
- During this time, Krateo is setting up the necessary controllers
- Success means you can now create FireworksApp instances
Creating Compositions
1. Setting Up GitHub Credentials
kubectl create secret generic github-repo-creds \
--namespace krateo-system \
--from-literal=token=YOUR_GITHUB_TOKEN
Why This Matters:
- Stores your GitHub token securely in the cluster
- Enables the system to interact with your repositories
- The secret will be referenced by compositions for Git operations
3. Creating the FireworksApp Instance
cat <<EOF | kubectl apply -f -
apiVersion: composition.krateo.io/v1-1-13
kind: FireworksApp
metadata:
name: fireworksapp-composition-1
namespace: fireworksapp-system
spec:
app:
service:
port: 31180
type: NodePort
argocd:
application:
destination:
namespace: fireworks-app
server: https://kubernetes.default.svc
project: default
source:
path: chart/
syncPolicy:
automated:
prune: true
selfHeal: true
namespace: krateo-system
git:
fromRepo:
branch: main
credentials:
authMethod: generic
secretRef:
key: token
name: github-repo-creds
namespace: krateo-system
name: krateo-v2-template-fireworksapp
org: krateoplatformops
path: skeleton/
scmUrl: https://github.com
insecure: true
toRepo:
apiUrl: https://api.github.com
branch: main
credentials:
authMethod: generic
secretRef:
key: token
name: github-repo-creds
namespace: krateo-system
initialize: true
org: your-organization
name: fireworksapp-test-v2
path: /
private: false
scmUrl: https://github.com
unsupportedCapabilities: true
EOF
What Occurs:
- Krateo creates a new FireworksApp resource
- The controller begins provisioning the application
- ArgoCD is configured to manage the deployment
- A new GitHub repository is created (if specified)
4. Monitoring Composition Progress
kubectl wait fireworksapp fireworksapp-composition-1 \
--for condition=Ready=True \
--timeout=300s \
--namespace fireworksapp-system
Expected Workflow:
- Command waits until all resources are provisioned
- During this time, containers are pulled and started
- Services are created and become accessible
- Success means your application is fully deployed
5. Check Helm Release Status
helm list -n fireworksapp-system
What to Expect:
- Helm lists all releases in the
fireworksapp-system
namespace - You should see the
fireworksapp-composition-1
release with version 1.1.13 - This confirms that the Helm chart was successfully deployed
Advanced Operations
1. Deploying Multiple Versions
Scenario:
You need to run two different versions of the same application simultaneously for testing purposes.
cat <<EOF | kubectl apply -f -
apiVersion: core.krateo.io/v1alpha1
kind: CompositionDefinition
metadata:
name: fireworksapp-cd-v2
namespace: fireworksapp-system
spec:
chart:
repo: fireworks-app
url: https://charts.krateo.io
version: 1.1.14
EOF
System Response:
- A second controller is deployed for the new version
- Both versions can operate simultaneously
- Each version maintains its own CRD and controller
This will create a new CompositionDefinition
named fireworksapp-cd-v2
in the fireworksapp-system
namespace, which will manage resources of version 1.1.14 of the fireworksapp chart.
You can then deploy the new version of the chart by applying the CompositionDefinition
manifest. The core-provider
will add a new version to the existing CRD fireworksapps.composition.krateo.io
and deploy a new instance of the composition-dynamic-controller
to manage resources of version 1.1.14.
The core-provider
will leave the previous version of the chart (1.1.13) running along with its associated composition-dynamic-controller
instance. This allows you to run multiple versions of the same application simultaneously, each managed by its own composition-dynamic-controller
.
Verifying CompositionDefinition Status
kubectl wait compositiondefinition fireworksapp-cd-v2 --for condition=Ready=True --namespace fireworksapp-system --timeout=600s
System Behavior:
- Command waits until the CompositionDefinition is fully processed
- During this time, Krateo is setting up the necessary controllers
- Success means you can now create FireworksApp instances
Create a New FireworksApp Instance
cat <<EOF | kubectl apply -f -
apiVersion: composition.krateo.io/v1-1-14
kind: FireworksApp
metadata:
name: fireworksapp-composition-2
namespace: fireworksapp-system
spec:
app:
service:
port: 31180
type: NodePort
argocd:
application:
destination:
namespace: fireworks-app
server: https://kubernetes.default.svc
project: default
source:
path: chart/
syncPolicy:
automated:
prune: true
selfHeal: true
namespace: krateo-system
git:
fromRepo:
branch: main
credentials:
authMethod: generic
secretRef:
key: token
name: github-repo-creds
namespace: krateo-system
name: krateo-v2-template-fireworksapp
org: krateoplatformops
path: skeleton/
scmUrl: https://github.com
insecure: true
toRepo:
apiUrl: https://api.github.com
branch: main
credentials:
authMethod: generic
secretRef:
key: token
name: github-repo-creds
namespace: krateo-system
initialize: true
org: your-organization
name: fireworksapp-test-v2
path: /
private: false
scmUrl: https://github.com
unsupportedCapabilities: true
EOF
Check Helm Release Status
helm list -n fireworksapp-system
What to Expect:
- You should see both
fireworksapp-composition-1
andfireworksapp-composition-2
listed - Each release corresponds to its respective version
- This confirms that both versions are deployed and managed independently
2. Upgrading Compositions (massive migration)
Scenario:
You need to upgrade the existing version of the application to a newer version (1.1.14). To be sure the new version can be deployed, the values.schema.json
should not add any new required fields or remove any existing required fields. If you need to add new required fields, you should create a new CompositionDefinition
with the new version of the chart. Refer to the next section for more details.
kubectl patch compositiondefinition fireworksapp-cd \
-n fireworksapp-system \
--type=merge \
-p '{"spec":{"chart":{"version":"1.1.14"}}}'
Upgrade Process:
- The controller gradually reconciles existing resources
- New pods are rolled out using the updated version
- The system ensures zero-downtime during transition
- All components eventually reflect the new version
- The old version is marked for cleanup
Automatic Deletion of Unused composition-dynamic-controller
Deployments
Notice that the previously deployed instances (pods) of composition-dynamic-controller
that were configured to manage resources of version 1.1.14 no longer exist in the cluster.
This is due to the automatic cleanup mechanism that removes older and unused deployments along with their associated RBAC resources from the cluster:
kubectl get po -n fireworksapp-system
This automatic cleanup helps maintain cluster cleaniness by removing outdated controller instances when they are no longer needed.
Check Helm Release Status
helm list -n fireworksapp-system
What to Expect:
- You should see both
fireworksapp-composition-1
andfireworksapp-composition-2
listed - Each release corresponds to its respective version but now
fireworksapp-composition-1
will be updated to version 1.1.14
3. Upgrading Compositions with changes in the values.schema.json
Scenario:
This can be useful for upgrading a chart where you have changed the values.schema.json
(in particular adding required fields or modifying existing fields) and you want to ensure that the new version can be deployed without issues.
Suppose you have installed composition fireworksapp-cd
with version 1.1.13 and you want to upgrade it to version 1.1.14, but the new version has changes in the values.schema.json
that require you to create a new CompositionDefinition
with the new version of the chart.
The first step is to create a new CompositionDefinition
with the new version of the chart. You can do this by applying the following manifest as done in the previous section:
cat <<EOF | kubectl apply -f -
apiVersion: core.krateo.io/v1alpha1
kind: CompositionDefinition
metadata:
name: fireworksapp-cd-v2
namespace: fireworksapp-system
spec:
chart:
repo: fireworks-app
url: https://charts.krateo.io
version: 1.1.14
EOF
At this point, you have two CompositionDefinition
resources in the cluster: fireworksapp-cd
with version 1.1.13 and fireworksapp-cd-v2
with version 1.1.14.
At this point, suppose you need to upgrade an existing release of a composition to the new version. You can do this by following the steps below:
Step 1: Pause the Composition with the Old Version
kubectl annotate fireworksapp fireworksapp-composition-1 \
-n fireworksapp-system \
"krateo.io/paused=true"
This ensures that the old version of the composition is not reconciled while you are upgrading to the new version.
Step 2: Create a New FireworksApp Instance with the New Version
Create a new FireworksApp instance with the new version of the chart. Note that you need to add a label called krateo.io/release-name
to the new FireworksApp instance. This label is used by the composition-dynamic-controller
to identify the release name of the composition. The value of this label should be the same as the name of the old FireworksApp instance. In the following example, the value of the label is fireworksapp-composition-1
, which is the name of the release installed at step 1. Normally, this label is automatically added to the FireworksApp instance when it is created. However, in this case, you need to add it manually because you are creating a new FireworksApp instance with a different name.
You can do that by applying the following manifest:
cat <<EOF | kubectl apply -f -
apiVersion: composition.krateo.io/v1-1-14
kind: FireworksApp
metadata:
name: fireworksapp-composition-2
namespace: fireworksapp-system
labels:
krateo.io/release-name: fireworksapp-composition-1
spec:
app:
service:
port: 31180
type: NodePort
argocd:
application:
destination:
namespace: fireworks-app
server: https://kubernetes.default.svc
project: default
source:
path: chart/
syncPolicy:
automated:
prune: true
selfHeal: true
namespace: krateo-system
git:
fromRepo:
branch: main
credentials:
authMethod: generic
secretRef:
key: token
name: github-repo-creds
namespace: krateo-system
name: krateo-v2-template-fireworksapp
org: krateoplatformops
path: skeleton/
scmUrl: https://github.com
insecure: true
toRepo:
apiUrl: https://api.github.com
branch: main
credentials:
authMethod: generic
secretRef:
key: token
name: github-repo-creds
namespace: krateo-system
initialize: true
org: your-organization
name: fireworksapp-test-v2
path: /
private: false
scmUrl: https://github.com
unsupportedCapabilities: true
EOF
At this point, the helm release fireworksapp-composition-1
will be upgraded to version 1.1.14.
Step 3: Remove the old composition at version 1.1.13
kubectl delete fireworksapp fireworksapp-composition-1 \
-n fireworksapp-system
You also need to remove the finalizer in the old composition because the annotation krateo.io/paused=true
will prevent the old composition from being deleted. You can do this by applying the following command:
kubectl patch fireworksapp fireworksapp-composition-1 \
-n fireworksapp-system \
--type=merge \
-p '{"metadata":{"finalizers":null}}'
At this point, the old composition will be deleted and the new composition will be upgraded to version 1.1.14.
4. Pausing Composition Reconciliation
Use Case:
Temporarily stop automatic reconciliation during maintenance or troubleshooting.
Step 1: Pause Composition
kubectl annotate fireworksapp fireworksapp-composition-1 \
-n fireworksapp-system \
"krateo.io/paused=true"
Expected Result:
- Composition is annotated with
krateo.io/paused=true
- Controller stops reconciling this instance
- Verify with:
You could also check events:
kubectl get fireworksapp fireworksapp-composition-1 -n fireworksapp-system -o jsonpath='{.metadata.annotations}'
kubectl get events -n fireworksapp-system --sort-by='.metadata.creationTimestamp' | grep "fireworksapp-composition-1"
Step 2: Make Manual Changes
During paused state, you can:
- Manually modify resources
- Troubleshoot issues
- Perform maintenance
Step 3: Resume Reconciliation
kubectl annotate fireworksapp fireworksapp-composition-1 \
-n fireworksapp-system \
"krateo.io/paused-"
Expected Result:
- Pause annotation is removed
- Controller resumes normal operation
- Changes are reconciled according to desired state
4. Safely Deleting Compositions
You can safely delete all compositions and their associated resources using the following command:
kubectl delete compositiondefinition fireworksapp-cd \
-n fireworksapp-system
Expected Outcome:
- The
deletionTimestamp
is set in the generated CRD and in all the compositions related to this CRD. - The
compositiondefinition
and all associated resources are deleted. (It may take a few minutes because any release associated with the composition will be deleted first.)
Note: If some compositions are stuck in the deletion process, do not enforce their deletion by removing the finalizer in the CRD. Instead, you should check the logs of the composition-dynamic-controller
to understand why the deletion is stuck. If you need to force the deletion, you can do so by removing the finalizer in the composition CR. Then the core-provider will safely delete the CRD.
Deleting the finalizer in the CRD can cause problems with Kubernetes etcd, so it is not recommended. See section 5. Error after creating CompositionDefinition for troubleshooting.
Troubleshooting Guide
Common Issues and Diagnostic Procedures
1. CompositionDefinition Not Becoming Ready
Symptoms:
- CompositionDefinition remains in "Not Ready" state
- No corresponding controller pod created
- CRD not generated
Diagnostic Steps:
# Check CompositionDefinition status details
kubectl describe compositiondefinition <name> -n fireworksapp-system
# Examine core provider logs for processing errors
kubectl logs -n krateo-system -l app=core-provider --tail=100
What to Look For:
- In the describe output: Check "Status.Conditions" for error messages
- In core provider logs: Look for chart download or CRD generation failures
- Verify network connectivity to the chart repository
2. Compositions Failing to Deploy
Symptoms:
- FireworksApp resource stuck in "Not Ready" state
Diagnostic Steps:
# Get detailed status of the composition
kubectl describe fireworksapp <name> -n fireworksapp-system
# Check controller logs for reconciliation errors
kubectl logs -n fireworksapp-system -l app.kubernetes.io/name=fireworksapps-controller
# Verify ArgoCD application status
kubectl get application -n krateo-system
What to Examine:
- Events section in describe output for resource creation failures
- Controller logs for Git operations or resource deployment errors
- ArgoCD application health and sync status
3. Upgrade/Rollback Failures
Symptoms:
- Version transition stuck
- Mixed version resources
- Controller crash loops
Diagnostic Steps:
# Check rollout status of controller deployment
kubectl rollout status deployment/fireworksapps-v1-1-14-controller -n fireworksapp-system
# Verify resource versions
kubectl get all -n fireworks-app -l krateo.io/composition-version
# Compare desired vs actual state
kubectl get fireworksapp -n fireworksapp-system -o yaml
Investigation Areas:
- Resource version labels consistency
- Controller pod logs for reconciliation errors
- Helm release history for the composition
4. Certificate Issues: Mutating Webhook Configuration - Valid ONLY for versions of core-provider before 0.24.2
Note: This issue is only valid for versions of core-provider before 0.24.2. In versions after this, the management of certificates is automatically handled by the core-provider and you should not face this issue.
Symptoms:
- You receive an error like
Internal error occurred: failed calling webhook "core.provider.krateo.io": failed to call webhook: Post "https://core-provider-webhook- service.krateo-v2-system.svc:9443/mutate? timeout=10s": t|s: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "core-provider-webhook- service.krateo-v2-system.svc")
- You cannot create the related composition in the cluster
Diagnostic Steps:
# Check the caBundle in the core-provider-webhook-service
kubectl get mutatingWebhookConfiguration core-provider -o jsonpath='{.webhooks[0].clientConfig.caBundle}'
# Check if the base64 encoded caBundle is aligned with the core-provider tls secret
kubectl get secret core-provider-certs -n krateo-system -o json | jq -r '.data["tls.crt"]'
If you receive the error described above, the caBundle is not aligned with the core-provider tls secret. So you need to patch the caBundle in the core-provider mutatingWebhookConfiguration.
Remediation Steps:
- Patch the caBundle in the core-provider mutatingWebhookConfiguration using the command provided above.
caBundle=$(kubectl get secret core-provider-certs -n krateo-system -o json | jq -r '.data["tls.crt"]')
kubectl patch mutatingWebhookConfiguration core-provider \
--type='json' \
-p='[{"op": "replace", "path": "/webhooks/0/clientConfig/caBundle", "value": "'"$caBundle"'"}]'
4.1 Certificate Issues: Conversion Webhook Configuration
Symptoms:
- You receive an error like
conversion webhook for composition.krateo.io/v0-0-13, Kind=KrateoTemplateToolAwsGlueJobs failed: Post \"https://core-provider-webhook-service.krateo-v2-system.svc:9443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority
- You cannot create the related composition in the cluster
Diagnostic Steps:
# Check the caBundle in the core-provider-webhook-service
kubectl get crds <your-crd-name> -o jsonpath='{.spec.conversion.webhook.clientConfig.caBundle}'
# Check if the base64 encoded caBundle is aligned with the core-provider tls secret
kubectl get secret core-provider-certs -n krateo-system -o json | jq -r '.data["tls.crt"]'
Remediation Steps:
- Patch the caBundle in the core-provider mutatingWebhookConfiguration using the command provided above.
caBundle=$(kubectl get secret core-provider-certs -n krateo-system -o json | jq -r '.data["tls.crt"]')
kubectl patch crd <your-crd-name> \
--type='json' \
-p='[{"op": "replace", "path": "/spec/conversion/webhook/clientConfig/caBundle", "value": "'"$caBundle"'"}]'
You will need to replace <your-crd-name>
with the name of the CRD you are trying to create.
5. Error after creating CompositionDefinition
Symptoms:
- You receive an error like
request to convert CR from an invalid group/version: composition.krateo.io/vacuum
- You cannot create the related composition in the cluster
Diagnostic Steps:
# This error is usually present in the Kubernetes API server logs, in core-provider logs, and is also generally present when listing `compositions`
kubectl get compositions -A
Remediation Steps:
- To solve this issue, you need to force core-provider to re-create the
vacuum
version in the CRD. You can do this changing the version of the chart in theCompositionDefinition
.
General Diagnostic Tools
# Cluster-wide event inspection
kubectl get events -A --sort-by='.metadata.creationTimestamp' | grep -i "error\|fail"
# Cross-namespace pod status check
kubectl get pods -A -o wide | grep -E "krateo-system|fireworksapp-system"
# API resource verification
kubectl api-resources | grep fireworksapp
# Network connectivity tests
kubectl run -it --rm --restart=Never network-test \
--image=alpine -n fireworksapp-system -- \
ping charts.krateo.io
Common Solutions
-
Authentication Issues:
- Regenerate Service token with correct scopes
- Recreate the secret with proper formatting
- Verify network policies allow outbound connections
-
Version Conflicts:
- Manually clean up orphaned resources
- Delete and recreate the CompositionDefinition
- Verify chart compatibility between versions
-
Resource Constraints:
- Check pod resource limits and node availability
- Review pending pods with
kubectl get pods --field-selector=status.phase=Pending
- Increase cluster resources if needed