Helm defaults not even starting up - [CRITICAL] WORKER TIMEOUT #146
Comments
|
+1 for me, more context: I override version of postgresql (14) my values.yaml: postgresql:
image:
tag: "14"
persistence:
# custom settings for PVC
image:
# SL 6/6/23: latest version of Redash docker image
tag: 10.1.0.b50633redash pod logs: Using Database: postgresql://redash:******@redash-dev-postgresql:5432/redash
Using Redis: redis://:******@redash-dev-redis-master:6379/0
[2023-06-07 14:55:46 +0000] [7] [INFO] Starting gunicorn 20.0.4
[2023-06-07 14:55:46 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
[2023-06-07 14:55:46 +0000] [7] [INFO] Using worker: sync
[2023-06-07 14:55:46 +0000] [10] [INFO] Booting worker with pid: 10
[2023-06-07 14:55:46 +0000] [11] [INFO] Booting worker with pid: 11
[2023-06-07 14:55:46 +0000] [12] [INFO] Booting worker with pid: 12
[2023-06-07 14:55:46 +0000] [13] [INFO] Booting worker with pid: 13
[2023-06-07 14:56:17 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:10)
[2023-06-07 14:56:17 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:11)
[2023-06-07 14:56:17 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:12)
[2023-06-07 14:56:17 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:13)
[2023-06-07 14:56:17 +0000] [11] [INFO] Worker exiting (pid: 11)
[2023-06-07 14:56:17 +0000] [12] [INFO] Worker exiting (pid: 12)
[2023-06-07 14:56:17 +0000] [10] [INFO] Worker exiting (pid: 10)
[2023-06-07 14:56:17 +0000] [13] [INFO] Worker exiting (pid: 13)
[2023-06-07 14:56:18 +0000] [30] [INFO] Booting worker with pid: 30
[2023-06-07 14:56:18 +0000] [31] [INFO] Booting worker with pid: 31
[2023-06-07 14:56:18 +0000] [32] [INFO] Booting worker with pid: 32
[2023-06-07 14:56:18 +0000] [33] [INFO] Booting worker with pid: 33
[2023-06-07 14:56:48 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:30)
[2023-06-07 14:56:48 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:31)
[2023-06-07 14:56:48 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:32)
[2023-06-07 14:56:48 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:33)
[2023-06-07 14:56:48 +0000] [33] [INFO] Worker exiting (pid: 33)
[2023-06-07 14:56:49 +0000] [32] [INFO] Worker exiting (pid: 32)
[2023-06-07 14:56:49 +0000] [30] [INFO] Worker exiting (pid: 30)
[2023-06-07 14:56:49 +0000] [31] [INFO] Worker exiting (pid: 31)
[2023-06-07 14:56:50 +0000] [50] [INFO] Booting worker with pid: 50
[2023-06-07 14:56:50 +0000] [51] [INFO] Booting worker with pid: 51
[2023-06-07 14:56:50 +0000] [52] [INFO] Booting worker with pid: 52
[2023-06-07 14:56:50 +0000] [53] [INFO] Booting worker with pid: 53
[2023-06-07 14:57:20 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:50)
[2023-06-07 14:57:20 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:51)
[2023-06-07 14:57:20 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:52)
[2023-06-07 14:57:20 +0000] [7] [CRITICAL] WORKER TIMEOUT (pid:53)
[2023-06-07 14:57:20 +0000] [51] [INFO] Worker exiting (pid: 51)
[2023-06-07 14:57:20 +0000] [50] [INFO] Worker exiting (pid: 50)
[2023-06-07 14:57:20 +0000] [53] [INFO] Worker exiting (pid: 53)
[2023-06-07 14:57:21 +0000] [52] [INFO] Worker exiting (pid: 52)
[2023-06-07 14:57:22 +0000] [70] [INFO] Booting worker with pid: 70
[2023-06-07 14:57:22 +0000] [71] [INFO] Booting worker with pid: 71
[2023-06-07 14:57:22 +0000] [72] [INFO] Booting worker with pid: 72
[2023-06-07 14:57:22 +0000] [73] [INFO] Booting worker with pid: 73
|
|
update: here are attached the pod's logs with LOG_LEVEL=DEBUG |
|
update 2: reproduced with the default version of postgres, as well as with v. 14 or 15 
|
|
resource limits were the culprit! resources: {}
# limits:
# cpu: 500m
# memory: 3Gi
# requests:
# cpu: 100m
# memory: 500Mi
|
|
Alright I'll have a look on Friday, exceeding 3Gi memory sounds way too much though 😅 thanks already
…________________________________
From: Samuel ***@***.***>
Sent: Wednesday, June 7, 2023 6:45:39 PM
To: getredash/contrib-helm-chart ***@***.***>
Cc: Justin Güse ***@***.***>; Mention ***@***.***>
Subject: Re: [getredash/contrib-helm-chart] Helm defaults not even starting up - [CRITICAL] WORKER TIMEOUT (Issue #146)
resource limits were the culprit!
once I removed them, it worked a lot better !
@JustinGuese<https://github.com/JustinGuese>
these were the resource limits that I used. I'm still digging around to figure out what's the best fit (I can't let redash use all my cluster resources)
resources: {}
# limits:
# cpu: 500m
# memory: 3Gi
# requests:
# cpu: 100m
# memory: 500Mi
—
Reply to this email directly, view it on GitHub<#146 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACJFA2Y36JV4SGOYJUC4PM3XKCV3HANCNFSM6AAAAAAY5RC45M>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
nope, still nothing. the workers throw the following error |
|
so i would say redis doesnt work... i also can't see any redis pod, so i guess the redis pod isn't created? |
|
also this repo is 3 years old, so i guess they do not offer support anymore & therefore i won't use it anyways, thanks for your help though! |
|
The chart is working with the default values, so I am guessing this must be something with your local setup? |
|
hm, at least not only for me, but also @Samuel29 |
|
Oh you make a good point. I'm using a managed Kubernetes cluster (v1.25) + ArgoCD + Helm |
|
For the record here is my setup redash:
postgresql:
image:
# use postgres v15 instead of 9.6 (!)
tag: "15"
persistence:
# OVH managed K8s
storageClass: csi-cinder-high-speed
image:
# SL 6/6/23: latest version of Redash docker image
tag: 10.1.0.b50633
# (... some ingress variables kept for me)
server:
# server.resources -- Server resource requests and limits [ref](http://kubernetes.io/docs/user-guide/compute-resources/)
resources:
limits:
cpu: 1000m
memory: 4Gi
requests:
cpu: 100m
memory: 500Mi |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
error
redash main pod
genericworker
I guess redis does not deploy?
recreating
just helm install it like in the basic example, all default values
-> nothing shows up when port-forwarding
The text was updated successfully, but these errors were encountered: