Failed to Call Webhook when deploying Metallb in K3s

After installing metallb using helm, I hit the below error when applying the IP Adress Pool.

Error from server (InternalError): error when creating "ip-pool.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": context deadline exceeded
Error from server (InternalError): error when creating "ip-pool.yaml": Internal error occurred: failed calling webhook "l2advertisementvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-l2advertisement?timeout=10s": context deadline exceeded

The IPAdressPool yaml:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  - 172.16.1.201-172.16.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
  - default-pool

To resolve the issue, install metallb by setting validationFailurePolicy=Ignore

 helm upgrade --install metallb  metallb/metallb --create-namespace --namespace metallb-system --set crds.validationFailurePolicy=Ignore --wait

More information here

Few additional things to look at,

The metallb pods were in a running state:

k get pods -n metallb-system -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES
metallb-controller-648b76f565-kmdx2   1/1     Running   0          33m   10.42.1.4     worker1.gs.labs    <none>           <none>
metallb-speaker-9hm5c                 4/4     Running   0          33m   172.16.1.11   control1.gs.labs   <none>           <none>
metallb-speaker-rpgr9                 4/4     Running   0          33m   172.16.1.14   worker1.gs.labs    <none>           <none>

The metallb webhook service was running fine and it was pointing to the correct endpoint(metallb-controller-648b76f565-kmdx2)

k describe svc metallb-webhook-service -n metallb-system
Name:              metallb-webhook-service
Namespace:         metallb-system
Labels:            app.kubernetes.io/instance=metallb
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=metallb
                   app.kubernetes.io/version=v0.14.3
                   helm.sh/chart=metallb-0.14.3
Annotations:       meta.helm.sh/release-name: metallb
                   meta.helm.sh/release-namespace: metallb-system
Selector:          app.kubernetes.io/component=controller,app.kubernetes.io/instance=metallb,app.kubernetes.io/name=metallb
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.43.120.56
IPs:               10.43.120.56
Port:              <unset>  443/TCP
TargetPort:        9443/TCP
Endpoints:         10.42.1.4:9443
Session Affinity:  None
Events:            <none>

The metallb-controller-648b76f565-kmdx2 logs looked clean as the service was listening on port 9443

{"action":"webhooks enabled","caller":"webhook.go:57","level":"info","op":"startup","ts":"2024-02-16T03:51:14Z"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"cert-rotation","msg":"certs are ready in /tmp/k8s-webhook-server/serving-certs"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"cert-rotation","msg":"CA certs are injected to webhooks"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-metallb-io-v1beta1-ipaddresspool"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-metallb-io-v1beta2-bgppeer"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-metallb-io-v1beta1-bgpadvertisement"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-metallb-io-v1beta1-l2advertisement"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-metallb-io-v1beta1-community"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-metallb-io-v1beta1-bfdprofile"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Starting webhook server"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":9443}
{"level":"info","ts":"2024-02-16T03:51:14Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}