This job view page is being replaced by Spyglass soon. Check out the new job view.
ResultFAILURE
Tests 0 failed / 0 succeeded
Started2022-08-16 03:07
Elapsed4m35s
Revisionrelease-0.3

No Test Failures!


Error lines from build-log.txt

... skipping 773 lines ...
I0816 03:08:57.741457    8476 server.go:121] controller-runtime/webhook "msg"="registering webhook"  "path"="/validate-addons-cluster-x-k8s-io-v1alpha3-clusterresourceset"
I0816 03:08:57.742349    8476 server.go:141] controller-runtime/webhook/webhooks "msg"="starting webhook server"  
I0816 03:08:57.743618    8476 certwatcher.go:127] controller-runtime/certwatcher "msg"="Updated current TLS certificate"  
I0816 03:08:57.743867    8476 certwatcher.go:83] controller-runtime/certwatcher "msg"="Starting certificate watcher"  
I0816 03:08:57.744019    8476 server.go:199] controller-runtime/webhook "msg"="serving webhook server"  "host"="" "port"=37449
I0816 03:08:58.645592    8476 kubeadmconfig_controller.go:223]  "msg"="Cluster infrastructure is not ready, waiting" "kind"="Machine" "kubeadmconfig"={"Namespace":"default","Name":"my-machine-config"} "name"="my-machine" "version"="109" 
•E0816 03:08:58.656587    8476 certwatcher.go:143] controller-runtime/certwatcher "msg"="error re-watching file" "error"="no such file or directory"  
E0816 03:08:58.656714    8476 certwatcher.go:148] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-667585825/tls.crt: no such file or directory"  
E0816 03:08:58.656752    8476 certwatcher.go:143] controller-runtime/certwatcher "msg"="error re-watching file" "error"="no such file or directory"  
E0816 03:08:58.656773    8476 certwatcher.go:148] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-667585825/tls.crt: no such file or directory"  
I0816 03:08:58.657050    8476 server.go:208] controller-runtime/webhook "msg"="shutting down webhook server"  


Ran 1 of 1 Specs in 15.055 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestAPIs (15.06s)
PASS
ok  	sigs.k8s.io/cluster-api/bootstrap/kubeadm/controllers	53.457s
=== RUN   TestNewInitControlPlaneAdditionalFileEncodings
--- PASS: TestNewInitControlPlaneAdditionalFileEncodings (0.01s)
=== RUN   TestNewInitControlPlaneCommands
... skipping 316 lines ...
=== RUN   Test_clusterctlClient_GetProvidersConfig/Returns_default_providers_and_custom_providers_if_defined
--- PASS: Test_clusterctlClient_GetProvidersConfig (0.00s)
    --- PASS: Test_clusterctlClient_GetProvidersConfig/Returns_default_providers (0.00s)
    --- PASS: Test_clusterctlClient_GetProvidersConfig/Returns_default_providers_and_custom_providers_if_defined (0.00s)
=== RUN   Test_clusterctlClient_GetProviderComponents
=== RUN   Test_clusterctlClient_GetProviderComponents/Pass
=== RUN   Test_clusterctlClient_GetProviderComponents/Fail
--- PASS: Test_clusterctlClient_GetProviderComponents (0.00s)
    --- PASS: Test_clusterctlClient_GetProviderComponents/Pass (0.00s)
    --- PASS: Test_clusterctlClient_GetProviderComponents/Fail (0.00s)
=== RUN   Test_getComponentsByName_withEmptyVariables
--- PASS: Test_getComponentsByName_withEmptyVariables (0.00s)
=== RUN   Test_clusterctlClient_templateOptionsToVariables
=== RUN   Test_clusterctlClient_templateOptionsToVariables/pass_(using_KubernetesVersion_from_template_options)
=== RUN   Test_clusterctlClient_templateOptionsToVariables/pass_(using_KubernetesVersion_from_env_variables)
=== RUN   Test_clusterctlClient_templateOptionsToVariables/pass_(using_defaults_for_machine_counts)
... skipping 1398 lines ...
I0816 03:08:42.825418   10501 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machinehealthcheck" "worker count"=1
I0816 03:08:42.825424   10501 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="machinedeployment" 
I0816 03:08:42.825464   10501 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machinedeployment" "worker count"=1
I0816 03:08:42.826015   10501 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machineset" "worker count"=1
I0816 03:08:42.831226   10501 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="cluster" "worker count"=1
I0816 03:08:42.831747   10501 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machine" "worker count"=1
2022/08/16 03:08:42 http: TLS handshake error from 127.0.0.1:45226: EOF
=== RUN   TestClusterReconcilePhases
=== RUN   TestClusterReconcilePhases/reconcile_infrastructure
=== RUN   TestClusterReconcilePhases/reconcile_infrastructure/returns_no_error_if_infrastructure_ref_is_nil
=== RUN   TestClusterReconcilePhases/reconcile_infrastructure/returns_error_if_unable_to_reconcile_infrastructure_ref
I0816 03:08:42.925314   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "cluster"="test-cluster" "namespace"="test-namespace" "err"="cannot create a metadata client without a rest config"
I0816 03:08:42.928418   10501 cluster_controller_phases.go:76]  "msg"="Could not find external object for cluster, requeuing" "cluster"="test-cluster" "namespace"="test-namespace" "refGroupVersionKind"={"Group":"infrastructure.cluster.x-k8s.io","Version":"v1alpha3","Kind":"InfrastructureMachine"} "refName"="test"
... skipping 105 lines ...
    --- PASS: TestReconcileBootstrap/existing_machine,_machineset_owner_and_version_v1alpha2,_and_ownerref_updated (0.00s)
=== RUN   TestReconcileInfrastructure
=== RUN   TestReconcileInfrastructure/new_machine,_infrastructure_config_ready
I0816 03:08:43.015954   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
=== RUN   TestReconcileInfrastructure/ready_bootstrap,_infra,_and_nodeRef,_machine_is_running,_infra_object_is_deleted,_expect_failed
I0816 03:08:43.018076   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
E0816 03:08:43.018470   10501 machine_controller_phases.go:246]  "msg"="Machine infrastructure reference has been deleted after being ready, setting failure state" "error"="could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=InfrastructureMachine \"infra-config1\" for Machine \"machine-test\" in namespace \"default\", requeuing: requeue in 1s"  
=== RUN   TestReconcileInfrastructure/infrastructure_ref_is_paused
I0816 03:08:43.018884   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
--- PASS: TestReconcileInfrastructure (0.01s)
    --- PASS: TestReconcileInfrastructure/new_machine,_infrastructure_config_ready (0.01s)
    --- PASS: TestReconcileInfrastructure/ready_bootstrap,_infra,_and_nodeRef,_machine_is_running,_infra_object_is_deleted,_expect_failed (0.00s)
    --- PASS: TestReconcileInfrastructure/infrastructure_ref_is_paused (0.00s)
=== RUN   TestWatches
I0816 03:08:43.391386   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machine" "source"={}
E0816 03:08:43.393708   10501 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine"  "node"="node-1"
I0816 03:08:43.400388   10501 tracker.go:55] controllers/Machine "msg"="Adding watcher on external object" "machine"="machine-created-7c288" "namespace"="test-machine-watches-glm4d" "GroupVersionKind"="bootstrap.cluster.x-k8s.io/v1alpha3, Kind=BootstrapMachine"
I0816 03:08:43.400476   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machine" "source"={"Type":{"apiVersion":"bootstrap.cluster.x-k8s.io/v1alpha3","kind":"BootstrapMachine"}}
I0816 03:08:43.517428   10501 tracker.go:55] controllers/Machine "msg"="Adding watcher on external object" "machine"="machine-created-7c288" "namespace"="test-machine-watches-glm4d" "GroupVersionKind"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=InfrastructureMachine"
I0816 03:08:43.517516   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machine" "source"={"Type":{"apiVersion":"infrastructure.cluster.x-k8s.io/v1alpha3","kind":"InfrastructureMachine"}}
I0816 03:08:43.618045   10501 machine_controller_noderef.go:86] controllers/Machine "msg"="Set Machine's NodeRef" "machine"="machine-created-7c288" "namespace"="test-machine-watches-glm4d" "noderef"="node-1"
E0816 03:08:43.630598   10501 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine"  "node"="node-1"
E0816 03:08:43.630662   10501 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine"  "node"="node-1"
I0816 03:08:43.680899   10501 machine_controller_noderef.go:86] controllers/Machine "msg"="Set Machine's NodeRef" "machine"="machine-created-7c288" "namespace"="test-machine-watches-glm4d" "noderef"="node-1"
E0816 03:08:43.708240   10501 machine_controller_phases.go:246] controllers/Machine "msg"="Machine infrastructure reference has been deleted after being ready, setting failure state" "error"="could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=InfrastructureMachine \"infra-config1\" for Machine \"machine-created-7c288\" in namespace \"test-machine-watches-glm4d\", requeuing: requeue in 1s"  
E0816 03:08:43.733200   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="[could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=InfrastructureMachine \"infra-config1\" for Machine \"machine-created-7c288\" in namespace \"test-machine-watches-glm4d\", requeuing: requeue in 1s, no matching Node for Machine \"machine-created-7c288\" in namespace \"test-machine-watches-glm4d\": cannot find node with matching ProviderID]" "controller"="machine" "name"="machine-created-7c288" "namespace"="test-machine-watches-glm4d"
--- PASS: TestWatches (0.74s)
=== RUN   TestIndexMachineByNodeName
=== RUN   TestIndexMachineByNodeName/when_the_machine_has_no_NodeRef
=== RUN   TestIndexMachineByNodeName/when_the_machine_has_valid_a_NodeRef
--- PASS: TestIndexMachineByNodeName (0.00s)
    --- PASS: TestIndexMachineByNodeName/when_the_machine_has_no_NodeRef (0.00s)
... skipping 2 lines ...
=== RUN   TestMachine_Reconcile/reconcile_create
I0816 03:08:43.859616   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="machine-reconcile-rwzhz" "namespace"="test-machine-watches-glm4d" "count"=1
I0816 03:08:43.859697   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="machine-reconcile-rwzhz" "namespace"="test-machine-watches-glm4d" "descendants"="Worker machines: machine-created-7c288" "indirect descendants count"=0
I0816 03:08:43.877704   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="machine-reconcile-rwzhz" "namespace"="test-machine-watches-glm4d" "count"=1
I0816 03:08:43.877777   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="machine-reconcile-rwzhz" "namespace"="test-machine-watches-glm4d" "descendants"="Worker machines: machine-created-7c288" "indirect descendants count"=0
I0816 03:08:44.734703   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="machine-reconcile-rwzhz" "machine"="machine-created-7c288" "namespace"="test-machine-watches-glm4d" "cause"="cluster is being deleted" "node"={"kind":"Node","name":"node-1","uid":"9d562411-652a-4f26-8895-020decbed87f","apiVersion":"v1"}
E0816 03:08:44.800690   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"machine-created-7c288\" not found" "controller"="machine" "name"="machine-created-7c288" "namespace"="test-machine-watches-glm4d"
E0816 03:08:45.849379   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="[retrieved empty dataSecretName from bootstrap provider for Machine \"machine-created-f4dbn\" in namespace \"default\", error creating client and cache for remote cluster: error fetching REST client config for remote cluster \"default/machine-reconcile-v8ss9\": failed to retrieve kubeconfig secret for Cluster default/machine-reconcile-v8ss9: secrets \"machine-reconcile-v8ss9-kubeconfig\" not found]" "controller"="machine" "name"="machine-created-f4dbn" "namespace"="default"
I0816 03:08:46.849982   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="machine-reconcile-v8ss9" "machine"="machine-created-f4dbn" "namespace"="default" "cause"="noderef is nil" "node"=null
I0816 03:08:46.919027   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="machine-reconcile-v8ss9" "machine"="machine-created-f4dbn" "namespace"="default" "cause"="noderef is nil" "node"=null
I0816 03:08:46.987682   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="machine-reconcile-v8ss9" "machine"="machine-created-f4dbn" "namespace"="default" "cause"="noderef is nil" "node"=null
E0816 03:08:47.022497   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"machine-created-f4dbn\" not found" "controller"="machine" "name"="machine-created-f4dbn" "namespace"="default"
--- PASS: TestMachine_Reconcile (3.27s)
    --- PASS: TestMachine_Reconcile/reconcile_create (3.27s)
=== RUN   TestMachineFinalizer
=== RUN   TestMachineFinalizer/should_add_a_machine_finalizer_to_the_machine_if_it_doesn't_have_one
=== RUN   TestMachineFinalizer/should_append_the_machine_finalizer_to_the_machine_if_it_already_has_a_finalizer
--- PASS: TestMachineFinalizer (0.00s)
... skipping 15 lines ...
    --- PASS: TestMachineOwnerReference/should_not_add_cluster_owner_reference_if_machine_has_a_controller_owner (0.00s)
=== RUN   TestReconcileRequest
=== RUN   TestReconcileRequest/machine_should_be_created
I0816 03:08:47.048747   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="created" "namespace"="default" "err"="cannot create a metadata client without a rest config"
=== RUN   TestReconcileRequest/machine_should_be_updated
I0816 03:08:47.057503   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="updated" "namespace"="default" "err"="cannot create a metadata client without a rest config"
E0816 03:08:47.072328   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusters.cluster.x-k8s.io \"machine-reconcile-v8ss9\" not found" "controller"="cluster" "name"="machine-reconcile-v8ss9" "namespace"="default"
=== RUN   TestReconcileRequest/machine_should_be_deleted
I0816 03:08:47.074579   10501 machine_controller.go:305]  "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster" "machine"="deleted" "namespace"="default" "cause"="noderef is nil" "node"=null
--- PASS: TestReconcileRequest (0.03s)
    --- PASS: TestReconcileRequest/machine_should_be_created (0.01s)
    --- PASS: TestReconcileRequest/machine_should_be_updated (0.02s)
    --- PASS: TestReconcileRequest/machine_should_be_deleted (0.01s)
... skipping 110 lines ...
    --- PASS: TestMachineDeploymentSyncStatus/machine_set_failed (0.00s)
=== RUN   TestMachineHealthCheck_Reconcile
=== RUN   TestMachineHealthCheck_Reconcile/it_should_ensure_the_correct_cluster-name_label_when_no_existing_labels_exist
I0816 03:08:47.216281   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-tq5dd" "namespace"="test-mhc-2x78b" 
=== RUN   TestMachineHealthCheck_Reconcile/it_should_ensure_the_correct_cluster-name_label_when_the_label_has_the_wrong_value
I0816 03:08:47.237558   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
E0816 03:08:47.239188   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="Cluster.cluster.x-k8s.io \"test-cluster-cnmhv\" not found"  "cluster"="test-mhc-2x78b/test-cluster-cnmhv"
E0816 03:08:47.246777   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="[MachineHealthCheck.cluster.x-k8s.io \"test-mhc-tq5dd\" not found, machinehealthchecks.cluster.x-k8s.io \"test-mhc-tq5dd\" not found]" "controller"="machinehealthcheck" "name"="test-mhc-tq5dd" "namespace"="test-mhc-2x78b"
=== RUN   TestMachineHealthCheck_Reconcile/it_should_ensure_the_correct_cluster-name_label_when_other_labels_are_present
=== RUN   TestMachineHealthCheck_Reconcile/it_should_ensure_an_owner_reference_is_present_when_no_existing_ones_exist
error cannot retrieve mhc in ctx: MachineHealthCheck.cluster.x-k8s.io "test-mhc-qczvd" not foundI0816 03:08:48.247119   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-tq5dd" "namespace"="test-mhc-2x78b" 
I0816 03:08:48.247237   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-drfmf" "namespace"="test-mhc-lq5xn" 
I0816 03:08:48.247271   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-j8wn8" "namespace"="test-mhc-fkvtp" 
I0816 03:08:48.247299   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-qczvd" "namespace"="test-mhc-p6f7h" 
I0816 03:08:48.261963   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:08:48.282551   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-qczvd" "namespace"="test-mhc-p6f7h" 
I0816 03:08:48.372876   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-qczvd" "namespace"="test-mhc-p6f7h" 
=== RUN   TestMachineHealthCheck_Reconcile/it_should_ensure_an_owner_reference_is_present_when_modifying_existing_ones
E0816 03:08:48.382168   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-p6f7h/test-cluster-7xc8k"
E0816 03:08:48.388231   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusters.cluster.x-k8s.io \"test-cluster-7xc8k\" not found" "controller"="cluster" "name"="test-cluster-7xc8k" "namespace"="test-mhc-p6f7h"
I0816 03:08:48.498530   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-8q6xc" "namespace"="test-mhc-5dcb6" 
I0816 03:08:48.524681   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:08:48.646563   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-8q6xc" "namespace"="test-mhc-5dcb6" 
I0816 03:08:48.647418   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-8q6xc" "namespace"="test-mhc-5dcb6" 
E0816 03:08:48.703300   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-5dcb6/test-cluster-tjw26"
I0816 03:08:48.707165   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-8q6xc" "namespace"="test-mhc-5dcb6" 
=== RUN   TestMachineHealthCheck_Reconcile/it_doesn't_mark_anything_unhealthy_when_all_Machines_are_healthy
I0816 03:08:48.723731   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
inframachine created: test-mhc-machine-infra-zp6kh
machine created: test-mhc-machine-d88zp
I0816 03:08:48.738777   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:08:48.839347   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-v55fz/test-mhc-96h54/test-mhc-machine-d88zp/"
E0816 03:08:48.850962   10501 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine"  "node"="test-mhc-node-mgmmx"
E0816 03:08:48.850962   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-mgmmx, got []"  "node"="test-mhc-node-mgmmx"
node created: test-mhc-node-mgmmx
inframachine created: test-mhc-machine-infra-77h7p
I0816 03:08:48.872768   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
machine created: test-mhc-machine-zbbt6
I0816 03:08:48.895410   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:48.901720   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
... skipping 40 lines ...
I0816 03:08:49.019180   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.022819   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.025017   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.026807   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.028370   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.032115   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.032654   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-v55fz/test-mhc-96h54/test-mhc-machine-zbbt6/"
I0816 03:08:49.048567   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.049053   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-v55fz/test-mhc-96h54/test-mhc-machine-zbbt6/"
I0816 03:08:49.052017   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.052852   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-v55fz/test-mhc-96h54/test-mhc-machine-zbbt6/"
I0816 03:08:49.068025   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.068995   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-v55fz/test-mhc-96h54/test-mhc-machine-zbbt6/"
E0816 03:08:49.096416   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-lv9jx, got []"  "node"="test-mhc-node-lv9jx"
E0816 03:08:49.096867   10501 machine_controller.go:685] controllers/Machine "msg"="Unable to retrieve machine from node" "error"="no matching Machine"  "node"="test-mhc-node-lv9jx"
node created: test-mhc-node-lv9jx
I0816 03:08:49.101801   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.103075   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-v55fz/test-mhc-96h54/test-mhc-machine-zbbt6/"
I0816 03:08:49.109296   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
Cleaning up nodes, machines and infra machines.
I0816 03:08:49.166796   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:49.169934   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-v55fz/test-mhc-96h54/test-mhc-machine-d88zp/"
E0816 03:08:49.195264   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="no matching Node for Machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": cannot find node with matching ProviderID" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
E0816 03:08:49.206798   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-v55fz/test-cluster-zcf2h"
=== RUN   TestMachineHealthCheck_Reconcile/it_marks_unhealthy_machines_for_remediation_when_there_is_one_unhealthy_Machine
inframachine created: test-mhc-machine-infra-pb9mt
machine created: test-mhc-machine-mghw9
E0816 03:08:49.386611   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machinehealthchecks.cluster.x-k8s.io \"test-mhc-96h54\" not found" "controller"="machinehealthcheck" "name"="test-mhc-96h54" "namespace"="test-mhc-v55fz"
E0816 03:08:49.399974   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-machine-watches-glm4d/machine-reconcile-rwzhz"
E0816 03:08:50.195744   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:08:50.386913   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:50.421307   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:08:50.444912   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-96h54" "namespace"="test-mhc-v55fz" 
I0816 03:08:50.444980   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:50.445900   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:50.447822   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 270 lines ...
I0816 03:08:51.176885   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.178831   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.182116   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.185880   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.191694   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.194071   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
E0816 03:08:51.196556   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:08:51.196759   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.198553   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.202324   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.205880   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.207942   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:51.211078   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 355 lines ...
I0816 03:08:52.209376   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:52.211701   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:52.212393   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:52.214084   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:52.215929   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:52.218648   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
E0816 03:08:52.219221   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:08:52.220831   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-mghw9/"
I0816 03:08:52.237078   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:52.237465   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-mghw9/"
E0816 03:08:52.275362   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-f95ng, got []"  "node"="test-mhc-node-f95ng"
node created: test-mhc-node-f95ng
I0816 03:08:52.284629   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
inframachine created: test-mhc-machine-infra-8nhbp
machine created: test-mhc-machine-8nppb
I0816 03:08:52.314391   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:52.321921   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 264 lines ...
I0816 03:08:53.203836   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.206064   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.208260   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.212458   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.215103   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.217809   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
E0816 03:08:53.219629   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:08:53.220053   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.222332   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.224444   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.226881   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.229291   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:53.231516   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 393 lines ...
I0816 03:08:54.246234   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.253577   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.255028   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.259540   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.262003   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.264280   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
E0816 03:08:54.264741   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:08:54.265849   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.266972   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.268432   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.269548   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.282371   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:54.283870   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 272 lines ...
I0816 03:08:55.246762   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.259739   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.264213   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.269222   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.272115   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.276312   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
E0816 03:08:55.282840   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:08:55.283864   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.284531   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-8nppb/"
I0816 03:08:55.299640   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.300458   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-8nppb/"
E0816 03:08:55.341497   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-w7q9z, got []"  "node"="test-mhc-node-w7q9z"
node created: test-mhc-node-w7q9z
I0816 03:08:55.352634   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
inframachine created: test-mhc-machine-infra-r8hfv
machine created: test-mhc-machine-f7frj
I0816 03:08:55.378830   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:55.386649   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 182 lines ...
I0816 03:08:56.281034   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.283576   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.285947   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.288517   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.291026   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.293774   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
E0816 03:08:56.295580   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:08:56.297030   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.298738   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.305033   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.307277   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.310665   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:56.313170   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 276 lines ...
I0816 03:08:57.317276   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.319028   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.329298   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.334132   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.336793   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.339087   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
E0816 03:08:57.346307   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:08:57.347275   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.350139   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.352547   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.355745   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.358306   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:57.361122   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
... skipping 235 lines ...
I0816 03:08:58.339277   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.342338   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.345320   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.350068   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.352954   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.358707   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.359185   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-f7frj/"
E0816 03:08:58.365161   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:08:58.373830   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.375066   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-f7frj/"
E0816 03:08:58.411334   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-drrg7, got []"  "node"="test-mhc-node-drrg7"
node created: test-mhc-node-drrg7
I0816 03:08:58.419817   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
Cleaning up nodes, machines and infra machines.
I0816 03:08:58.420480   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Condition Ready on node is reporting status Unknown for more than 5m0s" "reason"="UnhealthyNode" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-f7frj/test-mhc-node-drrg7"
I0816 03:08:58.454643   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.455536   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-f7frj/"
Cleaning up nodes, machines and infra machines.
I0816 03:08:58.468627   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.469254   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-f7frj/"
I0816 03:08:58.471535   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.472116   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-f7frj/"
I0816 03:08:58.472481   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-mghw9/"
I0816 03:08:58.521155   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:58.521836   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-mghw9/"
I0816 03:08:58.522310   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-8nppb/"
I0816 03:08:58.529103   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="test-cluster-vfs9f" "namespace"="test-mhc-gnjxx" "count"=2
I0816 03:08:58.529187   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-vfs9f" "namespace"="test-mhc-gnjxx" "descendants"="Worker machines: test-mhc-machine-mghw9,test-mhc-machine-8nppb,test-mhc-machine-f7frj" "indirect descendants count"=1
I0816 03:08:58.534653   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="test-cluster-vfs9f" "namespace"="test-mhc-gnjxx" "count"=2
I0816 03:08:58.534828   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-vfs9f" "namespace"="test-mhc-gnjxx" "descendants"="Worker machines: test-mhc-machine-mghw9,test-mhc-machine-8nppb,test-mhc-machine-f7frj" "indirect descendants count"=1
=== RUN   TestMachineHealthCheck_Reconcile/it_marks_unhealthy_machines_for_remediation_when_the_unhealthy_Machines_exceed_MaxUnhealthy
I0816 03:08:58.566881   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-gnjxx/test-mhc-fqdcb/test-mhc-machine-f7frj/"
E0816 03:08:58.580158   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machinehealthchecks.cluster.x-k8s.io \"test-mhc-fqdcb\" not found" "controller"="machinehealthcheck" "name"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx"
inframachine created: test-mhc-machine-infra-9mhcq
machine created: test-mhc-machine-2txf7
I0816 03:08:59.366137   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-vfs9f" "machine"="test-mhc-machine-8nppb" "namespace"="test-mhc-gnjxx" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-w7q9z"}
E0816 03:08:59.437139   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-8nppb\" not found" "controller"="machine" "name"="test-mhc-machine-8nppb" "namespace"="test-mhc-gnjxx"
I0816 03:08:59.580468   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-fqdcb" "namespace"="test-mhc-gnjxx" 
I0816 03:08:59.580594   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:08:59.599653   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:08:59.738283   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:08:59.744579   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:08:59.765721   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:08:59.806837   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:08:59.888399   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:00.050684   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:00.375745   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
E0816 03:09:00.437564   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:09:01.016856   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:01.438092   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-vfs9f" "machine"="test-mhc-machine-f7frj" "namespace"="test-mhc-gnjxx" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-drrg7"}
E0816 03:09:01.502493   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-f7frj\" not found" "controller"="machine" "name"="test-mhc-machine-f7frj" "namespace"="test-mhc-gnjxx"
I0816 03:09:02.298024   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:02.503167   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-vfs9f" "machine"="test-mhc-machine-mghw9" "namespace"="test-mhc-gnjxx" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-f95ng"}
E0816 03:09:02.558911   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-mghw9\" not found" "controller"="machine" "name"="test-mhc-machine-mghw9" "namespace"="test-mhc-gnjxx"
E0816 03:09:03.559889   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
E0816 03:09:03.562639   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-gnjxx/test-cluster-vfs9f"
I0816 03:09:04.569730   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
E0816 03:09:04.572813   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:09:04.573896   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:04.606432   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
node created: test-mhc-node-mq2rv
E0816 03:09:04.661981   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-mq2rv, got []"  "node"="test-mhc-node-mq2rv"
I0816 03:09:04.670263   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
inframachine created: test-mhc-machine-infra-gqznk
machine created: test-mhc-machine-wlxzr
I0816 03:09:04.704977   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:04.725799   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:04.730895   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:04.753029   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:04.794391   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:04.876527   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:05.038067   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:05.359455   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
E0816 03:09:05.580408   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:09:06.001167   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:06.610134   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:06.618292   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:06.636482   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:06.650398   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:06.664008   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
E0816 03:09:06.669155   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:09:06.669172   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:06.701096   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
node created: test-mhc-node-lkh9p
E0816 03:09:06.710373   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-lkh9p, got []"  "node"="test-mhc-node-lkh9p"
I0816 03:09:06.719905   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
inframachine created: test-mhc-machine-infra-2rk9d
I0816 03:09:06.742559   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
machine created: test-mhc-machine-62pmn
I0816 03:09:06.755270   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:07.282522   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
E0816 03:09:07.670125   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:09:08.682081   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:08.702058   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:08.712060   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:08.731797   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:08.753395   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:08.766926   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:08.773615   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
E0816 03:09:08.773731   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:09:08.820087   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
E0816 03:09:08.861225   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-nqn4q, got []"  "node"="test-mhc-node-nqn4q"
node created: test-mhc-node-nqn4q
I0816 03:09:08.869075   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
Cleaning up nodes, machines and infra machines.
I0816 03:09:08.895288   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:08.930377   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
Cleaning up nodes, machines and infra machines.
... skipping 2 lines ...
I0816 03:09:08.959000   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-7bz9w" "namespace"="test-mhc-mmjxq" "descendants"="Worker machines: test-mhc-machine-wlxzr,test-mhc-machine-62pmn,test-mhc-machine-2txf7" "indirect descendants count"=1
I0816 03:09:08.963798   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="test-cluster-7bz9w" "namespace"="test-mhc-mmjxq" "count"=2
I0816 03:09:08.963862   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-7bz9w" "namespace"="test-mhc-mmjxq" "descendants"="Worker machines: test-mhc-machine-2txf7,test-mhc-machine-wlxzr,test-mhc-machine-62pmn" "indirect descendants count"=1
=== RUN   TestMachineHealthCheck_Reconcile/when_a_Machine_has_no_Node_ref_for_less_than_the_NodeStartupTimeout
inframachine created: test-mhc-machine-infra-j2v4p
machine created: test-mhc-machine-nhhgm
E0816 03:09:09.094541   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="[MachineHealthCheck.cluster.x-k8s.io \"test-mhc-g2rjd\" not found, machinehealthchecks.cluster.x-k8s.io \"test-mhc-g2rjd\" not found]" "controller"="machinehealthcheck" "name"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq"
I0816 03:09:09.774270   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-7bz9w" "machine"="test-mhc-machine-2txf7" "namespace"="test-mhc-mmjxq" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-mq2rv"}
E0816 03:09:09.828441   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-2txf7\" not found" "controller"="machine" "name"="test-mhc-machine-2txf7" "namespace"="test-mhc-mmjxq"
I0816 03:09:10.094857   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:10.145692   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:09:10.183574   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-g2rjd" "namespace"="test-mhc-mmjxq" 
I0816 03:09:10.183660   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:10.189100   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:10.828995   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-7bz9w" "machine"="test-mhc-machine-wlxzr" "namespace"="test-mhc-mmjxq" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-lkh9p"}
E0816 03:09:10.872715   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-wlxzr\" not found" "controller"="machine" "name"="test-mhc-machine-wlxzr" "namespace"="test-mhc-mmjxq"
I0816 03:09:11.883886   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-7bz9w" "machine"="test-mhc-machine-62pmn" "namespace"="test-mhc-mmjxq" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-nqn4q"}
E0816 03:09:11.980041   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-62pmn\" not found" "controller"="machine" "name"="test-mhc-machine-62pmn" "namespace"="test-mhc-mmjxq"
E0816 03:09:12.980466   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
E0816 03:09:13.979018   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-mmjxq/test-cluster-7bz9w"
I0816 03:09:13.989976   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
E0816 03:09:13.995233   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:09:13.995635   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
E0816 03:09:14.027215   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-pqx4j, got []"  "node"="test-mhc-node-pqx4j"
node created: test-mhc-node-pqx4j
I0816 03:09:14.035632   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
inframachine created: test-mhc-machine-infra-9kw8h
machine created: test-mhc-machine-bfvhk
I0816 03:09:14.060957   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:14.067290   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
... skipping 4 lines ...
I0816 03:09:15.068435   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.073752   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.089100   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.111190   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.133521   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.140713   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
E0816 03:09:15.170005   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-jpjvl, got []"  "node"="test-mhc-node-jpjvl"
node created: test-mhc-node-jpjvl
I0816 03:09:15.172621   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.181393   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.187819   10501 machine_controller_noderef.go:86] controllers/Machine "msg"="Set Machine's NodeRef" "machine"="test-mhc-machine-bfvhk" "namespace"="test-mhc-cttjt" "noderef"="test-mhc-node-jpjvl"
inframachine created: test-mhc-machine-infra-w29nm
machine created: test-mhc-machine-kcnhk
... skipping 5 lines ...
Cleaning up nodes, machines and infra machines.
I0816 03:09:15.315848   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
Cleaning up nodes, machines and infra machines.
I0816 03:09:15.334023   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.349697   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-2wmtv" "machine"="test-mhc-machine-kcnhk" "namespace"="test-mhc-cttjt" "cause"="noderef is nil" "node"=null
I0816 03:09:15.349741   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.350243   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-cttjt/test-mhc-wfsxx/test-mhc-machine-nhhgm/"
I0816 03:09:15.444288   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:15.444898   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-cttjt/test-mhc-wfsxx/test-mhc-machine-nhhgm/"
I0816 03:09:15.445307   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-cttjt/test-mhc-wfsxx/test-mhc-machine-bfvhk/"
I0816 03:09:15.461626   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="test-cluster-2wmtv" "namespace"="test-mhc-cttjt" "count"=3
I0816 03:09:15.461716   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-2wmtv" "namespace"="test-mhc-cttjt" "descendants"="Worker machines: test-mhc-machine-nhhgm,test-mhc-machine-bfvhk,test-mhc-machine-kcnhk" "indirect descendants count"=0
I0816 03:09:15.467883   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="test-cluster-2wmtv" "namespace"="test-mhc-cttjt" "count"=3
I0816 03:09:15.467961   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-2wmtv" "namespace"="test-mhc-cttjt" "descendants"="Worker machines: test-mhc-machine-nhhgm,test-mhc-machine-bfvhk,test-mhc-machine-kcnhk" "indirect descendants count"=0
=== RUN   TestMachineHealthCheck_Reconcile/when_a_Machine_has_no_Node_ref_for_longer_than_the_NodeStartupTimeout
=== RUN   TestMachineHealthCheck_Reconcile/when_a_Machine's_Node_has_gone_away
=== RUN   TestMachineHealthCheck_Reconcile/should_react_when_a_Node_transitions_to_unhealthy
inframachine created: test-mhc-machine-infra-pj7bh
machine created: test-mhc-machine-ktpxp
E0816 03:09:15.593423   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machinehealthchecks.cluster.x-k8s.io \"test-mhc-wfsxx\" not found" "controller"="machinehealthcheck" "name"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt"
E0816 03:09:15.606178   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-kcnhk\" not found" "controller"="machine" "name"="test-mhc-machine-kcnhk" "namespace"="test-mhc-cttjt"
I0816 03:09:16.594865   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.614162   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-2wmtv" "machine"="test-mhc-machine-nhhgm" "namespace"="test-mhc-cttjt" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-pqx4j"}
I0816 03:09:16.616403   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:09:16.655464   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-wfsxx" "namespace"="test-mhc-cttjt" 
I0816 03:09:16.655566   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.662728   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.663818   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.665578   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
E0816 03:09:16.667103   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-nhhgm\" not found" "controller"="machine" "name"="test-mhc-machine-nhhgm" "namespace"="test-mhc-cttjt"
I0816 03:09:16.667286   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.669650   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.671540   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.673367   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.678911   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:16.680761   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
... skipping 500 lines ...
I0816 03:09:17.830426   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.832315   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.834158   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.838402   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.841181   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.843064   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
E0816 03:09:17.844389   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-bfvhk\" not found" "controller"="machine" "name"="test-mhc-machine-bfvhk" "namespace"="test-mhc-cttjt"
I0816 03:09:17.844992   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.846817   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.848705   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.850513   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.858926   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:17.860799   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
... skipping 404 lines ...
I0816 03:09:18.852260   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.854048   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.854655   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.855783   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.857547   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.858569   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
E0816 03:09:18.858652   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
I0816 03:09:18.858823   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-qzk87/test-mhc-c8lrx/test-mhc-machine-ktpxp/"
I0816 03:09:18.872043   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.872439   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "target"="test-mhc-qzk87/test-mhc-c8lrx/test-mhc-machine-ktpxp/"
node created: test-mhc-node-fm9nm
E0816 03:09:18.960093   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-fm9nm, got []"  "node"="test-mhc-node-fm9nm"
E0816 03:09:18.960107   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-fm9nm, got []"  "node"="test-mhc-node-fm9nm"
I0816 03:09:18.969738   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.993736   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.994804   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.998362   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:18.998830   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Condition Ready on node is reporting status Unknown for more than 5m0s" "reason"="UnhealthyNode" "target"="test-mhc-qzk87/test-mhc-c8lrx/test-mhc-machine-ktpxp/test-mhc-node-fm9nm"
I0816 03:09:19.037246   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:19.037815   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Condition Ready on node is reporting status Unknown for more than 5m0s" "reason"="UnhealthyNode" "target"="test-mhc-qzk87/test-mhc-c8lrx/test-mhc-machine-ktpxp/test-mhc-node-fm9nm"
Cleaning up nodes, machines and infra machines.
I0816 03:09:19.045573   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:19.046009   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="NodeNotFound" "target"="test-mhc-qzk87/test-mhc-c8lrx/test-mhc-machine-ktpxp/"
I0816 03:09:19.060051   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-tmvj2" "namespace"="test-mhc-qzk87" "descendants"="Worker machines: test-mhc-machine-ktpxp" "indirect descendants count"=1
I0816 03:09:19.064549   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-tmvj2" "namespace"="test-mhc-qzk87" "descendants"="Worker machines: test-mhc-machine-ktpxp" "indirect descendants count"=1
=== RUN   TestMachineHealthCheck_Reconcile/when_in_a_MachineSet,_unhealthy_machines_should_be_deleted
I0816 03:09:19.224519   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-c8lrx" "namespace"="test-mhc-qzk87" 
I0816 03:09:19.235033   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" "creating"=1 "need"=1
I0816 03:09:19.235092   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(1) \u003e currentMachineCount(0) )" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" 
I0816 03:09:19.251073   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"mhc-ms-kzsxr-8qg5j\"" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" 
I0816 03:09:19.318257   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:19.343577   10501 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinehealthcheck" "source"={}
I0816 03:09:19.467653   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:19.468467   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
E0816 03:09:19.859149   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:09:20.467796   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
E0816 03:09:20.487148   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-cttjt/test-cluster-2wmtv"
I0816 03:09:20.859727   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-tmvj2" "machine"="test-mhc-machine-ktpxp" "namespace"="test-mhc-qzk87" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-fm9nm"}
E0816 03:09:20.922456   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-ktpxp\" not found" "controller"="machine" "name"="test-mhc-machine-ktpxp" "namespace"="test-mhc-qzk87"
I0816 03:09:21.468986   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:21.932848   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:21.932852   10501 machineset_controller.go:139] controllers/MachineSet "msg"="Reconciliation is paused for this object" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" 
I0816 03:09:21.938089   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:21.938380   10501 machineset_controller.go:139] controllers/MachineSet "msg"="Reconciliation is paused for this object" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" 
I0816 03:09:21.968756   10501 machine_controller_phases.go:278] controllers/Machine "msg"="Infrastructure provider is not ready, requeuing" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
... skipping 4 lines ...
I0816 03:09:21.999291   10501 machineset_controller.go:139] controllers/MachineSet "msg"="Reconciliation is paused for this object" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.006744   10501 machine_controller_phases.go:278] controllers/Machine "msg"="Infrastructure provider is not ready, requeuing" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.006792   10501 machine_controller_noderef.go:45] controllers/Machine "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.012240   10501 machine_controller_phases.go:278] controllers/Machine "msg"="Infrastructure provider is not ready, requeuing" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.012281   10501 machine_controller_noderef.go:45] controllers/Machine "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.470086   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.470540   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1s" "reason"="NodeStartupTimeout" "target"="test-mhc-g5nml/test-mhc-74m66/mhc-ms-kzsxr-8qg5j/"
I0816 03:09:22.477806   10501 machineset_controller.go:139] controllers/MachineSet "msg"="Reconciliation is paused for this object" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.483876   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.484232   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1s" "reason"="NodeStartupTimeout" "target"="test-mhc-g5nml/test-mhc-74m66/mhc-ms-kzsxr-8qg5j/"
I0816 03:09:22.485576   10501 machine_controller_phases.go:278] controllers/Machine "msg"="Infrastructure provider is not ready, requeuing" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.485606   10501 machine_controller_noderef.go:45] controllers/Machine "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.494710   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.495037   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1s" "reason"="NodeStartupTimeout" "target"="test-mhc-g5nml/test-mhc-74m66/mhc-ms-kzsxr-8qg5j/"
I0816 03:09:22.495786   10501 machineset_controller.go:139] controllers/MachineSet "msg"="Reconciliation is paused for this object" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.504429   10501 machine_controller_phases.go:278] controllers/Machine "msg"="Infrastructure provider is not ready, requeuing" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.504545   10501 machine_controller_noderef.go:45] controllers/Machine "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.560494   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="test-cluster-2tn8m" "namespace"="test-mhc-g5nml" "count"=1
I0816 03:09:22.560563   10501 cluster_controller.go:241] controllers/Cluster "msg"="Deleting child" "cluster"="test-cluster-2tn8m" "namespace"="test-mhc-g5nml" "gvk"="cluster.x-k8s.io/v1alpha3, Kind=MachineSet" "name"="mhc-ms-kzsxr"
I0816 03:09:22.590767   10501 machineset_controller.go:237] controllers/MachineSet "msg"="Deleting unhealthy machine" "machineset"="mhc-ms-kzsxr" "namespace"="test-mhc-g5nml" "machine"="mhc-ms-kzsxr-8qg5j"
I0816 03:09:22.617172   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-2tn8m" "namespace"="test-mhc-g5nml" "descendants"="Machine sets: mhc-ms-kzsxr;Worker machines: mhc-ms-kzsxr-8qg5j" "indirect descendants count"=1
I0816 03:09:22.622619   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-2tn8m" "machine"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml" "cause"="cluster is being deleted" "node"=null
I0816 03:09:22.624595   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.625082   10501 machinehealthcheck_controller.go:387] controllers/MachineHealthCheck "msg"="Target has failed health check, marking for remediation"  "message"="Node failed to report startup in 1s" "reason"="NodeStartupTimeout" "target"="test-mhc-g5nml/test-mhc-74m66/mhc-ms-kzsxr-8qg5j/"
I0816 03:09:22.627338   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-74m66" "namespace"="test-mhc-g5nml" 
I0816 03:09:22.628473   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-2tn8m" "namespace"="test-mhc-g5nml" "descendants"="Worker machines: mhc-ms-kzsxr-8qg5j" "indirect descendants count"=1
=== RUN   TestMachineHealthCheck_Reconcile/when_a_machine_is_paused
=== RUN   TestMachineHealthCheck_Reconcile/When_remediationTemplate_is_set_and_node_transitions_to_unhealthy,_new_Remediation_Request_should_be_created
I0816 03:09:22.835540   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
inframachine created: test-mhc-machine-infra-bhcsp
... skipping 3 lines ...
I0816 03:09:22.972302   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:22.979947   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:22.980730   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:22.981309   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:22.983114   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:22.991704   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
E0816 03:09:22.995721   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"mhc-ms-kzsxr-8qg5j\" not found" "controller"="machine" "name"="mhc-ms-kzsxr-8qg5j" "namespace"="test-mhc-g5nml"
I0816 03:09:22.999807   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:23.002307   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:23.004206   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:23.006082   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:23.008651   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:23.010634   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
... skipping 461 lines ...
I0816 03:09:24.002639   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.004496   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.005724   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.006344   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.008187   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.010440   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
E0816 03:09:24.011293   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
node created: test-mhc-node-d2qn6
E0816 03:09:24.014859   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-d2qn6, got []"  "node"="test-mhc-node-d2qn6"
E0816 03:09:24.015055   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-d2qn6, got []"  "node"="test-mhc-node-d2qn6"
E0816 03:09:24.015346   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-d2qn6, got []"  "node"="test-mhc-node-d2qn6"
I0816 03:09:24.109136   10501 machinehealthcheck_controller.go:379] controllers/MachineHealthCheck "msg"="Target has failed health check, creating an external remediation request"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "remediation request name"="test-mhc-machine-vwx4k" "target"="test-mhc-m4w4r/test-mhc-mdkcz/test-mhc-machine-vwx4k/"
E0816 03:09:24.123396   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-qzk87/test-cluster-tmvj2"
I0816 03:09:24.133621   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.157661   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.161225   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.166174   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.168846   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.172612   10501 machinehealthcheck_controller.go:379] controllers/MachineHealthCheck "msg"="Target has failed health check, creating an external remediation request"  "message"="Condition Ready on node is reporting status Unknown for more than 5m0s" "reason"="UnhealthyNode" "remediation request name"="test-mhc-machine-vwx4k" "target"="test-mhc-m4w4r/test-mhc-mdkcz/test-mhc-machine-vwx4k/test-mhc-node-d2qn6"
I0816 03:09:24.203845   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
Cleaning up nodes, machines and infra machines.
I0816 03:09:24.213893   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.224500   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.233820   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-mdkcz" "namespace"="test-mhc-m4w4r" 
I0816 03:09:24.241777   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-8pzr4" "namespace"="test-mhc-m4w4r" "descendants"="Worker machines: test-mhc-machine-vwx4k" "indirect descendants count"=1
... skipping 214 lines ...
I0816 03:09:25.073039   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.074918   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.076734   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.078911   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.081172   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.082968   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
E0816 03:09:25.084166   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-vwx4k\" not found" "controller"="machine" "name"="test-mhc-machine-vwx4k" "namespace"="test-mhc-m4w4r"
I0816 03:09:25.084743   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.089473   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.091548   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.093699   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.096075   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:25.099114   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
... skipping 406 lines ...
I0816 03:09:26.093304   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.093972   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.095710   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.097629   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.099759   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.100772   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
E0816 03:09:26.102073   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
I0816 03:09:26.105408   10501 machinehealthcheck_controller.go:379] controllers/MachineHealthCheck "msg"="Target has failed health check, creating an external remediation request"  "message"="Node failed to report startup in 1ms" "reason"="NodeStartupTimeout" "remediation request name"="test-mhc-machine-mxl6r" "target"="test-mhc-97brf/test-mhc-5gj7k/test-mhc-machine-mxl6r/"
node created: test-mhc-node-vbmw7
E0816 03:09:26.125375   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-vbmw7, got []"  "node"="test-mhc-node-vbmw7"
E0816 03:09:26.125375   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-vbmw7, got []"  "node"="test-mhc-node-vbmw7"
E0816 03:09:26.125396   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node test-mhc-node-vbmw7, got []"  "node"="test-mhc-node-vbmw7"
I0816 03:09:26.130453   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.140821   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.172461   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.177235   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.181918   10501 machinehealthcheck_controller.go:379] controllers/MachineHealthCheck "msg"="Target has failed health check, creating an external remediation request"  "message"="Condition Ready on node is reporting status Unknown for more than 5m0s" "reason"="UnhealthyNode" "remediation request name"="test-mhc-machine-mxl6r" "target"="test-mhc-97brf/test-mhc-5gj7k/test-mhc-machine-mxl6r/test-mhc-node-vbmw7"
I0816 03:09:26.203521   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.212424   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.216127   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.243719   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
Cleaning up nodes, machines and infra machines.
I0816 03:09:26.251040   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
I0816 03:09:26.258601   10501 machinehealthcheck_controller.go:379] controllers/MachineHealthCheck "msg"="Target has failed health check, creating an external remediation request"  "message"="" "reason"="NodeNotFound" "remediation request name"="test-mhc-machine-mxl6r" "target"="test-mhc-97brf/test-mhc-5gj7k/test-mhc-machine-mxl6r/"
I0816 03:09:26.277333   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-bnbr7" "namespace"="test-mhc-97brf" "descendants"="Worker machines: test-mhc-machine-mxl6r" "indirect descendants count"=1
I0816 03:09:26.283731   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster-bnbr7" "namespace"="test-mhc-97brf" "descendants"="Worker machines: test-mhc-machine-mxl6r" "indirect descendants count"=1
E0816 03:09:26.293507   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machinehealthchecks.cluster.x-k8s.io \"test-mhc-5gj7k\" not found" "controller"="machinehealthcheck" "name"="test-mhc-5gj7k" "namespace"="test-mhc-97brf"
--- PASS: TestMachineHealthCheck_Reconcile (39.11s)
    --- PASS: TestMachineHealthCheck_Reconcile/it_should_ensure_the_correct_cluster-name_label_when_no_existing_labels_exist (0.04s)
    --- PASS: TestMachineHealthCheck_Reconcile/it_should_ensure_the_correct_cluster-name_label_when_the_label_has_the_wrong_value (0.05s)
    --- PASS: TestMachineHealthCheck_Reconcile/it_should_ensure_the_correct_cluster-name_label_when_other_labels_are_present (0.04s)
    --- PASS: TestMachineHealthCheck_Reconcile/it_should_ensure_an_owner_reference_is_present_when_no_existing_ones_exist (1.06s)
    --- PASS: TestMachineHealthCheck_Reconcile/it_should_ensure_an_owner_reference_is_present_when_modifying_existing_ones (0.33s)
... skipping 10 lines ...
    --- SKIP: TestMachineHealthCheck_Reconcile/when_a_machine_is_paused (0.00s)
        machinehealthcheck_controller_test.go:933: skipping until made stable
    --- PASS: TestMachineHealthCheck_Reconcile/When_remediationTemplate_is_set_and_node_transitions_to_unhealthy,_new_Remediation_Request_should_be_created (1.62s)
    --- PASS: TestMachineHealthCheck_Reconcile/When_remediationTemplate_is_set_and_node_transitions_back_to_healthy,_new_Remediation_Request_should_be_deleted (2.04s)
=== RUN   TestClusterToMachineHealthCheck
=== RUN   TestClusterToMachineHealthCheck/when_the_object_passed_isn't_a_cluster
E0816 03:09:26.294686   10501 machinehealthcheck_controller.go:416]  "msg"="expected a Cluster" "error"="incorrect type"  "type"="handler.MapObject"
=== RUN   TestClusterToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_the_Cluster_in_the_same_namespace
=== RUN   TestClusterToMachineHealthCheck/when_2_MachineHealthChecks_exists_for_the_Cluster_in_the_same_namespace
=== RUN   TestClusterToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_another_Cluster_in_the_same_namespace
=== RUN   TestClusterToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_another_Cluster_in_another_namespace
--- PASS: TestClusterToMachineHealthCheck (0.00s)
    --- PASS: TestClusterToMachineHealthCheck/when_the_object_passed_isn't_a_cluster (0.00s)
    --- PASS: TestClusterToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_the_Cluster_in_the_same_namespace (0.00s)
    --- PASS: TestClusterToMachineHealthCheck/when_2_MachineHealthChecks_exists_for_the_Cluster_in_the_same_namespace (0.00s)
    --- PASS: TestClusterToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_another_Cluster_in_the_same_namespace (0.00s)
    --- PASS: TestClusterToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_another_Cluster_in_another_namespace (0.00s)
=== RUN   TestMachineToMachineHealthCheck
=== RUN   TestMachineToMachineHealthCheck/when_the_object_passed_isn't_a_machine
E0816 03:09:26.297242   10501 machinehealthcheck_controller.go:445]  "msg"="expected a Machine" "error"="incorrect type"  "type"="handler.MapObject"
=== RUN   TestMachineToMachineHealthCheck/when_a_MachineHealthCheck_matches_labels_for_the_Machine_in_the_same_namespace
=== RUN   TestMachineToMachineHealthCheck/when_2_MachineHealthChecks_match_labels_for_the_Machine_in_the_same_namespace
=== RUN   TestMachineToMachineHealthCheck/when_a_MachineHealthCheck_does_not_match_labels_for_the_Machine_in_the_same_namespace
=== RUN   TestMachineToMachineHealthCheck/when_a_MachineHealthCheck_matches_labels_for_the_Machine_in_another_namespace
--- PASS: TestMachineToMachineHealthCheck (0.00s)
    --- PASS: TestMachineToMachineHealthCheck/when_the_object_passed_isn't_a_machine (0.00s)
    --- PASS: TestMachineToMachineHealthCheck/when_a_MachineHealthCheck_matches_labels_for_the_Machine_in_the_same_namespace (0.00s)
    --- PASS: TestMachineToMachineHealthCheck/when_2_MachineHealthChecks_match_labels_for_the_Machine_in_the_same_namespace (0.00s)
    --- PASS: TestMachineToMachineHealthCheck/when_a_MachineHealthCheck_does_not_match_labels_for_the_Machine_in_the_same_namespace (0.00s)
    --- PASS: TestMachineToMachineHealthCheck/when_a_MachineHealthCheck_matches_labels_for_the_Machine_in_another_namespace (0.00s)
=== RUN   TestNodeToMachineHealthCheck
=== RUN   TestNodeToMachineHealthCheck/when_the_object_passed_isn't_a_Node
E0816 03:09:26.300361   10501 machinehealthcheck_controller.go:474]  "msg"="expected a Node" "error"="incorrect type"  "type"="handler.MapObject"
=== RUN   TestNodeToMachineHealthCheck/when_no_Machine_exists_for_the_Node
E0816 03:09:26.300842   10501 machinehealthcheck_controller.go:480]  "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node node1, got []"  "node"="node1"
=== RUN   TestNodeToMachineHealthCheck/when_two_Machines_exist_for_the_Node
E0816 03:09:26.301515   10501 machinehealthcheck_controller.go:480]  "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node node1, got [machine1 machine2]"  "node"="node1"
=== RUN   TestNodeToMachineHealthCheck/when_no_MachineHealthCheck_exists_for_the_Node_in_the_Machine's_namespace
=== RUN   TestNodeToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_the_Node_in_the_Machine's_namespace
=== RUN   TestNodeToMachineHealthCheck/when_two_MachineHealthChecks_exist_for_the_Node_in_the_Machine's_namespace
=== RUN   TestNodeToMachineHealthCheck/when_a_MachineHealthCheck_exists_for_the_Node,_but_not_in_the_Machine's_cluster
--- PASS: TestNodeToMachineHealthCheck (0.01s)
    --- PASS: TestNodeToMachineHealthCheck/when_the_object_passed_isn't_a_Node (0.00s)
... skipping 31 lines ...
    --- PASS: TestGetMaxUnhealthy/when_maxUnhealthy_is_nil (0.00s)
    --- PASS: TestGetMaxUnhealthy/when_maxUnhealthy_is_not_an_int_or_percentage (0.00s)
    --- PASS: TestGetMaxUnhealthy/when_maxUnhealthy_is_an_int (0.00s)
    --- PASS: TestGetMaxUnhealthy/when_maxUnhealthy_is_a_40%_(of_5) (0.00s)
    --- PASS: TestGetMaxUnhealthy/when_maxUnhealthy_is_a_60%_(of_7) (0.00s)
=== RUN   TestPatchTargets
I0816 03:09:26.305630   10501 machinehealthcheck_controller.go:387]  "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="" "target"="default/mhc/machine1/"
I0816 03:09:26.306394   10501 machinehealthcheck_controller.go:387]  "msg"="Target has failed health check, marking for remediation"  "message"="" "reason"="" "target"="default/mhc/machine2/"
E0816 03:09:26.309103   10501 machinehealthcheck_controller.go:320]  "msg"="failed to patch healthy machine status for machine" "error"="machines.cluster.x-k8s.io \"machine1\" not found"  "machine"="machine1"
--- PASS: TestPatchTargets (0.01s)
=== RUN   TestGetTargetsFromMHC
=== RUN   TestGetTargetsFromMHC/with_no_matching_machines
=== RUN   TestGetTargetsFromMHC/when_a_machine's_node_is_missing
=== RUN   TestGetTargetsFromMHC/when_a_machine's_labels_do_not_match_the_selector
=== RUN   TestGetTargetsFromMHC/with_multiple_machines,_should_match_correct_nodes
... skipping 26 lines ...
--- PASS: TestMachineSetOwnerReference (0.01s)
    --- PASS: TestMachineSetOwnerReference/should_add_cluster_owner_reference_to_machine_set (0.00s)
    --- PASS: TestMachineSetOwnerReference/should_not_add_cluster_owner_reference_if_machine_is_owned_by_a_machine_deployment (0.00s)
=== RUN   TestMachineSetReconcile
=== RUN   TestMachineSetReconcile/ignore_machine_sets_marked_for_deletion
=== RUN   TestMachineSetReconcile/records_event_if_reconcile_fails
E0816 03:09:26.340145   10501 machineset_controller.go:151]  "msg"="Failed to reconcile MachineSet" "error"="failed to calculate MachineSet's Status: failed to calculate status for MachineSet default/machineset1: invalid label key \"--$-invalid\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')" "machineset"="machineset1" "namespace"="default" 
--- PASS: TestMachineSetReconcile (0.00s)
    --- PASS: TestMachineSetReconcile/ignore_machine_sets_marked_for_deletion (0.00s)
    --- PASS: TestMachineSetReconcile/records_event_if_reconcile_fails (0.00s)
=== RUN   TestMachineSetToMachines
=== RUN   TestMachineSetToMachines/should_return_empty_request_when_controller_is_set
=== RUN   TestMachineSetToMachines/should_return_nil_if_machine_has_no_owner_reference
... skipping 8 lines ...
--- PASS: TestAdoptOrphan (0.00s)
=== RUN   TestHasMatchingLabels
=== RUN   TestHasMatchingLabels/machine_set_and_machine_have_matching_labels
=== RUN   TestHasMatchingLabels/machine_set_and_machine_do_not_have_matching_labels
=== RUN   TestHasMatchingLabels/machine_set_has_empty_selector
=== RUN   TestHasMatchingLabels/machine_set_has_bad_selector
E0816 03:09:26.342981   10501 machineset_controller.go:569]  "msg"="Unable to convert selector" "error"="\"bad-operator\" is not a valid pod selector operator" "machine"="match" "machineset"="" "namespace"="" 
--- PASS: TestHasMatchingLabels (0.00s)
    --- PASS: TestHasMatchingLabels/machine_set_and_machine_have_matching_labels (0.00s)
    --- PASS: TestHasMatchingLabels/machine_set_and_machine_do_not_have_matching_labels (0.00s)
    --- PASS: TestHasMatchingLabels/machine_set_has_empty_selector (0.00s)
    --- PASS: TestHasMatchingLabels/machine_set_has_bad_selector (0.00s)
=== RUN   TestMachineToDelete
... skipping 65 lines ...
I0816 03:09:26.526380   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 2 with name \"ms-57879-tcp9m\"" "machineset"="ms-57879" "namespace"="ms-test" 
I0816 03:09:26.526445   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 2 of 2, ( spec.replicas(2) \u003e currentMachineCount(0) )" "machineset"="ms-57879" "namespace"="ms-test" 
I0816 03:09:26.552274   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 2 of 2 with name \"ms-57879-phvcl\"" "machineset"="ms-57879" "namespace"="ms-test" 
I0816 03:09:26.676088   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="ms-57879" "namespace"="ms-test" "creating"=1 "need"=2
I0816 03:09:26.676145   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(2) \u003e currentMachineCount(1) )" "machineset"="ms-57879" "namespace"="ms-test" 
I0816 03:09:26.702108   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"ms-57879-6klnl\"" "machineset"="ms-57879" "namespace"="ms-test" 
E0816 03:09:26.808079   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.808470   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.808496   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.923252   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.923252   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.923324   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.923366   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.923428   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.923468   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-6klnl-lwr5s, got []"  "node"="ms-57879-6klnl-lwr5s"
E0816 03:09:26.981880   10501 machineset_controller.go:625] controllers/MachineSet "msg"="Unable to retrieve Node status" "error"="error retrieving node ms-57879-6klnl-lwr5s for machine ms-test/ms-57879-6klnl: the cache is not started, can not read objects" "machineset"="ms-57879" "namespace"="ms-test" 
E0816 03:09:26.992141   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:26.992407   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:26.993470   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:26.998857   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:26.998943   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:27.001683   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:27.001862   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:27.001939   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
E0816 03:09:27.002043   10501 machinehealthcheck_controller.go:480] controllers/MachineHealthCheck "msg"="Unable to retrieve machine from node" "error"="expecting one machine for node ms-57879-tcp9m-2qzr9, got []"  "node"="ms-57879-tcp9m-2qzr9"
I0816 03:09:27.102711   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test-cluster-bnbr7" "machine"="test-mhc-machine-mxl6r" "namespace"="test-mhc-97brf" "cause"="cluster is being deleted" "node"={"name":"test-mhc-node-vbmw7"}
I0816 03:09:27.178510   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster" "namespace"="ms-test" "descendants"="Worker machines: ms-57879-tcp9m,ms-57879-6klnl" "indirect descendants count"=2
•I0816 03:09:27.186419   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster" "namespace"="ms-test" "descendants"="Worker machines: ms-57879-tcp9m,ms-57879-6klnl" "indirect descendants count"=2
E0816 03:09:27.219358   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machines.cluster.x-k8s.io \"test-mhc-machine-mxl6r\" not found" "controller"="machine" "name"="test-mhc-machine-mxl6r" "namespace"="test-mhc-97brf"
I0816 03:09:27.294051   10501 machinehealthcheck_controller.go:109] controllers/MachineHealthCheck "msg"="Reconciling" "machinehealthcheck"="test-mhc-5gj7k" "namespace"="test-mhc-97brf" 
•E0816 03:09:27.340021   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusters.cluster.x-k8s.io \"test1-7cfh6\" not found" "controller"="cluster" "name"="test1-7cfh6" "namespace"="default"
E0816 03:09:28.372743   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-g5nml/test-cluster-2tn8m"
•I0816 03:09:28.471645   10501 conversion.go:64] controllers/Cluster "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "cluster"="test2-2dptz" "namespace"="default" "err"="you requested PartialObjectMetadata, but the requested object is a list (*apiextensions.CustomResourceDefinitionList)"
I0816 03:09:28.591888   10501 conversion.go:64] controllers/Cluster "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "cluster"="test2-2dptz" "namespace"="default" "err"="you requested PartialObjectMetadata, but the requested object is a list (*apiextensions.CustomResourceDefinitionList)"
E0816 03:09:28.598223   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to find a CustomResourceDefinition for /, Kind= with contract \"cluster.x-k8s.io/v1alpha3\"" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
E0816 03:09:29.622492   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test-too\" for Cluster default/test2-2dptz: failed to retrieve  external object \"default\"/\"test-too\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
•E0816 03:09:30.640935   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-m4w4r/test-cluster-8pzr4"
E0816 03:09:30.660279   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusters.cluster.x-k8s.io \"test3-dq645\" not found" "controller"="cluster" "name"="test3-dq645" "namespace"="default"
E0816 03:09:31.661204   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test-too\" for Cluster default/test2-2dptz: failed to retrieve  external object \"default\"/\"test-too\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
E0816 03:09:32.686281   10501 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="test-mhc-97brf/test-cluster-bnbr7"
E0816 03:09:32.686462   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test-too\" for Cluster default/test2-2dptz: failed to retrieve  external object \"default\"/\"test-too\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
•I0816 03:09:33.687022   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster" "namespace"="ms-test" "descendants"="Worker machines: ms-57879-tcp9m,ms-57879-6klnl" "indirect descendants count"=2
E0816 03:09:33.691432   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test\" for Cluster default/test4-6lx7h: failed to retrieve  external object \"default\"/\"test\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test4-6lx7h" "namespace"="default"
E0816 03:09:34.251928   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-zbbt6\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-zbbt6" "namespace"="test-mhc-v55fz"
E0816 03:09:34.702807   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test-too\" for Cluster default/test2-2dptz: failed to retrieve  external object \"default\"/\"test-too\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
•E0816 03:09:35.703632   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test\" for Cluster default/test4-6lx7h: failed to retrieve  external object \"default\"/\"test\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test4-6lx7h" "namespace"="default"
E0816 03:09:36.342476   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get cluster \"test-cluster-zcf2h\" for machine \"test-mhc-machine-d88zp\" in namespace \"test-mhc-v55fz\": Cluster.cluster.x-k8s.io \"test-cluster-zcf2h\" not found" "controller"="machine" "name"="test-mhc-machine-d88zp" "namespace"="test-mhc-v55fz"
E0816 03:09:36.713912   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test-too\" for Cluster default/test2-2dptz: failed to retrieve  external object \"default\"/\"test-too\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
I0816 03:09:37.377949   10501 conversion.go:64] controllers/Machine "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="test6-rh9jw" "namespace"="default" "err"="you requested PartialObjectMetadata, but the requested object is a list (*apiextensions.CustomResourceDefinitionList)"
E0816 03:09:37.389606   10501 machine_controller_noderef.go:73] controllers/Machine "msg"="Failed to retrieve Node by ProviderID" "error"="the cache is not started, can not read objects" "machine"="test6-rh9jw" "namespace"="default" 
E0816 03:09:37.406643   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="[failed to find a CustomResourceDefinition for /, Kind= with contract \"cluster.x-k8s.io/v1alpha3\", the cache is not started, can not read objects]" "controller"="machine" "name"="test6-rh9jw" "namespace"="default"
E0816 03:09:37.714691   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test\" for Cluster default/test4-6lx7h: failed to retrieve  external object \"default\"/\"test\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test4-6lx7h" "namespace"="default"
I0816 03:09:38.423884   10501 conversion.go:64] controllers/Machine "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="test6-rh9jw" "namespace"="default" "err"="you requested PartialObjectMetadata, but the requested object is a list (*apiextensions.CustomResourceDefinitionList)"
I0816 03:09:38.424147   10501 machine_controller_noderef.go:86] controllers/Machine "msg"="Set Machine's NodeRef" "machine"="test6-rh9jw" "namespace"="default" "noderef"="id-node-1"
E0816 03:09:38.440087   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to find a CustomResourceDefinition for /, Kind= with contract \"cluster.x-k8s.io/v1alpha3\"" "controller"="machine" "name"="test6-rh9jw" "namespace"="default"
E0816 03:09:38.719925   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test-too\" for Cluster default/test2-2dptz: failed to retrieve  external object \"default\"/\"test-too\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
•I0816 03:09:38.844650   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "creating"=2 "need"=2
I0816 03:09:38.844712   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 2, ( spec.replicas(2) \u003e currentMachineCount(0) )" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" 
I0816 03:09:38.857323   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 2 with name \"md-9r9kz-6657c7fddb-4rmvb\"" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" 
I0816 03:09:38.857378   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 2 of 2, ( spec.replicas(2) \u003e currentMachineCount(0) )" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" 
I0816 03:09:38.870868   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 2 of 2 with name \"md-9r9kz-6657c7fddb-fkj97\"" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" 
I0816 03:09:38.931828   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "creating"=2 "need"=2
... skipping 4 lines ...
I0816 03:09:39.048926   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "creating"=1 "need"=3
I0816 03:09:39.048969   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(3) \u003e currentMachineCount(2) )" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" 
I0816 03:09:39.060880   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"md-9r9kz-6657c7fddb-s72cn\"" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" 
I0816 03:09:39.160917   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "creating"=1 "need"=1
I0816 03:09:39.160982   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(1) \u003e currentMachineCount(0) )" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" 
I0816 03:09:39.174231   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"md-9r9kz-cdfc6fd6c-gdnjz\"" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" 
E0816 03:09:39.383268   10501 machineset_controller.go:625] controllers/MachineSet "msg"="Unable to retrieve Node status" "error"="error retrieving node md-9r9kz-cdfc6fd6c-gdnjz-5j64d for machine md-test/md-9r9kz-cdfc6fd6c-gdnjz: the cache is not started, can not read objects" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" 
I0816 03:09:39.414313   10501 machineset_controller.go:388] controllers/MachineSet "msg"="Too many replicas" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "deleting"=1 "need"=2
I0816 03:09:39.414368   10501 machineset_controller.go:394] controllers/MachineSet "msg"="Found delete policy" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "delete-policy"="Random"
I0816 03:09:39.419429   10501 machineset_controller.go:405] controllers/MachineSet "msg"="Deleted machine" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "machine"="md-9r9kz-6657c7fddb-67jz5"
I0816 03:09:39.440843   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test6-drftz" "machine"="test6-rh9jw" "namespace"="default" "cause"="cluster is being deleted" "node"={"kind":"Node","name":"id-node-1","uid":"e786328b-0681-4dd2-8c6c-afff33f40bb2","apiVersion":"v1"}
E0816 03:09:39.464673   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get /, Kind= \"\" for Machine \"test6-rh9jw\" in namespace \"default\": failed to retrieve  external object \"default\"/\"\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="machine" "name"="test6-rh9jw" "namespace"="default"
I0816 03:09:39.531344   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "creating"=1 "need"=2
I0816 03:09:39.531407   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(2) \u003e currentMachineCount(1) )" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" 
I0816 03:09:39.549392   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"md-9r9kz-cdfc6fd6c-tl6fq\"" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" 
I0816 03:09:39.645623   10501 machineset_controller.go:388] controllers/MachineSet "msg"="Too many replicas" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "deleting"=1 "need"=1
I0816 03:09:39.645664   10501 machineset_controller.go:394] controllers/MachineSet "msg"="Found delete policy" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "delete-policy"="Random"
I0816 03:09:39.658244   10501 machineset_controller.go:405] controllers/MachineSet "msg"="Deleted machine" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "machine"="md-9r9kz-6657c7fddb-7lb5q"
I0816 03:09:39.680484   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "creating"=1 "need"=3
I0816 03:09:39.680542   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(3) \u003e currentMachineCount(2) )" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" 
I0816 03:09:39.690026   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"md-9r9kz-cdfc6fd6c-vb9dv\"" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" 
E0816 03:09:39.720676   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test\" for Cluster default/test4-6lx7h: failed to retrieve  external object \"default\"/\"test\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test4-6lx7h" "namespace"="default"
I0816 03:09:39.796662   10501 machineset_controller.go:388] controllers/MachineSet "msg"="Too many replicas" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "deleting"=1 "need"=0
I0816 03:09:39.796772   10501 machineset_controller.go:394] controllers/MachineSet "msg"="Found delete policy" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "delete-policy"="Random"
I0816 03:09:39.801517   10501 machineset_controller.go:405] controllers/MachineSet "msg"="Deleted machine" "machineset"="md-9r9kz-6657c7fddb" "namespace"="md-test" "machine"="md-9r9kz-6657c7fddb-s72cn"
I0816 03:09:40.057311   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" "creating"=1 "need"=1
I0816 03:09:40.057368   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(1) \u003e currentMachineCount(0) )" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" 
I0816 03:09:40.070676   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"md-9r9kz-74d45c49c5-wxpwh\"" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" 
... skipping 4 lines ...
I0816 03:09:40.315566   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(2) \u003e currentMachineCount(1) )" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" 
I0816 03:09:40.327171   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"md-9r9kz-74d45c49c5-6xl8l\"" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" 
I0816 03:09:40.473773   10501 machineset_controller.go:388] controllers/MachineSet "msg"="Too many replicas" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "deleting"=1 "need"=1
I0816 03:09:40.473813   10501 machineset_controller.go:394] controllers/MachineSet "msg"="Found delete policy" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "delete-policy"="Random"
I0816 03:09:40.477713   10501 machineset_controller.go:405] controllers/MachineSet "msg"="Deleted machine" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "machine"="md-9r9kz-cdfc6fd6c-gdnjz"
I0816 03:09:40.509546   10501 machine_controller.go:305] controllers/Machine "msg"="Deleting Kubernetes Node associated with Machine is not allowed" "cluster"="test6-drftz" "machine"="test6-rh9jw" "namespace"="default" "cause"="cluster is being deleted" "node"={"kind":"Node","name":"id-node-1","uid":"e786328b-0681-4dd2-8c6c-afff33f40bb2","apiVersion":"v1"}
E0816 03:09:40.510436   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get /, Kind= \"\" for Machine \"test6-rh9jw\" in namespace \"default\": failed to retrieve  external object \"default\"/\"\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="machine" "name"="test6-rh9jw" "namespace"="default"
I0816 03:09:40.588992   10501 machineset_controller.go:316] controllers/MachineSet "msg"="Too few replicas" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" "creating"=1 "need"=3
I0816 03:09:40.589035   10501 machineset_controller.go:324] controllers/MachineSet "msg"="Creating machine 1 of 1, ( spec.replicas(3) \u003e currentMachineCount(2) )" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" 
I0816 03:09:40.600256   10501 machineset_controller.go:378] controllers/MachineSet "msg"="Created machine 1 of 1 with name \"md-9r9kz-74d45c49c5-d7kgn\"" "machineset"="md-9r9kz-74d45c49c5" "namespace"="md-test" 
I0816 03:09:40.721397   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test-cluster" "namespace"="ms-test" "descendants"="Worker machines: ms-57879-tcp9m,ms-57879-6klnl" "indirect descendants count"=2
I0816 03:09:40.721876   10501 cluster_controller.go:223] controllers/Cluster "msg"="Cluster still has children - deleting them first" "cluster"="test6-drftz" "namespace"="default" "count"=1
I0816 03:09:40.721930   10501 cluster_controller.go:256] controllers/Cluster "msg"="Cluster still has descendants - need to requeue" "cluster"="test6-drftz" "namespace"="default" "descendants"="Control plane machines: test6-rh9jw" "indirect descendants count"=0
E0816 03:09:40.751060   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get  \"test-too\" for Cluster default/test2-2dptz: failed to retrieve  external object \"default\"/\"test-too\": Object 'Kind' is missing in 'unstructured object has no kind'" "controller"="cluster" "name"="test2-2dptz" "namespace"="default"
I0816 03:09:40.776244   10501 machineset_controller.go:388] controllers/MachineSet "msg"="Too many replicas" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "deleting"=1 "need"=0
I0816 03:09:40.776305   10501 machineset_controller.go:394] controllers/MachineSet "msg"="Found delete policy" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "delete-policy"="Random"
I0816 03:09:40.784311   10501 machineset_controller.go:405] controllers/MachineSet "msg"="Deleted machine" "machineset"="md-9r9kz-cdfc6fd6c" "namespace"="md-test" "machine"="md-9r9kz-cdfc6fd6c-tl6fq"
E0816 03:09:40.851670   10501 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="machinedeployments.cluster.x-k8s.io \"md-9r9kz\" not found" "controller"="machinedeployment" "name"="md-9r9kz" "namespace"="md-test"
•I0816 03:09:40.861895   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
I0816 03:09:40.863269   10501 machine_controller_phases.go:219]  "msg"="Bootstrap provider is not ready, requeuing" "machine"="machine-test" "namespace"="default" 
I0816 03:09:40.863354   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
I0816 03:09:40.864375   10501 machine_controller_phases.go:278]  "msg"="Infrastructure provider is not ready, requeuing" "machine"="machine-test" "namespace"="default" 
I0816 03:09:40.864424   10501 machine_controller_noderef.go:45]  "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="machine-test" "namespace"="default" 
•I0816 03:09:40.865365   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
... skipping 13 lines ...
I0816 03:09:40.880136   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
•I0816 03:09:40.882665   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
I0816 03:09:40.883761   10501 conversion.go:64]  "msg"="Cannot retrieve CRD with metadata only client, falling back to slower listing" "machine"="machine-test" "namespace"="default" "err"="cannot create a metadata client without a rest config"
I0816 03:09:40.885328   10501 machine_controller_phases.go:278]  "msg"="Infrastructure provider is not ready, requeuing" "machine"="machine-test" "namespace"="default" 
I0816 03:09:40.885400   10501 machine_controller_noderef.go:45]  "msg"="Cannot reconcile Machine's Node, no valid ProviderID yet" "machine"="machine-test" "namespace"="default" 
•I0816 03:09:40.886359   10501 machine_controller.go:327]  "msg"="Draining node" "cluster"="test-cluster" "machine"="machine-test" "namespace"="default" "node"="machine-test-node"
E0816 03:09:40.889115   10501 machine_controller.go:494]  "msg"="Error creating a remote client while deleting Machine, won't retry" "error"="failed to create REST configuration for Cluster default/test-cluster: invalid configuration: no configuration has been provided" "cluster"="test-cluster" "machine"="machine-test" "namespace"="default" "node"="machine-test-node" 
•

Ran 16 of 16 Specs in 14.546 seconds
SUCCESS! -- 16 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestGinkgoSuite (14.55s)
PASS
Tearing down test suite
I0816 03:09:40.890942   10501 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="machinehealthcheck" 
I0816 03:09:40.890997   10501 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="cluster" 
I0816 03:09:40.891087   10501 server.go:208] controller-runtime/webhook "msg"="shutting down webhook server"  
I0816 03:09:40.891177   10501 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="machine" 
I0816 03:09:40.891200   10501 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="cluster" 
I0816 03:09:40.891224   10501 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="machineset" 
I0816 03:09:40.891251   10501 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="machinedeployment" 
E0816 03:09:40.932803   10501 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105: Failed to watch *v1.Node: Get http://127.0.0.1:39579/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=1251&timeout=10s&timeoutSeconds=522&watch=true: dial tcp 127.0.0.1:39579: connect: connection refused
E0816 03:09:40.932806   10501 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105: Failed to watch *v1.Node: Get http://127.0.0.1:39579/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=1251&timeout=10s&timeoutSeconds=540&watch=true: dial tcp 127.0.0.1:39579: connect: connection refused
E0816 03:09:40.933374   10501 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105: Failed to watch *v1.Node: Get http://127.0.0.1:39579/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=1251&timeout=10s&timeoutSeconds=519&watch=true: dial tcp 127.0.0.1:39579: connect: connection refused
ok  	sigs.k8s.io/cluster-api/controllers	72.648s
=== RUN   TestGetResourceFound
--- PASS: TestGetResourceFound (0.00s)
=== RUN   TestGetResourceNotFound
--- PASS: TestGetResourceNotFound (0.00s)
=== RUN   TestCloneTemplateResourceNotFound
... skipping 261 lines ...
I0816 03:08:42.998599   11060 certwatcher.go:83] controller-runtime/certwatcher "msg"="Starting certificate watcher"  
I0816 03:08:43.023727   11060 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinedeployment" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"clusterName":"","selector":{},"template":{"metadata":{},"spec":{"clusterName":"","bootstrap":{},"infrastructureRef":{}}}},"status":{}}}
I0816 03:08:43.224269   11060 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="machinedeployment" 
I0816 03:08:43.224575   11060 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="machinedeployment" "worker count"=1
I0816 03:08:43.469767   11060 controller.go:132] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinedeployment" "source"={}
•I0816 03:08:43.922674   11060 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="machinedeployment" 
E0816 03:08:45.981923   11060 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="timed out waiting for the condition"  "cluster"="cluster-cache-test-8h99l/test-cluster"
•E0816 03:08:46.455188   11060 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="the server could not find the requested resource"  "cluster"="cluster-cache-test-h9p4h/test-cluster"
•E0816 03:08:46.993879   11060 cluster_cache.go:366]  "msg"="Error health checking cluster" "error"="Get http://127.0.0.1:34511/?timeout=50ms: dial tcp 127.0.0.1:34511: connect: connection refused"  "cluster"="cluster-cache-test-vfzd4/test-cluster"
•I0816 03:08:47.123396   11060 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="cluster" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}
I0816 03:08:47.224157   11060 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="cluster" 
I0816 03:08:47.224215   11060 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="cluster" "worker count"=1
•E0816 03:08:47.401646   11060 certwatcher.go:143] controller-runtime/certwatcher "msg"="error re-watching file" "error"="no such file or directory"  
E0816 03:08:47.401707   11060 certwatcher.go:148] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-341478554/tls.crt: no such file or directory"  
I0816 03:08:47.401731   11060 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="cluster" 
I0816 03:08:47.401764   11060 server.go:208] controller-runtime/webhook "msg"="shutting down webhook server"  
E0816 03:08:47.457670   11060 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.9/tools/cache/reflector.go:105: Failed to watch *v1alpha3.Cluster: Get http://127.0.0.1:34065/apis/cluster.x-k8s.io/v1alpha3/clusters?allowWatchBookmarks=true&resourceVersion=143&timeout=10s&timeoutSeconds=478&watch=true: dial tcp 127.0.0.1:34065: connect: connection refused


Ran 5 of 5 Specs in 16.375 seconds
SUCCESS! -- 5 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestGinkgoSuite (16.38s)
PASS
ok  	sigs.k8s.io/cluster-api/controllers/remote	16.533s
?   	sigs.k8s.io/cluster-api/controllers/remote/fake	[no test files]
?   	sigs.k8s.io/cluster-api/controlplane/kubeadm	[no test files]
=== RUN   TestKubeadmControlPlaneDefault
... skipping 208 lines ...
=== RUN   TestReconcileNoClusterOwnerRef
I0816 03:08:55.466218   11609 controller.go:130]  "msg"="Cluster Controller has not yet set OwnerRef" "kubeadmControlPlane"="foo" "namespace"="test" 
--- PASS: TestReconcileNoClusterOwnerRef (0.00s)
=== RUN   TestReconcileNoKCP
--- PASS: TestReconcileNoKCP (0.00s)
=== RUN   TestReconcileNoCluster
E0816 03:08:55.467668   11609 controller.go:126]  "msg"="Failed to retrieve owner Cluster from the API Server" "error"="clusters.cluster.x-k8s.io \"foo\" not found" "kubeadmControlPlane"="foo" "namespace"="test" 
--- PASS: TestReconcileNoCluster (0.00s)
=== RUN   TestReconcilePaused
I0816 03:08:55.468327   11609 controller.go:136]  "msg"="Reconciliation is paused for this object" "cluster"="foo" "kubeadmControlPlane"="foo" "namespace"="test" 
I0816 03:08:55.468481   11609 controller.go:136]  "msg"="Reconciliation is paused for this object" "cluster"="foo" "kubeadmControlPlane"="foo" "namespace"="test" 
--- PASS: TestReconcilePaused (0.00s)
=== RUN   TestReconcileClusterNoEndpoints
... skipping 94 lines ...
I0816 03:09:10.752587   11609 remediation.go:248]  "msg"="etcd cluster projected after remediation of m1-unhealthy-jd526" "cluster"="" "kubeadmControlPlane"="" "namespace"="" "canSafelyRemediate"=true "healthyMembers"=["node-m2-healthy-jl9sn (m2-healthy-jl9sn)","node-m3-healthy-58r6n (m3-healthy-58r6n)"] "targetQuorum"=2 "targetTotalMembers"=2 "targetUnhealthyMembers"=0 "unhealthyMembers"=[]
I0816 03:09:10.757301   11609 remediation.go:154]  "msg"="Remediating unhealthy machine" "cluster"="" "kubeadmControlPlane"="" "namespace"="" "UnhealthyMachine"="m1-unhealthy-jd526"
=== RUN   TestReconcileUnhealthyMachines/Remediation_deletes_unhealthy_machine_-_4_CP_(during_3_CP_rolling_upgrade)
I0816 03:09:10.878201   11609 remediation.go:195]  "msg"="etcd cluster before remediation" "cluster"="" "kubeadmControlPlane"="" "namespace"="" "currentMembers"=["node-m3-healthy-ttd9q","node-m4-healthy-rdslv","node-m1-unhealthy-46twg","node-m2-healthy-fzcp4"] "currentTotalMembers"=4
I0816 03:09:10.878286   11609 remediation.go:248]  "msg"="etcd cluster projected after remediation of m1-unhealthy-46twg" "cluster"="" "kubeadmControlPlane"="" "namespace"="" "canSafelyRemediate"=true "healthyMembers"=["node-m3-healthy-ttd9q (m3-healthy-ttd9q)","node-m4-healthy-rdslv (m4-healthy-rdslv)","node-m2-healthy-fzcp4 (m2-healthy-fzcp4)"] "targetQuorum"=2 "targetTotalMembers"=3 "targetUnhealthyMembers"=0 "unhealthyMembers"=[]
I0816 03:09:10.882998   11609 remediation.go:154]  "msg"="Remediating unhealthy machine" "cluster"="" "kubeadmControlPlane"="" "namespace"="" "UnhealthyMachine"="m1-unhealthy-46twg"
--- FAIL: TestReconcileUnhealthyMachines (0.81s)
    --- PASS: TestReconcileUnhealthyMachines/Remediation_does_not_happen_if_there_are_no_unhealthy_machines (0.00s)
    --- PASS: TestReconcileUnhealthyMachines/reconcileUnhealthyMachines_return_early_if_the_machine_to_be_remediated_is_marked_for_deletion (0.00s)
    --- PASS: TestReconcileUnhealthyMachines/Remediation_does_not_happen_if_desired_replicas_<=_1 (0.05s)
    --- PASS: TestReconcileUnhealthyMachines/Remediation_does_not_happen_if_number_of_machines_lower_than_desired (0.06s)
    --- FAIL: TestReconcileUnhealthyMachines/Remediation_does_not_happen_if_there_is_a_deleting_machine (0.04s)
        remediation_test.go:841: 
            Expected
                <string>: 
            to equal
                <string>: KCP waiting for control plane machine deletion to complete before triggering remediation
    --- PASS: TestReconcileUnhealthyMachines/Remediation_does_not_happen_if_there_is_at_least_one_additional_unhealthy_etcd_member_on_a_3_machine_CP (0.10s)
... skipping 85 lines ...
    --- PASS: TestSelectMachineForScaleDown/when_there_is_an_up_to_date_machine_with_delete_annotation,_while_there_are_any_outdated_machines_without_annotatio_that_still_exist,_it_returns_oldest_marked_machine_first (0.00s)
=== RUN   TestPreflightChecks
=== RUN   TestPreflightChecks/control_plane_without_machines_(not_initialized)_should_pass
=== RUN   TestPreflightChecks/control_plane_with_a_deleting_machine_should_requeue
I0816 03:09:13.035814   11609 scale.go:173]  "msg"="Waiting for machines to be deleted" "cluster"="" "kubeadmControlPlane"="" "namespace"="" "Machines"=""
=== RUN   TestPreflightChecks/control_plane_with_an_unhealthy_machine_condition_should_requeue
I0816 03:09:13.036030   11609 scale.go:212]  "msg"="Waiting for control plane to pass preflight checks" "cluster"="" "kubeadmControlPlane"="" "namespace"="" "failures"="machine  reports APIServerPodHealthy condition is false (Error, )"
=== RUN   TestPreflightChecks/control_plane_with_an_healthy_machine_and_an_healthy_kcp_condition_should_pass
--- PASS: TestPreflightChecks (0.00s)
    --- PASS: TestPreflightChecks/control_plane_without_machines_(not_initialized)_should_pass (0.00s)
    --- PASS: TestPreflightChecks/control_plane_with_a_deleting_machine_should_requeue (0.00s)
    --- PASS: TestPreflightChecks/control_plane_with_an_unhealthy_machine_condition_should_requeue (0.00s)
    --- PASS: TestPreflightChecks/control_plane_with_an_healthy_machine_and_an_healthy_kcp_condition_should_pass (0.00s)
... skipping 18 lines ...
=== RUN   TestAPIs
Running Suite: Controller Suite
===============================
Random Seed: 1660619322
Will run 1 of 1 specs

E0816 03:09:13.061732   11609 controller.go:126]  "msg"="Failed to retrieve owner Cluster from the API Server" "error"="Cluster.cluster.x-k8s.io \"kcp-foo-lmv54g\" not found" "kubeadmControlPlane"="kcp-foo-lmv54g" "namespace"="test" 
•

Ran 1 of 1 Specs in 0.021 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestAPIs (0.02s)
=== RUN   TestKubeadmControlPlaneReconciler_upgradeControlPlane
I0816 03:09:13.063155   11609 controller.go:246]  "msg"="Reconcile KubeadmControlPlane" "cluster"="kcp-foo-g00xww" "kubeadmControlPlane"="kcp-foo-g00xww" "namespace"="test" 
I0816 03:09:14.466501   11609 controller.go:331]  "msg"="Rolling out Control Plane machines" "cluster"="kcp-foo-g00xww" "kubeadmControlPlane"="kcp-foo-g00xww" "namespace"="test" "needRollout"=["kcp-foo-g00xww-qzcrd"]
I0816 03:09:14.466651   11609 scale.go:212]  "msg"="Waiting for control plane to pass preflight checks" "cluster"="kcp-foo-g00xww" "kubeadmControlPlane"="kcp-foo-g00xww" "namespace"="test" "failures"="[machine kcp-foo-g00xww-mvbcs does not have APIServerPodHealthy condition, machine kcp-foo-g00xww-mvbcs does not have ControllerManagerPodHealthy condition, machine kcp-foo-g00xww-mvbcs does not have SchedulerPodHealthy condition, machine kcp-foo-g00xww-mvbcs does not have EtcdPodHealthy condition, machine kcp-foo-g00xww-mvbcs does not have EtcdMemberHealthy condition]"
--- PASS: TestKubeadmControlPlaneReconciler_upgradeControlPlane (1.41s)
FAIL
I0816 03:09:14.467536   11609 server.go:208] controller-runtime/webhook "msg"="shutting down webhook server"  
FAIL	sigs.k8s.io/cluster-api/controlplane/kubeadm/controllers	31.979s
I0816 03:08:53.799802   11555 webhook.go:102] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha3","Kind":"Cluster"} "path"="/mutate-cluster-x-k8s-io-v1alpha3-cluster"
I0816 03:08:53.799952   11555 server.go:121] controller-runtime/webhook "msg"="registering webhook"  "path"="/mutate-cluster-x-k8s-io-v1alpha3-cluster"
I0816 03:08:53.799982   11555 webhook.go:123] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha3","Kind":"Cluster"} "path"="/validate-cluster-x-k8s-io-v1alpha3-cluster"
I0816 03:08:53.800002   11555 server.go:121] controller-runtime/webhook "msg"="registering webhook"  "path"="/validate-cluster-x-k8s-io-v1alpha3-cluster"
I0816 03:08:53.800083   11555 webhook.go:102] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"cluster.x-k8s.io","Version":"v1alpha3","Kind":"Machine"} "path"="/mutate-cluster-x-k8s-io-v1alpha3-machine"
I0816 03:08:53.800099   11555 server.go:121] controller-runtime/webhook "msg"="registering webhook"  "path"="/mutate-cluster-x-k8s-io-v1alpha3-machine"
... skipping 51 lines ...
==================================
Random Seed: 1660619321
Will run 7 of 7 specs

•••••••
Ran 7 of 7 Specs in 0.001 seconds
SUCCESS! -- 7 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestControlPlane (0.00s)
=== RUN   TestHasUnhealthyMachine
--- PASS: TestHasUnhealthyMachine (0.00s)
=== RUN   TestNewFailureDomainPicker
=== RUN   TestNewFailureDomainPicker/simple
=== RUN   TestNewFailureDomainPicker/no_machines
... skipping 111 lines ...
=======================================
Random Seed: 1660619321
Will run 7 of 7 specs

•••••••
Ran 7 of 7 Specs in 0.001 seconds
SUCCESS! -- 7 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestMachineCollection (0.00s)
=== RUN   TestUpdateEtcdConditions
=== RUN   TestUpdateEtcdConditions/if_list_nodes_return_an_error_should_report_all_the_conditions_Unknown
=== RUN   TestUpdateEtcdConditions/node_without_machine_should_be_ignored_if_there_are_provisioning_machines
=== RUN   TestUpdateEtcdConditions/node_without_machine_should_report_a_problem_at_KCP_level_if_there_are_no_provisioning_machines
=== RUN   TestUpdateEtcdConditions/failure_creating_the_etcd_client_should_report_unknown_condition
... skipping 297 lines ...
    --- PASS: TestUpdateKubeProxyImageInfo/returns_error_if_image_repository_is_invalid (0.00s)
    --- PASS: TestUpdateKubeProxyImageInfo/does_not_update_image_repository_when_no_kube-proxy_update_is_requested (0.00s)
=== RUN   TestRemoveMachineFromKubeadmConfigMap
=== RUN   TestRemoveMachineFromKubeadmConfigMap/does_not_panic_if_machine_is_nil
=== RUN   TestRemoveMachineFromKubeadmConfigMap/does_not_panic_if_machine_noderef_is_nil
=== RUN   TestRemoveMachineFromKubeadmConfigMap/returns_error_if_unable_to_find_kubeadm-config_for_Kubernetes_version_<_1.22.0
E0816 03:08:55.712315   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:09:04.035370   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:09:13.021358   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:09:24.151662   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:09:40.624906   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:09:53.633524   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:10:11.280944   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:10:32.204285   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:11:03.220834   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
E0816 03:11:48.378283   11555 workload_cluster.go:354]  "msg"="unable to get kubeadmConfigMap" "error"="error getting kube-system/kubeadm-config configmap from target cluster: configmaps \"kubeadm-config\" not found"  
=== RUN   TestRemoveMachineFromKubeadmConfigMap/returns_error_if_unable_to_remove_api_endpoint_for_Kubernetes_version_<_1.22.0
=== RUN   TestRemoveMachineFromKubeadmConfigMap/removes_the_machine_node_ref_from_kubeadm_config_for_Kubernetes_version_<_1.22.0
=== RUN   TestRemoveMachineFromKubeadmConfigMap/no_op_for_Kubernetes_version_>=_1.22.0
--- PASS: TestRemoveMachineFromKubeadmConfigMap (172.67s)
    --- PASS: TestRemoveMachineFromKubeadmConfigMap/does_not_panic_if_machine_is_nil (0.00s)
    --- PASS: TestRemoveMachineFromKubeadmConfigMap/does_not_panic_if_machine_noderef_is_nil (0.00s)
... skipping 383 lines ...
I0816 03:09:06.366486   11924 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="clusterresourceset" "source"={"Type":{"kind":"ConfigMap","apiVersion":"v1","metadata":{"creationTimestamp":null}}}
I0816 03:09:06.366510   11924 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="clusterresourcesetbinding" "worker count"=1
I0816 03:09:06.467026   11924 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="clusterresourceset" "source"={"Type":{"kind":"Secret","apiVersion":"v1","metadata":{"creationTimestamp":null}}}
I0816 03:09:06.569219   11924 controller.go:162] controller-runtime/controller "msg"="Starting Controller" "controller"="clusterresourceset" 
I0816 03:09:06.569293   11924 controller.go:189] controller-runtime/controller "msg"="Starting workers" "controller"="clusterresourceset" "worker count"=1
I0816 03:09:06.655847   11924 clusterresourcesetbinding_controller.go:84]  "msg"="deleting ClusterResourceSetBinding because the owner Cluster no longer exists" "clusterresourcesetbinding"={"Namespace":"default","Name":"cluster-n749i0"} 
E0816 03:09:06.684999   11924 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusterresourcesets.addons.cluster.x-k8s.io \"test-clusterresourceset\" not found" "controller"="clusterresourceset" "name"="test-clusterresourceset" "namespace"="default"
•I0816 03:09:07.730971   11924 clusterresourcesetbinding_controller.go:84]  "msg"="deleting ClusterResourceSetBinding because the owner Cluster no longer exists" "clusterresourcesetbinding"={"Namespace":"default","Name":"cluster-kls1d3"} 
E0816 03:09:07.773974   11924 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusterresourcesets.addons.cluster.x-k8s.io \"test-clusterresourceset\" not found" "controller"="clusterresourceset" "name"="test-clusterresourceset" "namespace"="default"
•E0816 03:09:08.860301   11924 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="failed to get data field from the resource" "controller"="clusterresourceset" "name"="test-clusterresourceset" "namespace"="default"
•E0816 03:09:09.881978   11924 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusterresourcesets.addons.cluster.x-k8s.io \"test-clusterresourceset\" not found" "controller"="clusterresourceset" "name"="test-clusterresourceset" "namespace"="default"
E0816 03:09:11.067813   11924 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusterresourcesets.addons.cluster.x-k8s.io \"test-clusterresourceset\" not found" "controller"="clusterresourceset" "name"="test-clusterresourceset" "namespace"="default"
•E0816 03:09:12.102669   11924 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusterresourcesets.addons.cluster.x-k8s.io \"test-clusterresourceset2\" not found" "controller"="clusterresourceset" "name"="test-clusterresourceset2" "namespace"="default"
E0816 03:09:13.118520   11924 controller.go:257] controller-runtime/controller "msg"="Reconciler error" "error"="clusterresourcesets.addons.cluster.x-k8s.io \"test-clusterresourceset\" not found" "controller"="clusterresourceset" "name"="test-clusterresourceset" "namespace"="default"
•I0816 03:09:13.122154   11924 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="clusterresourcesetbinding" 
I0816 03:09:13.122196   11924 server.go:208] controller-runtime/webhook "msg"="shutting down webhook server"  
I0816 03:09:13.122436   11924 controller.go:203] controller-runtime/controller "msg"="Stopping workers" "controller"="clusterresourceset" 
E0816 03:09:13.122674   11924 certwatcher.go:143] controller-runtime/certwatcher "msg"="error re-watching file" "error"="inotify instance already closed"  
E0816 03:09:13.122730   11924 certwatcher.go:148] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-211847493/tls.crt: no such file or directory"  


Ran 5 of 5 Specs in 20.747 seconds
SUCCESS! -- 5 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestAPIs (20.75s)
PASS
ok  	sigs.k8s.io/cluster-api/exp/addons/controllers	20.988s
?   	sigs.k8s.io/cluster-api/exp/addons/controllers/predicates	[no test files]
=== RUN   TestMachinePoolDefault
=== RUN   TestMachinePoolDefault/for_MachinePool
... skipping 61 lines ...
    --- PASS: TestReconcileMachinePoolBootstrap/new_machinepool,_no_bootstrap_config_or_data (0.00s)
    --- PASS: TestReconcileMachinePoolBootstrap/existing_machinepool,_bootstrap_data_should_not_change (0.00s)
    --- PASS: TestReconcileMachinePoolBootstrap/existing_machinepool,_bootstrap_provider_is_to_not_ready (0.01s)
=== RUN   TestReconcileMachinePoolInfrastructure
=== RUN   TestReconcileMachinePoolInfrastructure/new_machinepool,_infrastructure_config_ready
=== RUN   TestReconcileMachinePoolInfrastructure/ready_bootstrap,_infra,_and_nodeRef,_machinepool_is_running,_infra_object_is_deleted,_expect_failed
E0816 03:08:57.909087   12092 machinepool_controller_phases.go:237]  "msg"="MachinePool infrastructure reference has been deleted after being ready, setting failure state" "error"="could not find infrastructure.cluster.x-k8s.io/v1alpha3, Kind=InfrastructureConfig \"infra-config1\" for MachinePool \"machinepool-test\" in namespace \"default\", requeuing: failed to retrieve InfrastructureConfig external object \"default\"/\"infra-config1\": infrastructureconfigs.infrastructure.cluster.x-k8s.io \"infra-config1\" not found" "cluster"="test-cluster" "machinepool"="machinepool-test" "namespace"="default" 
=== RUN   TestReconcileMachinePoolInfrastructure/infrastructure_ref_is_paused
--- PASS: TestReconcileMachinePoolInfrastructure (0.00s)
    --- PASS: TestReconcileMachinePoolInfrastructure/new_machinepool,_infrastructure_config_ready (0.00s)
    --- PASS: TestReconcileMachinePoolInfrastructure/ready_bootstrap,_infra,_and_nodeRef,_machinepool_is_running,_infra_object_is_deleted,_expect_failed (0.00s)
    --- PASS: TestReconcileMachinePoolInfrastructure/infrastructure_ref_is_paused (0.00s)
=== RUN   TestMachinePoolFinalizer
... skipping 96 lines ...
•I0816 03:09:09.352828   12092 machinepool_controller_noderef.go:90]  "msg"="Cannot assign NodeRefs to MachinePool, no matching Nodes"  
••I0816 03:09:09.363633   12092 server.go:208] controller-runtime/webhook "msg"="shutting down webhook server"  
I0816 03:09:09.364421   12092 controller.go:155] controller-runtime/controller "msg"="Starting EventSource" "controller"="machinepool" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"controlPlaneEndpoint":{"host":"","port":0}},"status":{"infrastructureReady":false,"controlPlaneInitialized":false}}}


Ran 9 of 9 Specs in 11.364 seconds
SUCCESS! -- 9 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestAPIs (11.36s)
PASS
ok  	sigs.k8s.io/cluster-api/exp/controllers	11.680s
?   	sigs.k8s.io/cluster-api/exp/util	[no test files]
?   	sigs.k8s.io/cluster-api/feature	[no test files]
?   	sigs.k8s.io/cluster-api/hack/boilerplate/test	[no test files]
... skipping 266 lines ...
--- PASS: TestGetFirstReasonAndMessage (0.00s)
=== RUN   TestNewConditionsGroup
--- PASS: TestNewConditionsGroup (0.00s)
=== RUN   TestMergeRespectPriority
=== RUN   TestMergeRespectPriority/aggregate_nil_list_return_nil
=== RUN   TestMergeRespectPriority/aggregate_empty_list_return_nil
=== RUN   TestMergeRespectPriority/When_there_is_false/error_it_returns_false/error
=== RUN   TestMergeRespectPriority/When_there_is_false/warning_and_no_false/error,_it_returns_false/warning
=== RUN   TestMergeRespectPriority/When_there_is_false/info_and_no_false/error_or_false/warning,_it_returns_false/info
=== RUN   TestMergeRespectPriority/When_there_is_true_and_no_false/*,_it_returns_info
=== RUN   TestMergeRespectPriority/When_there_is_unknown_and_no_true_or_false/*,_it_returns_unknown
=== RUN   TestMergeRespectPriority/nil_conditions_are_ignored
--- PASS: TestMergeRespectPriority (0.00s)
    --- PASS: TestMergeRespectPriority/aggregate_nil_list_return_nil (0.00s)
    --- PASS: TestMergeRespectPriority/aggregate_empty_list_return_nil (0.00s)
    --- PASS: TestMergeRespectPriority/When_there_is_false/error_it_returns_false/error (0.00s)
    --- PASS: TestMergeRespectPriority/When_there_is_false/warning_and_no_false/error,_it_returns_false/warning (0.00s)
    --- PASS: TestMergeRespectPriority/When_there_is_false/info_and_no_false/error_or_false/warning,_it_returns_false/info (0.00s)
    --- PASS: TestMergeRespectPriority/When_there_is_true_and_no_false/*,_it_returns_info (0.00s)
    --- PASS: TestMergeRespectPriority/When_there_is_unknown_and_no_true_or_false/*,_it_returns_unknown (0.00s)
    --- PASS: TestMergeRespectPriority/nil_conditions_are_ignored (0.00s)
=== RUN   TestNewPatch
=== RUN   TestNewPatch/No_changes_return_empty_patch
... skipping 200 lines ...
I0816 03:09:35.022159   15080 webhook.go:123] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"addons.cluster.x-k8s.io","Version":"v1alpha3","Kind":"ClusterResourceSet"} "path"="/validate-addons-cluster-x-k8s-io-v1alpha3-clusterresourceset"
I0816 03:09:35.022188   15080 server.go:121] controller-runtime/webhook "msg"="registering webhook"  "path"="/validate-addons-cluster-x-k8s-io-v1alpha3-clusterresourceset"
I0816 03:09:35.022544   15080 server.go:141] controller-runtime/webhook/webhooks "msg"="starting webhook server"  
I0816 03:09:35.023569   15080 certwatcher.go:127] controller-runtime/certwatcher "msg"="Updated current TLS certificate"  
I0816 03:09:35.023776   15080 server.go:199] controller-runtime/webhook "msg"="serving webhook server"  "host"="" "port"=43597
I0816 03:09:35.023874   15080 certwatcher.go:83] controller-runtime/certwatcher "msg"="Starting certificate watcher"  
••••••••••••••E0816 03:09:35.991222   15080 certwatcher.go:143] controller-runtime/certwatcher "msg"="error re-watching file" "error"="no such file or directory"  
E0816 03:09:35.991281   15080 certwatcher.go:148] controller-runtime/certwatcher "msg"="error re-reading certificate" "error"="open /tmp/envtest-serving-certs-107831740/tls.crt: no such file or directory"  
I0816 03:09:35.991292   15080 server.go:208] controller-runtime/webhook "msg"="shutting down webhook server"  


Ran 14 of 14 Specs in 7.885 seconds
SUCCESS! -- 14 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestPatch (7.88s)
=== RUN   TestToUnstructured
=== RUN   TestToUnstructured/with_a_typed_object
=== RUN   TestToUnstructured/with_an_unstructured_object
--- PASS: TestToUnstructured (0.00s)
    --- PASS: TestToUnstructured/with_a_typed_object (0.00s)
... skipping 105 lines ...
    --- PASS: TestToUnstructured/returns_error_for_invalid_yaml#01 (0.00s)
    --- PASS: TestToUnstructured/returns_error_for_invalid_yaml#02 (0.00s)
=== RUN   TestFromUnstructured
--- PASS: TestFromUnstructured (0.00s)
PASS
ok  	sigs.k8s.io/cluster-api/util/yaml	0.066s
FAIL
make: *** [Makefile:116: test] Error 1