/kind bug

After completion of bayesianoptimization automated training, the corresponding indicator results cannot be collected. Could you please tell me how to collect the indicator of training results. My yaml file is as follows: apiVersion: "kubeflow.org/v1alpha3" kind: Experiment metadata: namespace: kubeflow labels: controller-tools.k8s.io: "1.0" name: bayesianoptimization-example spec: objective: type: maximize goal: 0.99 objectiveMetricName: Validation-accuracy additionalMetricNames: - accuracy algorithm: algorithmName: bayesianoptimization algorithmSettings: - name: "random_state" value: "10" parallelTrialCount: 3 maxTrialCount: 12 maxFailedTrialCount: 3 MetricsCollectorSpec: Collector: Kind: stdOut parameters: - name: --lr parameterType: double feasibleSpace: min: "0.01" max: "0.03" - name: --num-layers parameterType: int feasibleSpace: min: "2" max: "5" - name: --optimizer parameterType: categorical feasibleSpace: list: - sgd - adam - ftrl trialTemplate: goTemplate: rawTemplate: |- apiVersion: batch/v1 kind: Job metadata: name: {{.Trial}} namespace: {{.NameSpace}} spec: template: spec: containers: - name: {{.Trial}} image: docker.io/katib/mxnet-mnist-example command: - "python" - "/mxnet/example/image-classification/train_mnist.py" - "--batch-size=64" {{- with .HyperParameters}} {{- range .}} - "{{.Name}}={{.Value}}" {{- end}} {{- end}} restartPolicy: Never

What steps did you take and what happened: [A clear and concise description of what the bug is.]

What did you expect to happen:

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Kubeflow version:0.7.0
• Minikube version:
• Kubernetes version: (use kubectl version):1.15.5
• OS (e.g. from /etc/os-release):CentOS Linux release 7.7.1908

Fixes: https://github.com/kubeflow/katib/issues/1405.

This PR introduces new mechanism to get certificate for webhooks. I updated YAMLs for our webhooks. I added initContainer to Katib controller which executes cert-generator.sh script. This script creates CertificateSigningRequest, katib-webhook-cert secret and patches webhooks configurations with appropriate caBundle. Since we have katib-webhook-cert secret in the manifest, cleanup process should delete everything.

So we don't need to deploy cert-manager for Katib.

@gaocegege @johnugeorge @yanniszark @kuikuikuizzZ @knkski What do you think about this approach ?

Also I updated controller-runtime to v0.8.2 and k8s.io deps to v0.20.4. That requires some changes:

• Change some packages location
• Change the arguments for client calls (List, Get, etc.)
• In the newer Kubernetes versions we can't add owner reference for cluster-scoped objects (e.g. PV) with namespace-scoped object (e.g. Suggestion). Thus, I have to disable owner reference for the PV which is created when Experiment has FromVolume resume policy. For that reason, I added PersistentVolumeReclaimPolicy: Delete for the PV and once PVC is garbage collected, PV should also be deleted.
• I removed PyTorch operator from the dependencies because of this problem.

I still need to make some tests and create new image for cert generator. It would be great if you can start to review this.

/cc @gaocegege @johnugeorge

/kind feature

Describe the solution you'd like [A clear and concise description of what you want to happen.]

We need to marshal the TFJob to JSON string then use it to create experiments if we are using K8s client-go. It is not good. And, go template is ugly, too.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Related: https://github.com/kubeflow/katib/issues/1332. I will debug the infra in this PR.

I also made few changes to improve CI/CD quality.

/cc @gaocegege @johnugeorge /cc @Jeffwan @PatrickXYS @jlewi @Bobgy

@YujiOshima @gaocegege @johnugeorge @alexandraj777 @hougangliu @xyhuang

This is an initial proposal for the Katib v1alpha2 API. The changes here reflect the discussion in https://github.com/kubeflow/katib/issues/370.

Comments and suggestions are welcome.

Please note that the NAS APIs are not included here since the feature is still in early phase.

This change is 

Add StudyController CRD: studycontroller.kubeflow.org Operator: StudyController

Update examples. This implementation is polling workers status in go process of StudyController. Though I understand this is not an elegant implementation, this is the least impact to existing codes.

Next step we should make worker CRD and its controller and support multi-type jobs (k8s, TF-Job..). Assign @gaocegege

This change is 

What this PR does / why we need it:

Support the discovery of modulated hyperparameters rather than attempting to find a fixed set over the entire training process. The paper has more details about the technique.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

This PR provides some initial support for PBT within Katib (#1382).

Checklist:

• [ ] Docs included if any changes are user facing

Related: #1332. I will debug the infra in this PR.

• [x] This is the PR to see if we can trigger AWS Presubmit.
• [x] This is the PR to see if Github UI integrate aws-kf-ci-bot

when I deploy katib_v1alpha3 with scripts/v1alpha3/deploy.sh, the katib-controller pod gives the following error: {"level":"info","ts":1578296376.3173876,"logger":"entrypoint","msg":"Config:","experiment-suggestion-name":"default","cert-local-filesystem":false} {"level":"info","ts":1578296376.375878,"logger":"entrypoint","msg":"Registering Components."} {"level":"info","ts":1578296376.3765948,"logger":"entrypoint","msg":"Setting up controller"} {"level":"info","ts":1578296376.3766346,"logger":"experiment-controller","msg":"Using the default suggestion implementation"} {"level":"info","ts":1578296376.3767953,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"experiment-controller","source":"kind source: /, Kind="} {"level":"error","ts":1578296376.3768966,"logger":"kubebuilder.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":{"Group":"kubeflow.org","Kind":"Experiment"},"error":"no matches for kind "Experiment" in version "kubeflow.org/v1alpha3"","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:89\ngithub.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Watch\n\t/go/src/github.com/kubeflow/katib/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.addWatch\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:119\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:107\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.Add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:62\ngithub.com/kubeflow/katib/pkg/controller%2ev1alpha3.AddToManager\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/controller.go:28\nmain.main\n\t/go/src/github.com/kubeflow/katib/cmd/katib-controller/v1alpha3/main.go:90\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"} {"level":"error","ts":1578296376.377135,"logger":"experiment-controller","msg":"Experiment watch failed","error":"no matches for kind "Experiment" in version "kubeflow.org/v1alpha3"","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.addWatch\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:121\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:107\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.Add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:62\ngithub.com/kubeflow/katib/pkg/controller%2ev1alpha3.AddToManager\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/controller.go:28\nmain.main\n\t/go/src/github.com/kubeflow/katib/cmd/katib-controller/v1alpha3/main.go:90\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"} {"level":"error","ts":1578296376.3772092,"logger":"experiment-controller","msg":"Trial watch failed","error":"no matches for kind "Experiment" in version "kubeflow.org/v1alpha3"","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:108\ngithub.com/kubeflow/katib/pkg/controller.v1alpha3/experiment.Add\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/experiment/experiment_controller.go:62\ngithub.com/kubeflow/katib/pkg/controller%2ev1alpha3.AddToManager\n\t/go/src/github.com/kubeflow/katib/pkg/controller.v1alpha3/controller.go:28\nmain.main\n\t/go/src/github.com/kubeflow/katib/cmd/katib-controller/v1alpha3/main.go:90\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"} {"level":"error","ts":1578296376.377267,"logger":"entrypoint","msg":"unable to register controllers to the manager","error":"no matches for kind "Experiment" in version "kubeflow.org/v1alpha3"","stacktrace":"github.com/kubeflow/katib/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubeflow/katib/vendor/github.com/go-logr/zapr/zapr.go:128\nmain.main\n\t/go/src/github.com/kubeflow/katib/cmd/katib-controller/v1alpha3/main.go:91\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

And the ui pod gives the following error: 2020/01/06 06:56:46 CreateExperiment from YAML failed: no matches for kind "Experiment" in version "kubeflow.org/v1alpha3"

/kind bug

Hi, I'm setting a Katib job through the Kale deployment panel - after creating a Kale pipeline. The pipeline builds successfully but the Katib experiments run forever and don't complete a single trial.

I expect the Katib jobs to run successfully, but to no avail.

Any way/suggestion to go about this?

Environment:

• Kubeflow version (kfctl version):
• Minikube version (minikube version):
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug

Hi, I'm having trouble using katib v1alpha3. First, I installed katib by the followings

1. git clone https://github.com/kubeflow/katib
2. sh katib/scripts/v1alpha3/deploy.sh

And I tried to apply random-example.yaml kubectl apply -f random-example.yaml (example in katib/examples/v1alpha3)

Results: kubectl get pods -n kubeflow NAME READY STATUS RESTARTS AGE katib-controller-6c6974678d-zsnlc 1/1 Running 1 24m katib-db-558f649dc6-8cd9t 1/1 Running 0 24m katib-manager-5f74bdff84-4d78z 1/1 Running 0 24m katib-ui-6568bd6b44-qbq5k 1/1 Running 0 24m random-example-random-846dc99654-bxb8j 1/1 Running 0 23m

kubectl get trials -n kubeflow NAME TYPE STATUS AGE random-example-drpkvb4b Running True 23m random-example-k7xv6ktt Running True 23m random-example-w6jlwdp2 Running True 23m

kubectl get experiment -n kubeflow -oyaml apiVersion: v1 items:

• apiVersion: kubeflow.org/v1alpha3 kind: Experiment metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"kubeflow.org/v1alpha3","kind":"Experiment","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name":"random-example","namespace":"kubeflow"},"spec":{"algorithm":{"algorithmName":"random"},"maxFailedTrialCount":3,"maxTrialCount":12,"objective":{"additionalMetricNames":["accuracy"],"goal":0.99,"objectiveMetricName":"Validation-accuracy","type":"maximize"},"parallelTrialCount":3,"parameters":[{"feasibleSpace":{"max":"0.03","min":"0.01"},"name":"--lr","parameterType":"double"},{"feasibleSpace":{"max":"5","min":"2"},"name":"--num-layers","parameterType":"int"},{"feasibleSpace":{"list":["sgd","adam","ftrl"]},"name":"--optimizer","parameterType":"categorical"}],"trialTemplate":{"goTemplate":{"rawTemplate":"apiVersion: batch/v1\nkind: Job\nmetadata:\n name: {{.Trial}}\n namespace: {{.NameSpace}}\nspec:\n template:\n spec:\n containers:\n - name: {{.Trial}}\n image: docker.io/kubeflowkatib/mxnet-mnist-example\n command:\n - "python"\n - "/mxnet/example/image-classification/train_mnist.py"\n - "--batch-size=64"\n {{- with .HyperParameters}}\n {{- range .}}\n - "{{.Name}}={{.Value}}"\n {{- end}}\n {{- end}}\n restartPolicy: Never"}}}} creationTimestamp: "2019-12-20T07:58:52Z" finalizers:
  • update-prometheus-metrics generation: 2 labels: controller-tools.k8s.io: "1.0" name: random-example namespace: kubeflow resourceVersion: "11682124" selfLink: /apis/kubeflow.org/v1alpha3/namespaces/kubeflow/experiments/random-example uid: 9005bab0-22fe-11ea-8cf0-0679676001a5 spec: algorithm: algorithmName: random algorithmSettings: null maxFailedTrialCount: 3 maxTrialCount: 12 metricsCollectorSpec: collector: kind: StdOut objective: additionalMetricNames:
    • accuracy goal: 0.99 objectiveMetricName: Validation-accuracy type: maximize parallelTrialCount: 3 parameters:
  • feasibleSpace: max: "0.03" min: "0.01" name: --lr parameterType: double
  • feasibleSpace: max: "5" min: "2" name: --num-layers parameterType: int
  • feasibleSpace: list:
    • sgd
    • adam
    • ftrl name: --optimizer parameterType: categorical trialTemplate: goTemplate: rawTemplate: |- apiVersion: batch/v1 kind: Job metadata: name: {{.Trial}} namespace: {{.NameSpace}} spec: template: spec: containers: - name: {{.Trial}} image: docker.io/kubeflowkatib/mxnet-mnist-example command: - "python" - "/mxnet/example/image-classification/train_mnist.py" - "--batch-size=64" {{- with .HyperParameters}} {{- range .}} - "{{.Name}}={{.Value}}" {{- end}} {{- end}} restartPolicy: Never status: conditions:
  • lastTransitionTime: "2019-12-20T07:58:52Z" lastUpdateTime: "2019-12-20T07:58:52Z" message: Experiment is created reason: ExperimentCreated status: "True" type: Created
  • lastTransitionTime: "2019-12-20T08:00:22Z" lastUpdateTime: "2019-12-20T08:00:22Z" message: Experiment is running reason: ExperimentRunning status: "True" type: Running currentOptimalTrial: observation: metrics: null parameterAssignments: null startTime: "2019-12-20T07:58:52Z" trials: 3 trialsRunning: 3 kind: List metadata: resourceVersion: "" selfLink: ""

kubectl logs -n kubeflow random-example-random-846dc99654-bxb8j INFO:hyperopt.utils:Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support. INFO:hyperopt.fmin:Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support. ERROR:grpc._server:Exception calling application: Method not implemented! Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/usr/src/app/github.com/kubeflow/katib/pkg/apis/manager/v1alpha3/python/api_pb2_grpc.py", line 135, in ValidateAlgorithmSettings raise NotImplementedError('Method not implemented!') NotImplementedError: Method not implemented!

What can I do to fix it? Thank you for your help in solving this problem.

• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5+icp", GitCommit:"903c3b31caddc675ce2d8bddf62aa0f875c2a3bc", GitTreeState:"clean", BuildDate:"2019-05-08T06:16:32Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5+icp", GitCommit:"903c3b31caddc675ce2d8bddf62aa0f875c2a3bc", GitTreeState:"clean", BuildDate:"2019-05-08T06:16:32Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release): CentOS Linux release 7.7.1908 (Core)

/kind feature

Describe the solution you'd like [A clear and concise description of what you want to happen.] We need to support Kubernetes v1.26 since that version was released on 2022-12-9.

https://kubernetes.io/releases/#release-v1-26

Maybe, we can support that version after the next katib release. This means supporting v1.26 is out of scope in katib v0.15.0.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Love this feature? Give it a 👍 We prioritize the features with the most 👍

/kind discussion

Describe the solution you'd like [A clear and concise description of what you want to happen.] The operators directory corresponds to katib v0.12.0, which is a bit out of date.

Also, it looks like the latest Charmed katib-operator exists at https://github.com/canonical/katib-operators. Those Charmed katib-operators don't seem to sync. This situation is likely to be confusing for users.

@DomFleischmann @DnPlas @ca-scribner @knkski Would you like to keep maintaining both kubeflow/katib/operators and canonical/katib-operators? Or would you like to remove Charmed katib-operator from this repository (katib repo)?

/cc @kubeflow/wg-automl-leads

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Love this feature? Give it a 👍 We prioritize the features with the most 👍

Signed-off-by: Yuki Iwai [email protected]

What this PR does / why we need it: I removed all coded related Chocolate Suggestion Service.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Part-of #2058

Checklist:

• [ ] Docs included if any changes are user facing

In this PR: https://github.com/kubeflow/katib/pull/2018#issuecomment-1330713221, I proposed to introduce label for disabling Katib Webhooks (validator, defaulter, mutator). For example: katib.kubeflow.org/webhooks: disabled. Let's discuss if that would be useful for the users with large-scale environment.

Currently, if user's namespace has katib.kubeflow.org/metrics-collector-injection: enabled label, Katib Mutation Webhook runs for every Pod in that namespace. That might increase latency in the Kubernetes API server. Some users might want to use Katib Experiments and run other pods in their namespaces without Webhook execution.

What do you think @gaocegege @johnugeorge @tenzen-y @anencore94 @terrytangyuan ?

/kind discussion

Love this feature? Give it a 👍 We prioritize the features with the most 👍

This is a follow-up PR to this and updates the COMMIT file to checkout to the latest KF commit and have the conditions table sorted by timestamp by default.