Updates to the controller logic to better handle failures in etc updates

[z103cb]

Updated the ETCD update functions.
Added error handling.
Reduce the calls to update.

[asm582]

Thanks for the PR, please address the review

[asm582]

Something is off with this PR, I built an image from your branch and submitted just 1 AW, it does not run, log message:

0627 11:04:45.616792 1 queuejob_controller_ex.go:1752] [Informer-addQJ] enqueue defaultaw-schd-spec-with-timeout-191 &qj=0xc002a8a500 Version=4842895 Status={Pending:0 Running:0 Succeeded:0 Failed:0 MinAvailable:0 CanRun:false IsDispatched:false State: Message: SystemPriority:9 QueueJobState:Init ControllerFirstTimestamp:2023-06-27 11:04:45.616785091 +0000 UTC m=+8.773521703 ControllerFirstDispatchTimestamp:0001-01-01 00:00:00 +0000 UTC FilterIgnore:false Sender: Local:false Conditions:[{Type:Init Status:True LastUpdateMicroTime:2023-06-27 11:04:45.616785429 +0000 UTC m=+8.773522036 LastTransitionMicroTime:2023-06-27 11:04:45.616785482 +0000 UTC m=+8.773522079 Reason: Message:}] PendingPodConditions:[]}
I0627 11:04:45.616856 1 queuejob_controller_ex.go:2403] [getAppWrapper] geting a copy of 'default/defaultaw-schd-spec-with-timeout-191' when called by '[syncQueueJob] get fresh appwrapper '.
I0627 11:04:45.616861 1 queuejob_controller_ex.go:2410] [getAppWrapper] get a copy of 'default/defaultaw-schd-spec-with-timeout-191' suceeded when called by '[syncQueueJob] get fresh appwrapper '
I0627 11:04:45.616895 1 queuejob_controller_ex.go:1475] [updateStatusInEtcd] trying to update 'default/defaultaw-schd-spec-with-timeout-191' called by 'manageQueueJob - setQueueing'
E0627 11:04:45.618282 1 queuejob_controller_ex.go:2091] [manageQueueJob] Failed to updated etcd for AppWrapper Job 'default/defaultaw-schd-spec-with-timeout-191', err=appwrappers.mcad.ibm.com "defaultaw-schd-spec-with-timeout-191" not found
E0627 11:04:45.618295 1 queuejob_controller_ex.go:1924] [worker] Failed to sync AppWrapper 'default/defaultaw-schd-spec-with-timeout-191', err &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"appwrappers.mcad.ibm.com \"defaultaw-schd-spec-with-timeout-191\" not found", Reason:"NotFound", Details:(*v1.StatusDetails)(0xc00668e1e0), Code:404}}
W0627 11:04:45.618314 1 queuejob_controller_ex.go:1933] [worker] Fail to process item from eventQueue, err appwrappers.mcad.ibm.com "defaultaw-schd-spec-with-timeout-191" not found. Attempting to re-enqueque...
W0627 11:04:45.618321 1 queuejob_controller_ex.go:1937] [worker] Item re-enqueued%!(EXTRA *errors.StatusError=appwrappers.mcad.ibm.com "defaultaw-schd-spec-with-timeout-191" not found)
I0627 11:04:45.618342 1 queuejob_controller_ex.go:2403] [getAppWrapper] geting a copy of 'default/defaultaw-schd-spec-with-timeout-191' when called by '[syncQueueJob] get fresh appwrapper '.
I0627 11:04:45.618350 1 queuejob_controller_ex.go:2410] [getAppWrapper] get a copy of 'default/defaultaw-schd-spec-with-timeout-191' suceeded when called by '[syncQueueJob] get fresh appwrapper '

image used: podman pull quay.io/asmalvan/z103cb_issue297

[asm582]

I restarted the build, it was still failing with some updateEtcd failures.

[asm582]

All test cases pass on the laptop, Scheduling fail fast test case is now passing post changes.

[asm582]

All except 1 test pass. the 1 test is pending by design, it is a bad AW and the system rejects, below is the test:

  Create AppWrapper  - Bad Generic Item Only
  /home/travis/gopath/src/github.com/project-codeflare/multi-cluster-app-dispatcher/test/e2e/queue.go:356
------------------------------
P [PENDING]

The test should be changed, it should pass when such AW is never dispatched. We need to figure stop failing builds for tests that are pending until the test is changed.