add endpoints for crud operations & update hardware data model #64

kqdeng · 2020-04-14T18:25:35Z

Changes:

Added endpoints for the following operations (unofficial api):
- tink hardware push/all/mac/ip/id
- tink template create/get/list/delete
- tink workflow create/get/list/delete/state/events
Added basic auth to above endpoints
- Added two new envvars TINK_AUTH_USERNAME and TINK_AUTH_PASSWORD to set basic auth username and password
Changed the data field in WorkflowTemplate to be a string instead of bytes (in template proto)
Updated Hardware struct to new data model
Updated docker-compose.yml to reflect additions of new envvars across all three prs

TODO:

update setup.sh and docker-compose.yml to account for new envvars
- setup.sh
- docker-compose.yml

Related PRs:
boots: tinkerbell/smee#27
hegel: tinkerbell/hegel#13

gauravgahlot · 2020-04-24T15:13:23Z

@kdeng3849 Can you please rebase the branch?

gauravgahlot · 2020-04-27T15:48:44Z

cli/tink/cmd/hardware/push.go

@@ -34,7 +34,13 @@ var pushCmd = &cobra.Command{
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 		for _, j := range args {
-			if _, err := client.HardwareClient.Push(context.Background(), &hardware.PushRequest{Data: j}); err != nil {
+			fmt.Println(j)


Do we really need this?

gauravgahlot · 2020-04-27T16:47:54Z

cli/tink/cmd/template/create.go

@@ -62,7 +62,7 @@ func readTemplateData() []byte {
 }

 func createTemplate(c *cobra.Command, args []string) {
-	req := template.WorkflowTemplate{Name: templateName, Data: readTemplateData()}
+	req := template.WorkflowTemplate{Name: templateName, Data: string(readTemplateData())}


Since the returned value of readTemplateData is converted to string in most of the places, I think it would make sense to change the return type to string from []byte.

alexellis · 2020-05-18T12:11:37Z

Looks like another rebase is required on this PR @kdeng3849

mmlb · 2020-06-02T20:22:35Z

cmd/tink-cli/cmd/hardware/ip.go

@@ -28,11 +29,15 @@ var ipCmd = &cobra.Command{
 	},
 	Run: func(cmd *cobra.Command, args []string) {
 		for _, ip := range args {
-			hw, err := client.HardwareClient.ByIP(context.Background(), &hardware.GetRequest{IP: ip})
+			hw, err := client.HardwareClient.ByIP(context.Background(), &hardware.GetRequest{Ip: ip})


Is there a way for the generated code to influence the name for the generated struct fields? Looking at Ip -> IP and Mac -> MAC and maybe others?

I didn't really look into it too much since it seemed to work fine with boots without any issue. Is there a reason/necessity for the names to be capitalized? I could change them back to uppercase in the proto if needed

its a code nit, but acronyms should be in all caps. It also matches the types in the stdlib so not being an oddball would be good.

okay, I'll change it back then

I remember why I decided on all lowercase, it's because there are problems when the message name is the same as the field name. For example:

message RAID { string name = 1; string level = 2; repeated string devices = 3; int64 spare = 4; } repeated RAID RAID = 5;

this is giving me the error:

Generating hardware.pb.go... hardware/hardware.proto:164:41: "RAID" is already defined in "github.com.tinkerbell.tink.protos.hardware.Hardware.Metadata.Instance.Storage". hardware/hardware.proto:175:42: "RAID" is not defined.

So I decided on all lowercase for the field name (which is also the convention for protobuf).
I could still rename the message name to something like "Raid", so I guess it all boils down to which convention to follow (go's or protobuf's). Which one should I go with?

ah ok, sigh. lets go with protobuf then. I'm curious is there an equivalent of json:"SomeKeyName" metadata we can add so we can both?

looks like no. golang/protobuf#555. In that case lets stay with protobuf norms. Transforming protoc generated structs into human readable/meaningful structs is something that I think always makes sense and we could address then.

okay, gotcha

yetang-equinix · 2020-06-10T22:46:34Z

cmd/tink-cli/cmd/hardware/all.go

-			fmt.Println(hw.JSON)
+			b, err := json.Marshal(hw)
+			if err != nil {
+				log.Fatal(err)


This is just to retrieve hardware info, may be one hardware data cannot be converted to json properly, we can just log an error and go on for next. Why do we have to log fatal, which will crash the cli agent and make it harder to debug.

yetang-equinix · 2020-06-10T22:57:14Z

grpc-server/hardware.go

 		metrics.CacheTotals.With(labels).Inc()
 		metrics.CacheErrors.With(labels).Inc()
-		err = errors.Wrap(err, "unmarshal json")
+		err := errors.New("id must be set to a UUID")


We can indicate that current hw.Id is nil string in this case (add hw.Id to error log). This will give extra info for debugging.

yetang-equinix · 2020-06-10T23:23:13Z

grpc-server/hardware.go

 		default:
 			metrics.WatchMissTotal.Inc()
-			logger.With("id", h.ID).Info("skipping blocked watcher")
+			logger.With("id", hw.Id ).Info("skipping blocked watcher")
 		}
 	}
 	s.watchLock.RUnlock()


Logging is slow, it should not be done between watch lock/unlock. Critical logging for error case is fine. But for information logging, we should do it after unlock.
This comment is for all places in this file that has info logging inside locked sections

yetang-equinix · 2020-06-10T23:26:22Z

http-server/http-server.go

 )

 var (
 	gitRev         = "unknown"
 	gitRevJSON     []byte
+	grpcListenAddr = "localhost:42113"


this port number should be in a config file, not hard coded here.

How should the config file look like? Would it suffice if I just have the port be read from an environment variable and set a default port for when the variable unset?

Yes, there shall be a default value if there is no configured value. If we have quite some values, a file will be easy to manage.

I think the closest to the config file you were referring to is the docker-compose.yml that is currently used to run all of the services.
I have added two new environment variables to allow the user to specify their own ports.

Please refer to this commit here: aee3b9f

…/pass envvars are set

…e port

yetang-equinix · 2020-06-22T16:24:53Z

deploy/docker-compose.yml

@@ -128,32 +131,6 @@ services:
    volumes:
      - ./state/webroot:/usr/share/nginx/html/

-  cacher:


So we totally get ride of cacher from tinkerbell deployment. But do we still deploy cacher from other place? I can still see Hegel still interact with cacher in the other PR.

yeah, cacher is still used in other places, just not in tinkerbell

fixed

mmlb force-pushed the master branch from 5ef2433 to 4905261 Compare April 21, 2020 19:15

kqdeng force-pushed the add-endpoints branch from 5c87407 to 42d84a8 Compare April 21, 2020 20:27

kqdeng requested a review from gauravgahlot April 23, 2020 19:51

kqdeng force-pushed the add-endpoints branch 2 times, most recently from 2f9e9fd to 2630172 Compare April 27, 2020 15:42

gauravgahlot previously requested changes Apr 27, 2020

View reviewed changes

kqdeng force-pushed the add-endpoints branch 2 times, most recently from e3c814a to b4ec49c Compare April 27, 2020 16:33

gauravgahlot reviewed Apr 27, 2020

View reviewed changes

kqdeng force-pushed the add-endpoints branch from 7288c61 to 235da3e Compare April 29, 2020 23:58

mrmrcoleman mentioned this pull request May 19, 2020

REST API for Tink tinkerbell/tinkerbell.org#56

Closed

kqdeng force-pushed the add-endpoints branch 3 times, most recently from 9fb0d30 to 11965b0 Compare May 27, 2020 15:52

kqdeng requested review from gauravgahlot and nathangoulding May 27, 2020 23:18

nathangoulding requested review from mikemrm and mmlb May 29, 2020 17:42

kqdeng changed the title ~~added endpoints for crud operations~~ add endpoints for crud operations & update hardware data model Jun 2, 2020

This was referenced Jun 2, 2020

Add compatibility with tink tinkerbell/hegel#13

Merged

Gauravgahlot/new data model tinkerbell/smee#27

Merged

kqdeng force-pushed the add-endpoints branch from 242db84 to d913e6f Compare June 2, 2020 19:04

mmlb reviewed Jun 2, 2020

View reviewed changes

kqdeng requested a review from yetang-equinix June 10, 2020 19:09

yetang-equinix reviewed Jun 10, 2020

View reviewed changes

kqdeng added 24 commits June 22, 2020 11:35

updated netboot struct

6fa5a8f

readded bootstrapper struct

0b144c1

update dhcp ip

c6388e5

update hardware proto

7f09fbc

added hardware & workflow endpoints

960739c

updated post endpoints to match get endpoints; fixup

ba5f9ef

changed bymac and byip to use post

a2d8201

scrubbed traces of target

12003f7

removed hardware network.default

377f95b

filled out hardware storage struct

1a0b67e

updated storage filesystem struct

a602855

added/updated envvars in docker-compose file

a082fac

fixup hardware & template protos

b54b1d8

added hardware version, updated docker-compose.yml

ec14f11

changed log.Fatal to log.Println; updated basic auth to check if user…

c06bf19

…/pass envvars are set

allow default grpc and http port to be overridden by envvars

0b60b77

brought info log outside of lock/unlock

53872e9

prettier

9b64c8f

fixup

b98e2ac

fixed grpc connection refused; removed cacher from docker-compose.yml

0e994ef

updated http server to handle if TINKERBELL_GRPC_AUTHORITY is just th…

aab2e1c

…e port

fixup

1ddde35

updated hardware proto field names

a2944c1

updated basic auth envvars to default to empty

bc1a5a9

kqdeng force-pushed the add-endpoints branch from f19f327 to bc1a5a9 Compare June 22, 2020 15:35

yetang-equinix reviewed Jun 22, 2020

View reviewed changes

yetang-equinix approved these changes Jun 22, 2020

View reviewed changes

kqdeng merged commit 5e66098 into tinkerbell:master Jun 22, 2020

fransvanberckel mentioned this pull request Jul 12, 2020

Update generate-envrc.sh #215

Merged

add endpoints for crud operations & update hardware data model #64

add endpoints for crud operations & update hardware data model #64

Uh oh!

Conversation

kqdeng commented Apr 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gauravgahlot commented Apr 24, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexellis commented May 18, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yetang-equinix Jun 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kqdeng commented Apr 14, 2020 •

edited

Loading

yetang-equinix Jun 10, 2020 •

edited

Loading