gRPC support for TorchServe #687

harshbafna · 2020-09-17T17:31:51Z

Description

Added gRPC support for inference and management APIs
- inference.proto
- management.proto
Added sample python grpc client in sanity suite and regression suite
Updated sanity suite to run APIs through both REST and gRPC APIs.
Added pytest cases in regresstion suite to validate all the inference and management APIs which were tested in newman suite, through gRPC apis
- test_gRPC_inference_api.py
- test_gRPC_management_apis.py

gRPC vs REST performance report: https://docs.google.com/spreadsheets/d/1Vo6ZmeMgWY1TPiNWPtM41tV8H7Vy8mUuBnG42hsPnbg/edit#gid=1514552587

Fixes #656

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)
This change requires a documentation update

Feature/Issue validation/testing

gRPC_regression.log

Checklist:

Have you added tests that prove your fix is effective or that this feature works?
New and existing unit tests pass locally with these changes?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

sagemaker-neo-ci-bot · 2020-09-17T17:41:30Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 14cb1eb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T17:41:33Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 199811c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T17:41:47Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 199811c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T17:41:59Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 14cb1eb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T18:09:34Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 151cdcc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T18:09:48Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 151cdcc
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T18:24:08Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: d1abb5c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T18:24:17Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: d1abb5c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T18:38:13Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 48b049c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-17T18:38:18Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 48b049c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-09T04:13:41Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 26cfa0a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-09T04:20:39Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 26cfa0a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-09T07:11:13Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: ccdd310
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-09T07:17:11Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: ccdd310
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Resolved

maaquib

@harshbafna Failing regression tests

## DLAMI 38, g4dn.xl
git clone https://github.com/pytorch/serve/
cd serve/
git checkout issue_656
python ts_scripts/install_dependencies.py --environment dev
python torchserve_sanity.py
python test/regression_tests.py

## Successfully started TorchServe
newman

management_api_collection

→ management request
  {{METHOD}} http://localhost:8081/{{path}} [errored]
     Method must be a valid HTTP token ["{{METHOD}}"]
  2. Successful  request

┌─────────────────────────┬──────────┬──────────┐
│                         │ executed │   failed │
├─────────────────────────┼──────────┼──────────┤
│              iterations │        1 │        0 │
├─────────────────────────┼──────────┼──────────┤
│                requests │        1 │        1 │
├─────────────────────────┼──────────┼──────────┤
│            test-scripts │        1 │        0 │
├─────────────────────────┼──────────┼──────────┤
│      prerequest-scripts │        0 │        0 │
├─────────────────────────┼──────────┼──────────┤
│              assertions │        1 │        1 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 75ms                      │
├───────────────────────────────────────────────┤
│ total data received: 0B (approx)              │
└───────────────────────────────────────────────┘

  #  failure                          detail

 1.  TypeError                        Method must be a valid HTTP token ["{{METHOD}}"]
                                      at request
                                      inside "management request"

 2.  AssertionError                   Successful  request
                                      expected undefined to be one of [ 200 ]
                                      at assertion:0 in test-script
                                      inside "management request"
## Stopping TorchServe
## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop
TorchServe has stopped.

sagemaker-neo-ci-bot · 2020-12-10T05:05:56Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: d9bf3a7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T05:07:24Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: d9bf3a7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T05:09:22Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 557f51e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T05:14:45Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 557f51e
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T05:42:59Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: b21a174
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T05:48:47Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: b21a174
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

harshbafna · 2020-12-10T06:00:01Z

@harshbafna Failing regression tests

## DLAMI 38, g4dn.xl
git clone https://github.com/pytorch/serve/
cd serve/
git checkout issue_656
python ts_scripts/install_dependencies.py --environment dev
python torchserve_sanity.py
python test/regression_tests.py

## Successfully started TorchServe
newman

management_api_collection

→ management request
  {{METHOD}} http://localhost:8081/{{path}} [errored]
     Method must be a valid HTTP token ["{{METHOD}}"]
  2. Successful  request

┌─────────────────────────┬──────────┬──────────┐
│                         │ executed │   failed │
├─────────────────────────┼──────────┼──────────┤
│              iterations │        1 │        0 │
├─────────────────────────┼──────────┼──────────┤
│                requests │        1 │        1 │
├─────────────────────────┼──────────┼──────────┤
│            test-scripts │        1 │        0 │
├─────────────────────────┼──────────┼──────────┤
│      prerequest-scripts │        0 │        0 │
├─────────────────────────┼──────────┼──────────┤
│              assertions │        1 │        1 │
├─────────────────────────┴──────────┴──────────┤
│ total run duration: 75ms                      │
├───────────────────────────────────────────────┤
│ total data received: 0B (approx)              │
└───────────────────────────────────────────────┘

  #  failure                          detail

 1.  TypeError                        Method must be a valid HTTP token ["{{METHOD}}"]
                                      at request
                                      inside "management request"

 2.  AssertionError                   Successful  request
                                      expected undefined to be one of [ 200 ]
                                      at assertion:0 in test-script
                                      inside "management request"
## Stopping TorchServe
## In directory: /home/ubuntu/serve/test | Executing command: torchserve --stop
TorchServe has stopped.

@maaquib Fixed the issue.
Logs: gRPC regression.txt

dk19y · 2020-12-10T13:06:20Z

Can confirm that the regression tests are passing for both CPU & GPU in CodeBuild env.

dk19y · 2020-12-10T13:07:20Z

@maaquib i can go ahead and merge this if it looks good to you.

sagemaker-neo-ci-bot · 2020-12-10T17:12:12Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 5e015f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T17:18:36Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 5e015f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T17:44:48Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 5e015f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-12-10T17:57:06Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 5e015f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

harshbafna added 7 commits September 17, 2020 22:49

refactored torchserve job

2142715

added grpc server side implementation

77b1356

added protobuff files

b75b81a

added grpc server startup

d46d56a

fixed valid port test case

fec11bf

automated server stub generation through gradle

22734ba

enhanced sanity script to validate grpc inference api

14cb1eb

harshbafna self-assigned this Sep 17, 2020

harshbafna linked an issue Sep 17, 2020 that may be closed by this pull request

Add support for gRPC #656

Closed

harshbafna added this to the v0.3.0 milestone Sep 17, 2020

Merge branch 'master' into issue_656

199811c

Added grpcio-tools package

151cdcc

fixed path issue in grpc client

d1abb5c

fixed incorrect exit logic in client script

48b049c

harshbafna added 6 commits September 18, 2020 07:18

removed json parse in python gRPC client

897d5d7

removed unnecessary file checkin

0ffc689

added regression test cases for gRPC regression APIs

3770272

added tolerance check

585c03f

added python client stub cleanup

79dd23b

enhanced error handling for inference APIs

7037e6c

Merge branch 'master' into issue_656

ccdd310

maaquib requested changes Dec 9, 2020

View reviewed changes

harshbafna and others added 2 commits December 10, 2020 10:21

fixed management api newman command

557f51e

Merge branch 'master' into issue_656

d9bf3a7

harshbafna added 2 commits December 10, 2020 10:54

fixed import issues

4dfdecc

fixed regression pytest issues

b21a174

harshbafna mentioned this pull request Dec 10, 2020

Add gRPC support for explain model API #854

Open

dk19y approved these changes Dec 10, 2020

View reviewed changes

Merge branch 'master' into issue_656

5e015f4

maaquib approved these changes Dec 10, 2020

View reviewed changes

maaquib merged commit de42bcf into master Dec 10, 2020

harshbafna deleted the issue_656 branch December 22, 2020 07:47

gRPC support for TorchServe #687

gRPC support for TorchServe #687

Uh oh!

Conversation

harshbafna commented Sep 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Feature/Issue validation/testing

Checklist:

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 17, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 9, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 9, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 9, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 9, 2020

AWS CodeBuild CI Report

Uh oh!

maaquib left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sagemaker-neo-ci-bot commented Dec 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Dec 10, 2020

AWS CodeBuild CI Report

Uh oh!

harshbafna commented Dec 10, 2020

Uh oh!

dk19y commented Dec 10, 2020

Uh oh!

dk19y commented Dec 10, 2020

harshbafna commented Sep 17, 2020 •

edited

Loading

maaquib left a comment •

edited

Loading