(feat) opensearch serverless #1922

kukushking · 2023-01-04T16:03:11Z

Feature or Bugfix

Feature

Detail

Update existing client to support serverless
Add wr.opensearch.create_collection
Add helpers to generate default encryption and network policies for collections
Update tests to run against serverless opensearch

Relates

Add integration with OpenSearch Serverless #1917

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

malachi-constant · 2023-01-04T16:27:07Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 984d458
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2023-01-04T18:33:42Z

FYI tests are failing because of missing CodeBuild permissions - in progress.

malachi-constant · 2023-01-04T19:31:55Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 6a41aed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-05T10:43:06Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 6a41aed
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido

Looks good, left some comments

awswrangler/opensearch/_utils.py

jaidisido · 2023-01-05T11:17:57Z

awswrangler/opensearch/_utils.py


 def _get_distribution(client: OpenSearch) -> Any:
+    if getattr(client, "_serverless", False):


Might be mistaken here but isn't the OpenSearch client the same for both serverless and none serverless? In which case the exisiting client.info... line should already do the job?

This is a workaround to avoid calling .info() because info endpoint isn't available in OpenSearch serverless (more here). Sadly there isn't an endpoint to query cluster status or to get the version even - only a few collection/index operations are permitted

Let us ask the OpenSearch team as it feels like an ugly workaround

Sure.

An alternative at this point is to let it send a request and come back with NotFoundError but I think it's better to save on extra requests if we know it's going to fail.

tests/test_opensearch.py

malachi-constant · 2023-01-05T13:16:38Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: c9c8034
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant

Great work @kukushking !

malachi-constant · 2023-01-05T18:30:11Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 65db3a3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-05T21:52:53Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: ef77428
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-05T22:34:47Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 381fa63
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-08T21:30:09Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 95db5f6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-09T10:58:22Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 3725a2b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-09T12:11:23Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 7dae722
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-09T13:01:45Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 79c6c84
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-09T14:43:22Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 41763e2
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

awswrangler/opensearch/_read.py

awswrangler/opensearch/_utils.py

malachi-constant · 2023-01-09T23:08:25Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 371d20d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-10T00:19:15Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 180be9b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-10T11:33:11Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 549dc40
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-10T12:26:27Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 0b76b53
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-10T12:31:44Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: dc40025
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-11T11:48:47Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 77f1151
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-11T11:56:12Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 6a550dd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2023-01-11T12:03:43Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: e2ad779
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking added 4 commits January 3, 2023 17:04

Add OpenSearch serverless support - checkpoint

b526eaa

Add create_collection

cfb8edd

Add default policies

11c3bf1

Checkpoint

984d458

kukushking added the feature label Jan 4, 2023

kukushking requested review from malachi-constant, LeonLuttenberger, jaidisido and cnfait January 4, 2023 16:03

kukushking self-assigned this Jan 4, 2023

pylint

158f1e7

Minor fixes

6a41aed

jaidisido reviewed Jan 5, 2023

View reviewed changes

kukushking commented Jan 5, 2023

View reviewed changes

tests/test_opensearch.py Show resolved Hide resolved

PR feedback - add data policy creation, rename params

c9c8034

kukushking marked this pull request as ready for review January 5, 2023 12:48

Merge branch 'main' into feat/opensearch-serverless

65db3a3

malachi-constant reviewed Jan 5, 2023

View reviewed changes

Remove wait_for refresh policy

ef77428

kukushking added 2 commits January 5, 2023 21:58

Add sleep to replace refresh policy missing in serverless opensearch

3f5db54

Merge branch 'main' into feat/opensearch-serverless

381fa63

Minor - remove unnecessary .sleep()

8ccf230

Upd filter path test to inc coverage

95db5f6

Upd search sql test adding fetch_size to inc coverage

3725a2b

Remove ports check & decrease num of retries

7dae722

Move xfail out as a decorator

79c6c84

Merge branch 'main' into feat/opensearch-serverless

41763e2

LeonLuttenberger reviewed Jan 9, 2023

View reviewed changes

awswrangler/opensearch/_read.py Outdated Show resolved Hide resolved

awswrangler/opensearch/_utils.py Outdated Show resolved Hide resolved

PR feedback - rename exceptions

371d20d

PR feedback - upd exceptions in tests

180be9b

Merge branch 'main' into feat/opensearch-serverless

549dc40

kukushking added 2 commits January 10, 2023 11:59

Upd tutorial

0b76b53

Upd tutorial 2

dc40025

malachi-constant approved these changes Jan 10, 2023

View reviewed changes

kukushking added 3 commits January 11, 2023 11:20

Tutorial - narrow down collection data permissions

77f1151

Tutorial - fix create_index output

6a550dd

Merge branch 'main' into feat/opensearch-serverless

e2ad779

kukushking merged commit d7b7bd6 into main Jan 11, 2023

jaidisido deleted the feat/opensearch-serverless branch January 26, 2023 18:07


		def _get_distribution(client: OpenSearch) -> Any:
		if getattr(client, "_serverless", False):

(feat) opensearch serverless #1922

(feat) opensearch serverless #1922

Uh oh!

Conversation

kukushking commented Jan 4, 2023

Feature or Bugfix

Detail

Relates

Uh oh!

malachi-constant commented Jan 4, 2023

AWS CodeBuild CI Report

Uh oh!

kukushking commented Jan 4, 2023

Uh oh!

malachi-constant commented Jan 4, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 5, 2023

AWS CodeBuild CI Report

Uh oh!

jaidisido left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaidisido Jan 5, 2023

Choose a reason for hiding this comment

Uh oh!

kukushking Jan 5, 2023

Choose a reason for hiding this comment

Uh oh!

jaidisido Jan 5, 2023

Choose a reason for hiding this comment

Uh oh!

kukushking Jan 5, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

malachi-constant commented Jan 5, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant left a comment

Choose a reason for hiding this comment

Uh oh!

malachi-constant commented Jan 5, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 5, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 5, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 8, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 9, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 9, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 9, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 9, 2023

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

malachi-constant commented Jan 9, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 10, 2023

AWS CodeBuild CI Report

Uh oh!

malachi-constant commented Jan 10, 2023

AWS CodeBuild CI Report

Uh oh!