Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions packages/zip-it-and-ship-it/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@
"toml": "^3.0.0",
"unixify": "^1.0.0",
"urlpattern-polyfill": "8.0.2",
"yargs": "^17.0.0"
"yargs": "^17.0.0",
"zod": "^3.23.8"
},
"devDependencies": {
"@types/archiver": "5.3.4",
Expand All @@ -92,7 +93,7 @@
"cardinal": "2.1.1",
"cpy": "9.0.1",
"decompress": "4.2.1",
"deepmerge": "4.3.1",
"deepmerge": "^4.3.1",
"get-stream": "8.0.1",
"is-ci": "3.0.1",
"lambda-local": "2.2.0",
Expand Down
39 changes: 21 additions & 18 deletions packages/zip-it-and-ship-it/src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,35 @@ import { basename, extname, dirname, join } from 'path'

import isPathInside from 'is-path-inside'
import mergeOptions from 'merge-options'
import { z } from 'zod'

import { FunctionSource } from './function.js'
import type { NodeBundlerName } from './runtimes/node/bundlers/types.js'
import type { ModuleFormat } from './runtimes/node/utils/module_format.js'
import { nodeBundler } from './runtimes/node/bundlers/types.js'
import { moduleFormat } from './runtimes/node/utils/module_format.js'
import { minimatch } from './utils/matching.js'

interface FunctionConfig {
externalNodeModules?: string[]
includedFiles?: string[]
includedFilesBasePath?: string
ignoredNodeModules?: string[]
nodeBundler?: NodeBundlerName
nodeSourcemap?: boolean
nodeVersion?: string
rustTargetDirectory?: string
schedule?: string
zipGo?: boolean
name?: string
generator?: string
timeout?: number
export const functionConfig = z.object({
externalNodeModules: z.array(z.string()).optional().catch([]),
generator: z.string().optional().catch(undefined),
includedFiles: z.array(z.string()).optional().catch([]),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not quite sure why, but using undefined as the fallback for an optional value makes more sense to me.

Suggested change
includedFiles: z.array(z.string()).optional().catch([]),
includedFiles: z.array(z.string()).optional().catch(undefined),

includedFilesBasePath: z.string().optional().catch(undefined),
ignoredNodeModules: z.array(z.string()).optional().catch([]),
name: z.string().optional().catch(undefined),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's a lot of duplication in this. does mit make sense to extract z.string().optional().... into a few constants, just so that there's fewer characters on the screen? It reads very dense right now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean, but at the same time I don't know what would be the criteria for extracting something into a constant. We could have optionalString = z.string()).optional(), but is that really saving a lot of characters? We could add .catch(undefined) to that, but what do we do about the strings where we don't want to have a catch? I'm not super convinced this would help a lot, tbh.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any strings where we don't have a catch, though?

Maybe it'd already help to add a few empty lines in between the different parameters. That'd make it less of a uniform wall of text.

nodeBundler: nodeBundler.optional().catch(undefined),
nodeSourcemap: z.boolean().optional().catch(undefined),
nodeVersion: z.string().optional().catch(undefined),
rustTargetDirectory: z.string().optional().catch(undefined),
schedule: z.string().optional().catch(undefined),
timeout: z.number().optional().catch(undefined),
zipGo: z.boolean().optional().catch(undefined),

// Temporary configuration property, only meant to be used by the deploy
// configuration API. Once we start emitting ESM files for all ESM functions,
// we can remove this.
nodeModuleFormat?: ModuleFormat
}
nodeModuleFormat: moduleFormat.optional().catch(undefined),
})

type FunctionConfig = z.infer<typeof functionConfig>

interface FunctionConfigFile {
config: FunctionConfig
Expand Down
4 changes: 2 additions & 2 deletions packages/zip-it-and-ship-it/src/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ export { ArchiveFormat, ARCHIVE_FORMAT } from './archive.js'
export { NodeBundlerName, NODE_BUNDLER } from './runtimes/node/bundlers/types.js'
export { RuntimeName, RUNTIME } from './runtimes/runtime.js'
export { ModuleFormat, MODULE_FORMAT } from './runtimes/node/utils/module_format.js'
export { TrafficRules, Manifest } from './manifest.js'
export { Manifest } from './manifest.js'
export { FunctionResult } from './utils/format_result.js'

export interface ListedFunction {
Expand Down Expand Up @@ -157,7 +157,7 @@ const getListedFunction = function ({
name,
runtime: runtime.name,
runtimeAPIVersion: staticAnalysisResult ? staticAnalysisResult?.runtimeAPIVersion ?? 1 : undefined,
schedule: staticAnalysisResult?.schedule ?? config.schedule,
schedule: staticAnalysisResult?.config?.schedule ?? config.schedule,
inputModuleFormat: staticAnalysisResult?.inputModuleFormat,
}
}
Expand Down
20 changes: 1 addition & 19 deletions packages/zip-it-and-ship-it/src/manifest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,10 @@ import { resolve } from 'path'
import { arch, platform } from 'process'

import type { InvocationMode } from './function.js'
import type { TrafficRules } from './rate_limit.js'
import type { FunctionResult } from './utils/format_result.js'
import type { Route } from './utils/routes.js'

export interface TrafficRules {
action: {
type: string
config: {
rateLimitConfig: {
algorithm: string
windowSize: number
windowLimit: number
}
aggregate: {
keys: {
type: string
}[]
}
to?: string
}
}
}

interface ManifestFunction {
buildData?: Record<string, unknown>
invocationMode?: InvocationMode
Expand Down
85 changes: 62 additions & 23 deletions packages/zip-it-and-ship-it/src/rate_limit.ts
Original file line number Diff line number Diff line change
@@ -1,30 +1,69 @@
export enum RateLimitAlgorithm {
SlidingWindow = 'sliding_window',
}
import { z } from 'zod'

export enum RateLimitAggregator {
Domain = 'domain',
IP = 'ip',
export interface TrafficRules {
action: {
type: string
config: {
rateLimitConfig: {
algorithm: string
windowSize: number
windowLimit: number
}
aggregate: {
keys: {
type: string
}[]
}
to?: string
}
}
}

export enum RateLimitAction {
Limit = 'rate_limit',
Rewrite = 'rewrite',
}
const rateLimitAction = z.enum(['rate_limit', 'rewrite'])
const rateLimitAlgorithm = z.enum(['sliding_window'])
const rateLimitAggregator = z.enum(['domain', 'ip'])
const slidingWindow = z.object({
windowLimit: z.number(),
windowSize: z.number(),
})
const rewriteActionConfig = z.object({
to: z.string(),
})

interface SlidingWindow {
windowLimit: number
windowSize: number
}
export const rateLimit = z
.object({
action: rateLimitAction.optional(),
aggregateBy: rateLimitAggregator.or(z.array(rateLimitAggregator)).optional(),
algorithm: rateLimitAlgorithm.optional(),
})
.merge(slidingWindow)
.merge(rewriteActionConfig.partial())

export type RewriteActionConfig = SlidingWindow & {
to: string
}
type RateLimit = z.infer<typeof rateLimit>

interface RateLimitConfig {
action?: RateLimitAction
aggregateBy?: RateLimitAggregator | RateLimitAggregator[]
algorithm?: RateLimitAlgorithm
}
/**
* Takes a rate limiting configuration object and returns a traffic rules
* object that is added to the manifest.
*/
export const getTrafficRulesConfig = (input: RateLimit): TrafficRules | undefined => {
const { windowSize, windowLimit, algorithm, aggregateBy, action, to } = input
const rateLimitAgg = Array.isArray(aggregateBy) ? aggregateBy : [rateLimitAggregator.Enum.domain]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this incorrectly turns rateLimitAggregator: "ip" into "domain". Should it be this instead?

Suggested change
const rateLimitAgg = Array.isArray(aggregateBy) ? aggregateBy : [rateLimitAggregator.Enum.domain]
const rateLimitAgg = Array.isArray(aggregateBy) ? aggregateBy : [aggregateBy]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going by the current implementation:

const rateLimitAgg = Array.isArray(aggregateBy) ? aggregateBy : [RateLimitAggregator.Domain]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, then let's keep the fixing of that to a different PR. @paulo I think you wrote this (and I reviewed ...), what are your thoughts on it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thought process at the time was to not fail the build and just use the default since it's not an obligatory field, but I'm now regretting it. If I could do it all over again I think I'd just fail the build if aggregateBy is not an array (if defined, not an obligatory field), but leaving the current logic is also ok

const rewriteConfig = to ? { to: input.to } : undefined

export type RateLimit = RateLimitConfig & (SlidingWindow | RewriteActionConfig)
return {
action: {
type: action || rateLimitAction.Enum.rate_limit,
config: {
...rewriteConfig,
rateLimitConfig: {
windowLimit,
windowSize,
algorithm: algorithm || rateLimitAlgorithm.Enum.sliding_window,
},
aggregate: {
keys: rateLimitAgg.map((agg) => ({ type: agg })),
},
},
},
}
}
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import type { Message } from 'esbuild'
import { z } from 'zod'

import type { FunctionConfig } from '../../../config.js'
import type { FeatureFlags } from '../../../feature_flags.js'
import type { FunctionSource } from '../../../function.js'
import { ObjectValues } from '../../../types/utils.js'
import type { RuntimeCache } from '../../../utils/cache.js'
import { Logger } from '../../../utils/logger.js'
import type { ModuleFormat } from '../utils/module_format.js'
Expand All @@ -16,7 +16,9 @@ export const NODE_BUNDLER = {
NONE: 'none',
} as const

export type NodeBundlerName = ObjectValues<typeof NODE_BUNDLER>
export const nodeBundler = z.nativeEnum(NODE_BUNDLER)

export type NodeBundlerName = z.infer<typeof nodeBundler>

// TODO: Create a generic warning type
type BundlerWarning = Message
Expand Down
Loading