diff --git a/source/benchmarking/odm-benchmarking.md b/source/benchmarking/odm-benchmarking.md new file mode 100644 index 0000000000..197d04f223 --- /dev/null +++ b/source/benchmarking/odm-benchmarking.md @@ -0,0 +1,377 @@ +# ODM Performance Benchmarking + +- Status: In progress +- Minimum Server Version: N/A + +## Abstract + +This document describes a standard benchmarking suite for MongoDB ODMs (Object Document Mappers). Much of this +document's structure and content is taken from the existing MongoDB driver benchmarking suite for consistency. + +## Overview + +### Name and purpose + +ODM performance will be measured by the MongoDB ODM Performance Benchmark. It will provide both "horizontal" insights +into how individual ODM performance evolves over time and two types of "vertical" insights: the relative performance of +different ODMs, and the relative performance of ODMs and their associated language drivers. + +We expect substantial performance differences between ODMs based on both their language families (e.g. static vs. +dynamic or compiled vs. virtual-machine-based) as well as their inherent design (e.g. web frameworks such as Django vs. +application-agnostic such as Mongoose). However we still expect "vertical" comparison within families of ODMs to expose +outlier behavior that can be optimized away. + +### Task Hierarchy + +The benchmark suite consists of two groups of small, independent benchmarks. This allows us to better isolate areas +within ODMs that are faster or slower. + +- Flat models -- reading and writing flat models of various sizes, to explore basic operation efficiency +- Nested models -- reading and writing nested models of various sizes, to explore basic operation efficiency for complex + data + +The suite is intentionally kept small for several reasons. One, ODM feature sets vary significantly across libraries. +This limits the number of benchmarks that can be run across the entire collection of extant ODMs. Two, several popular +MongoDB ODMs are actively maintained by third-parties, such as Mongoose. By limiting the benchmarking suite to a minimal +set of representative tests that are easy to implement, we encourage adoption of the suite by these third-party +maintainers. + +### Measurement + +In addition to latency data, all benchmark tasks will be measured in terms of "megabytes/second" (MB/s) of documents +processed, with higher scores being better. (In this document, "megabyte" refers to the SI decimal unit, i.e. 1,000,000 +bytes.) This makes cross-benchmark comparisons easier. + +To avoid various types of measurement skew, tasks will be measured over several iterations. Each iteration will have a +number of operations performed per iteration that depends on the task being benchmarked. The final score for a task will +be the median score of the iterations. A range of percentiles will also be recorded for diagnostic analysis. + +### Data sets + +Data sets will vary by task. In most cases, data sets will be synthetic line-delimited JSON files to be constructed by +the ODM being benchmarked into the appropriate model. Some tasks will require additional modifications to these +constructed models, such as adding generated ObjectIds. + +### Versioning + +The MongoDB ODM Performance Benchmark will have vX.Y versioning. Minor updates and clarifications will increment "Y" and +should have little impact on score comparison. Major changes, such as task modifications, MongoDB version tested +against, or hardware used, will increment "X" to indicate that older version scores are unlikely to be comparable. + +## Benchmark execution phases and measurement + +All benchmark tasks will be conducted via a number of iterations. Each iteration will be timed and will generally +include a large number of individual ODM operations. + +The measurement is broken up this way to better isolate the benchmark from external volatility. If we consider the +problem of benchmarking an operation over many iterations, such as 100,000 model insertions, we want to avoid two +extreme forms of measurement: + +- measuring a single insertion 100,000 times -- in this case, the timing code is likely to be a greater proportion of + executed code, which could routinely evict the insertion code from CPU caches or mislead a JIT optimizer and throw + off results +- measuring 100,000 insertions one time -- in this case, the longer the timer runs, the higher the likelihood that an + external event occurs that affects the time of the run + +Therefore, we choose a middle ground: + +- measuring the same 10,000 insertions over 10 iterations -- each timing run includes enough operations that insertion + code dominates timing code; unusual system events are likely to affect only a fraction of the 10 timing measurements + +With 10 timings of inserting the same 10,000 models, we build up a statistical distribution of the operation timing, +allowing a more robust estimate of performance than a single measurement. (In practice, the number of iterations could +exceed 10, but 10 is a reasonable minimum goal.) + +Because a timing distribution is bounded by zero on one side, taking the mean would allow large positive outlier +measurements to skew the result substantially. Therefore, for the benchmark score, we use the median timing measurement, +which is robust in the face of outliers. + +Each benchmark is structured into discrete setup/execute/teardown phases. Phases are as follows, with specific details +given in a subsequent section: + +- setup -- (ONCE PER TASK) something to do once before any benchmarking, e.g. construct a model object, load test data, + insert data into a collection, etc. +- before operation -- (ONCE PER ITERATION) something to do before every task iteration, e.g. drop a collection, or + reload test data (if the test run modifies it), etc. +- do operation -- (ONCE PER ITERATION) smallest amount of code necessary to execute the task; e.g. insert 10,000 models + one by one into the database, or retrieve 10,000 models of test data from the database, etc. +- after operation -- (ONCE PER ITERATION) something to do after every task iteration (if necessary) +- teardown -- (ONCE PER TASK) something done once after all benchmarking is complete (if necessary); e.g. drop the test + database + +The wall-clock execution time of each "do operation" phase will be recorded. We use wall clock time to model user +experience and as a lowest-common denominator across ODMs. Iteration timing should be done with a high-resolution +monotonic timer (or best language approximation). + +Unless otherwise specified, the number of iterations to measure per task is variable: + +- iterations should loop for at least 30 seconds cumulative execution time +- iterations should stop after 10 iterations or 1 minute cumulative execution time, whichever is shorter + +This balances measurement stability with a timing cap to ensure all tasks can complete in a reasonable time. + +For each task, the 10th, 25th, 50th, 75th, 90th, 95th, 98th and 99th percentiles will be recorded using the following +algorithm: + +- Given a 0-indexed array A of N iteration wall clock times +- Sort the array into ascending order (i.e. shortest time first) +- Let the index i for percentile p in the range [1,100] be defined as: `i = int(N * p / 100) - 1` + +*N.B. This is the [Nearest Rank](https://en.wikipedia.org/wiki/Percentile#The_Nearest_Rank_method) algorithm, chosen for +its utter simplicity given that it needs to be implemented identically across a wide variety of ODMs and languages.* + +The 50th percentile (i.e. the median) will be used for score composition. Other percentiles will be stored for +visualizations and analysis. + +Each task will have defined for it an associated size in megabytes (MB). This size will be calculated using the task's +dataset size and the number of documents processed per iteration. The benchmarking score for each task will be the task +size in MB divided by the median wall clock time. + +## Benchmark task definitions + +Datasets are available in the `odm-data` directory adjacent to this spec. + +Note: The term "LDJSON" means "line-delimited JSON", which should be understood to mean a collection of UTF-8 encoded +JSON documents (without embedded CR or LF characters), separated by a single LF character. (Some Internet definition of +line-delimited JSON use CRLF delimiters, but this benchmark uses only LF.) + +### Flat models + +Datasets are in the `flat_models` tarball. + +Flat model tests focus on flatly-structured model reads and writes across data sizes. They are designed to give insights +into the efficiency of the ODM's implementation of basic data operations. + +The data will be stored as strict JSON with no extended types. These JSON representations must be converted into +equivalent models as part of each benchmark task. + +Flat model benchmark tasks include:s + +- Small model creation +- Small model update +- Small model find by filter +- Small model find foreign key by filter (if joins are supported) +- Large model creation +- Large model update + +#### Small model creation + +Summary: This benchmark tests ODM performance creating a single small model. + +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 +operations, which equals 2,250,000 bytes or 2.5 MB. + +This benchmark uses a comparable dataset to the driver `small doc insertOne` benchmark, allowing for direct comparisons. + +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory. | +| Before task | n/a. | +| Do task | Create an ODM-appropriate model instance for the SMALL_DOC document and save it to the database. Repeat this 10,000 times. | +| After task | Drop the collection associated with the SMALL_DOC model. | +| Teardown | n/a. | + +#### Small model update + +Summary: This benchmark tests ODM performance updating fields on a single small model. + +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `updated_value` string file (13 bytes) times +10,000 operations, which equals 130,000 bytes or 130 KB. + +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | +| Before task | n/a. | +| Do task | Update the `field1` field for each instance of the model to equal `updated_value` in an ODM-appropriate manner. | +| After task | Drop the collection associated with the SMALL_DOC model. | +| Teardown | n/a. | + +#### Small model find by filter + +Summary: This benchmark tests ODM performance finding documents using a basic filter. + +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 +operations, which equals 2,250,000 bytes or 2.5 MB. + +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the inserted `id` field for each into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 `id` values, perform a filter operation to find the corresponding SMALL_DOC model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the SMALL_DOC model. | + +#### Small model find foreign key by filter + +Summary: This benchmark tests ODM performance finding documents by foreign keys. This benchmark must only be run by ODMs +that support join ($lookup) operations. + +Dataset: The dataset (SMALL_DOC) is contained within `small_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 250 bytes. An additional model (FOREIGN_KEY) representing the foreign key, +consisting of only a string field called `name`, must also be created. + +Dataset size: For scoring purposes, the dataset size is the size of the `small_doc` source file (250 bytes) times 10,000 +operations, which equals 2,250,000 bytes or 2.5 MB. + +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the SMALL_DOC dataset into memory as an ODM-appropriate model object. For each SMALL_DOC model, create and assign a FOREIGN_KEY instance to the `field_fk` field. Insert 10,000 instances of both models into the database, saving the inserted `id` field for each FOREIGN_KEY into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 FOREIGN_KEY `id` values, perform a filter operation in an ODM-appropriate manner to find the corresponding SMALL_DOC model. | +| After task | n/a. | +| Teardown | Drop the collections associated with the SMALL_DOC and FOREIGN_KEY models. | + +#### Large model creation + +Summary: This benchmark tests ODM performance creating a single large model. + +Dataset: The dataset (LARGE_DOC) is contained within `large_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc` source file (8,000 bytes) times +10,000 operations, which equals 80,000,000 bytes or 80 MB. + +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC dataset into memory. | +| Before task | n/a. | +| Do task | Create an ODM-appropriate model instance for the LARGE_DOC document and save it to the database. Repeat this 10,000 times. | +| After task | Drop the collection associated with the LARGE_DOC model. | +| Teardown | n/a. | + +#### Large model update + +Summary: This benchmark tests ODM performance updating fields on a single large model. + +Dataset: The dataset (LARGE_DOC) is contained within `large_doc.json` and consists of a sample document stored as strict +JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `updated_value` string file (13 bytes) times +10,000 operations, which equals 130,000 bytes or 130 KB. + +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | +| Before task | n/a. | +| Do task | Update the `field1` field for each instance of the model to `updated_value` in an ODM-appropriate manner. | +| After task | Drop the collection associated with the LARGE_DOC model. | +| Teardown | n/a. | + +### Nested models + +Datasets are in the `nested_models` tarball. + +Nested model tests focus performing reads and writes on models containing nested (embedded) documents. They are designed +to give insights into the efficiency of operations on the more complex data structures enabled by the document model. + +The data will be stored as strict JSON with no extended types. These JSON representations must be converted into +equivalent ODM models as part of each benchmark task. + +Nested model benchmark tasks include:s + +- Large model creation +- Large model update nested field +- Large model find nested field by filter +- Large model find nested array field by filter + +#### Large model creation + +Summary: This benchmark tests ODM performance creating a single large nested model. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) +times 10,000 operations, which equals 80,000,000 bytes or 80 MB. + +| Phase | Description | +| ----------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory. | +| Before task | n/a. | +| Do task | Create an ODM-appropriate model instance for the LARGE_DOC_NESTED document and save it to the database. Repeat this 10,000 times. | +| After task | Drop the collection associated with the LARGE_DOC_NESTED model. | +| Teardown | n/a. | + +#### Large model update nested + +Summary: This benchmark tests ODM performance updating nested fields on a single large model. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `updated_value` string file (13 bytes) times +10,000 operations, which equals 130,000 bytes or 130 KB. + +| Phase | Description | +| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Save 10,000 instances into the database. | +| Before task | n/a. | +| Do task | Update the value of the `embedded_str_doc_1.field1` field to `updated_value` in an ODM-appropriate manner for each instance of the model. | +| After task | Drop the collection associated with the LARGE_DOC_NESTED model. | +| Teardown | n/a. | + +#### Large nested model find nested by filter + +Summary: This benchmark tests ODM performance finding nested documents using a basic filter. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) +times 10,000 operations, which equals 80,000,000 bytes or 80 MB. + +| Phase | Description | +| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the `unique_id` field for each model's `embedded_str_doc_1` nested model into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 `embedded_str_doc_1.unique_id` values, perform a filter operation to search for the parent LARGE_DOC_NESTED model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | + +#### Large nested model find nested array by filter + +Summary: This benchmark tests ODM performance finding nested document arrays using a basic filter. + +Dataset: The dataset (LARGE_DOC_NESTED) is contained within `large_doc_nested.json` and consists of a sample document +stored as strict JSON with an encoded length of approximately 8,000 bytes. + +Dataset size: For scoring purposes, the dataset size is the size of the `large_doc_nested` source file (8,000 bytes) +times 10,000 operations, which equals 80,000,000 bytes or 80 MB. + +| Phase | Description | +| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Setup | Load the LARGE_DOC_NESTED dataset into memory as an ODM-appropriate model object. Insert 10,000 instances into the database, saving the value of the `unique_id` field for the first item in each model's `embedded_str_doc_array` nested model into a list. | +| Before task | n/a. | +| Do task | For each of the 10,000 `unique_id` values, perform a filter operation to search for the parent LARGE_DOC_NESTED model. | +| After task | n/a. | +| Teardown | Drop the collection associated with the LARGE_DOC_NESTED model. | + +## Benchmark platform, configuration and environments + +### Benchmark Client + +The benchmarks should be run with the most recent stable version of the ODM and the newest version of the driver it +supports. + +### Benchmark Server + +The MongoDB ODM Performance Benchmark must be run against a standalone MongoDB server running the latest stable database +version without authentication or SSL enabled. + +### Benchmark placement and scheduling + +The MongoDB ODM Performance Benchmark should be placed within the ODM's test directory as an independent test suite. Due +to the relatively long runtime of the benchmarks, including them as part of an automated suite that runs against every +PR is not recommended. Instead, scheduling benchmark runs on a regular cadence is the recommended method of automating +this suite of tests. + +## Changelog diff --git a/source/benchmarking/odm-data/flat-models/large_doc.json b/source/benchmarking/odm-data/flat-models/large_doc.json new file mode 100644 index 0000000000..8a931b89d7 --- /dev/null +++ b/source/benchmarking/odm-data/flat-models/large_doc.json @@ -0,0 +1,252 @@ +{ + "field1": "kj9$mxz#p2qw8r*vn4@h7c&u1s", + "field2": "x3@9zf#mk7w$qp2n8v*r6j4h&c1u5s0a", + "field3": "p9#m2x$k7z@w3q8v*n4r&j6h1c5u0s", + "field4": "z8@x3m#k9w$p2q7v*r4n6j&h1c5u0s", + "field5": "m7#k9x$z3w@p8q2v*n4r6j&h1c5u0s", + "field6": "k6$x9m#z7w3p@q8v2n*r4j6h&c1u5s0", + "field7": "x5@m9k#z6w$p3q7v*n2r8j&h4c1u0s", + "field8": "m4#x8k$z9w6p@q3v*n7r2j&h5c1u0s", + "field9": "k3$m7x#z8w9p@q6v*n2r4j&h1c5u0s", + "field10": "x2@k6m#z7w8p$q9v*n3r1j&h4c5u0s", + "field11": "m1#x5k$z6w7p@q8v*n9r2j&h3c4u0s", + "field12": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field13": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field14": "m8#k2x$z3w4p@q5v*n6r9j&h0c1u7s", + "field15": "k7$x1m#z2w3p@q4v*n5r8j&h9c0u6s", + "field16": "x6@m0k#z1w2p$q3v*n4r7j&h8c9u5s", + "field17": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s", + "field18": "k4$m8x#z9w0p@q1v*n2r5j&h6c7u3s", + "field19": "x3@k7m#z8w9p$q0v*n1r4j&h5c6u2s", + "field20": "m2#x6k$z7w8p@q9v*n0r3j&h4c5u1s", + "field21": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s", + "field22": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s", + "field23": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s", + "field24": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s", + "field25": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s", + "field26": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s", + "field27": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s", + "field28": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s", + "field29": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s", + "field30": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s", + "field31": "x1@m5k#z6w7p$q8v*n9r2j&h3c4u0s", + "field32": "m0#k4x$z5w6p@q7v*n8r1j&h2c3u9s", + "field33": "k9$x3m#z4w5p@q6v*n7r0j&h1c2u8s", + "field34": "x8@m2k#z3w4p$q5v*n6r9j&h0c1u7s", + "field35": "m7#x1k$z2w3p@q4v*n5r8j&h9c0u6s", + "field36": "k6$m0x#z1w2p@q3v*n4r7j&h8c9u5s", + "field37": "x5@k9m#z0w1p$q2v*n3r6j&h7c8u4s", + "field38": "m4#x8k$z9w0p@q1v*n2r5j&h6c7u3s", + "field39": "k3$x7m#z8w9p@q0v*n1r4j&h5c6u2s", + "field40": "x2@m6k#z7w8p$q9v*n0r3j&h4c5u1s", + "field41": "m1#k5x$z6w7p@q8v*n9r2j&h3c4u0s", + "field42": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field43": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field44": "m8#x2k$z3w4p@q5v*n6r9j&h0c1u7s", + "field45": "k7$m1x#z2w3p@q4v*n5r8j&h9c0u6s", + "field46": "x6@k0m#z1w2p$q3v*n4r7j&h8c9u5s", + "field47": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s", + "field48": "k4$m8x#z9w0p@q1v*n2r5j&h6c7u3s", + "field49": "x3@k7m#z8w9p$q0v*n1r4j&h5c6u2s", + "field50": "m2#x6k$z7w8p@q9v*n0r3j&h4c5u1s", + "field51": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s9f8d", + "field52": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s7g6e", + "field53": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s5b4a", + "field54": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s3v2t", + "field55": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s1y0w", + "field56": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s9i8o", + "field57": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s7l6p", + "field58": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s5m4n", + "field59": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s3q2r", + "field60": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s1x0z", + "field61": "p7#q8r$s9t@u0v*w1x&y2z3a4b5c6d", + "field62": "f3$g4h#i5j@k6l*m7n&o8p9q0r1s2t", + "field63": "v5@w6x#y7z$a8b*c9d&e0f1g2h3i4j", + "field64": "n2#o3p$q4r@s5t*u6v&w7x8y9z0a1b", + "field65": "j8$k9l#m0n@o1p*q2r&s3t4u5v6w7x", + "field66": "b4@c5d#e6f$g7h*i8j&k9l0m1n2o3p", + "field67": "x6#y7z$a8b@c9d*e0f&g1h2i3j4k5l", + "field68": "t0$u1v#w2x@y3z*a4b&c5d6e7f8g9h", + "field69": "l9@m0n#o1p$q2r*s3t&u4v5w6x7y8z", + "field70": "h5#i6j$k7l@m8n*o9p&q0r1s2t3u4v", + "field71": "d1@e2f#g3h$i4j*k5l&m6n7o8p9q0r", + "field72": "z7$a8b#c9d@e0f*g1h&i2j3k4l5m6n", + "field73": "v3@w4x#y5z$a6b*c7d&e8f9g0h1i2j", + "field74": "r9#s0t$u1v@w2x*y3z&a4b5c6d7e8f", + "field75": "n5$o6p#q7r@s8t*u9v&w0x1y2z3a4b", + "field76": "j1@k2l#m3n$o4p*q5r&s6t7u8v9w0x", + "field77": "f7#g8h$i9j@k0l*m1n&o2p3q4r5s6t", + "field78": "b3@c4d#e5f$g6h*i7j&k8l9m0n1o2p", + "field79": "x9$y0z#a1b@c2d*e3f&g4h5i6j7k8l", + "field80": "t5#u6v$w7x@y8z*a9b&c0d1e2f3g4h", + "field81": "p1@q2r#s3t$u4v*w5x&y6z7a8b9c0d", + "field82": "l7#m8n$o9p@q0r*s1t&u2v3w4x5y6z", + "field83": "h3@i4j#k5l$m6n*o7p&q8r9s0t1u2v", + "field84": "d9$e0f#g1h@i2j*k3l&m4n5o6p7q8r", + "field85": "z5#a6b$c7d@e8f*g9h&i0j1k2l3m4n", + "field86": "v1@w2x#y3z$a4b*c5d&e6f7g8h9i0j", + "field87": "r7#s8t$u9v@w0x*y1z&a2b3c4d5e6f", + "field88": "n3@o4p#q5r$s6t*u7v&w8x9y0z1a2b", + "field89": "j9$k0l#m1n@o2p*q3r&s4t5u6v7w8x", + "field90": "f5#g6h$i7j@k8l*m9n&o0p1q2r3s4t", + "field91": "b1@c2d#e3f$g4h*i5j&k6l7m8n9o0p", + "field92": "x7#y8z$a9b@c0d*e1f&g2h3i4j5k6l", + "field93": "t3@u4v#w5x$y6z*a7b&c8d9e0f1g2h", + "field94": "p9$q0r#s1t@u2v*w3x&y4z5a6b7c8d", + "field95": "l5#m6n$o7p@q8r*s9t&u0v1w2x3y4z", + "field96": "h1@i2j#k3l$m4n*o5p&q6r7s8t9u0v", + "field97": "d7$e8f#g9h@i0j*k1l&m2n3o4p5q6r", + "field98": "z3#a4b$c5d@e6f*g7h&i8j9k0l1m2n", + "field99": "v9@w0x#y1z$a2b*c3d&e4f5g6h7i8j", + "field100": "r5#s6t$u7v@w8x*y9z&a0b1c2d3e4f", + "field101": "n1@o2p#q3r$s4t*u5v&w6x7y8z9a0b", + "field102": "j7$k8l#m9n@o0p*q1r&s2t3u4v5w6x", + "field103": "f3#g4h$i5j@k6l*m7n&o8p9q0r1s2t", + "field104": "b9@c0d#e1f$g2h*i3j&k4l5m6n7o8p", + "field105": "x5$y6z#a7b@c8d*e9f&g0h1i2j3k4l", + "field106": "t1#u2v$w3x@y4z*a5b&c6d7e8f9g0h", + "field107": "p7@q8r#s9t$u0v*w1x&y2z3a4b5c6d", + "field108": "l3#m4n$o5p@q6r*s7t&u8v9w0x1y2z", + "field109": "h9$i0j#k1l@m2n*o3p&q4r5s6t7u8v", + "field110": "d5@e6f#g7h$i8j*k9l&m0n1o2p3q4r", + "field111": "z1#a2b$c3d@e4f*g5h&i6j7k8l9m0n", + "field112": "v7$w8x#y9z@a0b*c1d&e2f3g4h5i6j", + "field113": "r3@s4t#u5v$w6x*y7z&a8b9c0d1e2f", + "field114": "n9#o0p$q1r@s2t*u3v&w4x5y6z7a8b", + "field115": "j5$k6l#m7n@o8p*q9r&s0t1u2v3w4x", + "field116": "f1@g2h#i3j$k4l*m5n&o6p7q8r9s0t", + "field117": "b7#c8d$e9f@g0h*i1j&k2l3m4n5o6p", + "field118": "x3@y4z#a5b$c6d*e7f&g8h9i0j1k2l", + "field119": "t9$u0v#w1x@y2z*a3b&c4d5e6f7g8h", + "field120": "p5#q6r$s7t@u8v*w9x&y0z1a2b3c4d", + "field121": "l1@m2n#o3p$q4r*s5t&u6v7w8x9y0z", + "field122": "h7$i8j#k9l@m0n*o1p&q2r3s4t5u6v", + "field123": "d3@e4f#g5h$i6j*k7l&m8n9o0p1q2r", + "field124": "z9#a0b$c1d@e2f*g3h&i4j5k6l7m8n", + "field125": "v5$w6x#y7z@a8b*c9d&e0f1g2h3i4j", + "field126": 42, + "field127": 789, + "field128": 156, + "field129": 923, + "field130": 347, + "field131": 681, + "field132": 294, + "field133": 835, + "field134": 167, + "field135": 459, + "field136": 672, + "field137": 381, + "field138": 928, + "field139": 514, + "field140": 760, + "field141": 293, + "field142": 846, + "field143": 107, + "field144": 582, + "field145": 734, + "field146": 395, + "field147": 861, + "field148": 248, + "field149": 657, + "field150": 419, + "field151": 703, + "field152": 186, + "field153": 542, + "field154": 698, + "field155": 375, + "field156": 829, + "field157": 164, + "field158": 517, + "field159": 743, + "field160": 286, + "field161": 904, + "field162": 358, + "field163": 621, + "field164": 795, + "field165": 432, + "field166": 876, + "field167": 189, + "field168": 653, + "field169": 407, + "field170": 728, + "field171": 564, + "field172": 391, + "field173": 817, + "field174": 275, + "field175": 938, + "field176": 102, + "field177": 586, + "field178": 749, + "field179": 423, + "field180": 867, + "field181": 234, + "field182": 695, + "field183": 458, + "field184": 701, + "field185": 316, + "field186": 872, + "field187": 539, + "field188": 684, + "field189": 297, + "field190": 815, + "field191": 463, + "field192": 728, + "field193": 156, + "field194": 934, + "field195": 507, + "field196": 682, + "field197": 349, + "field198": 876, + "field199": 214, + "field200": 758, + "field201": 436, + "field202": 691, + "field203": 325, + "field204": 809, + "field205": 582, + "field206": 147, + "field207": 763, + "field208": 498, + "field209": 625, + "field210": 384, + "field211": 917, + "field212": 256, + "field213": 743, + "field214": 469, + "field215": 836, + "field216": 172, + "field217": 594, + "field218": 721, + "field219": 358, + "field220": 907, + "field221": 283, + "field222": 649, + "field223": 416, + "field224": 785, + "field225": 532, + "field226": 698, + "field227": 245, + "field228": 871, + "field229": 409, + "field230": 756, + "field231": 183, + "field232": 627, + "field233": 394, + "field234": 819, + "field235": 568, + "field236": 742, + "field237": 125, + "field238": 896, + "field239": 453, + "field240": 607, + "field241": 784, + "field242": 329, + "field243": 856, + "field244": 291, + "field245": 673, + "field246": 418, + "field247": 905, + "field248": 562, + "field249": 739, + "field250": 864 +} \ No newline at end of file diff --git a/source/benchmarking/odm-data/flat-models/small_doc.json b/source/benchmarking/odm-data/flat-models/small_doc.json new file mode 100644 index 0000000000..d0727271fd --- /dev/null +++ b/source/benchmarking/odm-data/flat-models/small_doc.json @@ -0,0 +1 @@ +{"field1":"miNVpaKW","field2":"CS5VwrwN","field3":"Oq5Csk1w","field4":"ZPm57dhu","field5":"gxUpzIjg","field6":"Smo9whci","field7":"TW34kfzq","field8":55336395,"field9":41992681,"field10":72188733,"field11":46660880,"field12":3527055,"field13":74094448} \ No newline at end of file diff --git a/source/benchmarking/odm-data/flat_models.tgz b/source/benchmarking/odm-data/flat_models.tgz new file mode 100644 index 0000000000..9c550f40e7 Binary files /dev/null and b/source/benchmarking/odm-data/flat_models.tgz differ diff --git a/source/benchmarking/odm-data/nested-models/large_doc_nested.json b/source/benchmarking/odm-data/nested-models/large_doc_nested.json new file mode 100644 index 0000000000..497a6e2551 --- /dev/null +++ b/source/benchmarking/odm-data/nested-models/large_doc_nested.json @@ -0,0 +1,242 @@ +{ + "embedded_str_doc_1": { + "field1": "kj9$mxz#p2qw8r*vn4@h7c&u1s", + "field2": "x3@9zf#mk7w$qp2n8v*r6j4h&c1u5s0a", + "field3": "p9#m2x$k7z@w3q8v*n4r&j6h1c5u0s", + "field4": "z8@x3m#k9w$p2q7v*r4n6j&h1c5u0s", + "field5": "m7#k9x$z3w@p8q2v*n4r6j&h1c5u0s", + "field6": "k6$x9m#z7w3p@q8v2n*r4j6h&c1u5s0", + "field7": "x5@m9k#z6w$p3q7v*n2r8j&h4c1u0s", + "field8": "m4#x8k$z9w6p@q3v*n7r2j&h5c1u0s", + "field9": "k3$m7x#z8w9p@q6v*n2r4j&h1c5u0s", + "field10": "x2@k6m#z7w8p$q9v*n3r1j&h4c5u0s", + "field11": "m1#x5k$z6w7p@q8v*n9r2j&h3c4u0s", + "field12": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field13": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field14": "m8#k2x$z3w4p@q5v*n6r9j&h0c1u7s", + "field15": "k7$x1m#z2w3p@q4v*n5r8j&h9c0u6s" + }, + "embedded_str_doc_2": { + "field1": "x6@m0k#z1w2p$q3v*n4r7j&h8c9u5s", + "field2": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s", + "field3": "k4$m8x#z9w0p@q1v*n2r5j&h6c7u3s", + "field4": "x3@k7m#z8w9p$q0v*n1r4j&h5c6u2s", + "field5": "m2#x6k$z7w8p@q9v*n0r3j&h4c5u1s", + "field6": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s", + "field7": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s", + "field8": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s", + "field9": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s", + "field10": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s", + "field11": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s", + "field12": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s", + "field13": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s", + "field14": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s", + "field15": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s" + }, + "embedded_str_doc_3": { + "field1": "k9$x3m#z4w5p@q6v*n7r0j&h1c2u8s", + "field2": "x8@m2k#z3w4p$q5v*n6r9j&h0c1u7s", + "field3": "m7#x1k$z2w3p@q4v*n5r8j&h9c0u6s", + "field4": "k6$m0x#z1w2p@q3v*n4r7j&h8c9u5s", + "field5": "x5@k9m#z0w1p$q2v*n3r6j&h7c8u4s", + "field6": "m4#x8k$z9w0p@q1v*n2r5j&h6c7u3s", + "field7": "k3$x7m#z8w9p@q0v*n1r4j&h5c6u2s", + "field8": "x2@m6k#z7w8p$q9v*n0r3j&h4c5u1s", + "field9": "m1#k5x$z6w7p@q8v*n9r2j&h3c4u0s", + "field10": "k0$x4m#z5w6p@q7v*n8r1j&h2c3u9s", + "field11": "x9@m3k#z4w5p$q6v*n7r0j&h1c2u8s", + "field12": "m8#x2k$z3w4p@q5v*n6r9j&h0c1u7s", + "field13": "k7$m1x#z2w3p@q4v*n5r8j&h9c0u6s", + "field14": "x6@k0m#z1w2p$q3v*n4r7j&h8c9u5s", + "field15": "m5#x9k$z0w1p@q2v*n3r6j&h7c8u4s" + }, + "embedded_str_doc_4": { + "field1": "k1$x5m#z6w7p@q8v*n9r2j&h3c4u0s9f8d", + "field2": "x0@m4k#z5w6p$q7v*n8r1j&h2c3u9s7g6e", + "field3": "m9#k3x$z4w5p@q6v*n7r0j&h1c2u8s5b4a", + "field4": "k8$x2m#z3w4p@q5v*n6r9j&h0c1u7s3v2t", + "field5": "x7@m1k#z2w3p$q4v*n5r8j&h9c0u6s1y0w", + "field6": "m6#x0k$z1w2p@q3v*n4r7j&h8c9u5s9i8o", + "field7": "k5$m9x#z0w1p@q2v*n3r6j&h7c8u4s7l6p", + "field8": "x4@k8m#z9w0p$q1v*n2r5j&h6c7u3s5m4n", + "field9": "m3#x7k$z8w9p@q0v*n1r4j&h5c6u2s3q2r", + "field10": "k2$x6m#z7w8p@q9v*n0r3j&h4c5u1s1x0z", + "field11": "p7#q8r$s9t@u0v*w1x&y2z3a4b5c6d", + "field12": "f3$g4h#i5j@k6l*m7n&o8p9q0r1s2t", + "field13": "v5@w6x#y7z$a8b*c9d&e0f1g2h3i4j", + "field14": "n2#o3p$q4r@s5t*u6v&w7x8y9z0a1b", + "field15": "j8$k9l#m0n@o1p*q2r&s3t4u5v6w7x" + }, + "embedded_str_doc_5": { + "field1": "d1@e2f#g3h$i4j*k5l&m6n7o8p9q0r", + "field2": "z7$a8b#c9d@e0f*g1h&i2j3k4l5m6n", + "field3": "v3@w4x#y5z$a6b*c7d&e8f9g0h1i2j", + "field4": "r9#s0t$u1v@w2x*y3z&a4b5c6d7e8f", + "field5": "n5$o6p#q7r@s8t*u9v&w0x1y2z3a4b", + "field6": "j1@k2l#m3n$o4p*q5r&s6t7u8v9w0x", + "field7": "f7#g8h$i9j@k0l*m1n&o2p3q4r5s6t", + "field8": "b3@c4d#e5f$g6h*i7j&k8l9m0n1o2p", + "field9": "x9$y0z#a1b@c2d*e3f&g4h5i6j7k8l", + "field10": "t5#u6v$w7x@y8z*a9b&c0d1e2f3g4h", + "field11": "p1@q2r#s3t$u4v*w5x&y6z7a8b9c0d", + "field12": "l7#m8n$o9p@q0r*s1t&u2v3w4x5y6z", + "field13": "h3@i4j#k5l$m6n*o7p&q8r9s0t1u2v", + "field14": "d9$e0f#g1h@i2j*k3l&m4n5o6p7q8r", + "field15": "z5#a6b$c7d@e8f*g9h&i0j1k2l3m4n" + }, + "embedded_str_doc_array": [ + { + "field1": "n3@o4p#q5r$s6t*u7v&w8x9y0z1a2b", + "field2": "j9$k0l#m1n@o2p*q3r&s4t5u6v7w8x", + "field3": "f5#g6h$i7j@k8l*m9n&o0p1q2r3s4t", + "field4": "b1@c2d#e3f$g4h*i5j&k6l7m8n9o0p", + "field5": "x7#y8z$a9b@c0d*e1f&g2h3i4j5k6l", + "field6": "t3@u4v#w5x$y6z*a7b&c8d9e0f1g2h", + "field7": "p9$q0r#s1t@u2v*w3x&y4z5a6b7c8d", + "field8": "l5#m6n$o7p@q8r*s9t&u0v1w2x3y4z", + "field9": "h1@i2j#k3l$m4n*o5p&q6r7s8t9u0v", + "field10": "d7$e8f#g9h@i0j*k1l&m2n3o4p5q6r", + "field11": "z3#a4b$c5d@e6f*g7h&i8j9k0l1m2n", + "field12": "v9@w0x#y1z$a2b*c3d&e4f5g6h7i8j", + "field13": "r5#s6t$u7v@w8x*y9z&a0b1c2d3e4f", + "field14": "n1@o2p#q3r$s4t*u5v&w6x7y8z9a0b", + "field15": "j7$k8l#m9n@o0p*q1r&s2t3u4v5w6x" + }, + { + "field1": "t1#u2v$w3x@y4z*a5b&c6d7e8f9g0h", + "field2": "p7@q8r#s9t$u0v*w1x&y2z3a4b5c6d", + "field3": "l3#m4n$o5p@q6r*s7t&u8v9w0x1y2z", + "field4": "h9$i0j#k1l@m2n*o3p&q4r5s6t7u8v", + "field5": "d5@e6f#g7h$i8j*k9l&m0n1o2p3q4r", + "field6": "z1#a2b$c3d@e4f*g5h&i6j7k8l9m0n", + "field7": "v7$w8x#y9z@a0b*c1d&e2f3g4h5i6j", + "field8": "r3@s4t#u5v$w6x*y7z&a8b9c0d1e2f", + "field9": "n9#o0p$q1r@s2t*u3v&w4x5y6z7a8b", + "field10": "j5$k6l#m7n@o8p*q9r&s0t1u2v3w4x", + "field11": "f1@g2h#i3j$k4l*m5n&o6p7q8r9s0t", + "field12": "b7#c8d$e9f@g0h*i1j&k2l3m4n5o6p", + "field13": "x3@y4z#a5b$c6d*e7f&g8h9i0j1k2l", + "field14": "t9$u0v#w1x@y2z*a3b&c4d5e6f7g8h", + "field15": "p5#q6r$s7t@u8v*w9x&y0z1a2b3c4d" + } + ], + "embedded_int_doc_8": { + "field1": 42, + "field2": 789, + "field3": 156, + "field4": 923, + "field5": 347, + "field6": 681, + "field7": 294, + "field8": 835, + "field9": 167, + "field10": 459, + "field11": 672, + "field12": 381, + "field13": 928, + "field14": 514, + "field15": 760 + }, + "embedded_int_doc_9": { + "field1": 846, + "field2": 107, + "field3": 582, + "field4": 734, + "field5": 395, + "field6": 861, + "field7": 248, + "field8": 657, + "field9": 419, + "field10": 703, + "field11": 186, + "field12": 542, + "field13": 698, + "field14": 375, + "field15": 829 + }, + "embedded_int_doc_10": { + "field1": 904, + "field2": 358, + "field3": 621, + "field4": 795, + "field5": 432, + "field6": 876, + "field7": 189, + "field8": 653, + "field9": 407, + "field10": 728, + "field11": 564, + "field12": 391, + "field13": 817, + "field14": 275, + "field15": 938 + }, + "embedded_int_doc_11": { + "field1": 867, + "field2": 234, + "field3": 695, + "field4": 458, + "field5": 701, + "field6": 316, + "field7": 872, + "field8": 539, + "field9": 684, + "field10": 297, + "field11": 815, + "field12": 463, + "field13": 728, + "field14": 156, + "field15": 934 + }, + "embedded_int_doc_12": { + "field1": 214, + "field2": 758, + "field3": 436, + "field4": 691, + "field5": 325, + "field6": 809, + "field7": 582, + "field8": 147, + "field9": 763, + "field10": 498, + "field11": 625, + "field12": 384, + "field13": 917, + "field14": 256, + "field15": 743 + }, + "embedded_int_doc_13": { + "field1": 721, + "field2": 358, + "field3": 907, + "field4": 283, + "field5": 649, + "field6": 416, + "field7": 785, + "field8": 532, + "field9": 698, + "field10": 245, + "field11": 871, + "field12": 409, + "field13": 756, + "field14": 183, + "field15": 627 + }, + "embedded_int_doc_14": { + "field1": 896, + "field2": 453, + "field3": 607, + "field4": 784, + "field5": 329, + "field6": 856, + "field7": 291, + "field8": 673, + "field9": 418, + "field10": 905, + "field11": 562, + "field12": 739, + "field13": 864, + "field14": 285, + "field15": 928 + } +} diff --git a/source/benchmarking/odm-data/nested_models.tgz b/source/benchmarking/odm-data/nested_models.tgz new file mode 100644 index 0000000000..390c66ff21 Binary files /dev/null and b/source/benchmarking/odm-data/nested_models.tgz differ diff --git a/source/index.md b/source/index.md index fcf33d1933..0dcbd2815e 100644 --- a/source/index.md +++ b/source/index.md @@ -35,6 +35,7 @@ - [Max Staleness Tests](max-staleness/max-staleness-tests.md) - [MongoDB Handshake](mongodb-handshake/handshake.md) - [OCSP Support](ocsp-support/ocsp-support.md) +- [ODM Performance Benchmarking](benchmarking/odm-benchmarking.md) - [OP_MSG](message/OP_MSG.md) - [Performance Benchmarking](benchmarking/benchmarking.md) - [Polling SRV Records for mongos Discovery](polling-srv-records-for-mongos-discovery/polling-srv-records-for-mongos-discovery.md)