Merge pull request #4165 from netbox-community/develop

Release v2.7.5

Author: jeremystretch

.github/stale.yml

@@ -1,5 +1,8 @@
 # Configuration for Stale (https://github.com/apps/stale)
 
+# Pull requests are exempt from being marked as stale
+only: issues
+
 # Number of days of inactivity before an issue becomes stale
 daysUntilStale: 14
 

docs/additional-features/custom-scripts.md

@@ -177,10 +177,11 @@ An IPv4 or IPv6 network with a mask. Returns a `netaddr.IPNetwork` object. Two a
 
 All variables support the following default options:
 
-* `label` - The name of the form field
-* `description` - A brief description of the field
 * `default` - The field's default value
+* `description` - A brief description of the field
+* `label` - The name of the form field
 * `required` - Indicates whether the field is mandatory (default: true)
+* `widget` - The class of form widget to use (see the [Django documentation](https://docs.djangoproject.com/en/stable/ref/forms/widgets/))
 
 ## Example
 

docs/configuration/optional-settings.md

@@ -109,6 +109,20 @@ In order to send email, NetBox needs an email server configured. The following i
 * TIMEOUT - Amount of time to wait for a connection (seconds)
 * FROM_EMAIL - Sender address for emails sent by NetBox
 
+Email is sent from NetBox only for critical events. If you would like to test the email server configuration please use the django function [send_mail()](https://docs.djangoproject.com/en/stable/topics/email/#send-mail):
+
+```
+# python ./manage.py nbshell
+>>> from django.core.mail import send_mail
+>>> send_mail(
+  'Test Email Subject',
+  'Test Email Body',
+  '[email protected]',
+  ['[email protected]'],
+  fail_silently=False
+)
+```
+
 ---
 
 ## EXEMPT_VIEW_PERMISSIONS

docs/configuration/required-settings.md

@@ -21,7 +21,7 @@ NetBox requires access to a PostgreSQL database service to store data. This serv
 * `PASSWORD` - PostgreSQL password
 * `HOST` - Name or IP address of the database server (use `localhost` if running locally)
 * `PORT` - TCP port of the PostgreSQL service; leave blank for default port (5432)
-* `CONN_MAX_AGE` - Number in seconds for Netbox to keep database connections open. 150-300 seconds is typically a good starting point ([more info](https://docs.djangoproject.com/en/stable/ref/databases/#persistent-connections)).
+* `CONN_MAX_AGE` - Lifetime of a [persistent database connection](https://docs.djangoproject.com/en/stable/ref/databases/#persistent-connections), in seconds (150-300 is recommended)
 
 Example:
 
@@ -36,6 +36,9 @@ DATABASE = {
 }
 ```
 
+!!! note
+    NetBox supports all PostgreSQL database options supported by the underlying Django framework. For a complete list of available parameters, please see [the Django documentation](https://docs.djangoproject.com/en/stable/ref/settings/#databases).
+
 ---
 
 ## REDIS
@@ -85,6 +88,48 @@ REDIS = {
     It is highly recommended to keep the webhook and cache databases separate. Using the same database number on the
     same Redis instance for both may result in webhook processing data being lost during cache flushing events.
 
+### Using Redis Sentinel
+
+If you are using [Redis Sentinel](https://redis.io/topics/sentinel) for high-availability purposes, there is minimal 
+configuration necessary to convert NetBox to recognize it. It requires the removal of the `HOST` and `PORT` keys from 
+above and the addition of two new keys.
+
+* `SENTINELS`: List of tuples or tuple of tuples with each inner tuple containing the name or IP address 
+of the Redis server and port for each sentinel instance to connect to
+* `SENTINEL_SERVICE`: Name of the master / service to connect to
+
+Example:
+
+```python
+REDIS = {
+    'webhooks': {
+        'SENTINELS': [('mysentinel.redis.example.com', 6379)],
+        'SENTINEL_SERVICE': 'netbox',
+        'PASSWORD': '',
+        'DATABASE': 0,
+        'DEFAULT_TIMEOUT': 300,
+        'SSL': False,
+    },
+    'caching': {
+        'SENTINELS': [
+            ('mysentinel.redis.example.com', 6379),
+            ('othersentinel.redis.example.com', 6379)
+        ],
+        'SENTINEL_SERVICE': 'netbox',
+        'PASSWORD': '',
+        'DATABASE': 1,
+        'DEFAULT_TIMEOUT': 300,
+        'SSL': False,
+    }
+}
+```
+
+!!! note:
+    It is possible to have only one or the other Redis configurations to use Sentinel functionality. It is possible
+    for example to have the webhook use sentinel via `HOST`/`PORT` and for caching to use Sentinel via 
+    `SENTINELS`/`SENTINEL_SERVICE`.
+
+
 ---
 
 ## SECRET_KEY

docs/development/style-guide.md

@@ -32,7 +32,7 @@ pycodestyle --ignore=W504,E501 netbox/
 
 The introduction of a new dependency is best avoided unless it is absolutely necessary. For small features, it's generally preferable to replicate functionality within the NetBox code base rather than to introduce reliance on an external project. This reduces both the burden of tracking new releases and our exposure to outside bugs and attacks.
 
-If there's a strong case for introducing a new depdency, it must meet the following criteria:
+If there's a strong case for introducing a new dependency, it must meet the following criteria:
 
 * Its complete source code must be published and freely accessible without registration.
 * Its license must be conducive to inclusion in an open source project.
@@ -45,10 +45,18 @@ When adding a new dependency, a short description of the package and the URL of
 
 * When in doubt, remain consistent: It is better to be consistently incorrect than inconsistently correct. If you notice in the course of unrelated work a pattern that should be corrected, continue to follow the pattern for now and open a bug so that the entire code base can be evaluated at a later point.
 
+* Prioritize readability over concision. Python is a very flexible language that typically gives us several options for expressing a given piece of logic, but some may be more friendly to the reader than others. (List comprehensions are particularly vulnerable to over-optimization.) Always remain considerate of the future reader who may need to interpret your code without the benefit of the context within which you are writing it.
+
 * No easter eggs. While they can be fun, NetBox must be considered as a business-critical tool. The potential, however minor, for introducing a bug caused by unnecessary logic is best avoided entirely.
 
 * Constants (variables which generally do not change) should be declared in `constants.py` within each app. Wildcard imports from the file are acceptable.
 
-* Every model should have a docstring. Every custom method should include an expalantion of its function.
+* Every model should have a docstring. Every custom method should include an explanation of its function.
 
 * Nested API serializers generate minimal representations of an object. These are stored separately from the primary serializers to avoid circular dependencies. Always import nested serializers from other apps directly. For example, from within the DCIM app you would write `from ipam.api.nested_serializers import NestedIPAddressSerializer`.
+
+## Branding
+
+* When referring to NetBox in writing, use the proper form "NetBox," with the letters N and B capitalized. The lowercase form "netbox" should be used in code, filenames, etc. But never "Netbox" or any other deviation.
+
+* There is an SVG form of the NetBox logo at [docs/netbox_logo.svg](../netbox_logo.svg). It is preferred to use this logo for all purposes as it scales to arbitrary sizes without loss of resolution. If a raster image is required, the SVG logo should be converted to a PNG image of the prescribed size.

docs/installation/index.md

@@ -4,7 +4,7 @@ The following sections detail how to set up a new instance of NetBox:
 
 1. [PostgreSQL database](1-postgresql.md)
 2. [NetBox components](2-netbox.md)
-3. [HTTP dameon](3-http-daemon.md)
+3. [HTTP daemon](3-http-daemon.md)
 4. [LDAP authentication](4-ldap.md) (optional)
 
 # Upgrading

docs/installation/upgrading.md

@@ -88,7 +88,7 @@ Finally, restart the WSGI services to run the new code. If you followed this gui
 
 ```no-highlight
 # sudo systemctl restart netbox
-# sudo systemctl restart netbox-rqworker
+# sudo systemctl restart netbox-rq
 ```
 
 !!! note

docs/release-notes/version-2.7.md

@@ -1,3 +1,38 @@
+# v2.7.5 (2020-02-13)
+
+**Note:** This release includes several database schema migrations that calculate and store copies of names for certain objects to improve natural ordering performance (see [#3799](https://github.com/netbox-community/netbox/issues/3799)). These migrations may take a few minutes to run if you have a very large number of objects defined in NetBox.
+
+## Enhancements
+
+* [#3766](https://github.com/netbox-community/netbox/issues/3766) - Allow custom script authors to specify the form widget for each variable
+* [#3799](https://github.com/netbox-community/netbox/issues/3799) - Greatly improve performance when ordering device components
+* [#3984](https://github.com/netbox-community/netbox/issues/3984) - Add support for Redis Sentinel
+* [#3986](https://github.com/netbox-community/netbox/issues/3986) - Include position numbers in SVG image when rendering rack elevations
+* [#4093](https://github.com/netbox-community/netbox/issues/4093) - Add more status choices for virtual machines
+* [#4100](https://github.com/netbox-community/netbox/issues/4100) - Add device filter to component list views
+* [#4113](https://github.com/netbox-community/netbox/issues/4113) - Add bulk edit functionality for device type components
+* [#4116](https://github.com/netbox-community/netbox/issues/4116) - Enable bulk edit and delete functions for device component list views
+* [#4129](https://github.com/netbox-community/netbox/issues/4129) - Add buttons to delete individual device type components
+
+## Bug Fixes
+
+* [#3507](https://github.com/netbox-community/netbox/issues/3507) - Fix filtering IP addresses by multiple devices
+* [#3995](https://github.com/netbox-community/netbox/issues/3995) - Make dropdown menus in the navigation bar scrollable on small screens
+* [#4083](https://github.com/netbox-community/netbox/issues/4083) - Permit nullifying applicable choice fields via API requests
+* [#4089](https://github.com/netbox-community/netbox/issues/4089) - Selection of power outlet type during bulk update is optional
+* [#4090](https://github.com/netbox-community/netbox/issues/4090) - Render URL custom fields as links under object view
+* [#4091](https://github.com/netbox-community/netbox/issues/4091) - Fix filtering of objects by custom fields using UI search form
+* [#4099](https://github.com/netbox-community/netbox/issues/4099) - Linkify interfaces on global interfaces list
+* [#4108](https://github.com/netbox-community/netbox/issues/4108) - Avoid extraneous database queries when rendering search forms
+* [#4134](https://github.com/netbox-community/netbox/issues/4134) - Device power ports and outlets should inherit type from the parent device type
+* [#4137](https://github.com/netbox-community/netbox/issues/4137) - Disable occupied terminations when connecting a cable to a circuit
+* [#4138](https://github.com/netbox-community/netbox/issues/4138) - Restore device bay counts in rack elevation diagrams
+* [#4146](https://github.com/netbox-community/netbox/issues/4146) - Fix enforcement of secret role assignment for secret decryption
+* [#4150](https://github.com/netbox-community/netbox/issues/4150) - Correct YAML rendering of config contexts
+* [#4159](https://github.com/netbox-community/netbox/issues/4159) - Fix implementation of Redis caching configuration
+
+---
+
 # v2.7.4 (2020-02-04)
 
 ## Enhancements

mkdocs.yml

@@ -41,7 +41,6 @@ pages:
         - Prometheus Metrics: 'additional-features/prometheus-metrics.md'
         - Reports: 'additional-features/reports.md'
         - Tags: 'additional-features/tags.md'
-        - Topology Maps: 'additional-features/topology-maps.md'
         - Webhooks: 'additional-features/webhooks.md'
     - Administration:
         - Replicating NetBox: 'administration/replicating-netbox.md'

netbox/circuits/api/urls.py

@@ -15,15 +15,15 @@ def get_view_name(self):
 router.APIRootView = CircuitsRootView
 
 # Field choices
-router.register(r'_choices', views.CircuitsFieldChoicesViewSet, basename='field-choice')
+router.register('_choices', views.CircuitsFieldChoicesViewSet, basename='field-choice')
 
 # Providers
-router.register(r'providers', views.ProviderViewSet)
+router.register('providers', views.ProviderViewSet)
 
 # Circuits
-router.register(r'circuit-types', views.CircuitTypeViewSet)
-router.register(r'circuits', views.CircuitViewSet)
-router.register(r'circuit-terminations', views.CircuitTerminationViewSet)
+router.register('circuit-types', views.CircuitTypeViewSet)
+router.register('circuits', views.CircuitViewSet)
+router.register('circuit-terminations', views.CircuitTerminationViewSet)
 
 app_name = 'circuits-api'
 urlpatterns = router.urls

netbox/circuits/forms.py

@@ -9,7 +9,8 @@
 from tenancy.models import Tenant
 from utilities.forms import (
     APISelect, APISelectMultiple, add_blank_choice, BootstrapMixin, CommentField, CSVChoiceField, DatePicker,
-    FilterChoiceField, SmallTextarea, SlugField, StaticSelect2, StaticSelect2Multiple, TagFilterField
+    DynamicModelChoiceField, DynamicModelMultipleChoiceField, SmallTextarea, SlugField, StaticSelect2,
+    StaticSelect2Multiple, TagFilterField,
 )
 from .choices import CircuitStatusChoices
 from .models import Circuit, CircuitTermination, CircuitType, Provider
@@ -107,7 +108,7 @@ class ProviderFilterForm(BootstrapMixin, CustomFieldFilterForm):
         required=False,
         label='Search'
     )
-    region = FilterChoiceField(
+    region = DynamicModelMultipleChoiceField(
         queryset=Region.objects.all(),
         to_field_name='slug',
         required=False,
@@ -119,9 +120,10 @@ class ProviderFilterForm(BootstrapMixin, CustomFieldFilterForm):
             }
         )
     )
-    site = FilterChoiceField(
+    site = DynamicModelMultipleChoiceField(
         queryset=Site.objects.all(),
         to_field_name='slug',
+        required=False,
         widget=APISelectMultiple(
             api_url="/api/dcim/sites/",
             value_field="slug",
@@ -164,6 +166,18 @@ class Meta:
 #
 
 class CircuitForm(BootstrapMixin, TenancyForm, CustomFieldModelForm):
+    provider = DynamicModelChoiceField(
+        queryset=Provider.objects.all(),
+        widget=APISelect(
+            api_url="/api/circuits/providers/"
+        )
+    )
+    type = DynamicModelChoiceField(
+        queryset=CircuitType.objects.all(),
+        widget=APISelect(
+            api_url="/api/circuits/circuit-types/"
+        )
+    )
     comments = CommentField()
     tags = TagField(
         required=False
@@ -180,12 +194,6 @@ class Meta:
             'commit_rate': "Committed rate",
         }
         widgets = {
-            'provider': APISelect(
-                api_url="/api/circuits/providers/"
-            ),
-            'type': APISelect(
-                api_url="/api/circuits/circuit-types/"
-            ),
             'status': StaticSelect2(),
             'install_date': DatePicker(),
         }
@@ -235,14 +243,14 @@ class CircuitBulkEditForm(BootstrapMixin, AddRemoveTagsForm, CustomFieldBulkEdit
         queryset=Circuit.objects.all(),
         widget=forms.MultipleHiddenInput
     )
-    type = forms.ModelChoiceField(
+    type = DynamicModelChoiceField(
         queryset=CircuitType.objects.all(),
         required=False,
         widget=APISelect(
             api_url="/api/circuits/circuit-types/"
         )
     )
-    provider = forms.ModelChoiceField(
+    provider = DynamicModelChoiceField(
         queryset=Provider.objects.all(),
         required=False,
         widget=APISelect(
@@ -255,7 +263,7 @@ class CircuitBulkEditForm(BootstrapMixin, AddRemoveTagsForm, CustomFieldBulkEdit
         initial='',
         widget=StaticSelect2()
     )
-    tenant = forms.ModelChoiceField(
+    tenant = DynamicModelChoiceField(
         queryset=Tenant.objects.all(),
         required=False,
         widget=APISelect(
@@ -290,17 +298,19 @@ class CircuitFilterForm(BootstrapMixin, TenancyFilterForm, CustomFieldFilterForm
         required=False,
         label='Search'
     )
-    type = FilterChoiceField(
+    type = DynamicModelMultipleChoiceField(
         queryset=CircuitType.objects.all(),
         to_field_name='slug',
+        required=False,
         widget=APISelectMultiple(
             api_url="/api/circuits/circuit-types/",
             value_field="slug",
         )
     )
-    provider = FilterChoiceField(
+    provider = DynamicModelMultipleChoiceField(
         queryset=Provider.objects.all(),
         to_field_name='slug',
+        required=False,
         widget=APISelectMultiple(
             api_url="/api/circuits/providers/",
             value_field="slug",
@@ -311,7 +321,7 @@ class CircuitFilterForm(BootstrapMixin, TenancyFilterForm, CustomFieldFilterForm
         required=False,
         widget=StaticSelect2Multiple()
     )
-    region = forms.ModelMultipleChoiceField(
+    region = DynamicModelMultipleChoiceField(
         queryset=Region.objects.all(),
         to_field_name='slug',
         required=False,
@@ -323,9 +333,10 @@ class CircuitFilterForm(BootstrapMixin, TenancyFilterForm, CustomFieldFilterForm
             }
         )
     )
-    site = FilterChoiceField(
+    site = DynamicModelMultipleChoiceField(
         queryset=Site.objects.all(),
         to_field_name='slug',
+        required=False,
         widget=APISelectMultiple(
             api_url="/api/dcim/sites/",
             value_field="slug",

netbox/circuits/tests/test_views.py

@@ -2,10 +2,10 @@
 
 from circuits.choices import *
 from circuits.models import Circuit, CircuitType, Provider
-from utilities.testing import StandardTestCases
+from utilities.testing import ViewTestCases
 
 
-class ProviderTestCase(StandardTestCases.Views):
+class ProviderTestCase(ViewTestCases.PrimaryObjectViewTestCase):
     model = Provider
 
     @classmethod
@@ -46,14 +46,9 @@ def setUpTestData(cls):
         }
 
 
-class CircuitTypeTestCase(StandardTestCases.Views):
+class CircuitTypeTestCase(ViewTestCases.OrganizationalObjectViewTestCase):
     model = CircuitType
 
-    # Disable inapplicable tests
-    test_get_object = None
-    test_delete_object = None
-    test_bulk_edit_objects = None
-
     @classmethod
     def setUpTestData(cls):
 
@@ -77,7 +72,7 @@ def setUpTestData(cls):
         )
 
 
-class CircuitTestCase(StandardTestCases.Views):
+class CircuitTestCase(ViewTestCases.PrimaryObjectViewTestCase):
     model = Circuit
 
     @classmethod