Skip to content

Commit 40ba17a

Browse files
djbwdjbw
committed
cxl/acpi: Introduce cxl_decoder objects
A cxl_decoder is a child of a cxl_port. It represents a hardware decoder configuration of an upstream port to one or more of its downstream ports. The decoder is either represented in CXL standard HDM decoder registers (see CXL 2.0 section 8.2.5.12 CXL HDM Decoder Capability Structure), or it is a static decode configuration communicated by platform firmware (see the CXL Early Discovery Table: Fixed Memory Window Structure). The firmware described and hardware described decoders differ slightly leading to 2 different sub-types of decoders, cxl_decoder_root and cxl_decoder_switch. At the root level the decode capabilities restrict what can be mapped beneath them. Mid-level switch decoders are configured for either acclerator (type-2) or memory-expander (type-3) operation, but they are otherwise agnostic to the type of memory (volatile vs persistent) being mapped. Here is an example topology from a single-ported host-bridge environment without CFMWS decodes enumerated. /sys/bus/cxl/devices/root0 ├── devtype ├── dport0 -> ../../../LNXSYSTM:00/LNXSYBUS:00/ACPI0016:00 ├── port1 │   ├── decoder1.0 │   │   ├── devtype │   │   ├── locked │   │   ├── size │   │   ├── start │   │   ├── subsystem -> ../../../../../../bus/cxl │   │   ├── target_list │   │   ├── target_type │   │   └── uevent │   ├── devtype │   ├── dport0 -> ../../../../pci0000:34/0000:34:00.0 │   ├── subsystem -> ../../../../../bus/cxl │   ├── uevent │   └── uport -> ../../../../LNXSYSTM:00/LNXSYBUS:00/ACPI0016:00 ├── subsystem -> ../../../../bus/cxl ├── uevent └── uport -> ../../ACPI0017:00 Reviewed-by: Jonathan Cameron <[email protected]> Link: https://lore.kernel.org/r/162325695128.2293823.17519927266014762694.stgit@dwillia2-desk3.amr.corp.intel.com Signed-off-by: Dan Williams <[email protected]>
1 parent 3b94ce7 commit 40ba17a

File tree

4 files changed

+417
-1
lines changed

4 files changed

+417
-1
lines changed

‎Documentation/ABI/testing/sysfs-bus-cxl

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,73 @@ Description:
5757
decode of CXL memory resources. The 'Y' integer reflects the
5858
hardware port unique-id used in the hardware decoder target
5959
list.
60+
61+
What: /sys/bus/cxl/devices/decoderX.Y
62+
Date: June, 2021
63+
KernelVersion: v5.14
64+
Contact: [email protected]
65+
Description:
66+
CXL decoder objects are enumerated from either a platform
67+
firmware description, or a CXL HDM decoder register set in a
68+
PCIe device (see CXL 2.0 section 8.2.5.12 CXL HDM Decoder
69+
Capability Structure). The 'X' in decoderX.Y represents the
70+
cxl_port container of this decoder, and 'Y' represents the
71+
instance id of a given decoder resource.
72+
73+
What: /sys/bus/cxl/devices/decoderX.Y/{start,size}
74+
Date: June, 2021
75+
KernelVersion: v5.14
76+
Contact: [email protected]
77+
Description:
78+
The 'start' and 'size' attributes together convey the physical
79+
address base and number of bytes mapped in the decoder's decode
80+
window. For decoders of devtype "cxl_decoder_root" the address
81+
range is fixed. For decoders of devtype "cxl_decoder_switch" the
82+
address is bounded by the decode range of the cxl_port ancestor
83+
of the decoder's cxl_port, and dynamically updates based on the
84+
active memory regions in that address space.
85+
86+
What: /sys/bus/cxl/devices/decoderX.Y/locked
87+
Date: June, 2021
88+
KernelVersion: v5.14
89+
Contact: [email protected]
90+
Description:
91+
CXL HDM decoders have the capability to lock the configuration
92+
until the next device reset. For decoders of devtype
93+
"cxl_decoder_root" there is no standard facility to unlock them.
94+
For decoders of devtype "cxl_decoder_switch" a secondary bus
95+
reset, of the PCIe bridge that provides the bus for this
96+
decoders uport, unlocks / resets the decoder.
97+
98+
What: /sys/bus/cxl/devices/decoderX.Y/target_list
99+
Date: June, 2021
100+
KernelVersion: v5.14
101+
Contact: [email protected]
102+
Description:
103+
Display a comma separated list of the current decoder target
104+
configuration. The list is ordered by the current configured
105+
interleave order of the decoder's dport instances. Each entry in
106+
the list is a dport id.
107+
108+
What: /sys/bus/cxl/devices/decoderX.Y/cap_{pmem,ram,type2,type3}
109+
Date: June, 2021
110+
KernelVersion: v5.14
111+
Contact: [email protected]
112+
Description:
113+
When a CXL decoder is of devtype "cxl_decoder_root", it
114+
represents a fixed memory window identified by platform
115+
firmware. A fixed window may only support a subset of memory
116+
types. The 'cap_*' attributes indicate whether persistent
117+
memory, volatile memory, accelerator memory, and / or expander
118+
memory may be mapped behind this decoder's memory window.
119+
120+
What: /sys/bus/cxl/devices/decoderX.Y/target_type
121+
Date: June, 2021
122+
KernelVersion: v5.14
123+
Contact: [email protected]
124+
Description:
125+
When a CXL decoder is of devtype "cxl_decoder_switch", it can
126+
optionally decode either accelerator memory (type-2) or expander
127+
memory (type-3). The 'target_type' attribute indicates the
128+
current setting which may dynamically change based on what
129+
memory regions are activated in this decode hierarchy.

‎drivers/cxl/acpi.c

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ static int add_host_bridge_uport(struct device *match, void *arg)
7070
struct device *host = root_port->dev.parent;
7171
struct acpi_pci_root *pci_root;
7272
struct cxl_walk_context ctx;
73+
struct cxl_decoder *cxld;
7374
struct cxl_port *port;
7475

7576
if (!bridge)
@@ -94,7 +95,24 @@ static int add_host_bridge_uport(struct device *match, void *arg)
9495

9596
if (ctx.count == 0)
9697
return -ENODEV;
97-
return ctx.error;
98+
if (ctx.error)
99+
return ctx.error;
100+
101+
/* TODO: Scan CHBCR for HDM Decoder resources */
102+
103+
/*
104+
* In the single-port host-bridge case there are no HDM decoders
105+
* in the CHBCR and a 1:1 passthrough decode is implied.
106+
*/
107+
if (ctx.count == 1) {
108+
cxld = devm_cxl_add_passthrough_decoder(host, port);
109+
if (IS_ERR(cxld))
110+
return PTR_ERR(cxld);
111+
112+
dev_dbg(host, "add: %s\n", dev_name(&cxld->dev));
113+
}
114+
115+
return 0;
98116
}
99117

100118
static int add_host_bridge_dport(struct device *match, void *arg)

‎drivers/cxl/core.c

Lines changed: 265 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,168 @@ static struct attribute_group cxl_base_attribute_group = {
3333
.attrs = cxl_base_attributes,
3434
};
3535

36+
static ssize_t start_show(struct device *dev, struct device_attribute *attr,
37+
char *buf)
38+
{
39+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
40+
41+
return sysfs_emit(buf, "%#llx\n", cxld->range.start);
42+
}
43+
static DEVICE_ATTR_RO(start);
44+
45+
static ssize_t size_show(struct device *dev, struct device_attribute *attr,
46+
char *buf)
47+
{
48+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
49+
50+
return sysfs_emit(buf, "%#llx\n", range_len(&cxld->range));
51+
}
52+
static DEVICE_ATTR_RO(size);
53+
54+
#define CXL_DECODER_FLAG_ATTR(name, flag) \
55+
static ssize_t name##_show(struct device *dev, \
56+
struct device_attribute *attr, char *buf) \
57+
{ \
58+
struct cxl_decoder *cxld = to_cxl_decoder(dev); \
59+
\
60+
return sysfs_emit(buf, "%s\n", \
61+
(cxld->flags & (flag)) ? "1" : "0"); \
62+
} \
63+
static DEVICE_ATTR_RO(name)
64+
65+
CXL_DECODER_FLAG_ATTR(cap_pmem, CXL_DECODER_F_PMEM);
66+
CXL_DECODER_FLAG_ATTR(cap_ram, CXL_DECODER_F_RAM);
67+
CXL_DECODER_FLAG_ATTR(cap_type2, CXL_DECODER_F_TYPE2);
68+
CXL_DECODER_FLAG_ATTR(cap_type3, CXL_DECODER_F_TYPE3);
69+
CXL_DECODER_FLAG_ATTR(locked, CXL_DECODER_F_LOCK);
70+
71+
static ssize_t target_type_show(struct device *dev,
72+
struct device_attribute *attr, char *buf)
73+
{
74+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
75+
76+
switch (cxld->target_type) {
77+
case CXL_DECODER_ACCELERATOR:
78+
return sysfs_emit(buf, "accelerator\n");
79+
case CXL_DECODER_EXPANDER:
80+
return sysfs_emit(buf, "expander\n");
81+
}
82+
return -ENXIO;
83+
}
84+
static DEVICE_ATTR_RO(target_type);
85+
86+
static ssize_t target_list_show(struct device *dev,
87+
struct device_attribute *attr, char *buf)
88+
{
89+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
90+
ssize_t offset = 0;
91+
int i, rc = 0;
92+
93+
device_lock(dev);
94+
for (i = 0; i < cxld->interleave_ways; i++) {
95+
struct cxl_dport *dport = cxld->target[i];
96+
struct cxl_dport *next = NULL;
97+
98+
if (!dport)
99+
break;
100+
101+
if (i + 1 < cxld->interleave_ways)
102+
next = cxld->target[i + 1];
103+
rc = sysfs_emit_at(buf, offset, "%d%s", dport->port_id,
104+
next ? "," : "");
105+
if (rc < 0)
106+
break;
107+
offset += rc;
108+
}
109+
device_unlock(dev);
110+
111+
if (rc < 0)
112+
return rc;
113+
114+
rc = sysfs_emit_at(buf, offset, "\n");
115+
if (rc < 0)
116+
return rc;
117+
118+
return offset + rc;
119+
}
120+
static DEVICE_ATTR_RO(target_list);
121+
122+
static struct attribute *cxl_decoder_base_attrs[] = {
123+
&dev_attr_start.attr,
124+
&dev_attr_size.attr,
125+
&dev_attr_locked.attr,
126+
&dev_attr_target_list.attr,
127+
NULL,
128+
};
129+
130+
static struct attribute_group cxl_decoder_base_attribute_group = {
131+
.attrs = cxl_decoder_base_attrs,
132+
};
133+
134+
static struct attribute *cxl_decoder_root_attrs[] = {
135+
&dev_attr_cap_pmem.attr,
136+
&dev_attr_cap_ram.attr,
137+
&dev_attr_cap_type2.attr,
138+
&dev_attr_cap_type3.attr,
139+
NULL,
140+
};
141+
142+
static struct attribute_group cxl_decoder_root_attribute_group = {
143+
.attrs = cxl_decoder_root_attrs,
144+
};
145+
146+
static const struct attribute_group *cxl_decoder_root_attribute_groups[] = {
147+
&cxl_decoder_root_attribute_group,
148+
&cxl_decoder_base_attribute_group,
149+
&cxl_base_attribute_group,
150+
NULL,
151+
};
152+
153+
static struct attribute *cxl_decoder_switch_attrs[] = {
154+
&dev_attr_target_type.attr,
155+
NULL,
156+
};
157+
158+
static struct attribute_group cxl_decoder_switch_attribute_group = {
159+
.attrs = cxl_decoder_switch_attrs,
160+
};
161+
162+
static const struct attribute_group *cxl_decoder_switch_attribute_groups[] = {
163+
&cxl_decoder_switch_attribute_group,
164+
&cxl_decoder_base_attribute_group,
165+
&cxl_base_attribute_group,
166+
NULL,
167+
};
168+
169+
static void cxl_decoder_release(struct device *dev)
170+
{
171+
struct cxl_decoder *cxld = to_cxl_decoder(dev);
172+
struct cxl_port *port = to_cxl_port(dev->parent);
173+
174+
ida_free(&port->decoder_ida, cxld->id);
175+
kfree(cxld);
176+
}
177+
178+
static const struct device_type cxl_decoder_switch_type = {
179+
.name = "cxl_decoder_switch",
180+
.release = cxl_decoder_release,
181+
.groups = cxl_decoder_switch_attribute_groups,
182+
};
183+
184+
static const struct device_type cxl_decoder_root_type = {
185+
.name = "cxl_decoder_root",
186+
.release = cxl_decoder_release,
187+
.groups = cxl_decoder_root_attribute_groups,
188+
};
189+
190+
struct cxl_decoder *to_cxl_decoder(struct device *dev)
191+
{
192+
if (dev_WARN_ONCE(dev, dev->type->release != cxl_decoder_release,
193+
"not a cxl_decoder device\n"))
194+
return NULL;
195+
return container_of(dev, struct cxl_decoder, dev);
196+
}
197+
36198
static void cxl_dport_release(struct cxl_dport *dport)
37199
{
38200
list_del(&dport->list);
@@ -138,6 +300,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,
138300

139301
port->uport = uport;
140302
port->component_reg_phys = component_reg_phys;
303+
ida_init(&port->decoder_ida);
141304
INIT_LIST_HEAD(&port->dports);
142305

143306
device_initialize(dev);
@@ -274,6 +437,108 @@ int cxl_add_dport(struct cxl_port *port, struct device *dport_dev, int port_id,
274437
}
275438
EXPORT_SYMBOL_GPL(cxl_add_dport);
276439

440+
static struct cxl_decoder *
441+
cxl_decoder_alloc(struct cxl_port *port, int nr_targets, resource_size_t base,
442+
resource_size_t len, int interleave_ways,
443+
int interleave_granularity, enum cxl_decoder_type type,
444+
unsigned long flags)
445+
{
446+
struct cxl_decoder *cxld;
447+
struct device *dev;
448+
int rc = 0;
449+
450+
if (interleave_ways < 1)
451+
return ERR_PTR(-EINVAL);
452+
453+
device_lock(&port->dev);
454+
if (list_empty(&port->dports))
455+
rc = -EINVAL;
456+
device_unlock(&port->dev);
457+
if (rc)
458+
return ERR_PTR(rc);
459+
460+
cxld = kzalloc(struct_size(cxld, target, nr_targets), GFP_KERNEL);
461+
if (!cxld)
462+
return ERR_PTR(-ENOMEM);
463+
464+
rc = ida_alloc(&port->decoder_ida, GFP_KERNEL);
465+
if (rc < 0)
466+
goto err;
467+
468+
*cxld = (struct cxl_decoder) {
469+
.id = rc,
470+
.range = {
471+
.start = base,
472+
.end = base + len - 1,
473+
},
474+
.flags = flags,
475+
.interleave_ways = interleave_ways,
476+
.interleave_granularity = interleave_granularity,
477+
.target_type = type,
478+
};
479+
480+
/* handle implied target_list */
481+
if (interleave_ways == 1)
482+
cxld->target[0] =
483+
list_first_entry(&port->dports, struct cxl_dport, list);
484+
dev = &cxld->dev;
485+
device_initialize(dev);
486+
device_set_pm_not_required(dev);
487+
dev->parent = &port->dev;
488+
dev->bus = &cxl_bus_type;
489+
490+
/* root ports do not have a cxl_port_type parent */
491+
if (port->dev.parent->type == &cxl_port_type)
492+
dev->type = &cxl_decoder_switch_type;
493+
else
494+
dev->type = &cxl_decoder_root_type;
495+
496+
return cxld;
497+
err:
498+
kfree(cxld);
499+
return ERR_PTR(rc);
500+
}
501+
502+
static void unregister_dev(void *dev)
503+
{
504+
device_unregister(dev);
505+
}
506+
507+
struct cxl_decoder *
508+
devm_cxl_add_decoder(struct device *host, struct cxl_port *port, int nr_targets,
509+
resource_size_t base, resource_size_t len,
510+
int interleave_ways, int interleave_granularity,
511+
enum cxl_decoder_type type, unsigned long flags)
512+
{
513+
struct cxl_decoder *cxld;
514+
struct device *dev;
515+
int rc;
516+
517+
cxld = cxl_decoder_alloc(port, nr_targets, base, len, interleave_ways,
518+
interleave_granularity, type, flags);
519+
if (IS_ERR(cxld))
520+
return cxld;
521+
522+
dev = &cxld->dev;
523+
rc = dev_set_name(dev, "decoder%d.%d", port->id, cxld->id);
524+
if (rc)
525+
goto err;
526+
527+
rc = device_add(dev);
528+
if (rc)
529+
goto err;
530+
531+
rc = devm_add_action_or_reset(host, unregister_dev, dev);
532+
if (rc)
533+
return ERR_PTR(rc);
534+
return cxld;
535+
536+
err:
537+
put_device(dev);
538+
return ERR_PTR(rc);
539+
}
540+
EXPORT_SYMBOL_GPL(devm_cxl_add_decoder);
541+
277542
/**
278543
* cxl_probe_component_regs() - Detect CXL Component register blocks
279544
* @dev: Host device of the @base mapping

0 commit comments

Comments
 (0)