Made possible through funding provided by the Michael & Susan Dell Foundation and the efforts of:
The Ed-Fi API Publisher is a utility that can be used to move all data (or just changes) from one Ed-Fi ODS API instance to another instance of the same version of Ed-Fi. It operates as a standard API client against both API endpoints (source and target) and thus it does not require any special network configuration, direct ODS database access or a particular database engine. From a data security/privacy perspective, it is also subject to all authorization performed by the Ed-Fi ODS API endpoints with which it communicates.
Operationally, it can be used in a "Pull" model where it is deployed alongside a target (central) API and gathers data from multiple source APIs.
Alternatively, it could also be used in a "Push" model where the utility is deployed alongside the source APIs and pushes data to the central target.
However, it can also be used in a "Publishing" model where it is installed alongside a source API and pushes data to multiple targets (e.g. to both a State Education Agency and a collaborative).
If a source API supports the "Change Queries" feature, the Ed-Fi API Publisher will perform a full publishing of the source data on the first run, and then will only publish changed data to the target on subsequent runs. The change versions that have been published to a particular target are maintained in a configuration store automatically for each source/target combination.
To demonstrate how the API Publisher works, this exercise copies all the data from the sample hosted Ed-Fi ODS API to a local sandbox Ed-Fi ODS API using the API client for the "minimal" template. (The database scripts are written for SQL Server.)
Before using the API Publisher on a target ODS, you must create and configure an API client with the appropriate permissions for publishing.
Create and assign a claim set for the API Publisher by running the following database scripts:
- Create-API-Publisher-Writer-Security-Metadata.sql
- Configure-Minimal-Sandbox-Client-as-API-Publisher-Writer.sql
Build the API Publisher solution by running the following command from the repository's root directory:
dotnet build
The API Publisher executable (EdFiApiPublisher.exe) will be located in the .\EdFi.Tools.ApiPublisher.Cli\bin\Debug\netcoreapp3.1 subfolder.
IMPORTANT: After changing security metadata for the API, YOU MUST RESTART the local sandbox Ed-Fi ODS API if it is already running.
Next, locate the key/secret for the API client for the minimal template sandbox. You can use the Sandbox Admin tool, or can just run the following query against the EdFi_Admin database:
SELECT [Key], [Secret]
FROM EdFi_Admin.dbo.ApiClients
WHERE Name = 'Minimal Demonstration Sandbox'The following table shows the command-line arguments that will be used for publishing.
NOTE: Due to the nature of the Quick Start configuration (assuming SQL Server and the Ed-Fi-ODS API are running on a local development machine), we'll limit the parallelism for POST requests to
5. Architectures with dedicated API and database resources should be able to accommodate much higher numbers.
| Parameter | Value | |
|---|---|---|
--sourceUrl |
= |
https://api.ed-fi.org/v5.2/api/ |
--sourceKey |
= |
RvcohKz9zHI4 |
--sourceSecret |
= |
E1iEFusaNf81xzCxwHfbolkC |
--targetUrl |
= |
http://localhost:54746/ |
--targetKey |
= |
(Minimal Sandbox API key) |
--targetSecret |
= |
(Minimal Sandbox API secret) |
--ignoreIsolation |
= |
true |
--maxDegreeOfParallelismForPostResourceItem |
= |
5 |
--maxDegreeOfParallelismForStreamResourcePages |
= |
3 |
--includeDescriptors |
= |
true |
--excludeResources |
= |
surveys |
Run the Ed-Fi API Publisher from the folder containing all the binaries by executing the following command (be sure to substitute your own sandbox API client's key/secret):
.\EdFiApiPublisher.exe --sourceUrl=https://api.ed-fi.org/v5.2/api/ --sourceKey=RvcohKz9zHI4 --sourceSecret=E1iEFusaNf81xzCxwHfbolkC --targetUrl=http://localhost:54746/ --targetKey=minimal_sandbox_API_key --targetSecret=minimal_sandbox_API_secret --ignoreIsolation=true --maxDegreeOfParallelismForPostResourceItem=5 --maxDegreeOfParallelismForStreamResourcePages=3 --includeDescriptors=true --excludeResources=surveys
NOTE: The
--excludeResourcesflag is used to prevent trying to move any survey data due to an issue with the security metadata (described in ODS-4974) in the Ed-Fi ODS API v5.2 release. If you remove this argument, the publishing operation will fail due to unsatisfied dependencies in the data.
Currently the Ed-Fi API Publisher has the following known limitations:
- API resource items deleted in source API cannot be published to the target API due to limitations of the current Change Queries implementation in the Ed-Fi ODS API.
- Even with delete support added by exposing the primary key values, tracking and publishing deletions of Descriptors will still not be possible due to internal implementation details within the API.
- Changes to primary keys in source API resources will currently result in stale copies of the "old" version of the resources (and all impacted dependencies) remaining in the target API.
- Student/Staff/Parent UniqueId changes in the source API could result in the inability of the Ed-Fi API Publisher to continue publishing to the target API.
- Profiles (for defining resource/property level data policies for API clients) are not yet supported by the Ed-Fi API Publisher.
Feedback on the need for resolution to these issues should be provided to the Ed-Fi Alliance through Ed-Fi Tracker.
More technical details on some of these issues can be found here.
When you're ready to look further, review these other topics:
Copyright (c) 2021 Ed-Fi Alliance, LLC and contributors.
Licensed under the Apache License, Version 2.0 (the "License").
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
See NOTICES for additional copyright and license notifications.