End-to-end SDK generation and publishing with Jenkins Pipelines
Supported SDK languages:
TypeScript / Javascript | Java / Kotlin | Python | C# | Go | PHP |
---|---|---|---|---|---|
✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
This tutorial shows you how to use Jenkins Pipelines with liblab to automate the SDK generation and publishing process for an existing project. You'll learn how to configure CI/CD workflows that automatically generate and release SDKs to public package managers like npm, PyPI, NuGet, Maven, Go Packages, and Packagist.
Ensure that you:
- Have Docker installed, which will be used to run Jenkins.
- Have an account on the package management platform(s) you want to deploy your SDKs to (ex. NPM, PyPi, Maven). This guide will help you with all of the other deployment details.
This guide doesn't cover setting up a new liblab SDK project.
If you don't have a liblab SDK project yet then you'll first want to follow one of our framework guides or general getting started guide.
When using Jenkins to control your CI/CD, you can choose to use GitHub, GitLab, or BitBucket.
This tutorial uses GitHub to host the necessary repos.
Structural Overview
Your completed project will consist of:
- Your existing liblab project repository which we'll refer to as the "Control Repo"
- One SDK repository per SDK language you want to publish which are your "SDK Repos"
When you complete the tutorial, the project repository structure will look like the example below.
Where each SDK repo will have the following organization:
- TypeScript
- Python
- C#
- Java
- Go
- PHP
Any change to the API spec or liblab configuration in the Control Repo will trigger SDK builds and create pull requests (PRs) in the respective SDK repositories.
Set up the Control Repo
First, you'll need to set up your Jenkins Pipeline. Run the following command from the root of your project:
cd your-liblab-project
curl --remote-name "https://raw.githubusercontent.com/liblaber/control-repo-template/main/Jenkinsfile"
This command downloads the official liblab Jenkins file (Jenkinsfile
) and places it in the root of your Control Repo. The Jenkinsfile
is a pipeline configuration file for automating SDK generation and pull request creation when specific files or directories, such as liblab.config.json
, spec/
, hooks/
, or customPlanModifiers/
, are modified. It defines steps to validate environment variables, detect file changes, build SDKs using liblab
, and create pull requests. You can customize it as needed.
This is what you should have so far:
Create SDK Repos
Your generated SDKs live in separate repos, one per SDK language.
Templates are available for each language. Click the links below and follow the instructions to create a new repository for each SDK language you want to publish.
Each liblab template includes a Jenkinsfile that automates the SDK build and deployment process to the package manager service.
This is what you should have so far:
Create liblab and GitHub tokens
To manage your CI/CD process with Jenkins, you must configure the following secrets to enable interactions with GitHub and access liblab features:
- A liblab token for authenticating the liblab CLI.
- A GitHub token to allow Jenkins and the liblab CLI to push to your SDK repositories.
If you're using Jenkins with GitLab or Bitbucket, use their respective tokens instead of the GitHub token to grant permissions to Jenkins and the liblab CLI.
Create a liblab token
By default, this token will expire after 90 days. You can increase this to up to 364 days by passing the --durationInDays
parameter.
Read more on the liblab token
command in the CLI documentation.
For guidance on setting up secrets in GitHub, refer to the GitHub secrets documentation.
To generate a liblab token, run the following command from your terminal:
liblab token create GITHUB_PR_TOKEN
You can replace GITHUB_PR_TOKEN
with any name you prefer. If you're generating SDKs for multiple APIs, you can either share the token between them or create separate tokens for each API.
Token successfully generated, it will be valid for 90 days
-TOKEN-----------------------------------------
liblab_HYeJzYb5CO3t7JaW4I6ZWKNKbOGp01MqqYSTS6yG
-----------------------------------------------
Copy the token (e.g., liblab_HYeJzYb5CO3t7JaW4I6ZWKNKbOGp01MqqYSTS6yG
) and save it somewhere safe. You'll use it to configure credentials in Jenkins.
Create a GitHub token
It's helpful to create a reminder for yourself such as a calendar item to refresh your liblab and Github tokens just before they expire!
To enable the Jenkins pipeline to identify changes in your Control Repo, run the liblab CLI, and push changes to your SDKs repos, you'll need to generate a GitHub Fine-Grained Access Token with the right permissions. Follow these steps:
- Open GitHub's fine-grained token settings and click Generate new token.
- Name the token GITHUB_TOKEN and set its expiration.
- Under Repository Access, select **Only select repositories **. Choose the repos created in Step 1 and Step 2.
- Set the following repo permissions:
- Commit Statuses: Read/Write.
- Contents: Read/Write.
- Metadata: Read.
- Pull Requests: Read/Write.
- Workflows: Read/Write.
- Click Generate token.
Access the Pull request creation documentation page for further information and detailed guidance on creating the token.
Once created, copy the GITHUB_TOKEN and save it in a secure location for use in the upcoming steps. For more details on how the GITHUB_TOKEN will be used to create PRs, access the liblab documentation.
Create package manager tokens
Depending on which SDK languages you're using, you'll need to create additional secrets.
- TypeScript
- Python
- C#
- Java
- Go
- PHP
To publish to npm, you will need an npm access token. You can generate a new access token from your user settings on npm.
-
Select the Access Tokens tab, drop down the Generate New Token button, then select Granular Access Token.
-
Fill in all the required details for the token such as the name and expiry.
-
Make sure that this token has read and write permission to publish packages. If the package already exists, you can scope this token to just that package, otherwise this token needs read and write for all packages.
-
Once the token has been created, make a copy of it.
-
In your TypeScript SDK repo, add this token as an Actions secret named
NPM_TOKEN
.
You can learn more about creating npm tokens in the npm access tokens documentation.
To publish to PyPI you will need an API token. You can generate a new API token from your account settings page on PyPI.
-
Select Account settings, scroll to the API tokens section, then select the Add API token button.
-
Give the token a name.
-
Select the scope for the token. If the package already exists, select that package for the scope, otherwise select Entire account (all packages).
-
Select the Create token button.
-
Once the token has been created, make a copy of it.
-
In your Python SDK repo, add this token as an Actions secret named
PYPI_TOKEN
.
You can learn more about creating PyPI tokens in the PyPI API tokens documentation.
To publish to NuGet, you will need a NuGet API key. You can generate an API key from your account on NuGet.
-
Select your name on the top right, then select the API Keys.
-
From the API Keys page, expand the Create section.
-
Give your API Key a name, and set the required expiry and package owner.
-
Set the scope for this API Key.
- If the package already exists, select Push and Push only new package versions. Then in the Select Packages section, select the package you want to publish updates for.
- If the package does not exist, select Push and Push new packages and package versions, then in the Glob pattern, put
*
. This will allow you to push a new package.
-
Select the Create button.
-
Once the API Key has been created, expand the Manage section, and make a copy of it using the Copy button.
-
In your C# SDK repo, add this token as an Actions secret named
NUGET_TOKEN
.
You can learn more about creating NuGet API Keys in the NuGet API Keys documentation.
To publish to the Maven Central Repository, you will need Maven credentials and a GPG key with a passphrase to sign your artifacts.
As of March 12th, 2024, the Central Portal became the default publishing server for Maven packages. At the moment, we do not support automatic publishing through the Legacy OSSRH.
Central Portal Setup
The Central Portal Account Documentation explains how to create an account. An account is required for claiming namespaces, generating credentials and managing package deployments.
Namespace Registration
Your namespace is the crucial prerequisite for publishing a package to the Central Repository as it is the groupId
of the package. The Central Repository Namespace Documentation provides detailed instructions for claiming a namespace.
The namespace verification process is not instant.
- For own domain namespaces (e.g.
com.liblab
), an additional step is required, which involves setting up the verification key as a DNS record in order for the domain name to be verified by the Central Repository. - For GitHub namespaces (namespace in the form of
io.github.githuborgname
), though, verification process is automatic if the user is registered using the GitHub SSO.
Once your namespace is verified, the next step is to generate your Maven credentials.
Generate Maven Credentials
-
Head to the Central Portal at central.sonatype.com, and either create an account, or sign in.
-
Select your email on the top right, then select View Account.
-
From the Account page, select the Generate User Token button.
-
When the dialog appears, confirm that you want to generate a new token, and credentials will be generated.
-
In your Java SDK repo, add value of the
Username
field as a new GitHub Actions secret calledMAVEN_USERNAME
and the value of thePassword
field as a secret calledMAVEN_PASSWORD
.
You can learn more about creating Central Portal credentials in the Central Portal Documentation.
Generate GPG Key
The Central Repository requires all artifacts to be signed with PGP, such as using a GPG key. You will need to generate your own key pair, distribute it to the key server and obtain the private key, then set this as an other GitHub Actions secret.
The following steps will guide you through the process of creating a GPG key using GnuPG. If you already have a GPG key, you can skip this step. If you want to use a different tool to generate your GPG key, you will need to follow the relevant documentation for that tool.
-
Install GnuPG if you don't have it installed.
-
Initialize the key generation prompt by running:
Terminalgpg --gen-key
-
Follow the instructions from
gpg
, and choose a passphrase -
Once the key is generated, you will need to get the key id. This is the 40-character hex string on the second line of the
pub
section in the output from generating the key:Terminalpublic and secret key created and signed.
pub ed25519 2024-05-07 [SC] [expires: 2027-05-07]
AB1C2D3456EF78A90BC12D34567890123456789E
uid Exciting Soda <[email protected]>
sub cv25519 2024-05-07 [E] [expires: 2027-05-07] -
Send the key to your key server, such as
openpgp
by running the following command:Terminalgpg --keyserver keys.openpgp.org --send-keys <key-id>
Replace
<key-id>
with the key id you obtained in the previous step. -
Get the private key using the following command:
Terminalgpg --export-secret-keys --armour <key-id>
Replace
<key-id>
with your key id.Terminal➜ ~ gpg --export-secret-keys --armour AB1C2D3456EF78A90BC12D34567890123456789E
-----BEGIN PGP PRIVATE KEY BLOCK-----
mDMEXEcE6RYJKwYBBAHaRw8BAQdArjWwk3FAqyiFbFBKT4TzXcVBqPTB3gmzlC/U
b7O1u120JkFsaWNlIExvdmVsYWNlIDxhbGljZUBvcGVucGdwLmV4YW1wbGU+iJAE
ExYIADgCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQTrhbtfozp14V6UTmPy
MVUMT0fjjgUCXaWfOgAKCRDyMVUMT0fjjukrAPoDnHBSogOmsHOsd9qGsiZpgRnO
dypvbm+QtXZqth9rvwD9HcDC0tC+PHAsO7OTh1S1TC9RiJsvawAfCPaQZoed8gK4
OARcRwTpEgorBgEEAZdVAQUBAQdAQv8GIa2rSTzgqbXCpDDYMiKRVitCsy203x3s
E9+eviIDAQgHiHgEGBYIACAWIQTrhbtfozp14V6UTmPyMVUMT0fjjgUCXEcE6QIb
DAAKCRDyMVUMT0fjjlnQAQDFHUs6TIcxrNTtEZFjUFm1M0PJ1Dng/cDW4xN80fsn
0QEA22Kr7VkCjeAEC08VSTeV+QFsmz55/lntWkwYWhmvOgE=
=iIGO
-----END PGP PRIVATE KEY BLOCK----- -
Copy the output from this command, and add it to a new GitHub Actions secret called
GPG_PRIVATE_KEY
. -
Create another GitHub Actions secret called
GPG_PASSPHRASE
, and set this to the passphrase you chose when generating the key.
Go package management differs from many other languages in that it does not require API tokens for publishing to the Go Packages repository. Unlike some ecosystems where API tokens are necessary to interact with package registries, Go relies on its module proxy system, which integrates seamlessly with GitHub.
As explained in the official publishing tutorial on Go's website, packages are published to the Go Packages repository by tagging a release on GitHub. The Create a release in GitHub section later in this tutorial provides a step-by-step guide on how to create a release in GitHub which will automatically trigger the publishing of the package to the Go Packages repository. Upon creating a release, it should not take longer than one hour for the package to become available on the Go Packages repository.
To publish your PHP SDK to Packagist, you will first need an account on Packagist and your Packagist API Token.
-
Create an Account on Packagist or login to Packagist using your GitHub account.
-
Obtain your Packagist API Token from your profile settings
-
In your PHP SDK repo, add this token as an Actions secret named
PACKAGIST_TOKEN
and set the secretPACKAGIST_USERNAME
to your packagist username as it appears on your profile. -
Submit your package
- Log in to your Packagist account.
- Go to the Submit tab.
- Enter the URL of your SDK's GitHub repository.
- Select Check to validate the repository and Submit to add your package to Packagist.
At this point, you should have a complete setup.
Configure Jenkins
You can install Jenkins using different approaches and host it locally or in the cloud.
Run Jenkins using Docker composer
For this tutorial, Jenkins will run in a Docker and a Docker composer is used to define the configurations. The following code snippet shows the necessary code to use in the docker-compose.yml
file.
version: '3.8'
services:
jenkins:
image: jenkins/jenkins:lts
container_name: jenkins
ports:
- '8080:8080'
- '50000:50000'
volumes:
- ${HOME}/jenkins:/var/jenkins_home
privileged: true
user: root
Replace ${HOME}
with an existing directory path on your machine to ensure Jenkins data is persisted locally.
In the code above, volume
links the host machine's ${HOME}/jenkins
directory to the container's /var/jenkins_home
. This ensures Jenkins data, such as jobs and configurations, remains intact even if the container is stopped or removed. The privileged: true
setting grants the container additional permissions. The user: root
setting allows the container to run as the root user, enabling more control for system-level operations.
To start the Jenkins service, follow these steps:
-
In a new directory, separate from the Control Repo, create a file named
docker-compose.yml
. -
Copy and paste the provided code into the file and save it.
-
Open a terminal in the directory containing the
docker-compose.yml
file and run the following command:docker-compose up -d
Your terminal should display output similar to this:
$ docker-compose up -d
[+] Running 1/1
✔ Container jenkins Started -
Once the Jenkins service is running, it will be accessible in your browser at http://localhost:8080.
-
To log in, you need the Jenkins Admin password. Follow these steps to retrieve it:
-
Access the Docker container terminal running Jenkins:
docker exec -it jenkins bash
-
Retrieve the Admin password:
cat /var/jenkins_home/secrets/initialAdminPassword
The password will be displayed in the terminal. Save it securely, as you'll need it to log in to Jenkins.
-
Run Jenkins and install the plugins
To get started with Jenkins, follow the steps:
-
Access Jenkins through your browser using the address http://localhost:8080.
-
Use the admin password from the previous step to unlock Jenkins. For future logins, you'll also need to use the admin user with the admin password.
Additional UsersYou don't need to create additional users for this tutorial. You can perform all actions using the Admin user.
-
Select the Install suggested plugins option. These are the essential plugins you must select:
-
Jenkins will download and install the selected plugins. Once the installation is complete, you'll be redirected to the Jenkins dashboard.
To set up your Jenkins environment, install the three additional plugins:
- From the Jenkins dashboard, navigate to Manage Jenkins -> Plugins -> Available plugins.
- Search for and select the following plugins:
- Docker plugin
- Docker Pipeline
- Basic Branch Build Strategies Plugin
- Once the plugins are selected, click Install.
- After the installation is complete, Jenkins will restart automatically.
During the restart process, the Jenkins container may stop temporarily. If you cannot access the dashboard at http://localhost:8080 and see a Page Not Found error, restart the Jenkins container by running:
docker-compose up -d
Add your credentials
Once you install the necessary plugins, add the LIBLAB_TOKEN
and GITHUB_TOKEN
as credentials in Jenkins. These credentials are used by Jenkins to interact with your GitHub repos (both the Control Repo and SDK repos) and to use the liblab CLI:
- From the Jenkins dashboard, go to Manage Jenkins -> Credentials.
- Under Stores scoped to Jenkins, select System.
- Under the Domain, choose Global credentials.
- Click + Add Credentials.
- Fill in the form for each of the credentials as described below:
- LIBLAB_TOKEN
- GITHUB_TOKEN
- github-credentials
- Kind: Select the Secret text option.
- Secret: Paste the LIBLAB_TOKEN from Step 3.
- ID: Use ID equal LIBLAB_TOKEN.
- Description: Add a meaningful description to help identify this credential later.
- Kind: Select the Secret text option.
- Secret: Paste the GITHUB_TOKEN from Step 3.
- ID: Use ID equal GITHUB_TOKEN.
- Description: Add a meaningful description to help identify this credential later.
- Kind: Select the Username with password option.
- Username: Use the GitHub organization or the GitHub user used to create the GITHUB_TOKEN in Step 3.
- Password: Paste the GITHUB_TOKEN from Step 3.
- ID: Use ID equal github-credentials.
- Description: Add a meaningful description to help identify this credential later.
This is what you should have:
Create a Jenkins pipeline
Now that your environment is configured, you can create a pipeline to manage your SDK projects' CI/CD process:
-
From the Jenkins dashboard, click New Item.
-
Enter a name for the pipeline, select Multibranch Pipeline, and click OK.
-
On the pipeline configuration page, do the following:
- In the Display Name field, add a name for the pipeline.
- In the Description field, provide a meaningful description of the pipeline’s purpose.
- For Branch Sources, select GitHub as the source. A new configuration form will appear.
- For Credentials, choose the username and password (github-credentials) you added in Add your credentials.
- Paste the Control Repo URL into the Repository HTTPS URL field.
- (Optional) Configure branch strategies if needed. For this tutorial, you can leave the default settings.
- For Build Configuration, select by Jenkins for the Mode and add Jenkins for the Script Path field.
- For the Scan Repository Triggers, set Interval to 10 minutes to define how often Jenkins should scan the repo for changes.
Webhooks and GitHub pluginsInstead of setting a scan interval, you can configure GitHub Webhooks for automatic triggering or use GitHub plugins to enhance Jenkins-GitHub integration.
-
Click Save.
Once saved, Jenkins will scan the repo for existing branches and pull requests. It will automatically trigger a build if it finds a Jenkinsfile.
Since your Control Repo has a Jenkinsfile, Jenkins will start the pipeline job. However, the pipeline will fail initially because the liblab.config.json
file has not yet been updated with the SDK repos and organizations.
Update the config file
Now that finished configuring Jenkins and all your repositories are set up, you'll need to update the liblab.config.json
file. Make sure to configure the publishing information and update the settings for each SDK language you plan to publish.
- In the
publishing
section, set thegithubOrg
option to the name of your GitHub organization. - In the
languageOptions
section:-
Update the
githubRepoName
for each SDK. This should be the repo name only (e.g.,typescript-sdk
), not the full GitHub URL (e.g.,https://github.com/myorg/typescript-sdk
). -
Set the
sdkVersion
to match your desired version number. This should be updated for each new release. -
Configure details for each package manager:
- TypeScript
- Python
- C#
- Java
- Go
- PHP
- Set the
pypiPackageName
option to the name of your PyPI package. This needs to be unique.
- Set the
packageId
option to the name of your NuGet package.
- Set the
groupId
option to your namespace verified by the Maven Central Portal. - Set the
homepage
option to the valid public URL of your SDK homepage. - Add at least one developer to the
developers
option. - (Recommended) Set the
artifactId
option. If not set, it defaults to the kebab-cased version ofsdkName
.
-
Set the
goModuleName
option to the name of your Go module. This should match the SDK repo name, for examplegithub.com/myorg/go-sdk
. -
Ensure your license is set.
noteThe Go Packages License Policy requires that all packages published to the Go Packages repository must include a recognized license file. It is important to note that if the package does not include a recognized license file, only limited package information will be displayed on the Go Packages repository.
For more information on how to specify a license for your SDK, see the license config file option.
- Set the
packageName
option to the name of your PHP package. It must be in the formatvendor/packageName
.
-
There are many other options you can configure for SDK deployment but these are optional. Refer to the config file documentation to learn more.
The following code snippet shows an example of a liblab.config.json
file, where SDKs for C#, Python, TypeScript, Go, PHP, and Java are configured for generation and publishing:
Example liblab.config.json configured for SDK deployment
{
"sdkName": "test-sdk",
"apiVersion": "1.0.0",
"apiName": "test-api",
"specFilePath": "spec.json",
"languages": [
"csharp",
"python",
"typescript",
"go",
"php",
"java"
],
"languageOptions": {
"csharp": {
"liblabVersion": "2",
"packageId": "Test.SDK",
"githubRepoName": "csharp-sdk",
"sdkVersion": "1.0.0"
},
"python": {
"liblabVersion": "2",
"pypiPackageName": "test-sdk",
"githubRepoName": "python-sdk",
"sdkVersion": "1.0.0"
},
"typescript": {
"npmName": "test-sdk",
"npmOrg": "myorg",
"githubRepoName": "typescript-sdk",
"sdkVersion": "1.0.0"
},
"java": {
"groupId": "com.myorg",
"artifactId": "test-sdk",
"homepage": "https://myorg.com",
"githubRepoName": "java-sdk",
"sdkVersion": "1.0.0",
"developers": [
{
"name": "John Doe",
"email": "[email protected]",
"organization": "My Organization",
"organizationUrl": "https://myorg.com"
}
]
},
"go": {
"goModuleName": "github.com/myorg/go-sdk",
"githubRepoName": "go-sdk",
"sdkVersion": "1.0.0",
},
"php": {
"packageName": "myorg/test-sdk",
"githubRepoName": "php-sdk",
"sdkVersion": "1.0.0"
},
},
"publishing": {
"githubOrg": "myorg"
}
}
Finally, commit and push your changes to your Control Repo. This will trigger the Jenkins pipeline to generate your SDKs and create PRs in your SDK repos.
Check the Jenkins pipeline
Once you pushes changes to the Control Repo, Jenkins will automatically scan the repo. If it detects changes, it will trigger a new build for the pipeline.
If you want to manually force Jenkins to scan your repo for changes, go to the pipeline you created previously and click Scan Repository Now.
To review the results of the latest Jenkins pipeline job:
- Navigate to the pipeline you created previously.
- Click Build History to view the results of previous builds.
- Select the build number of the most recent job to open its summary.
- Review the Pipeline Overview to inspect each pipeline stage. If errors arise, you can check the output of each stage to identify the problem.
The following GIF demonstrates how to inspect the results of your builds.
Merge SDK PRs
After Jenkins completes the pipeline, a pull request (PR) will be created automatically in each SDK repo by liblab.
To view the PRs, access each SDK repo on GitHub and navigate to the Pull Requests section to view the new PRs created by the liblab CLI.
If you don't see the PRs, review Jenkins' build history to check for any errors during the SDK generation process.
To finish, follow the steps:
- Open each PR to review the changes and ensure the SDK has been generated correctly.
- Verify that the modifications align with your expectations and meet project requirements.
- If everything is correct, approve the PR and merge it into the main branch.
Create a release in GitHub
To trigger the Action to publish your SDK to the package manager, you need to create a release in the SDK repo. Releases need a tag to tag the commit that you want to release.
-
From each SDK repo, go to Releases and click Create a new release.
-
Click Choose a tag, enter a tag (e.g.,
v1.0.0
), and select Create new tag.noteThe tag should follow semantic versioning and typically includes a
v
prefix, likev1.0.0
.Note on Versioning Go PackagesIt is important to note that the release tag version will be used as the version of the package in the Go Packages repository. Therefore, it is important to follow semantic versioning and ensure that the version in the
liblab.config.json
file corresponds to the version of the release tag. -
Give the release a title and description. The title can be the same as the tag.
-
Click Publish release. This will trigger the GitHub Action to publish your SDK.
Confirm SDK publication
Once the publishing Action is done, check the package manager to confirm your SDK was published.
Your setup is now fully automated to generate and publish SDKs with liblab and Jenkins pipelines, ensuring your latest API updates are quickly available to your users.
If you don’t see the SDK in the package manager, check the CI/CD > Pipelines or Jobs section in each SDK repo to troubleshoot any errors during the publishing process.