End-to-end SDK generation and publishing with Jenkins Pipelines

Supported SDK languages:

TypeScript / Javascript	Java / Kotlin	Python	C#	Go	PHP
✅	✅	✅	✅	✅	✅

This tutorial shows you how to use Jenkins Pipelines with liblab to automate the SDK generation and publishing process for an existing project. You'll learn how to configure CI/CD workflows that automatically generate and release SDKs to public package managers like npm, PyPI, NuGet, Maven, Go Packages, and Packagist.

Requirements

Ensure that you:

• Have Docker installed, which will be used to run Jenkins.
• Have an account on the package management platform(s) you want to deploy your SDKs to (ex. NPM, PyPi, Maven). This guide will help you with all of the other deployment details.

Starting from scratch

This guide doesn't cover setting up a new liblab SDK project.

If you don't have a liblab SDK project yet then you'll first want to follow one of our framework guides or general getting started guide.

Version Control Systems

When using Jenkins to control your CI/CD, you can choose to use GitHub, GitLab, or BitBucket.

This tutorial uses GitHub to host the necessary repos.

Structural Overview

Your completed project will consist of:

• Your existing liblab project repository which we'll refer to as the "Control Repo"
• One SDK repository per SDK language you want to publish which are your "SDK Repos"

When you complete the tutorial, the project repository structure will look like the example below.

Where each SDK repo will have the following organization:

TypeScript

Python

C#

Java

Go

PHP

Any change to the API spec or liblab configuration in the Control Repo will trigger SDK builds and create pull requests (PRs) in the respective SDK repositories.

Set up the Control Repo

First, you'll need to set up your Jenkins Pipeline. Run the following command from the root of your project:

Terminal

cd your-liblab-project
curl --remote-name "https://raw.githubusercontent.com/liblaber/control-repo-template/main/Jenkinsfile"

This command downloads the official liblab Jenkins file (Jenkinsfile) and places it in the root of your Control Repo. The Jenkinsfile is a pipeline configuration file for automating SDK generation and pull request creation when specific files or directories, such as liblab.config.json, spec/, hooks/, or customPlanModifiers/, are modified. It defines steps to validate environment variables, detect file changes, build SDKs using liblab, and create pull requests. You can customize it as needed.

This is what you should have so far:

Create SDK Repos

Your generated SDKs live in separate repos, one per SDK language.

Templates are available for each language. Click the links below and follow the instructions to create a new repository for each SDK language you want to publish.

• C#
• Go
• Java
• PHP
• Python
• TypeScript

Each liblab template includes a Jenkinsfile that automates the SDK build and deployment process to the package manager service.

This is what you should have so far:

Create liblab and GitHub tokens

To manage your CI/CD process with Jenkins, you must configure the following secrets to enable interactions with GitHub and access liblab features:

1. A liblab token for authenticating the liblab CLI.
2. A GitHub token to allow Jenkins and the liblab CLI to push to your SDK repositories.

Version Control Systems

If you're using Jenkins with GitLab or Bitbucket, use their respective tokens instead of the GitHub token to grant permissions to Jenkins and the liblab CLI.

Create a liblab token

Token expiration and secret management

By default, this token will expire after 90 days. You can increase this to up to 364 days by passing the --durationInDays parameter.

Read more on the liblab token command in the CLI documentation.

For guidance on setting up secrets in GitHub, refer to the GitHub secrets documentation.

To generate a liblab token, run the following command from your terminal:

liblab token create GITHUB_PR_TOKEN

You can replace GITHUB_PR_TOKEN with any name you prefer. If you're generating SDKs for multiple APIs, you can either share the token between them or create separate tokens for each API.

Token successfully generated, it will be valid for 90 days

-TOKEN-----------------------------------------
liblab_HYeJzYb5CO3t7JaW4I6ZWKNKbOGp01MqqYSTS6yG
-----------------------------------------------

Copy the token (e.g., liblab_HYeJzYb5CO3t7JaW4I6ZWKNKbOGp01MqqYSTS6yG) and save it somewhere safe. You'll use it to configure credentials in Jenkins.

Create a GitHub token

tip

It's helpful to create a reminder for yourself such as a calendar item to refresh your liblab and Github tokens just before they expire!

To enable the Jenkins pipeline to identify changes in your Control Repo, run the liblab CLI, and push changes to your SDKs repos, you'll need to generate a GitHub Fine-Grained Access Token with the right permissions. Follow these steps:

1. Open GitHub's fine-grained token settings and click Generate new token.
2. Name the token GITHUB_TOKEN and set its expiration.
3. Under Repository Access, select **Only select repositories **. Choose the repos created in Step 1 and Step 2.
4. Set the following repo permissions:
   • Commit Statuses: Read/Write.
   • Contents: Read/Write.
   • Metadata: Read.
   • Pull Requests: Read/Write.
   • Workflows: Read/Write.

8. Click Generate token.

CI/CD documentation

Access the Pull request creation documentation page for further information and detailed guidance on creating the token.

Once created, copy the GITHUB_TOKEN and save it in a secure location for use in the upcoming steps. For more details on how the GITHUB_TOKEN will be used to create PRs, access the liblab documentation.

Create package manager tokens

Depending on which SDK languages you're using, you'll need to create additional secrets.

TypeScript

To publish to npm, you will need an npm access token. You can generate a new access token from your user settings on npm.

1. Select the Access Tokens tab, drop down the Generate New Token button, then select Granular Access Token.

   

2. Fill in all the required details for the token such as the name and expiry.

3. Make sure that this token has read and write permission to publish packages. If the package already exists, you can scope this token to just that package, otherwise this token needs read and write for all packages.

4. Once the token has been created, make a copy of it.

5. In your TypeScript SDK repo, add this token as an Actions secret named NPM_TOKEN.

   

note

You can learn more about creating npm tokens in the npm access tokens documentation.

Python

To publish to PyPI you will need an API token. You can generate a new API token from your account settings page on PyPI.

1. Select Account settings, scroll to the API tokens section, then select the Add API token button.

   

2. Give the token a name.

3. Select the scope for the token. If the package already exists, select that package for the scope, otherwise select Entire account (all packages).

4. Select the Create token button.

5. Once the token has been created, make a copy of it.

6. In your Python SDK repo, add this token as an Actions secret named PYPI_TOKEN.

   

note

You can learn more about creating PyPI tokens in the PyPI API tokens documentation.

C#

To publish to NuGet, you will need a NuGet API key. You can generate an API key from your account on NuGet.

1. Select your name on the top right, then select the API Keys.

   

2. From the API Keys page, expand the Create section.

   

3. Give your API Key a name, and set the required expiry and package owner.

4. Set the scope for this API Key.

   • If the package already exists, select Push and Push only new package versions. Then in the Select Packages section, select the package you want to publish updates for.
   • If the package does not exist, select Push and Push new packages and package versions, then in the Glob pattern, put *. This will allow you to push a new package.

5. Select the Create button.

6. Once the API Key has been created, expand the Manage section, and make a copy of it using the Copy button.

   

7. In your C# SDK repo, add this token as an Actions secret named NUGET_TOKEN.

   

note

You can learn more about creating NuGet API Keys in the NuGet API Keys documentation.

Java

To publish to the Maven Central Repository, you will need Maven credentials and a GPG key with a passphrase to sign your artifacts.

note

As of March 12th, 2024, the Central Portal became the default publishing server for Maven packages. At the moment, we do not support automatic publishing through the Legacy OSSRH.

Central Portal Setup

The Central Portal Account Documentation explains how to create an account. An account is required for claiming namespaces, generating credentials and managing package deployments.

Namespace Registration

Your namespace is the crucial prerequisite for publishing a package to the Central Repository as it is the groupId of the package. The Central Repository Namespace Documentation provides detailed instructions for claiming a namespace.

The namespace verification process is not instant.

• For own domain namespaces (e.g. com.liblab), an additional step is required, which involves setting up the verification key as a DNS record in order for the domain name to be verified by the Central Repository.
• For GitHub namespaces (namespace in the form of io.github.githuborgname), though, verification process is automatic if the user is registered using the GitHub SSO.

Once your namespace is verified, the next step is to generate your Maven credentials.

Generate Maven Credentials

1. Head to the Central Portal at central.sonatype.com, and either create an account, or sign in.

2. Select your email on the top right, then select View Account.

   

3. From the Account page, select the Generate User Token button.

   

4. When the dialog appears, confirm that you want to generate a new token, and credentials will be generated.

   

5. In your Java SDK repo, add value of the Username field as a new GitHub Actions secret called MAVEN_USERNAME and the value of the Password field as a secret called MAVEN_PASSWORD.

   

note

You can learn more about creating Central Portal credentials in the Central Portal Documentation.

Generate GPG Key

The Central Repository requires all artifacts to be signed with PGP, such as using a GPG key. You will need to generate your own key pair, distribute it to the key server and obtain the private key, then set this as an other GitHub Actions secret.

The following steps will guide you through the process of creating a GPG key using GnuPG. If you already have a GPG key, you can skip this step. If you want to use a different tool to generate your GPG key, you will need to follow the relevant documentation for that tool.

1. Install GnuPG if you don't have it installed.

2. Initialize the key generation prompt by running:

   Terminal

   gpg --gen-key

3. Follow the instructions from gpg, and choose a passphrase

4. Once the key is generated, you will need to get the key id. This is the 40-character hex string on the second line of the pub section in the output from generating the key:

   Terminal

   public and secret key created and signed.
   
   pub   ed25519 2024-05-07 [SC] [expires: 2027-05-07]
         AB1C2D3456EF78A90BC12D34567890123456789E
   uid                      Exciting Soda <[email protected]>
   sub   cv25519 2024-05-07 [E] [expires: 2027-05-07]

5. Send the key to your key server, such as openpgp by running the following command:

   Terminal

   gpg --keyserver keys.openpgp.org --send-keys <key-id>

   Replace <key-id> with the key id you obtained in the previous step.

6. Get the private key using the following command:

   Terminal

   gpg --export-secret-keys --armour <key-id>

   Replace <key-id> with your key id.

   Terminal

   ➜  ~ gpg --export-secret-keys --armour AB1C2D3456EF78A90BC12D34567890123456789E
   -----BEGIN PGP PRIVATE KEY BLOCK-----
   
   mDMEXEcE6RYJKwYBBAHaRw8BAQdArjWwk3FAqyiFbFBKT4TzXcVBqPTB3gmzlC/U
   b7O1u120JkFsaWNlIExvdmVsYWNlIDxhbGljZUBvcGVucGdwLmV4YW1wbGU+iJAE
   ExYIADgCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQTrhbtfozp14V6UTmPy
   MVUMT0fjjgUCXaWfOgAKCRDyMVUMT0fjjukrAPoDnHBSogOmsHOsd9qGsiZpgRnO
   dypvbm+QtXZqth9rvwD9HcDC0tC+PHAsO7OTh1S1TC9RiJsvawAfCPaQZoed8gK4
   OARcRwTpEgorBgEEAZdVAQUBAQdAQv8GIa2rSTzgqbXCpDDYMiKRVitCsy203x3s
   E9+eviIDAQgHiHgEGBYIACAWIQTrhbtfozp14V6UTmPyMVUMT0fjjgUCXEcE6QIb
   DAAKCRDyMVUMT0fjjlnQAQDFHUs6TIcxrNTtEZFjUFm1M0PJ1Dng/cDW4xN80fsn
   0QEA22Kr7VkCjeAEC08VSTeV+QFsmz55/lntWkwYWhmvOgE=
   =iIGO
   -----END PGP PRIVATE KEY BLOCK-----

7. Copy the output from this command, and add it to a new GitHub Actions secret called GPG_PRIVATE_KEY.

8. Create another GitHub Actions secret called GPG_PASSPHRASE, and set this to the passphrase you chose when generating the key.

Go

Go package management differs from many other languages in that it does not require API tokens for publishing to the Go Packages repository. Unlike some ecosystems where API tokens are necessary to interact with package registries, Go relies on its module proxy system, which integrates seamlessly with GitHub.

As explained in the official publishing tutorial on Go's website, packages are published to the Go Packages repository by tagging a release on GitHub. The Create a release in GitHub section later in this tutorial provides a step-by-step guide on how to create a release in GitHub which will automatically trigger the publishing of the package to the Go Packages repository. Upon creating a release, it should not take longer than one hour for the package to become available on the Go Packages repository.

PHP

To publish your PHP SDK to Packagist, you will first need an account on Packagist and your Packagist API Token.

1. Create an Account on Packagist or login to Packagist using your GitHub account.

2. Obtain your Packagist API Token from your profile settings

   

3. In your PHP SDK repo, add this token as an Actions secret named PACKAGIST_TOKEN and set the secret PACKAGIST_USERNAME to your packagist username as it appears on your profile.

   

4. Submit your package

   1. Log in to your Packagist account.
   2. Go to the Submit tab.
   3. Enter the URL of your SDK's GitHub repository.
   4. Select Check to validate the repository and Submit to add your package to Packagist.

At this point, you should have a complete setup.

Configure Jenkins

You can install Jenkins using different approaches and host it locally or in the cloud.

Run Jenkins using Docker composer

For this tutorial, Jenkins will run in a Docker and a Docker composer is used to define the configurations. The following code snippet shows the necessary code to use in the docker-compose.yml file.

docker-compose.yml

version: '3.8'

services:
  jenkins:
    image: jenkins/jenkins:lts
    container_name: jenkins
    ports:
      - '8080:8080'
      - '50000:50000'
    volumes:
      - ${HOME}/jenkins:/var/jenkins_home
    privileged: true
    user: root

Volume Configuration

Replace ${HOME} with an existing directory path on your machine to ensure Jenkins data is persisted locally.

In the code above, volume links the host machine's ${HOME}/jenkins directory to the container's /var/jenkins_home. This ensures Jenkins data, such as jobs and configurations, remains intact even if the container is stopped or removed. The privileged: true setting grants the container additional permissions. The user: root setting allows the container to run as the root user, enabling more control for system-level operations.

To start the Jenkins service, follow these steps:

1. In a new directory, separate from the Control Repo, create a file named docker-compose.yml.

2. Copy and paste the provided code into the file and save it.

3. Open a terminal in the directory containing the docker-compose.yml file and run the following command:

   docker-compose up -d

   Your terminal should display output similar to this:

   $ docker-compose up -d
   [+] Running 1/1
   ✔ Container jenkins        Started 

4. Once the Jenkins service is running, it will be accessible in your browser at http://localhost:8080.

5. To log in, you need the Jenkins Admin password. Follow these steps to retrieve it:

   1. Access the Docker container terminal running Jenkins:

      docker exec -it jenkins bash

   2. Retrieve the Admin password:

      cat /var/jenkins_home/secrets/initialAdminPassword

   The password will be displayed in the terminal. Save it securely, as you'll need it to log in to Jenkins.

Run Jenkins and install the plugins

To get started with Jenkins, follow the steps:

1. Access Jenkins through your browser using the address http://localhost:8080.

2. Use the admin password from the previous step to unlock Jenkins. For future logins, you'll also need to use the admin user with the admin password.

   

   Additional Users

   You don't need to create additional users for this tutorial. You can perform all actions using the Admin user.

3. Select the Install suggested plugins option. These are the essential plugins you must select:

   

4. Jenkins will download and install the selected plugins. Once the installation is complete, you'll be redirected to the Jenkins dashboard.

   

To set up your Jenkins environment, install the three additional plugins:

1. From the Jenkins dashboard, navigate to Manage Jenkins -> Plugins -> Available plugins.
2. Search for and select the following plugins:
   • Docker plugin
   • Docker Pipeline
   • Basic Branch Build Strategies Plugin
3. Once the plugins are selected, click Install.
4. After the installation is complete, Jenkins will restart automatically.

note

During the restart process, the Jenkins container may stop temporarily. If you cannot access the dashboard at http://localhost:8080 and see a Page Not Found error, restart the Jenkins container by running:

docker-compose up -d

Add your credentials

Once you install the necessary plugins, add the LIBLAB_TOKEN and GITHUB_TOKEN as credentials in Jenkins. These credentials are used by Jenkins to interact with your GitHub repos (both the Control Repo and SDK repos) and to use the liblab CLI:

1. From the Jenkins dashboard, go to Manage Jenkins -> Credentials.
2. Under Stores scoped to Jenkins, select System.
3. Under the Domain, choose Global credentials.
4. Click + Add Credentials.
5. Fill in the form for each of the credentials as described below:

LIBLAB_TOKEN

• Kind: Select the Secret text option.
• Secret: Paste the LIBLAB_TOKEN from Step 3.
• ID: Use ID equal LIBLAB_TOKEN.
• Description: Add a meaningful description to help identify this credential later.

GITHUB_TOKEN

• Kind: Select the Secret text option.
• Secret: Paste the GITHUB_TOKEN from Step 3.
• ID: Use ID equal GITHUB_TOKEN.
• Description: Add a meaningful description to help identify this credential later.

github-credentials

• Kind: Select the Username with password option.
• Username: Use the GitHub organization or the GitHub user used to create the GITHUB_TOKEN in Step 3.
• Password: Paste the GITHUB_TOKEN from Step 3.
• ID: Use ID equal github-credentials.
• Description: Add a meaningful description to help identify this credential later.

This is what you should have:

Create a Jenkins pipeline

Now that your environment is configured, you can create a pipeline to manage your SDK projects' CI/CD process:

1. From the Jenkins dashboard, click New Item.

2. Enter a name for the pipeline, select Multibranch Pipeline, and click OK.

3. On the pipeline configuration page, do the following:

   1. In the Display Name field, add a name for the pipeline.
   2. In the Description field, provide a meaningful description of the pipeline’s purpose.
   3. For Branch Sources, select GitHub as the source. A new configuration form will appear.
      1. For Credentials, choose the username and password (github-credentials) you added in Add your credentials.
      2. Paste the Control Repo URL into the Repository HTTPS URL field.
      3. (Optional) Configure branch strategies if needed. For this tutorial, you can leave the default settings.
   4. For Build Configuration, select by Jenkins for the Mode and add Jenkins for the Script Path field.
   5. For the Scan Repository Triggers, set Interval to 10 minutes to define how often Jenkins should scan the repo for changes.
   Webhooks and GitHub plugins

   Instead of setting a scan interval, you can configure GitHub Webhooks for automatic triggering or use GitHub plugins to enhance Jenkins-GitHub integration.

4. Click Save.

Once saved, Jenkins will scan the repo for existing branches and pull requests. It will automatically trigger a build if it finds a Jenkinsfile.

Since your Control Repo has a Jenkinsfile, Jenkins will start the pipeline job. However, the pipeline will fail initially because the liblab.config.json file has not yet been updated with the SDK repos and organizations.

Update the config file

Now that finished configuring Jenkins and all your repositories are set up, you'll need to update the liblab.config.json file. Make sure to configure the publishing information and update the settings for each SDK language you plan to publish.

1. In the publishing section, set the githubOrg option to the name of your GitHub organization.
2. In the languageOptions section:

   1. Update the githubRepoName for each SDK. This should be the repo name only (e.g., typescript-sdk), not the full GitHub URL (e.g., https://github.com/myorg/typescript-sdk).

   2. Set the sdkVersion to match your desired version number. This should be updated for each new release.

   3. Configure details for each package manager:

      TypeScript

      • Set the npmName option to the name of your npm package.
      • Set the npmOrg option to the name of your npm organization.

      Python

      • Set the pypiPackageName option to the name of your PyPI package. This needs to be unique.

      C#

      • Set the packageId option to the name of your NuGet package.

      Java

      • Set the groupId option to your namespace verified by the Maven Central Portal.
      • Set the homepage option to the valid public URL of your SDK homepage.
      • Add at least one developer to the developers option.
      • (Recommended) Set the artifactId option. If not set, it defaults to the kebab-cased version of sdkName.

      Go

      • Set the goModuleName option to the name of your Go module. This should match the SDK repo name, for example github.com/myorg/go-sdk.

      • Ensure your license is set.

        note

        The Go Packages License Policy requires that all packages published to the Go Packages repository must include a recognized license file. It is important to note that if the package does not include a recognized license file, only limited package information will be displayed on the Go Packages repository.

        For more information on how to specify a license for your SDK, see the license config file option.

      PHP

      • Set the packageName option to the name of your PHP package. It must be in the format vendor/packageName.

Additional configuration options

There are many other options you can configure for SDK deployment but these are optional. Refer to the config file documentation to learn more.

The following code snippet shows an example of a liblab.config.json file, where SDKs for C#, Python, TypeScript, Go, PHP, and Java are configured for generation and publishing:

Example liblab.config.json configured for SDK deployment

{
  "sdkName": "test-sdk",
  "apiVersion": "1.0.0",
  "apiName": "test-api",
  "specFilePath": "spec.json",
  "languages": [
    "csharp",
    "python",
    "typescript",
    "go",
    "php",
    "java"
  ],
  "languageOptions": {
    "csharp": {
      "liblabVersion": "2",
      "packageId": "Test.SDK",
      "githubRepoName": "csharp-sdk",
      "sdkVersion": "1.0.0"
    },
    "python": {
      "liblabVersion": "2",
      "pypiPackageName": "test-sdk",
      "githubRepoName": "python-sdk",
      "sdkVersion": "1.0.0"
    },
    "typescript": {
      "npmName": "test-sdk",
      "npmOrg": "myorg",
      "githubRepoName": "typescript-sdk",
      "sdkVersion": "1.0.0"
    },
    "java": {
      "groupId": "com.myorg",
      "artifactId": "test-sdk",
      "homepage": "https://myorg.com",
      "githubRepoName": "java-sdk",
      "sdkVersion": "1.0.0",
      "developers": [
        {
          "name": "John Doe",
          "email": "[email protected]",
          "organization": "My Organization",
          "organizationUrl": "https://myorg.com"
        }
      ]
    },
    "go": {
      "goModuleName": "github.com/myorg/go-sdk",
      "githubRepoName": "go-sdk",
      "sdkVersion": "1.0.0",
    },
    "php": {
      "packageName": "myorg/test-sdk",
      "githubRepoName": "php-sdk",
      "sdkVersion": "1.0.0"
    },
  },
  "publishing": {
    "githubOrg": "myorg"
  }
}

Finally, commit and push your changes to your Control Repo. This will trigger the Jenkins pipeline to generate your SDKs and create PRs in your SDK repos.

Check the Jenkins pipeline

Once you pushes changes to the Control Repo, Jenkins will automatically scan the repo. If it detects changes, it will trigger a new build for the pipeline.

Triggering a New Build

If you want to manually force Jenkins to scan your repo for changes, go to the pipeline you created previously and click Scan Repository Now.

To review the results of the latest Jenkins pipeline job:

1. Navigate to the pipeline you created previously.
2. Click Build History to view the results of previous builds.
3. Select the build number of the most recent job to open its summary.
4. Review the Pipeline Overview to inspect each pipeline stage. If errors arise, you can check the output of each stage to identify the problem.

The following GIF demonstrates how to inspect the results of your builds.

Merge SDK PRs

After Jenkins completes the pipeline, a pull request (PR) will be created automatically in each SDK repo by liblab.

To view the PRs, access each SDK repo on GitHub and navigate to the Pull Requests section to view the new PRs created by the liblab CLI.

note

If you don't see the PRs, review Jenkins' build history to check for any errors during the SDK generation process.

To finish, follow the steps:

1. Open each PR to review the changes and ensure the SDK has been generated correctly.
2. Verify that the modifications align with your expectations and meet project requirements.
3. If everything is correct, approve the PR and merge it into the main branch.

Create a release in GitHub

To trigger the Action to publish your SDK to the package manager, you need to create a release in the SDK repo. Releases need a tag to tag the commit that you want to release.

1. From each SDK repo, go to Releases and click Create a new release.

   

2. Click Choose a tag, enter a tag (e.g., v1.0.0), and select Create new tag.

   note

   The tag should follow semantic versioning and typically includes a v prefix, like v1.0.0.

   Note on Versioning Go Packages

   It is important to note that the release tag version will be used as the version of the package in the Go Packages repository. Therefore, it is important to follow semantic versioning and ensure that the version in the liblab.config.json file corresponds to the version of the release tag.

3. Give the release a title and description. The title can be the same as the tag.

4. Click Publish release. This will trigger the GitHub Action to publish your SDK.

Confirm SDK publication

Once the publishing Action is done, check the package manager to confirm your SDK was published.

🎉🎉🎉   Congratulations! You have successfully generated and published your SDK!   🎉🎉🎉

Your setup is now fully automated to generate and publish SDKs with liblab and Jenkins pipelines, ensuring your latest API updates are quickly available to your users.

If you don’t see the SDK in the package manager, check the CI/CD > Pipelines or Jobs section in each SDK repo to troubleshoot any errors during the publishing process.